Traditional vs Jev
The same problems, the same simulation code, the same legal moves. The traditional column is the textbook solution for each problem; the Jev column is one Choice call per decision. Two LLM columns show GPT-5.6 Luna with reasoning off (the like-for-like comparison) and on (a reference); both get exactly Jev's input and must answer with one legal option.
- Model
- jev-1.13.0
- Jev calls
- 10164
- Median latency
- 367 ms
- Episodes per setting
- 10
- Runs
- 19 Sept, 09:50 UTC · ef02480 · eightrank, eightpuzzle, missionaries, vacuum, restaurant, wumpus, tictactoe, pos
| Problem | Measure | Traditional | Jev | Luna, reasoning off | Luna, reasoning on |
|---|---|---|---|---|---|
| 01Vacuum | Cleanliness over 24 stepsin scope mean of 10 seeded dirt schedules | 87% | 88% | 88% | 88% |
| Moves (energy)in scope | 15.7 | 3.7 | 9.1 | 15.5 | |
| 02Restaurant | Agrees with AIMA treein scope 40 seeded random scenarios; no ground truth exists | reference | 30 / 4060–86% | 31 / 4062–88% | 33 / 4068–91% |
| 03Wumpus | Escaped with goldin scope AIMA world + 49 seeded worlds | 29 / 5044–71% | 23 / 5033–60% | 16 / 5021–46% | 23 / 5033–60% |
| Mean scorein scope | 512.1 | 245.5 | 292.6 | 228.4 | |
| 04Missionaries | Solved, current state onlystress test 30-step limit | BFS: 11 moves | 0 / 100–28% | 0 / 100–28% | 8 / 1049–94% |
| Solved, with memoryin scope | — | 10 / 1072–100% | 10 / 1072–100% | 10 / 1072–100% | |
| Optimal, with lookaheadin scope | — | 10 / 1072–100% | 10 / 1072–100% | 10 / 1072–100% | |
| 05Tic-Tac-Toe | Not lost vs perfect play, board onlystress test plays X and O | minimax: always | 4 / 208–42% | 7 / 2018–57% | 18 / 2070–97% |
| Not lost vs perfect play, + tacticsin scope | — | 14 / 2048–85% | 16 / 2058–92% | 19 / 2076–99% | |
| Won vs random, board onlystress test | — | 12 / 2039–78% | 13 / 2043–82% | 16 / 2058–92% | |
| 07POS Tagging | Tagging accuracy vs human annotationin scope EWT test sample, 49 Penn tags | HMM 92.3% · MFT 88.4% | 92.9%91.7–94.0% | 93.6%92.5–94.6% | 94.5%93.5–95.4% |
| 068-Puzzle | Ranking + search in code: solvedin scope 30 puzzles (depths 8, 12, 16), widening beam 1→13, ≤300 rankings; Manhattan ranker in the classic column | 29 / 3083–99% | 16 / 3036–70% | 12 / 3025–58% | — |
| 068-Puzzle | A* nodes generated, depth 12stress test mean of 4 puzzles; Manhattan in the classic column | 43 | 69 | 119 | — |
| Estimates above truthstress test board + features; above = breaks A* optimality | 0% (Manhattan) | 19% | 5% | — |
- Luna with reasoning off (10042 calls): median 1026 ms and $0.00014 per call, about 2.8× slower and 3.7× the cost of Jev (367 ms, $0.000038).
- Luna with reasoning on (1868 calls): median 1563 ms and $0.00062 per call, about 4.3× slower and 16.2× the cost of Jev (367 ms, $0.000038).
- Planning without help from code (stress tests): Missionaries from the current state alone, solved 0/10 by Jev, 0/10 by Luna with reasoning off and 8/10 with reasoning on; Tic-Tac-Toe from the board alone against perfect play, not lost 4 / 20, 7 / 20 and 18 / 20. The advantage comes with reasoning; with memory or tactics from code the gap mostly closes.
- On its headline use case, many-label classification (POS tagging, in scope), accuracy is close: Jev 92.9%, Luna with reasoning off 93.6%, with reasoning on 94.5%, against 92.3% for the HMM tagger. The same tagging cost $0.1419 with Jev and $0.5479 with Luna, reasoning off. Jev − Luna off: -0.7 points (95% interval -1.5 to 0.1), not a clear difference. Jev − Luna on: -1.5 points (95% interval -2.4 to -0.8). Jev − the HMM: +0.6 points (95% interval -0.7 to 1.9), not a clear difference.
Same payloads, same legal options, small samples. Luna returns a choice without probabilities. “Reasoning off” is the like-for-like comparison with Jev, which does not reason; “reasoning on” is a reference. With reasoning on, a single 8-Puzzle call took more than three minutes, so the 8-Puzzle was run with reasoning off only.
Cost and speed
- Model
- jev-1.13.0
- Calls
- 10164
- Median latency
- 367 ms
- Tokens in / out
- 9,256,585 / 1,521,410
- Total cost
- $0.3888
- Per call
- $0.000038
- Model
- gpt-5.6-luna
- Calls
- 10042
- Median latency
- 1026 ms
- Tokens in / out
- 6,155,997 / 157,943
- Total cost
- $1.42
- Per call
- $0.00014
- Model
- gpt-5.6-luna
- Calls
- 1868
- Median latency
- 1563 ms
- Tokens in / out
- 3,356,103 / 407,287
- Total cost
- $1.16
- Per call
- $0.00062
What each solver is given
The current room's status and location. Two if-statements: dirty → clean, else move.
Its location and the status of both rooms. More than the rule sees, which is why it can choose to wait.
▸ Example payload, verbatim
{
state: {
agentLocation: "A",
roomA: "dirty",
roomB: "clean"
},
question: "You control a vacuum-cleaning robot in a world of two rooms, A (left) and B (right). Dirt can appear in either room at any time. Which action should the robot take next?",
actions: ["CLEAN", "MOVE_RIGHT", "WAIT"],
descriptions: {
CLEAN: "Vacuum the room the robot is currently in.",
MOVE_LEFT: "Move left, from room B into room A.",
MOVE_RIGHT: "Move right, from room A into room B.",
WAIT: "Stay put and do nothing this step."
}
}Eight attributes: wait bucket, hunger, alternative, reservation, bar, Fri/Sat, raining (and patrons, fixed at Full). No kids, no ratings.
Ten plain-language fields including kids and both ratings. Controls on the exhibit also ask with only the tree's attributes.
▸ Example payload, verbatim
{
state: {
quotedWaitMinutes: 45,
hunger: "very hungry",
raining: false,
withYoungKids: true,
hasReservation: false,
barToWaitAt: false,
fridayOrSaturdayNight: true,
thisRestaurantRating: 4.5,
alternativeRestaurantNearby: true,
alternativeRestaurantRating: 4.1
},
question: "You are advising a party that has just arrived at a full restaurant and been quoted a wait for a table. Given their situation, should they wait for this table or leave?",
actions: ["WAIT", "LEAVE"],
descriptions: {
WAIT: "Stay and wait for a table at this restaurant.",
LEAVE: "Leave now and eat somewhere else."
}
}The same exact risk estimates as Jev, plus the whole known map: it plans multi-step paths through visited cells to the nearest safe unexplored cell.
Its own cell, percepts and its neighbouring cells' risks (two to four, depending on position). No map beyond its neighbours. The classic agent has more context here.
▸ Example payload, verbatim
{
state: {
position: {
x: 1,
y: 0
},
entrance: {
x: 0,
y: 0
},
percepts: {
breeze: true,
stench: false,
glitter: false,
heardScream: false
},
hasArrow: true,
hasGold: false,
wumpusKilled: false,
stepsTaken: 1,
stepLimit: 40,
timesVisitedThisCell: 1,
north: {
visited: false,
timesVisited: 0,
pitRisk: 0.56,
wumpusRisk: 0,
knownSafe: false
},
east: {
visited: false,
timesVisited: 0,
pitRisk: 0.56,
wumpusRisk: 0,
knownSafe: false
},
west: {
visited: true,
timesVisited: 1,
pitRisk: 0,
wumpusRisk: 0,
knownSafe: true
}
},
question: "You are an agent exploring a dark 4×4 cave to find gold and get out alive. Some cells have bottomless pits or a Wumpus. Given your percepts and the estimated risks of neighbouring cells, which action should you take next?",
actions: [
"MOVE_NORTH",
"MOVE_EAST",
"MOVE_WEST",
"SHOOT_NORTH",
"SHOOT_EAST",
"SHOOT_WEST",
"RETREAT"
]
}The full transition model. BFS generates and remembers every state until it reaches the goal.
Depends on the context level: from bank counts and legal moves (with the state each leads to) up to memory, lookahead and distance to goal.
▸ Example payload, verbatim
{
state: {
leftBank: {
missionaries: 2,
cannibals: 2
},
rightBank: {
missionaries: 1,
cannibals: 1
},
boatIsOn: "right bank",
goal: "all 3 missionaries and all 3 cannibals on the right bank"
},
question: "Three missionaries and three cannibals must all cross a river from the left bank to the right bank. The boat carries one or two people and cannot cross empty. On either bank, if any missionaries are present, cannibals must never outnumber them. Here is the current state and the legal moves available right now. Which move should be made next?",
actions: ["MOVE_1_MISSIONARY", "MOVE_1_MISSIONARY_1_CANNIBAL"],
descriptions: {
MOVE_1_MISSIONARY: "Take 1 missionary from the right bank to the left bank. Afterwards: left bank 3 missionaries and 2 cannibals; right bank 0 missionaries and 1 cannibals; boat on the left bank.",
MOVE_1_MISSIONARY_1_CANNIBAL: "Take 1 missionary + 1 cannibal from the right bank to the left bank. Afterwards: left bank 3 missionaries and 3 cannibals; right bank 0 missionaries and 0 cannibals; boat on the left bank."
}
}▸ Example payload · + memory level
{
state: {
leftBank: {
missionaries: 2,
cannibals: 2
},
rightBank: {
missionaries: 1,
cannibals: 1
},
boatIsOn: "right bank",
goal: "all 3 missionaries and all 3 cannibals on the right bank",
previousMove: "1 missionary + 1 cannibal from the left bank",
statesVisitedSoFar: [
{
state: "3M 3C on the left, 0M 0C on the right, boat on the left",
times: 1
},
{
state: "2M 2C on the left, 1M 1C on the right, boat on the right",
times: 1
}
],
legalMoves: {
MOVE_1_MISSIONARY: {
leadsTo: "3M 2C on the left, 0M 1C on the right, boat on the left",
alreadyVisited: false,
timesVisited: 0
},
MOVE_1_MISSIONARY_1_CANNIBAL: {
leadsTo: "3M 3C on the left, 0M 0C on the right, boat on the left",
alreadyVisited: true,
timesVisited: 1
}
}
},
question: "Three missionaries and three cannibals must all cross a river from the left bank to the right bank. The boat carries one or two people and cannot cross empty. On either bank, if any missionaries are present, cannibals must never outnumber them. Here is the current state and the legal moves available right now. Which move should be made next? You are also told which states have already been visited.",
actions: ["MOVE_1_MISSIONARY", "MOVE_1_MISSIONARY_1_CANNIBAL"],
descriptions: {
MOVE_1_MISSIONARY: "Take 1 missionary from the right bank to the left bank. Afterwards: left bank 3 missionaries and 2 cannibals; right bank 0 missionaries and 1 cannibals; boat on the left bank. You have not been in that state before.",
MOVE_1_MISSIONARY_1_CANNIBAL: "Take 1 missionary + 1 cannibal from the right bank to the left bank. Afterwards: left bank 3 missionaries and 3 cannibals; right bank 0 missionaries and 0 cannibals; boat on the left bank. You have already been in that state 1 time."
}
}The whole game tree. Minimax searches every continuation to the end and knows the result of each move.
The board, its mark and the empty squares. At + tactics, one-move-ahead facts per square; at oracle, the perfect-play result per square.
▸ Example payload, verbatim
{
state: {
youPlay: "X",
opponentPlays: "O",
board: ["O . .", ". X .", ". . ."],
boardLegend: "rows top to bottom, '.' is empty",
squares: {
TOP_LEFT: "O",
TOP_CENTER: "empty",
TOP_RIGHT: "empty",
MIDDLE_LEFT: "empty",
CENTER: "X",
MIDDLE_RIGHT: "empty",
BOTTOM_LEFT: "empty",
BOTTOM_CENTER: "empty",
BOTTOM_RIGHT: "empty"
},
movesPlayedSoFar: 2
},
question: "You are playing tic-tac-toe. Players take turns placing their mark on an empty square of a 3×3 board; the first to get three in a row horizontally, vertically or diagonally wins, and a full board with no line is a draw. It is your turn. Which empty square should you take?",
actions: [
"TOP_CENTER",
"TOP_RIGHT",
"MIDDLE_LEFT",
"MIDDLE_RIGHT",
"BOTTOM_LEFT",
"BOTTOM_CENTER",
"BOTTOM_RIGHT"
],
descriptions: {
TOP_CENTER: "Place X on the top center square.",
TOP_RIGHT: "Place X on the top right square.",
MIDDLE_LEFT: "Place X on the middle left square.",
MIDDLE_RIGHT: "Place X on the middle right square.",
BOTTOM_LEFT: "Place X on the bottom left square.",
BOTTOM_CENTER: "Place X on the bottom center square.",
BOTTOM_RIGHT: "Place X on the bottom right square."
}
}▸ Example payload · + tactics level
{
state: {
youPlay: "O",
opponentPlays: "X",
board: ["O . X", ". X .", ". . ."],
boardLegend: "rows top to bottom, '.' is empty",
squares: {
TOP_LEFT: "O",
TOP_CENTER: "empty",
TOP_RIGHT: "X",
MIDDLE_LEFT: "empty",
CENTER: "X",
MIDDLE_RIGHT: "empty",
BOTTOM_LEFT: "empty",
BOTTOM_CENTER: "empty",
BOTTOM_RIGHT: "empty"
},
movesPlayedSoFar: 3
},
question: "You are playing tic-tac-toe. Players take turns placing their mark on an empty square of a 3×3 board; the first to get three in a row horizontally, vertically or diagonally wins, and a full board with no line is a draw. It is your turn. Which empty square should you take? Each option lists facts about what it does one move ahead.",
actions: [
"TOP_CENTER",
"MIDDLE_LEFT",
"MIDDLE_RIGHT",
"BOTTOM_LEFT",
"BOTTOM_CENTER",
"BOTTOM_RIGHT"
],
descriptions: {
TOP_CENTER: "Place O on the top center square. After this, X can win on their next move.",
MIDDLE_LEFT: "Place O on the middle left square. After this, X can win on their next move.",
MIDDLE_RIGHT: "Place O on the middle right square. After this, X can win on their next move.",
BOTTOM_LEFT: "Place O on the bottom left square. This blocks X from completing three in a row.",
BOTTOM_CENTER: "Place O on the bottom center square. After this, X can win on their next move.",
BOTTOM_RIGHT: "Place O on the bottom right square. After this, X can win on their next move."
}
}How much context does planning need?
Reading it. With the current state alone Jev locks into a two-state loop, no better than a random mover (6% solved). A record of visited states turns that into a solve, and most of the credit belongs to the memory: random moves that avoid visited states solve it 98% of the time. With lookahead facts Jev is optimal, but a no-model rule given the same facts is optimal 100% of the time, so at that level the facts do the planning.
The engine view. Search supplies the guarantee; Jev supplies the scores. Guided by Jev's scores the search expanded 14 states, against 15 for BFS and 15 for a one-line "people across" heuristic. On a 16-state puzzle that shows the split works, not that Jev adds efficiency. The 8-Puzzle exhibit tests the same idea on 181,440 states.
Against a perfect opponent
Reading it. The same shape as Missionaries. From the board alone Jev plays plausible tic-tac-toe, enough to beat a random player, but a perfect opponent punishes its oversights. One-move tactics computed by code change its blunder rate from 27% to 9%; with the perfect-play result per square it is 0%. Jev chooses well when the options are described well; it is not a search algorithm.
Jev as a search heuristic
| Depth | No heuristic | Misplaced tiles | Manhattan | Jev, board only | Jev + features | Luna, board only | Luna + features |
|---|---|---|---|---|---|---|---|
| 8 | 623 b* 2.06 | 39 b* 1.35 | 27 b* 1.26 | 33 b* 1.30 | 31 b* 1.30 | 50 b* 1.40 | 40 b* 1.35 |
| 12 | 4,109 b* 1.88 | 157 b* 1.37 | 43 b* 1.19 | 133 b* 1.33 | 69 b* 1.25 | 152 b* 1.33 1 not optimal | 119 b* 1.32 |
| 16 | 28,813 b* 1.80 | 1,151 b* 1.44 | 226 b* 1.25 | 670 b* 1.39 1 over budget | 408 b* 1.32 | 426 b* 1.32 1 over budget | 224 b* 1.26 1 over budget |
Mean nodes generated over 4 puzzles per depth; b* is AIMA's effective branching factor. Model runs stop after scoring 600 boards. Luna ran with reasoning switched off on this problem: at its default setting a single 8-puzzle call reasoned for more than three minutes. Its other problems used the default setting.
Reading it. The diagonal is a perfect heuristic; the shaded area below it is where an estimate is safe for A*. Given the Manhattan distance as a feature, the question is whether Jev adds anything beyond it: compare the Manhattan column with the Jev + features column. In “Jev decides” mode (no search), it solved 0 of 10 puzzles with the board alone and 1 of 10 with a memory of visited boards (30-move limit, 10 moves optimal).
Tagging real text
Jev's most common mistakes (gold → Jev): NN→NNP ×17 · ,→: ×11 · VBN→JJ ×9 · NN→JJ ×9 · IN→TO ×8 · NNP→NN ×7. Same sentences, same tokenization, for every tagger.
Ranking boards, code searches
| Ranker | Solved | Depth 8 | Depth 12 | Depth 16 | Optimal | Mean rankings |
|---|---|---|---|---|---|---|
| Perfect ranker true distance · ceiling | 30 / 3089–100% | 10/10 | 10/10 | 10/10 | 30/30 | 9 |
| Manhattan ranker no model | 29 / 3083–99% | 10/10 | 10/10 | 9/10 | 18/30 | 64 |
| Jev | 16 / 3036–70% | 8/10 | 5/10 | 3/10 | 9/30 | 183 |
| Luna, reasoning off | 12 / 3025–58% | 7/10 | 5/10 | 0/10 | 8/30 | 173 |
| All options equal no information · floor | 1 / 301–17% | 1/10 | 0/10 | 0/10 | 1/30 | 181 |
10 seeded puzzles per optimal depth. The perfect, Manhattan and uninformative rows use no model and are identical across runs. Luna returns a pick without probabilities, so its ranking gives 90% to its pick. Earlier versions of this page used a single width-3 beam, which cannot back up after one poor ranking; widening was adopted because it lets a good ranker succeed (the perfect ranker solves every puzzle).
Where Jev and the tree disagree
| Scenario | Tree | Jev | P(wait) | Luna off | Luna on | Tree-only | Repeat | Reversed | Strongest factor readings |
|---|---|---|---|---|---|---|---|---|---|
| Date night | WAIT WaitEstimate 30–60, Alternate yes, Fri/Sat yes → wait | LEAVE disagrees | 36% | WAIT tree-only: leave | WAIT | 13% leave | 42% | 45% | Weather argues for staying no 8% A good alternative exists yes 87% |
| Family with kids | WAIT WaitEstimate 30–60, Alternate yes, Fri/Sat yes → wait | LEAVE disagrees | 10% | LEAVE | LEAVE | 1% leave | 10% | 7% | Can wait comfortably no 5% Weather argues for staying no 7% |
| Tourist in rain | WAIT WaitEstimate 10–30, Hungry yes, Alternate no → wait | WAIT | 68% | WAIT | WAIT | 16% leave | 70% | 64% | Can wait comfortably no 7% Hard on someone in the party no 19% |
| Very hungry | LEAVE WaitEstimate 30–60, Alternate yes, Fri/Sat no → leave | LEAVE | 9% | LEAVE | LEAVE | 0% leave | 11% | 12% | Can wait comfortably no 6% Weather argues for staying no 8% |
| Great restaurant / long wait | LEAVE WaitEstimate >60 → leave | LEAVE | 36% | WAIT tree-only: leave | LEAVE | 5% leave | 37% | 37% | Weather argues for staying no 8% A good alternative exists yes 86% |
Reading it. Removing kids and ratings lowers Jev's P(wait) in 5 of 5 scenarios. In none of the disagreements does the reduced state move Jev onto the tree's side: Jev weighs the tree's own attributes differently, not just more of them. Open any scenario on the exhibit for the full rationale and single-change counterfactuals.
- Jev: agrees with the tree on 30 of 40 (60–86%); waits in 22, the tree in 24.
- Luna off: agrees with the tree on 31 of 40 (62–88%); waits in 29, the tree in 24.
- Luna on: agrees with the tree on 33 of 40 (68–91%); waits in 29, the tree in 24.
Every input drawn independently, so some combinations are unusual. Agreement with a 1995 tree, not accuracy.
Same beliefs, different agents
| Agent | Escaped with gold | No gold, alive | Died | Mean score |
|---|---|---|---|---|
| Traditional | 29 / 5044–71% | 18 | 3 | 512 |
| Jev | 23 / 5033–60% | 17 | 10 | 245 |
| Luna off | 16 / 5021–46% | 33 | 1 | 293 |
| Luna on | 23 / 5033–60% | 16 | 11 | 228 |
▸ Every world (50)
| World | Traditional | Score | Jev | Score | Luna off | Score | Luna on | Score |
|---|---|---|---|---|---|---|---|---|
| AIMA Fig. 7.2 | ✓ escaped with gold | 988 | ✓ escaped with gold | 979 | ✓ escaped with gold | 981 | ✓ escaped with gold | 981 |
| Seed 1000 | ↩ retreated | -1 | ✓ escaped with gold | 985 | ✓ escaped with gold | 985 | ✓ escaped with gold | 985 |
| Seed 1001 | ↩ retreated | -1 | ↩ retreated | -30 | ↩ retreated | -1 | ✓ escaped with gold | 992 |
| Seed 1002 | ✓ escaped with gold | 984 | ↩ retreated | -20 | ↩ retreated | -3 | ✕ fell into a pit | -1017 |
| Seed 1003 | ✓ escaped with gold | 994 | ↩ retreated | -11 | ↩ retreated | -11 | ↩ retreated | -11 |
| Seed 1004 | ↩ retreated | -1 | ✓ escaped with gold | 990 | ↩ retreated | -1 | ✓ escaped with gold | 994 |
| Seed 1005 | ✓ escaped with gold | 994 | ✓ escaped with gold | 996 | ✓ escaped with gold | 996 | ✓ escaped with gold | 996 |
| Seed 1006 | ✓ escaped with gold | 994 | ↩ retreated | -20 | ↩ retreated | -7 | ✓ escaped with gold | 967 |
| Seed 1007 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
| Seed 1008 | ✓ escaped with gold | 986 | ✓ escaped with gold | 990 | ✓ escaped with gold | 992 | ✓ escaped with gold | 992 |
| Seed 1009 | ✓ escaped with gold | 988 | ✓ escaped with gold | 994 | ↩ retreated | -28 | ✓ escaped with gold | 963 |
| Seed 1010 | ✓ escaped with gold | 992 | ↩ retreated | -22 | ✕ fell into a pit | -1017 | ✓ escaped with gold | 965 |
| Seed 1011 | ✓ escaped with gold | 978 | ✓ escaped with gold | 969 | ↩ retreated | -3 | ↩ retreated | -3 |
| Seed 1012 | ✓ escaped with gold | 994 | ✓ escaped with gold | 992 | ↩ retreated | -13 | ↩ retreated | -3 |
| Seed 1013 | ↩ retreated | -1 | ↩ retreated | -41 | ↩ retreated | -1 | ↩ retreated | -5 |
| Seed 1014 | ↩ retreated | -1 | ↩ retreated | -41 | ↩ retreated | -1 | ↩ retreated | -36 |
| Seed 1015 | ✓ escaped with gold | 990 | ✓ escaped with gold | 992 | ✓ escaped with gold | 988 | ✕ fell into a pit | -1006 |
| Seed 1016 | ✓ escaped with gold | 990 | ✓ escaped with gold | 996 | ✓ escaped with gold | 994 | ✓ escaped with gold | 996 |
| Seed 1017 | ↩ retreated | -1 | ✓ escaped with gold | 985 | ✓ escaped with gold | 985 | ✓ escaped with gold | 985 |
| Seed 1018 | ✓ escaped with gold | 982 | ↩ retreated | -28 | ✓ escaped with gold | 988 | ✓ escaped with gold | 990 |
| Seed 1019 | ↩ retreated | -1 | ↩ retreated | -39 | ↩ retreated | -1 | ✕ fell into a pit | -1015 |
| Seed 1020 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
| Seed 1021 | ✓ escaped with gold | 980 | ↩ retreated | -28 | ✓ escaped with gold | 990 | ✓ escaped with gold | 990 |
| Seed 1022 | ✓ escaped with gold | 990 | ✓ escaped with gold | 996 | ✓ escaped with gold | 996 | ✓ escaped with gold | 996 |
| Seed 1023 | ✓ escaped with gold | 992 | ↩ retreated | -28 | ✓ escaped with gold | 971 | ✓ escaped with gold | 961 |
| Seed 1024 | ✓ escaped with gold | 990 | ✓ escaped with gold | 992 | ✓ escaped with gold | 994 | ✓ escaped with gold | 994 |
| Seed 1025 | ✕ fell into a pit | -1004 | ✕ fell into a pit | -1006 | ↩ retreated | -3 | ↩ retreated | -3 |
| Seed 1026 | ✕ fell into a pit | -1004 | ↩ retreated | -5 | ↩ retreated | -7 | ↩ retreated | -3 |
| Seed 1027 | ✓ escaped with gold | 988 | ↩ retreated | -20 | ↩ retreated | -3 | ✕ fell into a pit | -1002 |
| Seed 1028 | ↩ retreated | -1 | ↩ retreated | -41 | ↩ retreated | -1 | ✓ escaped with gold | 994 |
| Seed 1029 | ↩ retreated | -1 | ✓ escaped with gold | 996 | ↩ retreated | -1 | ✓ escaped with gold | 996 |
| Seed 1030 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
| Seed 1031 | ✓ escaped with gold | 996 | ✓ escaped with gold | 994 | ↩ retreated | -3 | ↩ retreated | -3 |
| Seed 1032 | ✓ escaped with gold | 980 | ✓ escaped with gold | 977 | ✓ escaped with gold | 984 | ✓ escaped with gold | 973 |
| Seed 1033 | ✕ fell into a pit | -1010 | ↩ retreated | -13 | ↩ retreated | -3 | ↩ retreated | -3 |
| Seed 1034 | ✓ escaped with gold | 990 | ✓ escaped with gold | 996 | ✓ escaped with gold | 996 | ✓ escaped with gold | 996 |
| Seed 1035 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
| Seed 1036 | ✓ escaped with gold | 992 | ✕ fell into a pit | -1008 | ✓ escaped with gold | 975 | ↩ retreated | -28 |
| Seed 1037 | ✓ escaped with gold | 990 | ✓ escaped with gold | 994 | ↩ retreated | -20 | ↩ retreated | -34 |
| Seed 1038 | ↩ retreated | -1 | ✓ escaped with gold | 979 | ↩ retreated | -1 | ↩ retreated | -26 |
| Seed 1039 | ✓ escaped with gold | 984 | ✓ escaped with gold | 986 | ↩ retreated | -20 | ✓ escaped with gold | 975 |
| Seed 1040 | ↩ retreated | -1 | ✓ escaped with gold | 994 | ↩ retreated | -1 | ✓ escaped with gold | 994 |
| Seed 1041 | ✓ escaped with gold | 980 | ↩ retreated | -5 | ↩ retreated | -3 | ↩ retreated | -3 |
| Seed 1042 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
| Seed 1043 | ✓ escaped with gold | 982 | ✓ escaped with gold | 971 | ✓ escaped with gold | 988 | ✓ escaped with gold | 984 |
| Seed 1044 | ✓ escaped with gold | 988 | ✕ fell into a pit | -1018 | ↩ retreated | -5 | ↩ retreated | -18 |
| Seed 1045 | ✓ escaped with gold | 971 | ↩ retreated | -30 | ↩ retreated | -3 | ↩ retreated | -11 |
| Seed 1046 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
| Seed 1047 | ✓ escaped with gold | 994 | ✓ escaped with gold | 992 | ↩ retreated | -3 | ↩ retreated | -3 |
| Seed 1048 | ↩ retreated | -1 | ✕ fell into a pit | -1001 | ↩ retreated | -1 | ✕ fell into a pit | -1001 |
Consistency checks
Among other Jev demos
| Practice | Public demos doing it | This project |
|---|---|---|
Compares Jev against something Any baseline: another model, embeddings, a heuristic or an algorithm. | 5 / 16 | yes a textbook algorithm on every problem |
Compares against a non-LLM method Embeddings, a rule, a heuristic or a search algorithm. | 1 / 16 | yes rules, AIMA tree, minimax, BFS, random movers |
Varies what Jev is told Runs the same task with more or less context, to separate model from inputs. | 0 / 16 | yes context ladders on Missionaries and Tic-Tac-Toe |
Code does the search, Jev chooses Lookahead or filtering is done by ordinary code and handed to Jev. | 2 / 16 | yes engine mode, tactics and lookahead levels |
Reports where Jev fails Losses, errors or limits, not only successes. | 5 / 16 | yes 0/3 state-only, losses to minimax |
Quantified results Counts, rates or scores rather than a single anecdote. | 7 / 16 | yes small samples: 3–8 per setting |
Code or live demo to reproduce A repository or a playable page. | 5 / 16 | yes live exhibits + npm run benchmark |
Compares against an LLM The same task given to a general language model. | 2 / 16 | yes GPT-5.6 Luna on every problem, same payloads |
Reports cost per decision Dollars per call or per task. | 3 / 16 | yes tokens and list-price USD for every call |
Real-world task Production-like data rather than a teaching problem. | 7 / 16 | no deliberately textbook problems |
Where this stands. Most public Jev posts are single demonstrations of speed and cost on a real task. They are good at showing Jev is fast and cheap, and most do not compare it with anything. This project is the only one found that tests Jev against textbook algorithms and varies what Jev is told. Only Mario (+ lookahead) matches our finding that code-supplied lookahead is what makes the model look capable, and its author plans the obvious next test: Jev versus a cheap heuristic on the same simulations.
Where others are ahead. They use real data (moderation, payments, pull requests, retrieval). This project deliberately uses textbook problems, and its samples are small.
| Demo | Kind | What it does (as the post describes it) | Baseline | Failures shown |
|---|---|---|---|---|
| This project | Game / control · Classification · Planning | Five textbook AI problems, each against a classical solution, with context ladders and recorded runs. | classical algorithms | yes |
| @TheINAOG Sep 17 | Game / control | Super Mario Bros. 1-1 cleared for $0.04. RAM parsed into structured state; code simulates controller macros and filters predicted deaths; Jev picks among the survivors. | heuristic (planned) | yes |
| @kanemama_ Sep 17 | Game / control | Chess: Jev picks moves probabilistically from the legal move set. Playable. | none stated | — |
| @DineshDataAI Sep 17 | Game / control | Tetris: Jev returns a typed placement (rotation, column, drop); the game engine validates it. | none stated | yes |
| @akafukusou Sep 17 | Retrieval | Evidence retrieval on 34 QASPER questions against pgvector + OpenAI embeddings. | embeddings | yes |
| @frankiedigiac Sep 17 | Classification | Real-time payment-transaction screening compared with Claude Haiku 4.5 on the same streams. | LLM | — |
| @iamMrDuncan Sep 18 | Evaluation | Open benchmark of a small local model (Needle 3) against Jev, Qwen 27B and OSS 120B. | local models | — |
| @redp314 Sep 17 | Dev tooling | Pull-request review: one call returns 14 typed checks (secrets, SQL injection, …) as probabilities. | LLM | — |
| @tamarajtran Sep 18 | Dev tooling | Instant context compaction: score every tool call and drop the irrelevant ones, instead of summarising. | none stated | — |
| @altryne Sep 18 | Dev tooling | The compaction idea as a Claude plugin: a session of nearly 1M tokens cut to 86K in about a second. | none stated | — |
| @kacpersinilo Sep 17 | Classification | Comment moderation: 20 comments in 1.48 s of API time for $0.00044. | none stated | — |
| @stevekrouse Sep 16 | Playground | Live playground for trying Jev questions. | none stated | — |
| @waynesutton Sep 17 | Playground | askjev.ai: ask anything; instead of answering, Jev judges. | none stated | — |
| @sydneyrunkle Sep 18 | Dev tooling | Article on using Jev as the evaluation step inside an agent harness loop. | none stated | — |
| @justALEXWORTEGA Sep 17 | Critique | Claims an MLP head on a 4B open model behaves like Jev; weights published. | local models | — |
| @HiromTeachesAI Sep 16 | Critique | Thread arguing Jev is a fast classifier, not a chat-model competitor, and questioning the launch claims. | none stated | yes |
| @Salman_Bareesh Sep 17 | Critique | Tried wiring Jev into a scoring pipeline; reports SDK errors and a billing wall before any results. | none stated | yes |
Collected from X search on 18 September 2026, three days after Jev launched: 16 posts from the top results for Jev queries (demos, evaluations, critiques). Not exhaustive; launch announcements, questions and reposts are left out. Each row reflects only what the post itself says, so a practice counted as missing may exist elsewhere.
How this was run
- Every Jev decision is a single Choice question over the legal actions only. Probabilities are as returned.
- Questions were written once. Missionaries context levels add information computed by ordinary code; none adds instructions about strategy.
- Traditional solutions: the AIMA reflex rule, the AIMA restaurant tree, a hand-written probabilistic Wumpus agent, and breadth-first search.
- The LLM column: scripts/benchmark.ts sends GPT-5.6 Luna exactly the payload Jev receives, through the same server-side task builders, with the answer constrained to the legal labels by a JSON schema. Luna returns a choice but no probabilities; none are invented. Results are included above.
- Samples: 10 vacuum dirt schedules, 5 restaurant presets plus 40 random scenarios, 50 Wumpus worlds, 10 Missionaries episodes per level, 20 tic-tac-toe games per setting, 30 8-puzzles for ranking, 200 POS sentences (2593 words). Rates carry 95% Wilson intervals; POS accuracy a bootstrap interval over sentences. Jev is nearly deterministic, so repeated episodes of the same start add little information: Missionaries and Tic-Tac-Toe show consistency more than breadth.