Jev plays SameGame
samegame.app now has a button called Jev. Press it and a model plays the board for you, one move at a time: it lifts the group it picked in amber, clears it, and shows you what else it considered and how sure it was. Here it is clearing board C-0CI3R3 in 22 moves:
The video runs at the speed you see on the site, with every move waiting for Jev’s real answer.
Update, September 18: the next day I tried 21 ways of showing Jev a board, including the board itself as plain characters, and ended up with prompts that score 36 to 71 % more than this one and keep pace with the search further down, without looking ahead at all. That is part two, with every request and response published.
What Jev is
Jev comes from TypeSafe, and it is not a chat model. It does not write text at all. You send it a state, which is whatever your program knows right now, and a set of typed questions. It sends back typed answers your code can use directly, with no parsing:
- Choice: pick one option from a set you define. You get the pick, a probability for every option, and a confidence.
- Score: place the state on an ordered scale you describe, like calm, frustrated, angry.
- Noul: a yes/no question, answered as the probability of yes.
TypeSafe calls this a System One model, after the fast, intuitive kind of thinking: the gut-check judgment a knowledgeable person makes in a few seconds. The probabilities are trained to be calibrated, so 0.9 is supposed to mean nine times in ten. Several questions in one call are answered in parallel, each on its own, and it is priced per input token only: $0.042 per million.
What it is for
Decisions inside software. Their examples are routing a support ticket to the right team, deciding whether a retrieved passage is relevant, catching a prompt injection before it reaches an LLM, and mapping a request onto a function and its arguments. The pattern is always the same: your code stays in control and asks the model narrow questions, and the confidence tells the code whether to act on the answer or hand it to a person.
It is just as clear about what Jev is not for. It is not a calculator, it does not count reliably, it does not compare dates, and it does not generate text. Their advice is to keep all of that in code and send the model plain words instead of raw numbers. That advice shaped everything below.
Why SameGame
SameGame is a string of choices. Every turn there is a list of legal moves, the groups of two or more touching tiles of one colour, and you pick one. That is a Choice question with the moves as the options. The hard part of SameGame is not seeing the moves; it is judgment. A group of n scores n(n−1)×10, so ten tiles are worth 900 points and five pairs are worth 100. The skill is knowing when to spend a small move now so that a colour comes together into something big later.
Code does the counting
For every legal move, the game engine plays it on a copy of the board and writes down what happened, in words. How big the group is and what it scores. Whether clearing it drops tiles of one colour into a bigger group. Whether it leaves tiles stranded with no neighbour of their colour. Whether it leaves a single tile of a colour, which can never be cleared.
export function describe(c: Candidate, all: Candidate[]): Record<string, string> {
const color = COLOR_NAMES[c.color];
const d: Record<string, string> = {
clears: `a ${sizeWord(c.size)} ${color} group of ${c.size} tiles for ${c.points} points`,
};
if (c.colorLargestAfter > Math.max(c.colorLargestBefore, c.size))
d.builds = `joins ${color} tiles into a bigger ${color} group of ${c.colorLargestAfter}`;
const loneChange = c.loneAfter - c.loneBefore;
d.lone_tiles = loneChange > 0 ? `strands ${loneChange} more lone tiles`
: loneChange < 0 ? `connects ${-loneChange} lone tiles` : 'no change to lone tiles';
if (c.colorLeftAfter === 1) d.warning = `leaves a single ${color} tile that can never be cleared`;
if (c.endsGame) d.ends = c.tilesAfter === 0 ? 'clears the board and ends the game'
: `ends the game with ${c.tilesAfter} tiles left`;
return d;
}There is no search here and no simulated ending. The engine looks exactly one move ahead and reports what it sees. Choosing is left to Jev.
One question per move
This is the request for the first move on C-0CI3R3, with three of its 22 options shown:
{
"state": {
"board": {
"tiles_left": 96,
"moves_available": 22,
"tiles_by_colour": { "red": 14, "blue": 16, "green": 20, "yellow": 21, "purple": 25 },
"lone_tiles": 25
}
},
"model": "jev-latest",
"questions": {
"best_move": {
"type": "choice",
"instructions": {
"question": "Which move should be played now to finish this SameGame board with the highest total score?",
"scoring": "A group of n tiles scores n(n-1)x10, so one group of 10 (900 points) is worth far more than five groups of 2 (100 points). Clearing the whole board adds 2000; 1 to 5 tiles left add a smaller bonus.",
"strategy": [
"Prefer a move that joins tiles of one colour into a bigger group to clear later.",
"Clear small groups of other colours out of the way of the colour with the most tiles.",
"Avoid stranding lone tiles and never leave a single tile of a colour.",
"Take a large group when waiting will not make it bigger."
]
},
"criteria": {
"m8": {
"clears": "a small yellow group of 2 tiles for 20 points",
"builds": "joins yellow tiles into a bigger yellow group of 6",
"lone_tiles": "connects 3 lone tiles"
},
"m13": {
"clears": "a large green group of 9 tiles for 720 points",
"lone_tiles": "connects 1 lone tile"
},
"m18": {
"clears": "a small blue group of 2 tiles for 20 points",
"lone_tiles": "strands 1 more lone tile"
}
}
}
}
}Jev’s answer came back in 416 ms. It picked m8, the two yellow tiles that bring six yellows together, at 39 %, and put the nine greens worth 720 points right now second at 23 %. It passed up the bigger score to build a bigger group, which is exactly the kind of judgment the game is about.
The whole call is one fetch:
const response = await fetch('https://api.typesafe.ai/v1/systemone', {
method: 'POST',
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
body: JSON.stringify(body),
});
const { answers, usage } = await response.json();
const { choice, probabilities, confidence } = answers.best_move;A Classic board costs about 1,200 input tokens a move. A whole game is about a tenth of a cent.
The key stays on the server
The page never sees the API key. samegame.app is a Cloudflare Worker, and the key is a secret on that Worker. The page sends the board, and only the board: the Worker checks that it is a board, builds the question itself and calls TypeSafe. The endpoint cannot be used to ask Jev anything else, and each visitor is rate limited.
sequenceDiagram
participant Page as samegame.app page
participant Worker as Cloudflare Worker
participant Jev as TypeSafe Jev
Page->>Worker: POST /api/jev { board }
Worker->>Worker: list the legal moves, describe each one
Worker->>Jev: one Choice question over every move
Jev-->>Worker: pick, probabilities, confidence
Worker-->>Page: the move + every move's probability
Page->>Page: lift the pick in amber, then tap it
async function jev(request: Request, env: Env): Promise<Response> {
if (request.method !== 'POST') return json({ error: 'POST a board' }, 405);
const ip = request.headers.get('CF-Connecting-IP') ?? 'local';
if (!(await env.JEV_LIMIT.limit({ key: ip })).success) return json({ error: 'slow down' }, 429);
const board = parsePicture((await request.json()).board); // at most 12×18, colours 0–4
if (!board) return json({ error: 'not a board' }, 400);
return json(await askJev(board, env.TYPESAFE_API_KEY));
}On the page, Jev plays the way you do. It selects the group, waits a moment so you can see the choice, and taps it again to clear it. If you press New or Undo, or tap the board yourself, it stops. You can take over for a few moves and hand the board back.
while (running && !store.game.isOver) {
const answer = await (await fetch('/api/jev', {
method: 'POST', body: JSON.stringify({ board: store.game.board.picture }),
})).json();
const at = { column: answer.choice.column, row: answer.choice.row };
store.suggest(at); // lifted in amber
await sleep(450);
store.tap(at); // the same tap a player makes
await sleep(500);
}What did not work
My first version also gave Jev a lookahead: for each move, the score a simple largest-group-first finish would reach from there. Scores went up, but Jev picked the move with the best lookahead about nine times in ten. At that point the lookahead was playing, not Jev. I took it out. What is on the site now is Jev’s own judgment about one move ahead.
How well it plays
I ran the same 20 boards Jev had never seen, 16 Classic and 4 Expert, through four players:
| Player | Total points | Boards cleared |
|---|---|---|
| Jev, one question per move | 113,860 | 5 |
| Always take the biggest group | 77,420 | 0 |
| Random legal moves (average of 8 runs a board) | 75,695 | – |
| Search: try every move, finish each one greedily | 147,020 | 16 |
Jev scores 47 % more than taking the biggest group every time, and clears boards the simple rule never clears. A search that plays out every option still beats it, as it should: before each move it plays a whole game to the end for every legal move, while Jev answers one question. Jev took about a quarter to a third of a second per move and cost $0.0012 for a Classic game.
That is what I like about it. Jev is not trying to be a solver. It is a fast, cheap, calibrated judgment call that you wire into ordinary code, and the probabilities it hands back are useful on their own: they are the bars you see under the board while it plays.
Part two: what should Jev see?
Added September 18, 2026.
Everything above hands Jev my description of each move. The next morning I asked the obvious question: why describe anything? Why not give Jev the board itself, as characters in rows and columns, and let it look?
That question turned into a day of experiments: 21 ways of showing Jev a board, each played three times on the same 20 boards and then three times on 20 fresh ones. That is 76,795 calls and $5.81 of Jev. The short version: Jev cannot read a board, it reads words very closely, how a fact is worded matters more than the fact, and the best prompts score 36 to 71 % more than the one from part one and keep pace with the search, with no lookahead.
Every request and every response from these runs is published, so you can check any number here or draw your own conclusions: github.com/objectgraph/jev-samegame-bench.
How I measured
The same 20 boards as above, 16 Classic and 4 Expert. Jev’s answers vary a little from call to call, so each strategy plays all 20 boards three times and the number I quote is the mean of the three 20-board totals. For scale, on these boards:
| Player, no model involved | Points |
|---|---|
| Random legal moves | 74,715 |
| Always take the biggest group | 77,420 |
| Search: try every move, finish each one greedily (the Hint button) | 147,020 |
The prompt from part one scored 112,013 this time, against 113,860 in yesterday’s single run, so the measurement repeats within 2 %.
I adjusted prompts while looking at these 20 boards, which flatters whatever I ended up with. So at the end every strategy also played 20 fresh boards that nothing was tuned on. That table is further down, and it is the one to trust.
The board as characters
This is the whole state for the first move of board C-046191. One letter per tile, rows numbered from the bottom like a chessboard, columns lettered:
{
"legend": "R red, B blue, G green, Y yellow, P purple. A dot is an empty cell.",
"board": [
"12 | R Y R G Y P Y B",
"11 | R Y R B G Y R P",
"10 | R P R P P B P P",
" 9 | G G B P G B R P",
" 8 | G R Y P P G P R",
" 7 | B G Y P R P Y B",
" 6 | G P P B R Y G G",
" 5 | B P Y P Y G Y P",
" 4 | G P Y P Y G R P",
" 3 | P G P B B G R B",
" 2 | P G P Y B R R G",
" 1 | B R B Y P Y Y R",
" | a b c d e f g h"
]
}The instructions are the same as before, plus how to read the board and what a move does: a move clears the named tile and every tile of its colour connected to it, tiles above fall, an empty column closes up to the left. The options are nothing but cells:
"criteria": {
"a2": "the group that includes the tile at column a, row 2",
"a10": "the group that includes the tile at column a, row 10",
"d7": "the group that includes the tile at column d, row 7"
}Twenty-three options, one per legal group. Jev has to find each group on the board, see its colour and size, and imagine what clearing it does. Its answer:
a10 0.12 d4 0.08 d3 0.06
c10 0.09 d7 0.06 the other 18: 0.05 down to 0.02
confidence 0.08That is a shrug. A fair share of 23 options is 0.04 each, and nothing gets more than three times
that. The six purples at d7, the biggest group on the board, got 0.06.
Over three runs of 20 boards the grid scored 77,907. Random play scores 74,715. Jev never cleared a board, and its confidence averaged 0.10 for the whole game, against 0.44 with words.
I tried the same grid as one string with line breaks instead of a list of rows (77,560), and a version that spares Jev the coordinate lookup by writing each move’s label into the grid itself:
Rc Yf Rj G- Y- P- Y- B-
Rc Yf Rj B- G- Y- R- Pv
Rc P- Rj Pn Pn Bt Pv Pv
Gb Gb B- Pn G- Bt R- PvEach tile is its colour and the label of the move that clears it; - means no move clears it
now. The options are a, b, c and so on. That scored 77,440. Same as random.
Is the grid worth anything next to words?
Maybe the grid cannot carry the decision but still helps as context. Two comparisons say no.
| What Jev sees | Points | Picks the biggest group |
|---|---|---|
| The grid, and each option states its colour, size and points | 78,200 | 99 % |
| Colour, size and points only, no grid | 75,593 | 99 % |
| The words from part one | 112,013 | 58 % |
| The words from part one, plus the grid | 95,393 | 85 % |
As soon as an option states its points, Jev takes the biggest, with or without the board, and plays exactly like the always-take-the-biggest rule. And putting the grid beside the good descriptions cost 15 %. TypeSafe’s documentation warns that state unrelated to the decision lowers accuracy. A board Jev cannot read is 96 tiles of unrelated state.
What Jev does see in a grid
I logged the probability of every option on every move, so I can ask what Jev notices. The cleanest test: on the same move, take two groups of the same size, one lying in a row and one standing in a column, and compare what Jev gave them.
| Same move, same size | Row-shaped gets, against column-shaped | On the fresh boards |
|---|---|---|
| Pairs, plain grid | 1.35× | 1.40× |
| Triples, plain grid | 2.9× | 2.0× |
| Pairs, no grid at all (options say only colour and size) | 1.10× | 1.13× |
| Triples, no grid at all | 1.4× | 1.1× |
The bottom rows are the control: a small lean that has nothing to do with seeing a board. The grid
adds to it, and adds more the longer the run. G G G on one line registers. The same letter at
the same place on three lines, which to a person is the same thing, does not. Text goes in as
one sequence, and “directly below” is a whole line away.
In the labelled grid the row advantage for pairs falls back to the control (1.14×), and what counts instead is size: groups of seven or more got 3.8 times a fair share (2.4 on the fresh boards). A big group writes its label many times, and repetition is something a sequence reader can see.
In neither grid did Jev give anything extra to a move that would join a colour into a bigger group (1.00 of a fair share in the plain grid, 0.67 in the labelled one). That is the whole skill of the game, and it needs the board after gravity, which Jev would have to imagine.
Better words
So it has to be words. Two small changes to the prompt from part one:
No digits. TypeSafe says Jev does better with words than numbers, so I wrote every size, score and count as a word: “a large green group, worth many points now”, “strands two more lone tiles”. It scored 105,953, which is 5 % lower, and 6 % lower on the fresh boards. Neither gap is bigger than the boards disagree by, so call it no help. Exact sizes are fine as descriptions. What Jev cannot do is arithmetic, and I was not asking for any.
Instructions by stage. Same facts per move, but the four strategy sentences change with the board: build while more than 60 % of the tiles are left, clear the board when fewer than 30 % are.
"strategy": [
"It is early in the game: build. Prefer a move that joins tiles of one colour into a bigger group to clear later.",
"Do not take the colour with the most tiles yet. Clear small groups of other colours out of its way.",
"Avoid stranding lone tiles."
]That scored 128,447, 15 % more, for the same tokens. Jev reads what the instructions say now. It does not work out for itself that the advice for a full board and for a nearly empty one differ.
The same fact, said two ways
Good SameGame players save the colour with the most tiles. They clear everything else out of its way so it falls together into one huge group, because a group of 30 is worth 8,700 points. Played as a fixed rule, with no model and no judgment (never touch the colour with the most tiles while anything else can be cleared, otherwise pick at random) this scores 127,828. A one-line rule outscores my prompt from part one. Board by board it is closer than the totals (the rule wins 12 of 20), but it stung, and it is the most useful thing I learned.
So I told Jev. First version: every move got one more fact.
"m14": {
"clears": "a medium purple group of 6 tiles for 300 points",
"lone_tiles": "connects 6 lone tiles",
"saved_colour": "takes purple tiles now; purple is the colour with the most tiles, the one worth saving to grow into a single big group"
},
"m20": {
"clears": "a small blue group of 2 tiles for 20 points",
"builds": "leaves a group of 11 on the board",
"saved_colour": "leaves purple, the colour with the most tiles, to keep growing"
}Score: 101,687, 9 % worse than without the fact. The log shows why. Moves that spend the saved colour got 1.40 times their fair share before I added the sentence, and 1.66 times after. I wrote “worth saving” and “a single big group” into those options, and Jev liked them more. It read the good words. It did not take the step from “purple is worth saving” to “so this move, which takes purple, is bad”.
Second version: say nothing on most moves, and on the moves that break the rule say it as a fault.
"m13": {
"clears": "a small purple group of 2 tiles for 20 points",
"builds": "joins purple tiles into a bigger purple group of 9",
"lone_tiles": "connects 2 lone tiles",
"too_early": "spends purple tiles too early: purple is the colour to keep until the end"
}Score: 137,413, 23 % better than without the fact. Those moves now get 0.47 of a fair share. On the first move of C-046191 the three versions answer like this:
| Prompt | Jev’s pick | Probability |
|---|---|---|
| Part one | 6 purples for 300 points | 0.49 |
| “worth saving” | the same 6 purples | 0.42 |
| “too early” | 2 greens that connect 7 lone tiles | 0.33 |
Same board, same fact, same model. From 9 % worse to 23 % better on the wording alone.
With the staged instructions on top, the warning scores 152,120. The search behind the Hint button scores 147,020, and it plays a whole game to the end for every legal move before each move. Board by board the two split, 12 to 8, so call it level. Jev gets there with one question about one move ahead, in a quarter of a second, for $0.0013 a game.
More questions per move
Jev answers many questions in one call, in parallel, so I tried spending more questions on a move. A call with 22 questions came back in about the same quarter of a second as a call with one.
| Strategy | Points | Note |
|---|---|---|
| Part one, one question | 112,013 | |
| The same question three times with the options in three orders, averaged | 114,760 | 2.6× the tokens |
| Three narrow questions (best overall, builds most, safest), multiplied together | 105,087 | worse |
| Every move rated on its own against four described levels, best rating plays | 116,353 | cleared 9.7 boards of 20, the most of any prompt; 3.8× the tokens |
| Two calls: all moves, then only the top four again | 112,007 | twice the wait |
The three-orders run doubles as a test for position bias. The option Jev likes keeps most of its lead when the list is reversed (1.55 times a fair share listed first, 1.21 listed last), so being first is worth about 1.3 times. Real, small, and averaging it away bought 2 % here and nothing on the fresh boards.
Rating each move on its own is the careful player: it scores less from big groups but leaves the fewest tiles. I tried switching to it for the last third of a game after the saving prompts. No gain (157,653 and 147,060). By then the board is decided.
Code applies the rule, Jev judges the rest
TypeSafe’s advice is that code stays in control and the model gets the narrow judgment. The saving rule is something code can apply exactly. So: while any other colour can be cleared, code removes the moves that spend the saved colour, and Jev chooses among what is left, with the staged instructions.
| Player | Points |
|---|---|
| The rule alone, random among the moves it allows | 127,828 |
| The rule alone, biggest among the moves it allows | 118,080 |
| Search (the Hint button) | 147,020 |
| The rule in code, Jev chooses among the rest | 146,993 |
| The rule in code, Jev chooses, staged instructions | 161,047 |
Jev’s judgment adds 26 % on top of the random version of the rule and 36 % on top of the take-the-biggest version, winning 18 of 20 boards against the latter. Against the search the total is 10 % higher, but the boards split ten to ten: the lead comes from the four Expert boards, 80,180 against the search’s 56,640, where saving a colour pays enormously. It is a lopsided player though: it hoards one colour for a giant group and almost never clears a board, where the warning version, which lets Jev decide when to cash in, clears about five of twenty.
The fresh boards were kinder to the rule. There, the rule alone taking the biggest of the rest scores 152,760, which already outscores the search, and Jev choosing among the rest reaches 183,440, another 20 %. The staged instructions stop mattering once the rule is in code: 10 % better on these boards, 3 % worse on the fresh ones.
All 21, in one table
Mean of three runs of the 20 boards. “Cleared” is boards cleared out of 20, “confidence” is Jev’s own, averaged over every decision.
| Strategy | Points | vs part one | Cleared | Confidence |
|---|---|---|---|---|
| Rule in code + Jev + staged instructions | 161,047 | +44 % | 0.3 | 0.42 |
| The same, rating each move for the last third | 157,653 | +41 % | 0 | 0.42 |
| “Too early” warning + staged instructions | 152,120 | +36 % | 4.7 | 0.36 |
| The same, rating each move for the last third | 147,060 | +31 % | 5.3 | 0.37 |
| Search, no model | 147,020 | 16 | ||
| Rule in code + Jev | 146,993 | +31 % | 0.3 | 0.44 |
| “Too early” warning | 137,413 | +23 % | 2.7 | 0.34 |
| Staged instructions | 128,447 | +15 % | 6.7 | 0.44 |
| The saving rule alone, random among the rest | 127,828 | 0.1 | ||
| “Worth saving” fact + staged instructions | 118,740 | +6 % | 2 | 0.38 |
| Every move rated on its own | 116,353 | +4 % | 9.7 | 0.67 |
| Three option orders averaged | 114,760 | +2 % | 5.7 | 0.44 |
| Part one: one-move-ahead facts in words | 112,013 | 4.3 | 0.44 | |
| Two calls, shortlist of four | 112,007 | 0 % | 4 | 0.35 |
| No digits | 105,953 | −5 % | 4 | 0.37 |
| Three narrow questions multiplied | 105,087 | −6 % | 2.7 | 0.47 |
| “Worth saving” fact | 101,687 | −9 % | 2.3 | 0.38 |
| Words plus the grid | 95,393 | −15 % | 3 | 0.37 |
| Grid, options state size and points | 78,200 | −30 % | 0.7 | 0.32 |
| Grid, options are cells | 77,907 | −30 % | 0 | 0.10 |
| Grid as one string | 77,560 | −31 % | 0.3 | 0.09 |
| Labelled grid | 77,440 | −31 % | 0 | 0.16 |
| Always the biggest group, no model | 77,420 | 0 | ||
| Size and points only, no grid | 75,593 | −33 % | 0 | 0.43 |
| Random, no model | 74,715 | 0.1 |
Fresh boards
Twenty boards I had never looked at, 16 Classic and 4 Expert, three runs each, nothing changed in any prompt. This is the table to trust.
| Strategy | Points | vs part one | Cleared | Confidence |
|---|---|---|---|---|
| Rule in code + Jev | 183,440 | +71 % | 1 | 0.45 |
| Rule in code + Jev + staged, rating each move for the last third | 180,040 | +68 % | 1.7 | 0.42 |
| “Too early” warning + staged instructions | 178,907 | +67 % | 4 | 0.36 |
| Rule in code + Jev + staged instructions | 177,860 | +66 % | 1.7 | 0.42 |
| “Too early” + staged, rating each move for the last third | 169,940 | +59 % | 1.3 | 0.38 |
| The saving rule alone, biggest among the rest | 152,760 | 0 | ||
| “Too early” warning | 147,067 | +37 % | 2.3 | 0.35 |
| Search, no model | 146,500 | 18 | ||
| The saving rule alone, random among the rest | 139,445 | 0.1 | ||
| Staged instructions | 130,440 | +22 % | 4.7 | 0.43 |
| “Worth saving” fact + staged instructions | 118,487 | +11 % | 3.3 | 0.39 |
| Two calls, shortlist of four | 110,600 | +3 % | 3.7 | 0.36 |
| Every move rated on its own | 109,040 | +2 % | 6.3 | 0.67 |
| Part one: one-move-ahead facts in words | 107,100 | 4 | 0.46 | |
| Three option orders averaged | 105,333 | −2 % | 6 | 0.44 |
| “Worth saving” fact | 104,380 | −3 % | 2.7 | 0.37 |
| No digits | 100,873 | −6 % | 5 | 0.37 |
| Three narrow questions multiplied | 94,100 | −12 % | 1.3 | 0.46 |
| Words plus the grid | 90,973 | −15 % | 3 | 0.37 |
| Grid, options are cells | 83,367 | −22 % | 0.3 | 0.10 |
| Grid, options state size and points | 79,580 | −26 % | 0.3 | 0.30 |
| Grid as one string | 79,133 | −26 % | 1 | 0.10 |
| Random, no model | 78,468 | 0.4 | ||
| Size and points only, no grid | 77,100 | −28 % | 0.3 | 0.42 |
| Always the biggest group, no model | 76,080 | 0 | ||
| Labelled grid | 75,953 | −29 % | 1 | 0.16 |
What held up: every grid plays like random (random’s eight runs ranged from 72,260 to 86,980); the grid beside the words costs 15 %; “too early” beats “worth saving” for the same fact, 147,067 against 104,380; staged instructions are worth about 20 %; and the top group outscores the search by more than 20 %. What did not: “worth saving” is no longer worse than saying nothing, just no better (104,380 against 107,100); the top four are within each other’s run-to-run range, so I cannot say which of them is best; averaging option orders gained nothing; and rating each move on its own cleared 6.3 boards, still the most, but no longer by much.
How sure am I?
Twenty boards is not many, and the four Expert boards carry 40 % of the points, so totals can mislead. I compared strategies board by board and resampled the boards 10,000 times to put a 95 % interval on each ratio of totals. First boards, then fresh boards:
| Claim | Boards won | Ratio of totals |
|---|---|---|
| Words beat the grid | 19 of 20, 18 of 20 | 1.43× and 1.28× |
| The grid is no better than random | 12, 11 | 1.04× and 1.06×, both intervals include 1 |
| “Too early” beats “worth saving” | 18, 18 | 1.35× [1.22 to 1.49], 1.41× [1.27 to 1.53] |
| Staged instructions beat fixed ones | 17, 15 | 1.15× [1.06 to 1.24], 1.22× [1.13 to 1.32] |
| Warning + staged beats part one | 19, 17 | 1.36× [1.26 to 1.46], 1.67× [1.34 to 2.04] |
| Jev on top of the rule beats the rule alone | 18, 16 | 1.36× [1.18 to 1.56], 1.16× [1.07 to 1.28] |
| Warning + staged against the search | 12, 11 | 1.03× [0.93 to 1.13], 1.22× [1.02 to 1.42] |
| Rule + Jev + staged against the search | 10, 10 | 1.10× [0.89 to 1.30], 1.21× [0.94 to 1.47] |
So the last two rows are the ones to hold loosely. Against the search the best prompts win about half the boards and score more in total, mostly on Expert boards. “Keeps pace with a search, with no lookahead” is what the data supports. “Beats it” is not.
The data itself I am sure of. A second implementation of the rules, written in Python and sharing
nothing with the benchmark, replays all 2,520 logged games from the move logs and arrives at every
published total, and all 52,843 logged picks and probabilities match the raw API responses. Both
checks are in the repository as verify.py and compare.py.
What this says about how Jev was trained
TypeSafe says Jev is trained with what it calls reinforcement learning for calibrated decisions,
to return decisions and probabilities instead of text. It publishes nothing about the data, the
size or the architecture. So everything in this section is my guess from a day of poking at one
version of it (jev-1.13.0) with one game.
It judges what is said. It does not work things out. Reading a SameGame board means finding connected groups, then imagining gravity. Jev gave a move that would join a colour exactly a fair share when it had to see that on the board, and five times a fair share when a sentence said so. I think it answers in one pass, with nowhere to do intermediate work. That is what “System One” means, and it is a real limit: anything that needs a step of simulation has to happen in code.
Its uncertainty is honest. With the grid its confidence was 0.10 from the first move. With words it was 0.44. It told me it could not read the board before the scores did, which is what calibration training should buy. But confidence is about telling the options apart, not about being right: with only sizes and points in the options it was 0.43 confident while playing no better than random. It was sure which group was the biggest. The question was wrong, not the answer.
It learned from text that reads left to right. Equal letters next to each other register, more so the longer the run. Equal letters stacked on successive lines do not, and repeated labels register as repetition. I would guess a language model underneath that has read a lot of prose and records and very little that is laid out in two dimensions.
An option’s description is read as the case for choosing it. TypeSafe’s own examples describe
an option by what belongs in it: billing is “payments, invoicing, refunds”. If most of its
training looks like that, a description is a list of reasons to pick the option, and “worth saving
to grow into a single big group” is a reason, whatever I meant. A reason against has to be
written as a fault, under a name like too_early or warning. This one is worth remembering for
any Choice question: never put praise for something else inside an option you want avoided.
Instructions are followed as written, for now. Staging the instructions was worth 15 % with identical facts. Jev does not infer that advice for a full board expires. TypeSafe calls this literal reading. The fix is cheap: let code notice the situation and swap the sentences.
Extra state costs. The unreadable grid beside good words cost 15 %. The “worth saving” sentence repeated on every option lowered confidence from 0.44 to 0.38. It behaves like a model trained on short states where everything present is relevant.
Numbers are fine as facts, not as work. It ranks “for 720 points” above “for 20 points” every time, and taking the digits out did not help. It will not add, count or compare dates, but it can read a number someone else computed.
Questions are cheap once the state is read. Twenty-two questions took the same time as one. That fits a design where the state is processed once and each question is scored against it, which would also explain why output tokens are free: there is no output to generate.
What samegame.app plays now
The Jev button now uses the winner that code and Jev share: the saving rule applied in code, Jev choosing among the rest, instructions by stage. It is the request the benchmark measured, byte for byte. The panel under the score has a new Log button. It lists every move Jev made on the board: what it picked and the sentences it read about that move, the other moves with their probabilities, the stage of the game, how sure it was, and how many moves of the saved colour the rule held back. You can watch one colour being kept out of play move after move and then cashed in at the end, and copy the whole log as JSON.
If you are building with Jev
- Compute every fact in code and say it in a sentence. Do not send the raw thing and hope.
- Write a drawback as a fault on the option that has it. Say nothing on the others.
- Let code notice the situation and change the instructions. Jev follows them literally.
- If a rule is exact, apply it in code and give Jev only what is left.
- Log the probabilities, not just the pick. The fair-share numbers above found the “worth saving” mistake in a minute; the score alone only told me something was wrong.
- Watch the confidence. A flat 0.1 means Jev cannot see what you are asking about.
Try it
Open board C-0CI3R3 and press ✨ Jev, or the J key. Any board works, on a phone too. Play a few moves yourself first and see whether you agree with where it goes next, then open Log and read what it was told.