A thousand minds for a cent and a half
What a small judgment model can and cannot do as the mind of a simulated city. Nineteen findings from building one on TypeSafe's Jev, the failures included.
An independent experiment, not affiliated with TypeSafe. Model jev-1.13.0, September 2026.
Every number on this page is measured from logs in the project's repository and can be reproduced; where something
is a projection or was not run, it says so. Conceived and directed by elberacasa, who set the goals, the standards and every product and money decision; engineered with Claude (Anthropic's Fable 5.1) working in Claude Code.
Jev is not a chat model. It does not write. You give it a situation and a question with a fixed set of answers, and it returns a probability for each, in a few hundred milliseconds, for a price that rounds to nothing. One request can carry hundreds of independent questions about the same situation.
That shape suggested an experiment: make it the mind of a society. In Jevtown, code owns everything that can be computed (time, hunger, food, fire, injury, gossip, death, memory) and Jev supplies only judgment: for each inhabitant, each tick, the probability of each thing they might do next. Code samples an action from those probabilities and the world moves on. We ran it as a village of 80 and as a city of 1,000.
What worked
Scale is nearly free
A city tick is ten parallel requests of 100 citizens. Over a live session of 140 ticks and 60,258 decisions, the whole tick took 980 ms at the median (p95 1,321 ms) and cost $0.0073 on average, because citizens are only asked again when their situation changes. Asking four judgments per citizen instead of one cost 31% more tokens, not four times more, because the situation is shared. A thousand yes/no character judgments times eight took 1.9 seconds and $0.0117.
People stay in character, measurably
Against a hand-written rule policy and a random policy behind the same interface (five seeds each, ten village days, 95% confidence intervals), the rate at which inhabitants with different characters act differently, which we pre-registered as "individuality", was three times higher under Jev, with nobody dying in either.
Individuality: do different people act differently? (80 villagers, hard-week, 5 seeds)
All five pre-registered character effects were significant (p < 0.01) in five of five seeds: the cowardly flee at 0.96 against 0.41 for everyone else, the brave fight at 0.35 against 0.06, guards fight at 0.37 against 0.06, the devout pray at 0.27 against 0.01, thieves steal at 0.19 against 0.01. Three effects we did not pre-register (lazy, hardworking, greedy) point the expected way and do not reach significance; we report them as such.
Every ingredient of a person moves the answer
We replayed logged moments and asked again with one ingredient removed. Every removal shifted answers well beyond what asking the identical question twice does.
How far answers move when one thing is removed (mean total variation distance; 1,947 decisions)
A jury you can predict, because it is made of people
When a citizen is caught stealing, code seats twelve jurors, putting people tied to the accused or the victim first, and Jev judges each juror once: does this person vote guilty? We tested it before building it, as a 96-cell factorial with hypotheses and pass marks committed to the repository before the first request.
Shift in the probability of a guilty vote, everything else held fixed
With plain answer criteria the jury leaned guilty (0.59 on one person's word). We searched four wordings, chose one by a rule fixed in advance, and tested it on a different case it had never seen. A reasonable-doubt rule written into the "no" criterion passed four of five checks there: strong evidence +0.205, an incriminating find +0.126, rivals against friends 0.24, and a jury that is not a rubber stamp (0.425 on weak evidence). The alibi still failed: −0.067 against a required −0.10. So the game as shipped does not let the crowd submit evidence for the defence, because the model would mostly ignore it.
What broke, and what each failure taught
These are the part we think is most useful to anyone building on a judgment model. None of them raised an error. Each one produced confident, well-formed, wrong behaviour, and each was found by looking at what a whole population did, then replaying the exact moment and changing one field.
| What we did | What happened | What fixed it |
|---|---|---|
| Told Jev each villager's current action | 80 of 80 chose it again, at 0.92 confidence. The village froze. | Remove the field. Continuity belongs in code. |
| Described what the crowd nearby was doing | 73 of 78 joined in, including 77 who were starving or exhausted. By day 3 the whole village was praying in the chapel. | Remove the field: 1 of 78. |
| Relied on "hunger: starving" in the state | A calm, rested, starving guard chose work over eating. | Say the priority in the answer's criteria, in words. |
| Let a famine run | Starving farmers answered "eat" (0.81) when there was no food, never "farm". 34.6 ± 26.0 of 80 died; hand-written rules lost 0.2. Jev answers the one-step question; "no food, so I should farm" is two. | Code withdraws impossible options; the option text states the missing link. 80 of 80 alive in 5 of 5 seeds. |
| Screened several chat messages in one request | A harmless line scored 0.41 for abuse when it sat next to an abusive one, and was blocked. A troll could silence a neighbour. | One message per request. Harmless shown 20 of 20, harmful blocked 17 of 18. |
| Pointed at villagers by list position | See below. This one invalidated a whole evaluation. | Key collections by name. |
The most expensive mistake: villagers[37]
The documentation says Jev cannot count. We did not see that an array index is a count. Same 80 villagers, same question; the measured gap between greedy and loyal villagers, by where they sat in the list:
Does Jev still tell people apart? Gap between greedy and loyal, by list position
We had run a complete pre-registered evaluation this way. We re-ran it with names and published both: individuality doubled (0.073 to 0.160), the cowardly fled at 0.96 instead of 0.66, and the flawed referencing turned out to have hidden about a third of the effect of character. We noticed only because a new feature showed the same flat answer for everyone after the first few villagers.
Two smaller cautions
Wording moves answers. Two reasonable phrasings of the same jury question differed by 0.058 on average and by 0.30 at worst. Pin the wording with the model version, and version both.
A label can outweigh a state. In the city, a well-fed thief steals at 0.27 and a starving one at 0.04, because eating comes first. Jev reads "thief" as what this person does. That is defensible, and it is not what a designer who wrote "hunger drives theft" would expect.
Jev is an instrument, not an oracle
To plan this project's launch we pointed Jev at 241 public posts about itself: 1,391 requests, no failures, three cents. Asked small literal questions (does this post state a measured number? is it about computer use?), it was clean and cheap, and turned an impression into counts: 31.5% of posts explain Jev, 11.6% are computer use, 4.6% are a game or a simulated world. Asked directly which of two real posts did better, it was at chance: 43.8% against likes per view, 53.8% against likes (260 pairs, both orders). The same judgments used as features of a fitted model, scored only on posts the model had not seen, reached 66.8%, against 64.3% from media type and length alone. Jev's own favourite among our drafts came last in the fitted model. Ask Jev small questions at scale; let something fitted, or someone, decide what the answers mean.
Rules we would give anyone building on Jev
- Code owns everything that is not a judgment. Counting, arithmetic, time, memory, what is possible. If the right answer can be computed, compute it, and withdraw impossible options before asking.
- Key everything by name, never by position.
- Put priorities and decision rules in the answer criteria, in words. State alone is weak.
- Say who an option is for, not who it is not for.
- Never show the model its own last answer, or what the crowd is doing. Both are traps nothing leaves.
- Shared state must hold only what everyone in the request knows. Unrelated detail moves answers, which becomes a fairness bug the moment two people share a request.
- One step per question. If the right answer needs "therefore", code has to supply the link.
- Treat probabilities as probabilities. Sample from them, with a floor for rare harmful actions and a commitment rule so a thousand people do not all change their minds every tick.
- Treat Jev as an instrument, not an oracle. Its judgments are good features and a poor forecast. Fit something on real outcomes.
- Look at populations, not examples. Every failure above was invisible in any single answer and obvious in a tally of eighty.
Reliability, observed rather than staged
The design rule was: one attempt per tick with a timeout shorter than the tick; a failed tick is skipped, never queued; people carry on with what they were doing; a rate limit slows the clock. During a live city session the machine running the server lost its network for four minutes (timeouts, then DNS failures: our side, not TypeSafe's). 36 consecutive ticks were skipped, nothing piled up, the first tick after the network returned was ordinary, and the whole run, outage included, rebuilds from its logs to the identical state hash on all 140 ticks with zero model calls.
Across all 20,851 requests the project made: 20,475 succeeded; 339 got no response, 336 of them in that outage; 37 were rejected with HTTP 400, every one our own doing (requests over the 64k-token limit while we were finding where it was). We never received a rate limit (HTTP 429) or a server error.
Every dollar
Every request the project ever made to Jev went through one client that writes a ledger line: purpose, tokens as counted by the API, cost at the published $0.042 per million input tokens (output is free). As of the evening of 19 September 2026:
| What it paid for | Requests | Input tokens | USD |
|---|---|---|---|
| Evaluations against baselines: seven rounds, five seeds per condition | 12,450 | 219,223,474 | 9.21 |
| The live city: citizens deciding, tick by tick | 3,196 | 102,891,564 | 4.32 |
| Development runs of the 80-villager world | 991 | 22,600,099 | 0.95 |
| Fifteen benchmarks (every finding above) | 1,434 | 16,082,581 | 0.68 |
| Daily character sheets in the live city | 568 | 11,631,938 | 0.49 |
| First headless run of 1,000 citizens | 171 | 5,876,990 | 0.25 |
| Jev researching its own launch: 241 posts annotated, pairs judged, a tournament of drafts | 1,978 | 1,213,974 | 0.05 |
| Juries, chat screening, reading proposals, handle checks, smoke tests | 63 | 79,949 | 0.00 |
| Total | 20,851 | 379,600,569 | 15.94 |
Two things in that table are our own waste, and they are in it on purpose. Evaluation rounds one to five ($6.51) all pointed at villagers by list position and were superseded by round six ($1.87), which is the one reported here. And for a while our test suite made real requests whenever a key was present on the machine: about $0.22 before we caught it in the ledger. A test now fails if any test reaches the API.
Running the city live costs about $3.50 an hour at one tick per eight seconds (projected from the measured $0.0073 per tick plus sheets), whether one person is watching or ten thousand, and nothing when nobody is.
Method, and what this does not show
The evaluation plan, its metrics and its pass marks were committed before the runs and only amended with dated notes. Baselines sit behind the same interface as Jev. Conditions ran on five seeds and are reported with 95% confidence intervals. Benchmarks that choose a wording do so on one case and are confirmed on another. Failed hypotheses are reported next to the ones that passed.