How it works, and how well
Ask a language model for a few hundred random integers and it will not give you random integers. It gives you its habits: numbers it reaches for, numbers it avoids, a shape that survives rewording and is stable enough across sessions to tell models apart. This page describes the method, the two things this site adds on top of it, and the numbers that were measured rather than hoped for.
The protocol
Each challenge asks for between 292 and 333 integers from 1 to 355, chosen one at a time, with no tools and no repairs to the list afterwards. The identify tool hands out 3 such prompts and compares the replies against a bank of 24 models, each represented by 36 recorded answers.
The comparison is the ModelTrace method: each answer becomes a distribution over the 355 values, the distribution is transformed so that sampling noise and prompt wording matter less, and the result is scored against every model's centre. The scoring core and the 36 challenge templates come from ModelTrace unchanged. The versioned bank also includes separately collected fingerprints, with their provenance and limits recorded in the changelog.
Length matters more than wording. Answers of 292 to 333 numbers fit their model with a median strength of 0.398 on the scale the verdict uses; answers of 218 to 257 fit at 0.302 and fall below the verdict's floor six times in 28. That is why every prompt on this site asks for at least 292.
Input gates
The scoring core is a closed-set classifier: whatever it is given, it ranks the bank and picks a winner. Fed a plain counted list from 1 to 300, it named a GPT model with 86% confidence. So before anything is scored, 6 rules decide whether the input is a model's answer at all.
- Nothing parsed, or every number the same.
- Sorted: nine in ten steps go the same direction. Real answers never exceed seven in ten.
- Fixed step: half or more of the steps are the same size. Real answers never exceed a quarter.
- Too few distinct values: fewer than one in twenty. One real answer in the bank goes as low as one in eleven, so the line sits below it.
- Too short to score: fewer than 55% of the requested count, which is the core's own floor.
Every threshold sits outside the most extreme value seen across all 576 real recorded answers, and a test fails if a threshold ever crosses one. The gates reject structure, not randomness: a uniformly random list passes them, because structurally it is indistinguishable from a model's answer. Rejecting it is the verdict's job.
The open-set verdict
The core's confidence number is a share of the bank, so it always adds up to one hundred percent over the models it knows. Uniform random input gets a median "probability" of 78 to 90 percent. That number is not used here.
The result always names the model the core ranked first, because that ranking is the core's own output and, on the bank's own models, it is the part that is right most often. What the verdict adds is a status that says how far ahead that model was, computed from what the core discards: how well the pasted answers agree with the first-ranked model's centre in absolute terms (the fit), and the gap between that fit and the best fit of any other model (the separation). A separation of 0.08 or more is a clear match. Below that, if the family is clear at 0.9 or better, it is a close call: the family is clear, the version is not, and the runner-up is shown. A fit below 0.25, or a fit that passes with neither the model nor its family standing out, is a weak match: the name is only the nearest of the models in the bank, the ranking is not reliable enough to identify the actual model, and the model may not be in the bank at all. The status never reorders the ranking, and the separation is a fit gap, not a difference in the core's own score.
Thresholds were chosen on 3,120 evidence records: held-out answers from the bank's own models, answers from each model scored against a bank rebuilt without it, uniform random input, and answers from two models collected through a different channel than the one in the bank.
What was measured
Only 3 of the 12 prompt environments the bank was built from can be reproduced by pasting into a chat window; the others need a real system field or a prefix of several thousand characters. The table below is grouped numerical cross-validation: three held-out answers from one such environment, verdict applied, over every model in the bank. It does not measure whether every original chat reply passes the page's transport checks. Fresh Kimi replies have additional input-path limitations documented on its model page and in the changelog.
| Prompt style | Runs | Clear match, right | Close call | Weak match | Clear match, wrong |
|---|---|---|---|---|---|
| 中文 | 24 | 19 | 4 | 1 | 0 |
| English, plain text | 24 | 21 | 3 | 0 | 0 |
| English, JSON array | 24 | 18 | 2 | 4 | 0 |
| All | 72 | 58 | 9 | 5 | 0 |
58 of 72 runs named the right model with a clear margin, 9 were close calls inside the right family, and 0 named a wrong model with a clear margin. On the full held-out set across all twelve environments the right model was a clear match in 211 of 288 runs with 0 wrong. Uniform random input was a weak match in 600 of 600 runs and never a clear match: the nearest model is still named, and the status is what tells those runs apart from a real reply.
What it cannot do
An unlisted version of a listed model is reported as its nearest sibling. With Claude Opus 4.7 removed from the bank, 6 of 12 three-answer runs confidently named Claude Opus 4.8. Close versions share a fingerprint, and no threshold separates what the feature space does not. Across all models removed one at a time, 30 of 288 runs still named a sibling. Each model's page states its own figure.
With one answer, roughly one random input in a hundred crosses the fit floor and can be told it belongs to a family. It was never named as a model in 640 tries, and with three answers none crossed the floor at all. Three answers are the recommendation for a reason.
The channel is not determined. Answers from Claude Sonnet 5 and Claude Haiku 4.5 collected through a coding tool were scored against bank entries collected through an API: 19 of 24 three-answer runs named the right model and 0 named a wrong one. The fingerprint follows the model, which is what makes a pasted chat reply recognisable, and it is also why the verdict cannot tell you which interface the reply came from.
It does not measure answer quality. A reply that matches a model says nothing about whether that deployment is quantised, rate-limited or given a shorter context; none of those conditions was tested.
18 of the bank's entries are inherited. Their answers came through an API relay and the serving identity behind the label was never independently confirmed. They behave like the project's own entries in every test; the caveat is about provenance, not accuracy.
Credits
The fingerprint method, the scoring core, the challenge templates and 18 of the bank's fingerprints are from ModelTrace by xqy2006, MIT licensed and used unmodified. The single-number baseline idea is credited to hlwy-ai-checker, which is not used in any form on this site. Everything else here, including the input gates, the open-set verdict and the Claude Fable 5.1 fingerprint, is this project's.