The Small Model Board, 27 small language models measured the same way

When a new small language model comes out, its card usually compares it with other models using numbers measured with different tools, different settings and different versions. Those numbers are hard to compare. Before we released Laghu, we wanted a fair comparison, so we tested 27 open language models of up to 400 million parameters ourselves: same tool, same settings, the same kind of GPU, one session. Today we are publishing all of it: the table, the scores behind it, and what it shows about the MMLU benchmark.

What we released

• The Small Model Board, a sortable table of all 27 models in two size groups (up to 200M parameters, and 200M to 400M). Every model links to the exact version we tested.
• The scores: for each model and each of 46,098 test questions, whether it got the question right and which answer it picked. That is 1.15 million rows, with no question text. Two models, MobileLLM-R1-140M-base and nltower-100m-base, have summary scores only, for licence reasons.
• The method and its limits, written out on the board itself, including a section on where the board falls short.

How we measured

• One tool, one setup. Every model ran through lm-eval (version 0.4.13), the open-source test harness from EleutherAI, on the same kind of GPU, with the same settings and no chat formatting.
• Exact versions. Models on Hugging Face change over time, so we recorded the exact version of each one and link to it. Ten models needed their own code to run; we read that code before running it.
• Eight benchmarks: ARC-Easy and ARC-Challenge (school science questions), HellaSwag (picking a sensible ending to a sentence), PIQA (everyday physical common sense), WinoGrande (working out who or what a sentence refers to), MMLU (exam questions in 57 subjects, asked in two ways, see below) and GSM8K (school maths word problems). The board's main column is the average of the first five.
• Checked twice. After the run, we re-scored every one of the 1,244,646 answers from the raw logs. All of them matched what the test tool had recorded.

Where our models land

Scores are percentages correct. Two models less than about one point apart are not really distinguishable on these tests, so Laghu-Base is level with 4th and 6th place. SmolLM2-360M is well ahead of Gargi-Base, by 7.3 points. Our Udgam-384M models are 6th and 7th of the 8 in their group. The full table is on the board.

What the board shows about MMLU

MMLU can be asked in two ways. In the letter format, the model sees the question and four answers labelled A to D, and we check which letter it finds most likely. In the cloze format, it sees only the question, and we check which of the four full answers it finds most likely. The questions are the same 14,042 in both.

• Cloze scores higher for 23 of the 27 models.
• The letter format mostly tests whether a model can answer with a letter. The right answer is "D" for 26.9% of the questions, so always answering "D" scores 26.9. In letter format, 23 of the 27 models, including all five of ours, score at or below that. Several pick the same letter almost every time.
• Four models do better with letters: LFM2.5-230M (by 8.3 points), Falcon-H1-Tiny-90M-Instruct (4.3), Falcon-H1-Tiny-90M-Base (3.8) and nltower-100m-base (2.3). The nltower lead shrinks to within noise under a different way of scoring the cloze answers, so treat it as borderline.
• The skill can come from pretraining. Falcon-H1-Tiny-90M-Base, a base model, already does better with letters.
• Chat or instruction training did not measurably change the gap in any of the four base-and-tuned pairs we could compare. The scores themselves can move: SmolLM2-135M-Instruct scores lower than SmolLM2-135M in both formats.

So when you compare models this small on MMLU, use the cloze format, and check a letter-format score against the 26.9 of always answering "D" before reading it as knowledge. This agrees with the OLMES evaluation standard (Gu et al., 2024), which tests both formats because smaller models need the cloze one. What we add is the measurement, for 27 current models up to 400M.

Where the board falls short

• These are our own runs, not a certified leaderboard. Other tools, prompts or hardware give different numbers.
• Instruct and chat models were tested as plain text completers, without their chat formatting, which can understate them.
• It is dated 19 October 2026. Newer models are not in it, for example Boris-2-Preview-2, released on 21 October.
• Most models score near zero on the maths test, so small differences there mean little. One model, Q-164M, has a built-in calculator, so its maths score is tool-assisted.
• We did not check any model for contamination. These benchmarks are public and old.

Try it

• The board: huggingface.co/spaces/udgamlabs/small-model-board
• Every score, to check our numbers or run your own comparisons: huggingface.co/datasets/udgamlabs/small-model-board-scores
• Our models on it: Laghu and Gargi

Thanks to Dipankar Sarkar, who raised the letter-versus-cloze point in the comments on our Gargi post on Hugging Face.