# The Small Model Board, 27 small language models measured the same way

Canonical: https://udgamlabs.com/blog/small-model-board · Updated: 2026-10-23

> We tested 27 open language models up to 400M parameters on eight benchmarks with one setup and published the results. On MMLU, 23 do better without answer letters.

Release · 2026-10-23 · By Jitesh K Prasad

When a new small language model comes out, its card usually compares it with other models using numbers measured
with different tools, different settings and different versions. Those numbers are hard to compare. Before we
released Laghu, we wanted a fair comparison, so we tested 27 open language models of up to 400 million parameters
ourselves: same tool, same settings, the same kind of GPU, one session. Today we are publishing all of it: the table, the
scores behind it, and what it shows about the MMLU benchmark.

> **The short version.** 27 models, 8 benchmarks, 1,244,646 scored answers, measured on 19 October 2026. Our own models are in it like any other: Laghu-Base is 5th of the 19 models up to 200M. On MMLU, 23 of the 27 models score higher when the question is asked without answer letters.

## What we released

- **[The Small Model Board](https://huggingface.co/spaces/udgamlabs/small-model-board)**, a sortable table of all 27
  models in two size groups (up to 200M parameters, and 200M to 400M). Every model links to the exact version we
  tested.
- **[The scores](https://huggingface.co/datasets/udgamlabs/small-model-board-scores)**: for each model and each of
  46,098 test questions, whether it got the question right and which answer it picked. That is 1.15 million rows, with
  no question text. Two models, MobileLLM-R1-140M-base and nltower-100m-base, have summary scores only, for licence
  reasons.
- **The method and its limits**, written out on the board itself, including a section on where the board falls short.

## How we measured

- **One tool, one setup.** Every model ran through lm-eval (version 0.4.13), the open-source test harness from
  EleutherAI, on the same kind of GPU, with the same settings and no chat formatting.
- **Exact versions.** Models on Hugging Face change over time, so we recorded the exact version of each one and link to
  it. Ten models needed their own code to run; we read that code before running it.
- **Eight benchmarks:** ARC-Easy and ARC-Challenge (school science questions), HellaSwag (picking a sensible ending to
  a sentence), PIQA (everyday physical common sense), WinoGrande (working out who or what a sentence refers to), MMLU
  (exam questions in 57 subjects, asked in two ways, see below) and GSM8K (school maths word problems). The board's
  main column is the average of the first five.
- **Checked twice.** After the run, we re-scored every one of the 1,244,646 answers from the raw logs. All of them
  matched what the test tool had recorded.

## Where our models land

| Size group | Model | 5-task average | Place |
|---|---|---|---|
| Up to 200M | GPT-X3-150M | 53.0 | 1st of 19 |
| | Boris-2-Preview-1 | 52.0 | 2nd |
| | SmolLM2-135M | 51.8 | 3rd |
| | cagliostro-v3.5 | 50.1 | 4th |
| | **Laghu-Base (ours)** | **49.9** | **5th** |
| | BananaMind-2-Pro | 49.8 | 6th |
| 200M to 400M | SmolLM2-360M | 59.0 | 1st of 8 |
| | **Gargi-Base (ours)** | **51.7** | **2nd** |
| | **Gargi (ours)** | **51.5** | **3rd** |

Scores are percentages correct. Two models less than about one point apart are not really distinguishable on these
tests, so Laghu-Base is level with 4th and 6th place. SmolLM2-360M is well ahead of Gargi-Base, by 7.3 points. Our
Udgam-384M models are 6th and 7th of the 8 in their group. The full table is on the board.

## What the board shows about MMLU

MMLU can be asked in two ways. In the **letter** format, the model sees the question and four answers labelled A to D,
and we check which letter it finds most likely. In the **cloze** format, it sees only the question, and we check which
of the four full answers it finds most likely. The questions are the same 14,042 in both.

![Dot chart of MMLU cloze score minus letter score for 27 models, with 95% intervals. 23 models sit to the right of zero, from +0.6 (Supra2-100M-Instruct) to +8.3 (SmolLM2-360M); four sit to the left: nltower-100m-base −2.3, Falcon-H1-Tiny-90M-Base −3.8, Falcon-H1-Tiny-90M-Instruct −4.3 and LFM2.5-230M −8.3. Our five models are between +3.5 and +4.8.](https://udgamlabs.com/blog/small-model-board/letter-vs-cloze.png)

*MMLU cloze minus letter, in points, with 95% intervals. Our models in black.*

- **Cloze scores higher for 23 of the 27 models.**
- **The letter format mostly tests whether a model can answer with a letter.** The right answer is "D" for 26.9% of
  the questions, so always answering "D" scores 26.9. In letter format, 23 of the 27 models, including all five of
  ours, score at or below that. Several pick the same letter almost every time.
- **Four models do better with letters:** LFM2.5-230M (by 8.3 points), Falcon-H1-Tiny-90M-Instruct (4.3),
  Falcon-H1-Tiny-90M-Base (3.8) and nltower-100m-base (2.3). The nltower lead shrinks to within noise under a
  different way of scoring the cloze answers, so treat it as borderline.
- **The skill can come from pretraining.** Falcon-H1-Tiny-90M-Base, a base model, already does better with letters.
- **Chat or instruction training did not measurably change the gap** in any of the four base-and-tuned pairs we could
  compare. The scores
  themselves can move: SmolLM2-135M-Instruct scores lower than SmolLM2-135M in both formats.

So when you compare models this small on MMLU, use the cloze format, and check a letter-format score against the 26.9
of always answering "D" before reading it as knowledge. This agrees with the OLMES evaluation standard
([Gu et al., 2024](https://arxiv.org/abs/2406.08446)), which tests both formats because smaller models need the cloze
one. What we add is the measurement, for 27 current models up to 400M.

## Where the board falls short

- These are our own runs, not a certified leaderboard. Other tools, prompts or hardware give different numbers.
- Instruct and chat models were tested as plain text completers, without their chat formatting, which can understate
  them.
- It is dated 19 October 2026. Newer models are not in it, for example Boris-2-Preview-2, released on 21 October.
- Most models score near zero on the maths test, so small differences there mean little. One model, Q-164M, has a
  built-in calculator, so its maths score is tool-assisted.
- We did not check any model for contamination. These benchmarks are public and old.

## Try it

- The board: [huggingface.co/spaces/udgamlabs/small-model-board](https://huggingface.co/spaces/udgamlabs/small-model-board)
- Every score, to check our numbers or run your own comparisons:
  [huggingface.co/datasets/udgamlabs/small-model-board-scores](https://huggingface.co/datasets/udgamlabs/small-model-board-scores)
- Our models on it: [Laghu](https://udgamlabs.com/models/laghu) and [Gargi](https://udgamlabs.com/models/gargi)

Thanks to Dipankar Sarkar, who raised the letter-versus-cloze point in the comments on our Gargi post on Hugging Face.

---

## About Udgam Labs

Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Today that means open models such as Gargi, a 384M-parameter language model that comes within 2 points of Qwen2.5-0.5B on three reasoning benchmarks with about 1,000× less training data. Next come products, custom models for businesses, and AI in the physical world. Udgam (उद्गम) is Sanskrit for origin.

Udgam Labs · AI research from first principles · https://udgamlabs.com