Blog
Release · 22 October 2026 · 6 min read

How far does $146 go? Laghu, a small chat model trained from scratch

We trained Laghu, a 152M-parameter English chat model, from scratch for $146 of pretraining and released everything, including what failed and what it cost.

The language models most people use are trained on trillions of words, on thousands of chips, at a cost few teams can pay. We wanted to know how far a very small budget goes. So we trained a chat model ourselves, kept the bill, wrote down every step, and are now giving all of it away.

The result is Laghu, a small English chat model. Its main training run cost $146, and the whole project about $194. Laghu (लघु) is Sanskrit for "small, light".

What we built

  • Laghu, a chat model with 152 million parameters (the numbers a model learns during training). It answers in English.
  • A thinking switch. With thinking on, it writes out its reasoning before it answers. That helps a little on short arithmetic; for everything else, off is better.
  • Long inputs of up to about 16,000 tokens (a token is a piece of a word).
  • Trained from scratch: it started from random numbers, with our own vocabulary and our own data recipe, not from someone else's model.
  • Everything open under Apache-2.0: the weights, every checkpoint that survives, the training code, the full recipe, every test answer, what failed and what it cost.

Where the $146 went

Making Laghu took four steps.

  1. Gather text. 52.6 billion tokens from 14 public, openly licensed sources: educational web pages, maths, Wikipedia, permissively licensed code and synthetic text. We removed any document that overlapped the tests we report.
  2. Pretrain. The model reads the text and learns to predict what comes next. This is the expensive part: 13.2 hours on eight A100 GPUs rented in the cloud, for $146.26.
  3. Stretch to long inputs. About four more hours on one GPU, reading about a billion tokens of long books, papers and documents.
  4. Teach it to chat. 41,467 example conversations (everyday questions, follow-ups, maths, writing, tools), shown to it twice.

Everything else (building the data, small test runs, the cloud machines that checked our setup, and the AI model that wrote the example answers and judged the tests) brought the cash total to about $194. About $5 of it, 2.6%, bought nothing we kept: a failed first start and stopped runs.

Watching it learn

Laghu by Udgam Labs.

How do you tell that a model is learning? Every 2,000 training steps, we showed it text it had never seen and measured how surprised it was, in bits per byte: lower means it predicts the text better. The chart is the time-lapse of the run. The line falls steadily, with no spikes. At 80% of the run we switched to a "cool-down" mix with more maths, reasoning and chat text, and the curves kept falling.

Held-out bits per byte during pretraining, lower is better, every 2,000 steps from step 20,000 to the end: new text falls from about 1.03 to 0.875, web text and reasoning text fall alongside it, with no spikes; the cool-down data mix starts at 80% of the run
Held-out bits per byte during pretraining, from step 20,000 (the first quarter used a different training trick and is not comparable).

We saved a copy of the model every 2,000 steps, so that anyone could replay this run. Then, on 21 October, a storage clean-up deleted those copies. That was our mistake. The late checkpoints survive, with everything needed to continue training, and they are published.

What it can do

We tested Laghu against three chat models of its size, with the same questions and the same settings for every model.

  • Chat. A judge model compared Laghu's answers with SmolLM2-135M-Instruct's, side by side, on 324 questions it had never seen. Laghu won 51.0% of the comparisons, after adjusting for answer length: level. On follow-up questions in a conversation it got 41.7% right, against 37.5%, also level.
  • Maths, thinking first. 7.9% of school maths problems (GSM8K) right, against SmolLM2-135M-Instruct's 1.8%.
  • Against our own bigger model. It is level with our 384M Gargi on chat quality and follow-ups, at 40% of the size, and well behind it on maths.
  • New text. Before chat training, Laghu's base model predicts Wikipedia pages and research abstracts published in September and October 2026 better than SmolLM2-135M and Falcon-H1-Tiny-90M-Base: 0.919 bits per byte, against 0.941 and 0.979 (152M parameters against 135M and 91M). SmolLM2-135M was pretrained on about 2 trillion tokens, about 28× Laghu's 71 billion.

That lead is on new text. On five commonsense benchmarks, Laghu's base model trails SmolLM2-135M (49.9 against 51.8). And more data and compute still show in chat: Falcon-H1-Tiny-90M-Instruct, trained with about 12× Laghu's compute and an extra stage that teaches it which answers people prefer, is ahead on all three of our chat tests.

Laghu against size-class rivals, same harness and items. Maths with thinking (GSM8K strict, %): Laghu 7.9, SmolLM2-135M-Instruct 1.8, Falcon-H1-Tiny-90M-Instruct 23.3, MobileLLM-R1.5-140M 0.0. Multi-turn (follow-up right, %): Laghu 41.7, SmolLM2-135M-Instruct 37.5, Falcon-H1-Tiny-90M-Instruct 54.9, MobileLLM-R1.5-140M 3.5
Laghu is the dark bar; Falcon-H1-Tiny-90M-Instruct leads both charts. MobileLLM-R1.5-140M results are reported for noncommercial research under the FAIR Noncommercial Research License; we acknowledge Meta's materials.

What it can't do

  • Maths. 7.9% is low. Its step-by-step reasoning often gets the arithmetic wrong, even when the final answer is right.
  • Following instructions. On IFEval, a test of instructions like "answer in three bullet points", it scores 26.1; Falcon-H1-Tiny-90M-Instruct scores 60.8.
  • Facts. A model this small knows little, and it states wrong facts confidently. Don't use it for facts or advice.
  • Repeating itself. With the most predictable settings, 8.3% of its replies loop until they are cut off. The settings it ships with bring that to 1-2%.
  • Long inputs. It finds a hidden code in 96% of tries at 16,000 tokens, and never at 24,000 or 32,000.

Why we publish everything, including the failures

Before we measured Laghu, we wrote down which tests we would run, which models we would compare it with, and what would count as best in its size class: ahead of every rival in all three chat tests. Laghu won none of the three, because Falcon-H1-Tiny-90M-Instruct is ahead in each. We changed no result. We publish Laghu as a research release, with no claim to be the best.

Other things went wrong too, and the technical report describes each one:

  • Reinforcement learning, a further training stage meant to improve maths, made it no better and made looping worse. The released model does not use it.
  • A validation split leaked. The conversations we held back to check fine-tuning overlapped the training data: about 37% had a near-copy in it. We caught it and chose the final model with a separate, clean set of 200 conversations.
  • The run came in below our projection on new text: 0.921 bits per byte after the long-input stretch, against the 0.900-0.912 we expected.
  • The lost checkpoints above.

Why publish all of this? Because a small model that anyone can rebuild end to end, with its mistakes written down, is more useful to the next person than a single good-looking number. You can check every number in this post against the files we released.

Try it

Laghu stands on openly licensed data: FineWeb-Edu, DCLM-Edu, FinePhrase and FineMath from Hugging Face, Ultra-FineWeb from OpenBMB, SYNTH from PleIAs, NVIDIA's Nemotron datasets, Wikipedia and permissively licensed code from GitHub, and on Andrej Karpathy's nanochat, which our training code is built on. Thank you to everyone who built and shared them.