Gargi, a 384M language model trained from scratch on about 17B tokens

Most open language models under a billion parameters are pretrained on trillions of tokens. That works, but it makes them hard to reason about: when a 0.5B model has read 18 trillion tokens, how much of what it can do comes from the model and its training, and how much from sheer volume of data?

Gargi asks the opposite question: how far does a small, fixed amount of openly licensed data go? It is named after Gargi Vachaknavi, the Vedic philosopher remembered for asking the hardest questions.

What we built

• Gargi, a 384M-parameter English chat model with an optional thinking mode, and Gargi-Base, the pretrained model before chat tuning.
• Trained from scratch — random initialisation and our own 32,768-token tokenizer — on openly licensed, public-domain and synthetic text. Counting every token of every stage, fine-tuning included, Gargi read about 17 billion tokens.
• The Gemma 4 architecture and chat format, so the same weights run unchanged in transformers, llama.cpp, Ollama, LM Studio and MLX. It shares its architecture and tokenizer with Udgam-384M, but it is a new model trained again from random weights, at a 32,768-token context.
• Builds for laptops: a recommended 8-bit GGUF file of 410 MB, a 238 MB 4-bit file, and MLX builds for Apple silicon.

Training ran in three stages: pretraining on web text, books, papers, public records and synthetic text; continued training on long documents at 32,768 tokens, which gives Gargi-Base; and fine-tuning on chat, refusals and step-by-step maths solutions written by gpt-oss models, shown both with a written thought and without one, so the same weights can answer either way. We checked the pretraining and long-context documents against the evaluation sets and dropped any that shared several passages with them.

The result: data efficiency, with limits

We scored Gargi-Base and four other small base models in one harness (lm-evaluation-harness, 0-shot):

The result is uneven, and it is worth being precise about what it does and doesn't show:

• It is a claim about data, not rank. SmolLM2-360M and Granite 4.0 350M score higher than Gargi-Base almost everywhere. Gargi is not the strongest model of its size; what is notable is reaching this level with far less data.
• Reasoning-style multiple choice comes early; knowledge does not. MMLU, HellaSwag and PIQA reward facts and broad exposure, which come from reading more text. There, the gap to Qwen2.5-0.5B averages 11.9 points.
• Two of the close benchmarks sit near chance (50% on WinoGrande, about 25% on ARC-Challenge), so being level there says less than it seems. On ARC-Easy, Gargi-Base is 1.9 points behind Qwen2.5-0.5B.
• The counts are not like for like. Ours includes every training stage; other developers report pretraining tokens. And part of Gargi's text was written by much larger open models, trained on far more data themselves.

Thinking, and maths

With thinking on, Gargi writes a short thought before answering — mainly on maths problems, averaging 40.5 words on GSM8K. It helps, modestly:

That is far below newer small reasoning models: Qwen3-0.6B is reported at 79.2% by others, with different settings and trained on 36 trillion tokens. Gargi is not a maths specialist, and a plausible-looking thought can hide a wrong step — asked a train's speed for 60 km in 45 minutes, it wrote 60 × 4 instead of 60 ÷ 0.75 and answered 240 km/h (the right answer is 80).

Long inputs

Gargi was trained on 32,768-token inputs. To test recall, we hid a 5-digit code near the start of a long document and asked for it at the end, 30 times at each length:

Six of the eight other small models we tested recall 100% at 24K, including three smaller than Gargi. So we recommend inputs of up to about 16K tokens, and the Ollama setup uses that limit by default. This is also an easy test: it shows recall of a single fact, not understanding of a long document.

Small and fast

On an Apple M6 Mac mini that was also running other jobs, the recommended Q8_0 file generated 273 tokens per second on the GPU and 248 on four CPU threads; the MLX 4-bit build reached 480. Q8_0 stays close to full precision on GSM8K (37.5% against 39.0% on 200 problems); the 238 MB Q4_K_M file loses a few points (32.0%).

Where it falls short

• Facts. It answers factual questions fluently and often wrongly — it told us the capital of Australia is Sydney.
• Follow-ups. It keeps track of what a follow-up question refers to, but its answers are often wrong: 56.7% right with thinking off on 30 checkable follow-ups.
• Maths and long inputs, as above; English only.
• Safety. Gargi had no dedicated safety-training round beyond a small set of refusal examples, and in our tests it may follow harmful requests. Treat it as a research and hobby model: put moderation in front of anything user-facing, and don't use it for medical, legal, financial or safety-critical advice. The try box on our site has its own filter for clearly harmful requests; downloaded copies of the weights do not.

Try it

The quickest way is the try box on the model page, in your browser. To run it locally with Ollama:

ollama run hf.co/udgamlabs/Gargi-GGUF:Q8_0

Ollama turns thinking on by default; /set nothink turns it off. The weights are on Hugging Face as Gargi and Gargi-Base, with GGUF and MLX builds; the GitHub repository has runnable examples and checksums. The technical report has every number above with its settings, plus the architecture, training data and speed measurements in full. Weights and tokenizer are Apache-2.0.