In 1150 CE the mathematician Bhāskara II wrote Līlāvatī (लीलावती), a book of arithmetic word problems for a young student — merchants and markets, travellers and tanks of water. Today, that book has a new reader.
Lilavati is a 95M-parameter English model, trained from scratch, that answers grade-school maths word problems. It does not do the arithmetic in its head. It writes a short Python program, the program runs in a sandbox, and the value it returns is the answer — shown together with the program, so a person can check the working.
Why write programs
Small models are unreliable at arithmetic, but they can learn to plan it. Python does the arithmetic, so the model only has to get the reasoning right: which quantities matter, and in what order to combine them. That is a much smaller job, and one a 95M model can learn a good part of.
What we built
- Lilavati, the writer: 94.7M parameters, 12 layers, width 640, a 4,096-token context. It starts from random weights — trained from scratch, not fine-tuned from another model.
- Lilavati-Checker, the rater: a second model of the same size, architecture and tokenizer. Given a problem and a program, it estimates how likely the program is to be right.
- Training data: TinyGSM, about 12 million grade-school problems with Python solutions, a TinyGSM-style set with Mixtral-written solutions, and general English web and educational text; then the GSM8K training split and problems generated with large language models and kept only if their solutions executed correctly. No test-set problems were used. We checked every evaluation set for overlap with the training data: none for the GSM8K test or GSM-Hard, and small overlaps with GSM-Plus, SVAMP and ASDiv, which the evaluation notes report in full.
Many tries, and a checker
One greedy try is fast. For harder problems, Lilavati can write many programs at once — each slightly different — and let the answers vote. The checker weighs each vote by how much it trusts that program, and the answer with the most support wins. The package does eight tries by default and reports how strongly the tries agreed.
On the GSM8K test set with 48 tries per problem:
| How the answer is chosen | GSM8K test |
|---|---|
| One greedy try | 54.7% |
| Plain majority vote | 65.7% |
| The checker's single top-rated program | 66.7% |
| Checker-weighted vote (the default) | 72.3% |
| At least one of the 48 programs is right (an upper bound) | 89.5% |
With 16 tries, the checker-weighted vote scores 68.8%.
Results
Every number here is ours, measured with the program executed and its return value compared with the reference answer. With one greedy try:
| Test set | Problems | Score |
|---|---|---|
| GSM8K test | 1,319 | 54.7% |
| SVAMP | 1,000 | 69.9% |
| ASDiv, numeric answers | 2,159 | 74.0% |
| GSM-Hard | 1,319 | 43.7% |
| GSM-Plus testmini, the 7 answerable perturbation types | 2,100 | 39.2% |
We also ran the open models we could find up to 150M parameters through the same GSM8K harness, each answering in text, its native format.
The next-best models were cagliostro-v3.5-chat (146M) at 21.3% on the first 300 problems and Falcon-H1-Tiny-R (90M) at 18.7% on all of them; the rest of the field scored under 7%.
Where it falls short
Lilavati is weakest where a problem tries to trip it up: irrelevant numbers, working backwards, extra steps, unit conversions and some ratios. On GSM-Plus problems with a distracting number inserted, it scores only 20.7%. Three real misses:
| Question | Lilavati | Right answer |
|---|---|---|
| How many seconds are there in 3.5 hours? | 756,000 | 12,600 |
| A recipe for 4 people needs 3 cups of rice. How many cups for 10 people? | 30 | 7.5 |
| After spending half his money on a book and 30 rupees on lunch, Arjun has 70 rupees left. How much did he start with? | 340 | 200 |
In each, only one of eight tries agreed, so Lilavati flagged its answer as unsure. That signal matters: below 0.5 confidence, treat the answer as a guess. A program that runs is not proof that it is correct, and the checker can be confidently wrong when most tries share the same mistake.
Lilavati always returns a number. It is not a chatbot, it handles English grade-school problems only, and it should not be used for grading, financial or medical decisions without a person checking.
Running code safely
Because Lilavati's answers are programs, the package runs them in a restricted sandbox: only arithmetic, loops, lists,
dictionaries and the math module; no other imports, file access or eval; a separate process with CPU, memory and
time limits. It is built for Lilavati's programs, not as a general-purpose sandbox.
Try it
The quickest way is the try box on the model page: type a word problem in your browser, free and without signing in. To run it yourself:
git clone https://github.com/UdgamLabs/lilavati && cd lilavati
pip install -e .
lilavati "A shop sells pencils at 8 rupees each. Riya buys 6 pencils and 2 erasers at 5 rupees each. She pays with a 100 rupee note. How much change does she get?" --show-program
The first run downloads both models, about 190 MB each, with no login, and runs on a laptop CPU. The weights are on Hugging Face as Lilavati and Lilavati-Checker, and the evaluation notes have every number above with its conditions. Weights and code are Apache-2.0.