Lilavati
Pocket Mathematician
A 95M-parameter model, trained from scratch, that solves maths word problems by writing a short Python program and running it.
Try it now · free, no login
Ask a word problem. Get the answer and the program.
One maths word problem with numbers, in English. It is not a chatbot.
Answer
The answer appears here, with the Python program that produced it.
Try one of these
Answers come from a tiny research model and can be wrong. It shows the program behind every answer, so you can check the working. Best at everyday word problems; weak at numbers that should be ignored, working backwards and unit conversions. Free and without login, with a fair-use limit per visitor. Every program runs in a restricted sandbox on our own machine, so it may be slow or offline at times.
The result
Good at everyday word problems. Weak at traps, units and working backwards.
On GSM8K, the standard test of grade-school maths word problems, Lilavati solves 54.7% on its first try, with its program executed. When Lilavati-Checker, a second 95M model, weighs 48 tries, it solves 72.3%. It is weak at numbers that should be ignored, working backwards and unit conversions, and it always returns a number, even when a question has no answer.
How it answers
Small models are unreliable at arithmetic, but they can learn to plan it. Lilavati writes the plan as a short Python program. Python does the arithmetic, so the model only has to get the reasoning right.
- 01
Read the question
Lilavati reads an English maths word problem.
- 02
Write a program
It writes a short Python function whose return value is the answer.
- 03
Run it in a sandbox
The program passes a structure check (maths-only building blocks, no file or network access) and runs in a restricted sandbox with time and memory limits.
- 04
Read the answer
The return value is the answer. The program comes with it, so you can check the working.
Not sure? It writes many programs.
Lilavati can write several programs for the same question and run each one. Lilavati-Checker, a second 95M model, reads the question and each program with its result, and rates how likely it is to be right. Each answer gets the sum of the ratings of the programs that produced it, and the answer with the most support wins.
Over 48 tries, that lifts GSM8K from 54.7% to 72.3%. The try box above uses 8 tries in Best mode and one in Fast mode.
How good is it
- Parameters
- 95M
- Trained from scratch. Small enough to run on a laptop CPU
- GSM8K, first try (code executed)
- 54.7%
- One greedy try. The program is run and its return value is compared with the reference answer
- GSM8K, with the checker over 48 tries
- 72.3%
- Lilavati-Checker weighs 48 programs per problem. With 16 tries: 68.8%
| Test set | Items | Lilavati |
|---|---|---|
| GSM8K test | 1,319 | 54.7 |
| SVAMP | 1,000 | 69.9 |
| ASDiv, numeric answers | 2,159 | 74.0 |
| GSM-Hard | 1,319 | 43.7 |
| GSM-Plus testmini, the 7 answerable perturbation types | 2,100 | 39.2 |
| How the answer is chosen | Score |
|---|---|
| Checker-weighted vote (the default) | 72.3 |
| The checker’s single top-rated program | 66.7 |
| Plain majority vote | 65.7 |
| At least one of the 48 programs is right (an upper bound) | 89.5 |
With 16 tries, the checker-weighted vote scores 68.8.
All numbers were measured by us. “Code executed” means the model’s program is run and its return value is compared with the reference answer. No test-set problems were used for training; the model card has a contamination audit for every test set.
Compared with other open models up to 150M parameters
On GSM8K it solves 54.7% on its first try: about 2.7× the best open model of its size we tested.*
| Model | Params | GSM8K test | Notes |
|---|---|---|---|
| Lilavati | 95M | 54.7 | Code executed. 57.3 on the first 300 items |
| cagliostro-v3.5-chat | 146M | 21.3 | First 300 items |
| Falcon-H1-Tiny-R-90M | 90M | 18.7 | All items, 8K-token cap. SVAMP: 40.1 (Lilavati 69.9) |
| MobileLLM-R1.5-140M | 140M | 6.7 | First 300 items; 99% of its answers ran past our 8K-token cap. Its developers report 8.3 |
| GPT-X3-150M | 150M | 2.3 | |
| GPT-X2.5-135M | 135M | 1.7 | |
| Supra2-100M-Instruct | 100M | 1.6 | |
| JugnuLM-110M-R2plus | 110M | 1.3 |
Other models answer in text, their native format, and are scored on their final number.
Published results for models whose weights are not released: TinyGSM (Liu et al., 2023, arXiv:2312.09241) reports a 125M model, first pretrained on Phi-1.5 data, at 63.1% with one try and 68.9% when a 125M verifier picks one of 48 programs. Under that exact protocol (the checker’s single top pick), Lilavati scores 66.7%. TinyGSM’s 125M model trained from random initialisation reports 53.1%.
What it is good at, and what it is not
Where it does well
Grade-school word problems: shopping and change, rates and speeds, percentages, discounts and tax, averages, simple interest, areas and perimeters, and multi-step arithmetic.
Where it is weak
- Irrelevant information: a number that should be ignored often gets used anyway.
- Working backwards, and problems that add an extra step to a familiar pattern.
- Unit conversions, and some ratio questions.
- Unanswerable questions: it always returns a number, even when the question has no answer.
- Anything that is not a maths word problem: it is not a chatbot and has little general knowledge.
Every answer comes with its agreement (the share of tries that gave it) and the checker’s confidence. When either is below half, treat the answer as a guess; the try box then says it is not sure.
Limitations
- Not for grading, financial or medical calculations, or anything where a wrong number could cause harm, without a person checking the program. A program that runs is not proof that it is correct.
- It works best on English grade-school maths word problems. Expect lower accuracy on other styles, longer problems and other languages.
- Never run its programs without a sandbox. The
lilavatipackage checks and sandboxes every program for you. - Intended for research and education on small reasoning models, program-aided maths, offline maths help on small devices, and as a starting point for fine-tuning.
Three things it gets wrong
Unit conversion
“How many seconds are there in 3.5 hours?”
It said 756,000; the answer is 12,600. Only 1 of 8 tries agreed, so it said it was not sure.
Ratio
“A recipe for 4 people needs 3 cups of rice. How many cups of rice are needed for 10 people?”
It said 30; the answer is 7.5. Only 1 of 8 tries agreed, so it said it was not sure.
Working backwards
“After spending half of his money on a book and then 30 rupees on lunch, Arjun has 70 rupees left. How much money did he have at first?”
It said 340; the answer is 200. Only 1 of 8 tries agreed, so it said it was not sure.
The name
Līlāvatī (लीलावती)
Bhāskara II · 1150 CE
In 1150 CE, the mathematician Bhāskara II wrote Līlāvatī, a book of arithmetic word problems for a young student: merchants and markets, travellers and tanks of water.
Today, that book has a new reader.
The model
- Parameters
- 94.7M · 12 layers, width 640
- Initialisation
- Random weights; trained from scratch
- Training data
- Public synthetic maths datasets with Python solutions, general English text, and problems generated with large language models and verified by executing their solutions.
- Input → output
- An English word problem → a Python function whose return value is the answer
- Context
- 4,096 tokens
- Checker
- Lilavati-Checker (95M) rates each program so that many tries can be combined
- Licence
- Apache-2.0 (weights and code)
How to run it
It is small enough to run on a laptop CPU. The lilavati package writes the programs, runs them in a restricted sandbox and combines the answers.
pip install torch transformers
git clone https://github.com/UdgamLabs/lilavati && cd lilavati && pip install -e .
# from Python
from lilavati import Lilavati
lv = Lilavati() # downloads UdgamLabs/Lilavati and UdgamLabs/Lilavati-Checker
r = lv.solve("A shop sells pencils at 8 rupees each. Riya buys 6 pencils and 2 erasers at 5 rupees each. "
"She pays with a 100 rupee note. How much change does she get?")
print(r["answer"]) # 42
print(r["agreement"]) # share of tries that gave this answer
print(r["program"]) # the program behind the answerlv.solve(q, tries=1)is one greedy try: fastest, no checker.lv.solve(q, tries=8)(the default) writes 8 programs, runs them all, and picks the answer with the most checker-weighted support. More tries are slower but more accurate.
Weights, code and the evaluation
- Lilavati on Hugging Face
Weights and model card
- Lilavati-Checker on Hugging Face
The 95M model that rates programs
- GitHub repository
Code, sandbox and evaluation
Weights and code are released under Apache-2.0.