PhoneGPT
A 1.21M-parameter character-level language model, trained live on the CPU of a 2021 Samsung Galaxy F62, in flight mode with no cloud. It started from random weights and learned from about 1 MB of Shakespeare.
The result
It missed nanoGPT’s published 1.4697 by 0.0047.
The target was the best validation loss in the README of nanoGPT, reached on one A100 GPU in about 3 minutes. Our rule, set before the start, was at or below it at two checkpoints in a row. The best checkpoint scored 1.4663, but the closest pair was 1.4744 and 1.4663. Lower is better.
What we trained
- Parameters
- 1,214,976
- Architecture
- nanoGPT-style GPT: 6 layers, 4 attention heads, width 128; it is not nanoGPT itself
- Initialization
- Random weights; no pretrained checkpoint
- Context
- 128 characters
- Data
- The first 90% of Tiny Shakespeare, about 1 MB; no other data and no other model’s weights
- Hardware
- Samsung Galaxy F62 (Exynos 9825), 2 of its CPU cores, plugged in over USB
- Training
- 10 h 45 min, 24,000 steps
- Checkpoints
- About 5 MB each, in the trainer’s own format
The trainer is one C file adapted from Andrej Karpathy’s llm.c (MIT); the text is his Tiny Shakespeare. He is not involved in this. Training ran entirely on the phone; each checkpoint was scored on a Mac.
Measured results
| Step | Hours in | Score |
|---|---|---|
| 16,000 | 7.2 | 1.4782 |
| 17,000 | 7.6 | 1.4744 |
| 18,000 | 8.1 | 1.4663 |
| 19,000 | 8.5 | 1.4748 |
| 20,000 | 9.0 | 1.4785 |
| 24,000 | 10.75 | 1.4817 |
Each score is on every window of the held-out 10% of the text. nanoGPT’s target: 1.4697.
| Measure | nanoGPT’s README | PhoneGPT |
|---|---|---|
| Hardware | One A100 GPU | Galaxy F62, 2 CPU cores |
| Time | About 3 minutes | 10 h 45 min |
| Parameters | 10.65M | 1.21M |
| Context | 256 characters | 128 characters |
| Score | Best of its sampled checks | Every window of the same held-out split |
Run it, or train your own
The checkpoints need the small C programs from GitHub, not Transformers. Download a checkpoint from Hugging Face, build the sampler, and it writes text on your computer. The release post has a step-by-step tutorial for training your own on an Android phone.
A record of the run and a small object for learning: see how a character model improves between checkpoints, or check our scores. Not for any real writing task.
Scope and limitations
- It missed the target. One checkpoint under 1.4697 does not meet the rule we set before the start.
- A toy-scale model with no practical use: it reads and writes single characters, knows one small text, and writes Shakespeare-shaped nonsense.
- 1.4697 comes from a small teaching run, not a measure of a good language model, and a GPU reaches it in minutes.
- 1.4663 is the best of 24 scores on the same text, so it flatters the model a little. From step 18,000 to 24,000 the score rose by 0.0154, which looks like overfitting; we did not test that with another run.
- One run on one phone. Other phones have not been tested.
MIT licence, like the trainer it came from. PhoneGPT is only our project name, with no link to the iPhone app of the same name.
Checkpoints, code and the recording
- Hugging Face
All 25 checkpoints, scores and the run’s log
- GitHub repository
The trainer, sampler and scorer in C
- The stream recording
The live run on YouTube