We wanted to find out how far a small budget goes for a video model, so we built one and kept notes. This post is those notes: what we tried, what worked, what didn't, and what it all cost. We are releasing almost everything, failures included.
The result is Drishya Tiny (Drishya means "a scene, something seen"). Drishya Tiny is a 212M-parameter video generator trained from scratch, using the frozen Wan2.2 VAE and Flan-T5-Base text encoder. It turns a text prompt, or a text prompt plus a first frame, into a 2-second 256×256 clip. In plain words: give it a sentence, or a photo and a sentence, and it animates it. The model page shows both.
The question
Video models are usually big and expensive, so we asked a smaller question. With one GPU and a budget you could put on a credit card, can you train a video generator from random weights that makes something worth watching? And once it works, what actually moves its ability to follow the prompt?
We didn't train everything ourselves. Two parts are borrowed and stay frozen: Flan-T5-Base reads the prompt, and the Wan2.2 VAE squeezes video into a small grid of numbers (and back). The part we trained is the generator in between, 211.92M parameters, from random weights. It learns to turn noise into that grid of numbers, guided by the prompt.
Stage 1: skies first
Before spending on the real thing, we checked the method on something narrow: SkyTimelapse, clips of moving skies, with no prompts at all. Same 211.9M generator, two hours on one RTX 5090, $7.14.
It worked. FVD-2048 (a distance between generated and real clips; lower is better) came out at 167.7. For scale, real frames frozen still score 215.1 and real clips against other real clips 72.2. The motion was about right too, 0.89 of real. It also ran out of data: held-out loss bottomed about halfway through. Fine for a method check. The recipe works.
Stage 2: three ideas, none better than plain
Next we tried three techniques that papers say make training faster or better, each against the plain recipe and each with the same 60 GPU-minutes. Differences under ±25.1 FVD were noise, by our own measurement.
- The plain recipe (the control): 160.7.
- Training on images and video together: 182.4. A tie.
- TREAD (routing some tokens past layers): 171.2. Also a tie, though with 39% more throughput.
- MeanFlow: our implementation diverged. We listed the likely causes; it's not a verdict on the method.
None beat the plain recipe, so the plain recipe went into the release run unchanged. Cost: $7.20.
Stage 3: the release model
Then the real run. The data is VidGen-1M, whose clips come from Microsoft's HD-VILA-100M: about 300K clips, cut into 475,157 two-second windows, with VidGen-1M's own captions. No photos, no synthetic data. 2,048 clips were held out and never trained on.
The data step took about 6.2 hours on one RTX 5090 ($7.40). Training took 15 hours on one L40S on 20 October ($13.76): 180,080 steps at batch 64, 11.53M clips seen. A quarter of the steps practised image-to-video, with the first frame given.
We wrote the pass rule before the run and checked it on the held-out clips:
| Check | Result | Reference |
|---|---|---|
| Text-to-video FVD-2048 | 235.7 | real clips frozen on frame 0: 660.4 |
| Motion, against real clips | 0.73 | rule: 0.5 to 2.0 |
| CLIP gap, own caption minus shuffled | 20.15 | real clips: 25.73, so 78% of real |
| Image-to-video FVD-512 | 240.3 | motion 0.89 |
It passed. Stage 3 cost $24.71 in all; $3.55 of that went on five failed data-step attempts, failed training starts, a dry run and a speed probe.
Stage 4: VBench
VBench is a public benchmark for video models, with many separate scores. We ran it ourselves with the official code: 946 prompts, five videos each. A pre-registered sweep picked the sampling settings (CFG 5, shift 5, 50 steps); CFG 3 gave better FVD (373 against 432 FVD-512) but followed prompts slightly worse. These are our own sampling and scoring with the official VBench code, not a certified leaderboard entry. Cost: $8.48.
On video quality it scores 75.6, level with the 256px VideoCrafter-0.9 (74.9) and close to ModelScope (78.1). Both are built on Stable Diffusion; our generator was trained from scratch and is about a sixth the size of ModelScope's 1.35B UNet. On prompt-following (semantic) it scores 32.3, about half theirs (65-67).
The details make the split plain. It is smooth (motion smoothness 96.87%), steady (temporal flickering 98.78%), gets colours right (84.39%) and really moves (dynamic degree 80.56%). It draws two named objects in one clip 2.91% of the time, and gets spatial relations and named scenes right about 6% of the time.
Stage 5a: what moves prompt-following?
So we ran a controlled test. Three copies of the release model each trained 18,000 more steps with the same compute: one with the original recipe as the control, one with fixed captions, and one with fixed captions plus photos. We scored them on six VBench semantic dimensions (516 prompts, two seeds each), against the control, with paired 95% intervals.
| Change, against the control | Prompt-following | Held-out FVD-512 |
|---|---|---|
| Caption fix (128 tokens, plus 20% short captions) | +3.01 (+0.93 to +5.26) | 413 against 414: no cost |
| Caption fix + 30% of the steps on photos (CC12M) | +4.34 (+1.99 to +6.72) | 518.7 against 414.4: +104, worse |
These are paired, per-prompt statistics, the ones our pre-registered rule uses; the plain gaps between the arms' scores are +2.29 and +3.94.
So the photos' own share, on top of the caption fix, is +1.65, and it showed up in what things are: object class up 6.0 points and scene up 3.9. Composition got no better: multiple objects and spatial relations didn't improve. And the photos cost picture quality.
Our reading: at this size, photos teach a video model what things are, not how to arrange them, and data fixes move prompt-following by 3-4 points, not 30. Looking real is cheap; following the prompt is what costs. That is a hypothesis with evidence, at 256px, 2 seconds and one budget, and we'd like to see it challenged. None of these three arms is the release model. Cost: $9.90.
Every dollar
| Stage | GPU cost | What it bought |
|---|---|---|
| Day-0 checks (Mac, Colab probe, research) | $0.03 | is it feasible? |
| Stage 1: skies | $7.14 | the recipe works |
| Stage 2: three techniques vs the plain recipe | $7.20 | keep the plain recipe |
| Stage 3: data and training | $24.71 | the release model |
| Stage 4: sampling sweep and VBench | $8.48 | the evaluation |
| Stage 5a: three controlled arms | $9.90 | the data test |
| Total | $57.46 | $57.43 of GPU across 55 machine rows, plus $0.03 |
The release model's own path is about $21: $7.40 for the data step and $13.76 for training. Not included: pretraining the borrowed Wan2.2 VAE and Flan-T5 (not ours), our time, electricity and storage.
If you want a smaller bill, look at MiniDance: it trained a scenery-only model for even less (about $2). Drishya Tiny learns from general web video rather than scenery alone, and we publish its VBench scores.
Where it falls short
- Prompt-following is about half the peers'. It almost never draws two named objects (2.9%), rarely gets spatial relations (6.0%) or named scenes (6.0%), and gets the object class right about a quarter of the time (25.0%).
- Faces and hands smear. Text melts.
- Short and small: 2 seconds, 256×256, 8 frames a second, and the motion is small.
- Image-to-video drifts: later frames lose detail and blur over the 2 seconds.
- Visual quality is modest: aesthetic 34.8%, imaging 51.5%.
- Not certified: the VBench numbers are our own sampling and scoring.
- Non-commercial weights: the training clips come from HD-VILA-100M, whose terms allow research use only.
It works best on nature and scenery: water, waves, sky, snow, boats, fire. People degrade. That is our impression from looking at clips, not a measurement.
The model page has a short blooper reel: a cat and a dog that merge into one animal, a red cup and a blue book that never show up, and a whiteboard where HELLO never gets written.
What we'd try next
Our test says more data tricks won't close the gap at this size. The open question is where the gap lives: in how well the model understands the prompt, or in how big the generator is. A cheap way to split those would be to score a model that pairs a small generator with a bigger frozen text encoder on the same VBench semantic test. We haven't run that.
Try it, and break it
- Run it in your browser, on your own device, from a prompt or a photo: udgamlabs.com/models/drishya-tiny (a tiny decoder, TAEW2.2, keeps the download small, so clips are softer)
- Or on a GPU: huggingface.co/spaces/udgamlabs/Drishya-Tiny
- The model page, with the clips and every result: udgamlabs.com/models/drishya-tiny
- Weights (CC BY-NC 4.0): huggingface.co/udgamlabs/Drishya-Tiny
- Code (Apache-2.0), logs and the cost ledger: github.com/udgamlabs/drishya-tiny
Show us the best and the worst 2 seconds you can get out of it. Both are useful.
Thank you to the VidGen-1M and HD-VILA-100M authors for the clips, the Wan team for the Wan2.2 VAE, Google for Flan-T5, and the VBench team for a benchmark anyone can run.