# Drishya Tiny, a lab notebook from building a small video model

Canonical: https://udgamlabs.com/blog/drishya-tiny · Updated: 2026-10-28

> How far does about $21 of GPU time go for a video model? Five stages, the numbers from each, what failed and every dollar, in the open.

Release · 2026-10-28 · By Jitesh K Prasad

We wanted to find out how far a small budget goes for a video model, so we built one and kept notes. This post is
those notes: what we tried, what worked, what didn't, and what it all cost. We are releasing almost everything,
failures included.

The result is Drishya Tiny (Drishya means "a scene, something seen"). Drishya Tiny is a 212M-parameter video
generator trained from scratch, using the frozen Wan2.2 VAE and Flan-T5-Base text encoder. It turns a text prompt, or a
text prompt plus a first frame, into a 2-second 256×256 clip. In plain words: give it a sentence, or a photo and a
sentence, and it animates it. The model page shows both.

> **The short version.** The release model's own path cost about \$21 of GPU time; everything, failures included, cost \$57.43. Its video quality on VBench is close to larger 256px models built on Stable Diffusion. It follows the prompt about half as well. Our best guess at why: looking real is cheap, following the prompt is what costs.

## The question

Video models are usually big and expensive, so we asked a smaller question. With one GPU and a budget you could put on a
credit card, can you train a video generator from random weights that makes something worth watching? And once it
works, what actually moves its ability to follow the prompt?

We didn't train everything ourselves. Two parts are borrowed and stay frozen: Flan-T5-Base reads the prompt, and the
Wan2.2 VAE squeezes video into a small grid of numbers (and back). The part we trained is the generator in between,
211.92M parameters, from random weights. It learns to turn noise into that grid of numbers, guided by the prompt.

## Stage 1: skies first

Before spending on the real thing, we checked the method on something narrow: SkyTimelapse, clips of moving skies, with
no prompts at all. Same 211.9M generator, two hours on one RTX 5090, \$7.14.

It worked. FVD-2048 (a distance between generated and real clips; lower is better) came out at 167.7. For scale, real
frames frozen still score 215.1 and real clips against other real clips 72.2. The motion was about right too, 0.89 of
real. It also ran out of data: held-out loss bottomed about halfway through. Fine for a method check. The recipe works.

## Stage 2: three ideas, none better than plain

Next we tried three techniques that papers say make training faster or better, each against the plain recipe and each
with the same 60 GPU-minutes. Differences under ±25.1 FVD were noise, by our own measurement.

- **The plain recipe** (the control): 160.7.
- **Training on images and video together:** 182.4. A tie.
- **TREAD** (routing some tokens past layers): 171.2. Also a tie, though with 39% more throughput.
- **MeanFlow:** our implementation diverged. We listed the likely causes; it's not a verdict on the method.

None beat the plain recipe, so the plain recipe went into the release run unchanged. Cost: \$7.20.

## Stage 3: the release model

Then the real run. The data is VidGen-1M, whose clips come from Microsoft's HD-VILA-100M: about 300K clips, cut into
475,157 two-second windows, with VidGen-1M's own captions. No photos, no synthetic data. 2,048 clips were held out and
never trained on.

The data step took about 6.2 hours on one RTX 5090 (\$7.40). Training took 15 hours on one L40S on 20 October (\$13.76):
180,080 steps at batch 64, 11.53M clips seen. A quarter of the steps practised image-to-video, with the first frame given.

We wrote the pass rule before the run and checked it on the held-out clips:

| Check | Result | Reference |
|---|---|---|
| Text-to-video FVD-2048 | 235.7 | real clips frozen on frame 0: 660.4 |
| Motion, against real clips | 0.73 | rule: 0.5 to 2.0 |
| CLIP gap, own caption minus shuffled | 20.15 | real clips: 25.73, so 78% of real |
| Image-to-video FVD-512 | 240.3 | motion 0.89 |

It passed. Stage 3 cost \$24.71 in all; \$3.55 of that went on five failed data-step attempts, failed training starts, a
dry run and a speed probe.

![Ten stills from Drishya Tiny clips, two seeds for each of five prompts: a sea under clouds, a sunset over the sea, a ridge under a blue sky, waves breaking, a boat on the sea](https://udgamlabs.com/blog/drishya-tiny/stills.jpg)

*One frame from each of ten text-to-video clips: five of our own prompts, two seeds each, nothing else picked. The moving versions, with their prompts and the clips where it breaks, are on the model page.*

## Stage 4: VBench

VBench is a public benchmark for video models, with many separate scores. We ran it ourselves with the official code:
946 prompts, five videos each. A pre-registered sweep picked the sampling settings (CFG 5, shift 5, 50 steps); CFG 3
gave better FVD (373 against 432 FVD-512) but followed prompts slightly worse. These are our own sampling and scoring
with the official VBench code, not a certified leaderboard entry. Cost: \$8.48.

![VBench at 256×256, about 2 seconds, higher is better. Video quality: Drishya Tiny (212M generator) 75.59, ModelScope T2V (1.7B system) 78.05, VideoCrafter-0.9 74.91, InstructVideo 81.56. Prompt-following (semantic): Drishya Tiny 32.26, ModelScope T2V 66.54, VideoCrafter-0.9 65.46, InstructVideo 56.81.](https://udgamlabs.com/models/drishya-tiny/vbench.png)

*Drishya Tiny is the dark bar: our own sampling and scoring with the official VBench code, 21-22 October 2026. The grey bars are published numbers for the 256px models the VBench team scored themselves, checked on 25 October 2026.*

On video quality it scores 75.6, level with the 256px VideoCrafter-0.9 (74.9) and close to ModelScope (78.1). Both are
built on Stable Diffusion; our generator was trained from scratch and is about a sixth the size of ModelScope's 1.35B
UNet. On prompt-following (semantic) it scores 32.3, about half theirs (65-67).

The details make the split plain. It is smooth (motion smoothness 96.87%), steady (temporal flickering 98.78%), gets
colours right (84.39%) and really moves (dynamic degree 80.56%). It draws two named objects in one clip 2.91% of the
time, and gets spatial relations and named scenes right about 6% of the time.

## Stage 5a: what moves prompt-following?

So we ran a controlled test. Three copies of the release model each trained 18,000 more steps with the same compute:
one with the original recipe as the control, one with fixed captions, and one with fixed captions plus photos. We
scored them on six VBench semantic dimensions (516 prompts, two seeds each), against the control, with paired 95%
intervals.

| Change, against the control | Prompt-following | Held-out FVD-512 |
|---|---|---|
| Caption fix (128 tokens, plus 20% short captions) | +3.01 (+0.93 to +5.26) | 413 against 414: no cost |
| Caption fix + 30% of the steps on photos (CC12M) | +4.34 (+1.99 to +6.72) | 518.7 against 414.4: +104, worse |

These are paired, per-prompt statistics, the ones our pre-registered rule uses; the plain gaps between the arms' scores
are +2.29 and +3.94.

So the photos' own share, on top of the caption fix, is +1.65, and it showed up in *what* things are: object class up 6.0
points and scene up 3.9. Composition got no better: multiple objects and spatial relations didn't improve. And the
photos cost picture quality.

Our reading: at this size, photos teach a video model what things are, not how to arrange them, and data fixes move
prompt-following by 3-4 points, not 30. Looking real is cheap; following the prompt is what costs. That is a hypothesis
with evidence, at 256px, 2 seconds and one budget, and we'd like to see it challenged. None of these three arms is the
release model. Cost: \$9.90.

## Every dollar

| Stage | GPU cost | What it bought |
|---|---|---|
| Day-0 checks (Mac, Colab probe, research) | \$0.03 | is it feasible? |
| Stage 1: skies | \$7.14 | the recipe works |
| Stage 2: three techniques vs the plain recipe | \$7.20 | keep the plain recipe |
| Stage 3: data and training | \$24.71 | the release model |
| Stage 4: sampling sweep and VBench | \$8.48 | the evaluation |
| Stage 5a: three controlled arms | \$9.90 | the data test |
| **Total** | **\$57.46** | \$57.43 of GPU across 55 machine rows, plus \$0.03 |

The release model's own path is about \$21: \$7.40 for the data step and \$13.76 for training. Not included: pretraining
the borrowed Wan2.2 VAE and Flan-T5 (not ours), our time, electricity and storage.

If you want a smaller bill, look at MiniDance: it trained a scenery-only model for even less (about \$2). Drishya Tiny
learns from general web video rather than scenery alone, and we publish its VBench scores.

## Where it falls short

- **Prompt-following is about half the peers'.** It almost never draws two named objects (2.9%), rarely gets spatial
  relations (6.0%) or named scenes (6.0%), and gets the object class right about a quarter of the time (25.0%).
- **Faces and hands smear. Text melts.**
- **Short and small:** 2 seconds, 256×256, 8 frames a second, and the motion is small.
- **Image-to-video drifts:** later frames lose detail and blur over the 2 seconds.
- **Visual quality is modest:** aesthetic 34.8%, imaging 51.5%.
- **Not certified:** the VBench numbers are our own sampling and scoring.
- **Non-commercial weights:** the training clips come from HD-VILA-100M, whose terms allow research use only.

It works best on nature and scenery: water, waves, sky, snow, boats, fire. People degrade. That is our impression from
looking at clips, not a measurement.

The model page has a short blooper reel: a cat and a dog that merge into one animal, a red cup and a blue book that never
show up, and a whiteboard where HELLO never gets written.

## What we'd try next

Our test says more data tricks won't close the gap at this size. The open question is where the gap lives: in how well
the model understands the prompt, or in how big the generator is. A cheap way to split those would be to score a model
that pairs a small generator with a bigger frozen text encoder on the same VBench semantic test. We haven't run that.

## Try it, and break it

- Run it in your browser, on your own device, from a prompt or a photo:
  [udgamlabs.com/models/drishya-tiny](https://udgamlabs.com/models/drishya-tiny#try) (a tiny decoder, TAEW2.2, keeps the download small, so
  clips are softer)
- Or on a GPU: [huggingface.co/spaces/udgamlabs/Drishya-Tiny](https://huggingface.co/spaces/udgamlabs/Drishya-Tiny)
- The model page, with the clips and every result: [udgamlabs.com/models/drishya-tiny](https://udgamlabs.com/models/drishya-tiny)
- Weights (CC BY-NC 4.0): [huggingface.co/udgamlabs/Drishya-Tiny](https://huggingface.co/udgamlabs/Drishya-Tiny)
- Code (Apache-2.0), logs and the cost ledger: [github.com/udgamlabs/drishya-tiny](https://github.com/udgamlabs/drishya-tiny)

Show us the best and the worst 2 seconds you can get out of it. Both are useful.

Thank you to the VidGen-1M and HD-VILA-100M authors for the clips, the Wan team for the Wan2.2 VAE, Google for Flan-T5,
and the VBench team for a benchmark anyone can run.

---

## About Udgam Labs

Udgam Labs is an independent AI research lab. We work from first principles: before we build, we understand why things work, whether we are training a model, fine-tuning one or applying AI to a new problem. Today that means open models such as Gargi, a 384M-parameter language model that comes within 2 points of Qwen2.5-0.5B on three reasoning benchmarks with about 1,000× less training data. Next come products, custom models for businesses, and AI in the physical world. Udgam (उद्गम) is Sanskrit for origin.

Udgam Labs · AI research from first principles · https://udgamlabs.com