Many people in India type to their phones the way they speak: Hindi and English mixed, in Latin letters. "bed time ka alarm kab hai?" is a perfectly clear request — when is my bedtime alarm? — but most small language models are not trained to turn it into something an app can act on.
Udgam Hinglish Commands is our first experimental release aimed at that gap. It is a small adapter that reads a short English or Romanized Hinglish command and returns a structured interpretation: the intent (look up an alarm) and the labelled details (alarm name: bed time). The bundled Python code turns that into ordered JSON for your app.
What it is, and what it is not
The release is a task-specific LoRA adapter for Qwen3-0.6B. We did not pretrain a new 0.6B model: we trained about 10.1 million adapter parameters on top of Qwen's weights, which stay unchanged.
It covers the eight command domains in Google Research's Hinglish-TOP benchmark — alarms, events, messaging, music, navigation, reminders, timers and weather. It executes nothing. It proposes an interpretation; your application resolves dates, names and permissions and decides whether to act.
How we trained it
- Data: human-labelled pairs from Hinglish-TOP, which builds on Meta's TOPv2. After removing 679 pairs that overlapped the test set, training used 2,310 pairs (4,620 rows, each pair in English and Hinglish). No teacher-model labels and no synthetic data went into the selected run.
- Recipe: LoRA rank 16 on the attention and MLP projections, learning rate 1e-4, effective batch 32. We kept the epoch-2 checkpoint.
- Compute: one NVIDIA A40 in BF16, about 100 minutes including validation.
We also tried a variant with synthetic augmentation. It scored lower on validation (48.44% against 51.56%) and produced more malformed outputs, so we left it out.
Results
Every model below gets the same four retrieved training examples in its prompt, and is scored on complete-tree exact match — the whole structure has to be right — on the frozen Hinglish-TOP test set of 13,026 commands.
| Model | All commands | Hinglish | English |
|---|---|---|---|
| Qwen3-0.6B, original | 9.90% | 7.54% | 12.27% |
| Existing Hinglish checkpoint (KingNish) | 9.19% | 5.94% | 12.44% |
| Udgam adapter | 54.88% | 46.84% | 62.92% |
Two diagnostics show where the gain comes from: the root intent is right 86.55% of the time (78.77% for the original model), and slot F1 rises from 40.60% to 72.93%.
Where it still falls short
- Hinglish is still below 50%. More than half of the Hinglish commands are not parsed exactly right.
- Valid is not correct. 99.96% of outputs are syntactically valid trees, but a well-formed result can still have the wrong meaning. "mujhe subah 9 baje ke liye jaga do" (wake me at 9 in the morning) was labelled as a departure time, for example.
- Narrow scope. It is evaluated only on English and Romanized Hinglish, not Devanagari or other Indian languages, and only on the eight domains. Unsupported requests can be forced into an in-domain intent.
- We cannot rule out that the benchmark overlaps Qwen's pretraining data, and Hinglish-TOP is not representative of everyday speech across India.
For anything consequential, have the user confirm before your app acts.
Try it
Install from a source checkout of the GitHub repository
(python -m pip install -e '.[inference]'), download the adapter package from
Hugging Face, then:
from udgam_hinglish import CommandParser
parser = CommandParser.from_local("/path/to/adapter-package", device="auto")
result = parser.parse("bed time ka alarm kab hai?")
It runs on Apple silicon and CUDA GPUs with Python 3.11 or newer. The walkthrough shows recorded examples in plain language, and the benchmark notes have every number above with its conditions.
The adapter and prepared data are released under CC BY-SA 4.0 and our code under Apache 2.0, with the Qwen and Google notices preserved — so the complete package is not Apache-only.