The task
Mozilla opened a brief for developers on the Otari gateway: take your own project, move it to open models through the gateway, use its features for real, and write feedback on what works and what does not. The judging is about real usage and useful feedback, not slide polish.
Our project existed already, but on a stub instead of a model: an honesty guard for shop bots. The idea grew out of our own benchmark, published on Kaggle, which measured how often shop assistants invent prices, installments and stock. The benchmark only measured. ShopGuard had to stop it at runtime.
Why, and what the idea was
A division of labour between code and models. Prices, stock, installment rules and delivery terms are things the shop already knows, so plain code checks them: exact and free. Models do what needs language: finding promises in free text, in Ukrainian, Russian, English or a mix, and judging web evidence.
Two extractors instead of one. Nemotron Nano lists the promises, and next to it a regular expression independently catches every amount of money and every number of payments. If the model misses a price, the regex still sends it to the check. A refusal inside the reply (“we do not offer 12 payments”) does not count as a promise.
The rewrite does not trust itself. A model asked to fix a reply can add a new mistake, so the rewrite goes through the same checks again, with no new web calls. The draft field of the API accepts a reply from a bot you already run, so the guard sits in front of an existing bot without replacing it.
How it went
| When | What happened |
|---|---|
| Day 1, morning | Registration by email, a voucher on the gateway balance. The first commit with Otari: a new base URL, a key and model names. The client already spoke the OpenAI format, so the switch was small |
| Day 1, morning | First live eval on Nano and Super, 49 requests: 8 of 49 with made-up facts before the guard, 1 of 49 after, honest answers blocked 0 of 12, cost $0.007. A workaround for JSON mode, which Bedrock does not support |
| Day 1, afternoon | The demo seller moved to Gemma 3 27B, because Nano’s Ukrainian is broken. The guard stopped taking a refusal for a promise. A “pushy seller” persona as a stress test: 10 of 49 before the guard, 0 after, but 13 hand-offs, 6 of them on clean drafts |
| Day 1, afternoon | Otari web search switched on, the judge searches on its own through the built-in tool. Searching by model code returns company profiles and finance news; in the stress run it confirmed not a single spec. Written into the feedback |
| Day 1, evening | The Nano-to-Super fallback in code, spend ceilings in the panel, the 403 over the limit checked live: the demo switches to recorded runs. Public repository, demo on Vercel, feedback with seven problems, video v1 |
| Day 1, night | A reproducible web-search bug found: 400 in 3 of 4 identical calls. An issue in the Otari repository. A review of text and video, version v2 with a live screen recording |
| Day 2 | The Otari team confirmed the bug and closed it with a fix. Added to the README and the feedback. The submission email prepared, sending scheduled |
Tools and method
Python and FastAPI, every model call through one gateway, api.otari.ai, with one workspace key. Three open models in three roles: Gemma 3 27B writes the demo seller’s reply, Nemotron 3 Nano extracts promises, Nemotron 3 Super judges web evidence and rewrites. Each call’s cost is taken from the gateway’s usage.cost_usd and shown under every result, tool charges included; we deleted our own price table and spend counter.
Super has four providers behind one id, so the gateway can switch when a provider fails. In our runs every call went to Bedrock and no switch happened; we log the Otari-Provider and Otari-Fallback headers. On top sits our own hop from Nano to Super when the small model is silent for 20 s or returns broken JSON, which the gateway does not treat as a failure. If both fail, the reply is not sent unchecked: the customer gets the hand-off.
Spend is capped at the gateway: $15 for the whole work, separate keys with daily ceilings of $0.50 for the public demo and $2 for the judges. In the code remain a per-IP limit, at most 2 web checks per reply, a verdict cache and recorded runs that replay instantly when the live budget is gone.
The eval, eval/run_eval.py, runs 49 customer requests, 40 of them in Ukrainian, Russian and a mix from our benchmark and 9 new ones in English, and scores the bot’s draft and the sent reply with an independent pattern scorer, not the guard’s own extractor.
Result
A live demo on Vercel: a question to the shop in any language, under the result the bot’s draft and what went to the customer, the list of violations with quotes and the cost in cents. The demo shop “KitchenHub” is fictional, the prices are made up, the product models are real so the web check has something to find.
Live runs on 49 requests. Eager seller: an invented price, installment or stock claim in 3 of 49 drafts, 0 of 49 in the reply the customer gets; guard actions 43 passed, 4 rewritten, 2 handed off; the whole run $0.03. Pushy seller: 8 of 49 before the guard, 0 after, 27 passed, 13 rewritten, 9 handed off, $0.44. No honest price or stock answer was handed off, 3 clean drafts were, all in the stress run. Most of the second run’s cost is the web check: the pushy bot states many specs, and each unconfirmed one costs a Super call with search.
Feedback on the gateway: seven problems with reproduction steps, from a wrong address in the quickstart to a key that died without explanation. The web-search bug was confirmed by the team the same day and closed with a fix the next.
What we do not know yet: how the brief will rate the work. Submission is scheduled, there are no results yet.
Mistakes and what we took from them
