Papers usually report what worked. We think the failures are as useful, because they show where the limit of a system is. This post lists the negative results in Table 5 and Section 5.4 of our paper, with the reasoning behind each one.
Key points
- A paired bootstrap puts the noise floor at about 0.012 to 0.016 BERTScore F1. Smaller differences are ties.
- Every post-hoc rewrite of a generated answer lost score.
- Adding content from outside the model traded a small recall gain for a larger precision loss.
- More frames, more pixels, vision-tower LoRA and higher rank tied or regressed.
- Our best local validation setting for scene description lost 0.077 on the leaderboard.
How we judged a result
All deltas below are BERTScore F1 against the base configuration on validation data, unless marked LB (leaderboard). We used a paired bootstrap on the validation data to estimate how much a score moves by chance. That gave a noise floor of roughly 0.012 to 0.016. A lever that moves the score by less than that is a tie, not a win.
The rule matters because many changes look like small wins on one run. Without a noise floor, a team can stack several ties and believe it has made progress.
The eleven levers
| Task | Lever | Delta | Failure mode |
|---|---|---|---|
| Rewriting the answer after generation | |||
| Open QA | Text-only LLM rewrites the answers | −0.071 | Polishes away detail that matched the reference |
| BCQ with explanation | LLM clean-up of drafts | −0.012 | Deletes words that match the reference |
| BCQ with explanation | LLM fusion of four candidates | −0.016 | Invents detail the drafts agree is absent |
| Scene | Trim last sentence / keep first three | −0.028 / −0.074 | The metric is recall-driven |
| Injecting content from outside the model | |||
| Scene | Vehicle-enumeration prompting | −0.014 | Recall +0.04, precision −0.07 |
| Scene | Appending gold open-QA content | −0.083 | Event narration is not a scene inventory |
| Open QA | Sibling-description hints in the prompt | −0.016 | Spreads the hints' errors |
| Adding capacity or input resolution | |||
| Scene | Vision-tower LoRA (last 4 / 8 blocks) | +0.002 / +0.003 | Moves hallucinations around, does not reduce them |
| Scene | rsLoRA, rank 128 | −0.012 | Shorter outputs drop marginal attributes |
| Scene | 32 to 64 frames; larger pixel budget | about 0 | Input levers saturated |
| Scene | Few frames at very high resolution | −0.077 (LB) | Transcribes distant incidental signage |
Lesson 1: you cannot paraphrase towards a reference you do not have
Every post-hoc transform that reworded a generated answer lost score: rewriting, clean-up, fusion and every truncation. The metric rewards token-level closeness to one specific reference. A paraphrase made without that reference moves away from it on average.
Closeness has to be trained into the weights instead. That is why our answer specialist learns from the gold answers directly, and why our judge selects answers instead of editing them. The mechanism is explained in our BERTScore post.
Two edits did survive. Both added verifiable content and left the surrounding text untouched. We transcribed the location overlay burned into each test frame, read from a native-resolution crop with Gemini 3.1 Pro, into the scene descriptions (+0.004 LB). We also anchored open-ended answers to event windows that Gemini localised from cropped clips and that we checked against facts from sibling tasks (+0.008 LB).
Lesson 2: the headroom is in perception, not capacity
On scene description, headroom exists. If an oracle grafts the reference's vehicle sentences into our answer, the score rises by 0.06. Every route we could deploy towards that gain added a little recall and lost more precision. Every capacity lever (vision-tower LoRA, higher rank, more frames, more pixels) tied or regressed.
Our reading is that the binding constraint is the precision of what the model claims it saw. Only fine-tuning improved it.
Lesson 3: local validation is not a leaderboard proxy
Our best-ever validation configuration for scene description scored 0.077 below the shipped model on the leaderboard. The two sets are drawn differently. The organisers' test set is dominated by short, collision-heavy clips with a median length of about 10 seconds. Our local split follows the released training distribution.
The configuration tuned on our split used few frames at very high resolution. In the sign-dense test clips, the model read and transcribed storefront signage, and precision collapsed. After this inversion we trusted validation only for the component it measures directly, and we checked configurations on the evaluation server, as the rules allow.
What this means outside the benchmark
The same failure modes appear in field systems. A model that reads distant signs in a benchmark will also read billboards beside a highway. A metric that rewards matching one reference will mislead you about a system that describes scenes in its own words. We now treat a validation set as a tool for one decision, not as a forecast of field accuracy. Our post on measuring ANPR accuracy applies the same idea to plate reading.
The full method is in our paper explainer, and the training set-up is in our single-GPU fine-tuning post. For the vision language model in our products, see AI Search and the Research page.
Frequently asked questions
Why publish negative results?
They save other teams time, and they show which parts of our system are at a limit. A reader can also check that our positive results clear the same noise floor.
Did any lever help on the leaderboard?
Two edits that injected verifiable content helped: overlay transcription (+0.004) and event-window anchoring (+0.008).
Is a delta of −0.012 a real loss?
It sits at the low end of the noise floor of 0.012 to 0.016, so we treat it as a tie at best. It gave us no reason to ship the change.
Share this article