Machine-generated labels are now common in video datasets. They are cheap and they scale, but they carry the habits of the pipeline that wrote them. This post describes how we audited one such dataset, what we measured and how we used the result. The numbers come from Section 3.2 and Table 1 of our paper, arXiv:2610.34993.
Key points
- The training annotations came from a three-stage VLM labelling pipeline. The test references are human-verified.
- 10.1% of scene-description traces cite inputs that do not exist at inference time.
- 66.1% of annotations contain timestamps, written in at least four different notations.
- Some annotations contradict the video, and one bad event description spreads to every task for that video.
- 33.1% of traces anchor no observation to a timestamp, and 46.1% of "No" verdicts cite no moment at all.
Figure 1 of the paper. Four defect classes, each with verbatim training examples.
Where the labels came from
The challenge overview paper describes the pipeline. Gemini 3.1 Pro wrote a caption and a structured event description for each video. Gemma-4 then turned these into question and answer pairs with chain-of-thought reasoning. For 910 of the 3,670 training videos, existing NVIDIA human annotations served as extra context. The 960 test references, in contrast, are human-verified.
This matters in two ways. A model trained on the reasoning inherits the pipeline's shortcuts. A model trained on the answers learns the style that the free-text tasks are scored against. We use both facts in the final system.
How we audited
We ran programmatic checks across the full training set and inspected a small sample of videos by hand. The programmatic checks were plain string and pattern tests:
- phrase matching for references to the labelling inputs ("ground truth", "the provided description" and similar),
- a parser for every timestamp notation we could find, with counts per field,
- counts of timestamp-anchored observations per reasoning trace,
- trace length and token-level ROUGE-L between each trace and its gold answer.
The manual sample found the defects that string tests cannot see, such as a label that describes an event the video does not contain.
The four defect classes
(a) Reasoning that cites sources the model cannot see
Many traces justify an answer from the captions and notes the teacher models were given, not from the video. Conservative phrase matching flags 10.1% of scene-description reasoning and about 1% overall. This is a lower bound, because only explicit wording is detectable. Fine-tuning on such traces teaches a model to appeal to an authority it will never have.
(b) Timestamps in no consistent format
66.1% of all annotations contain at least one timestamp, mostly inside the reasoning. At least four notations coexist, often in one item. 911 timestamps use a colon as a decimal separator, and 113 gold answers mix two notations inside one JSON object. The labelling prompt asked for MM:SS, yet only 63% of gold timestamps comply.
(c) Annotations that contradict the video
Manual inspection found annotations whose premise or timing is wrong. In one theft clip, a whole family of questions asks about a collision that never happens. Because every task type for a video derives from the same event description, one upstream error reaches every task for that video.
(d) Verdicts that do not inspect the timeline
Excluding scene description, a trace anchors a median of two observations to a timestamp, and 33.1% anchor none. A "No" answer to a binary question needs a scan of the whole video, yet 46.1% of "No" traces cite no timestamp.
The traces are also shallow
Mean trace length per task ranges from 109 to 244 words. The traces largely restate the answer they should derive: token-level ROUGE-L between trace and answer reaches 0.33. The table shows a subset of Table 1 in the paper, with our rewritten traces (TAR-RE) next to the official ones.
| Task | Items | With timestamps | Words, official / TAR-RE | ROUGE-L vs answer, official / TAR-RE |
|---|---|---|---|---|
| BCQ with explanation | 7,340 | 59.8% | 109 / 429 | 0.128 / 0.058 |
| Open QA | 3,670 | 57.2% | 126 / 471 | 0.246 / 0.151 |
| Scene description | 3,670 | 3.0% | 118 / 483 | 0.326 / 0.212 |
| Video summarisation | 3,670 | 89.0% | 117 / 482 | 0.289 / 0.233 |
| Causal linkage | 3,670 | 100.0% | 244 / 505 | 0.276 / 0.195 |
What we did about it, and what we did not do
We did not relabel. Our rewrite keeps every gold outcome fixed, including the wrong ones, so it cannot change the data silently. Label noise of type (c) therefore remains, and it bounds what any method can learn from this training set.
We did repair the supervision for the reasoning model. First, a parser turns every timestamp into one form, <X.Y seconds>. Decimal seconds need no base-60 arithmetic, and the form matches the textual timestamps that Qwen3-VL places next to video tokens. Second, Qwen 3.5 9B rewrites each trace into five sections (scene, query, timestamped evidence, reason, conclusion). A validator rejects traces that miss a tag, fall below the word floors or copy the gold answer verbatim.
The effect on structure is large. Over 99% of rewritten traces use the unified timestamp form only. The median number of time-anchored observations per trace rises from two to between 16 and 30. The share of traces that cite no moment falls from 33.1% to 0.2% (scene description excluded).
These statistics measure structure, not truth. Defects (a), (b) and (d) are removed by construction, but nothing here proves that each cited observation is correct. We measured the effect on accuracy separately: on a subset of our internal test split, multiple choice rose from 0.9429 to 1.0000. The method is described in our paper explainer.
A short audit checklist
- Find out who or what wrote the labels, and with which inputs.
- Search the reasoning for references to inputs the model will not have.
- Parse every number with a unit, especially time, and count the formats.
- Watch a sample of videos against their labels before you trust any aggregate.
- Measure how much a trace restates its answer.
- Decide in advance whether you will relabel, and write the reason down.
The same provenance helps in one place. The machine-written answers are the only sample of the style that the free-text tasks are scored against. Our post on BERTScore and better answers explains how we used that. For field systems we apply the same habit to our own data; see the Research page and our on-camera AI Search.
Frequently asked questions
Why not fix the wrong labels?
The rewrite uses the gold outcome as its anchor. If we changed outcomes, we could no longer show that the rewrite did not alter the task. We kept the outcomes and accepted the noise.
Are the test labels affected too?
The paper treats the 960 test references as human-verified, as the organisers describe them. The audit covers the training annotations.
Is the rewritten dataset public?
The repository contains the scripts that build it from the official data. We have not linked a released copy of the rewritten traces.
Share this article