Why a better answer can score lower under single-reference BERTScore

Seven of the nine scored task types in the Traffic Anomaly Reasoning (TAR) benchmark are free text. Each answer is compared with one reference answer using BERTScore F1. The training references were written by an automatic labelling pipeline. This post shows what that scoring rewards, with two examples from Figure 3 of our paper.

Key points

• BERTScore rewards token-level semantic overlap with the reference. Correct content that the reference omits earns nothing.
• In two validation examples, our accurate 9B answer scored 0.34 and 0.27. Our style-matched 27B answer scored 0.53 and 0.58.
• A false yes/no verdict and a true one can score almost the same under BERTScore (0.57 vs 0.61).
• Reasoning supervision helps exact-match tasks and costs score on similarity-scored tasks.
• If you evaluate video models this way, separate "correct" from "close to the reference".

Figure 3 of the paper. Green marks phrasing shared with the reference. Amber marks accurate content that the reference does not contain.

How BERTScore works

BERTScore embeds every token of the candidate and the reference with a contextual model. It then matches each token to its most similar token on the other side by cosine similarity:

Precision P = mean over candidate tokens of  max cosine(candidate token, reference token)
Recall    R = mean over reference tokens of  max cosine(reference token, candidate token)
F1          = 2 P R / (P + R)
Rescaled    = (score - b) / (1 - b),  b = baseline from random sentence pairs

TAR uses roberta-large with rescaling. The rescaling note explains why: raw scores from RoBERTa-large often fall between 0.85 and 0.95, so a baseline is subtracted to spread them out. Rescaling does not change the ranking of answers.

Two properties follow from the formula. Content in the answer that is not in the reference lowers precision. Content in the reference that is not in the answer lowers recall. Neither term asks whether the content is true.

Example 1: one vehicle or three

The question asks what happened in the first two seconds of a clip. The reference describes one black SUV in the leftmost lane. Our 9B reasoning specialist describes three vehicles: a white car, a black car and a red truck, each in its lane. Our 27B answer specialist describes the black SUV, almost clause for clause in the reference's words.

The reference narrates one of three vehicles. Matching its selection outscores reporting all three.

Example 2: both correct, different words

The second question asks for the road layout and surroundings. Both models perceive the scene correctly: a curved multi-lane road, dense trees, power lines and a date overlay. The 9B also reports dashed lane dividers, a street sign and a parallel road behind a grassy median, all of which are accurate and absent from the reference. The 27B follows the reference's order and phrasing, and it scores more than twice as high.

Neither example is about perception. The gap is about register and content selection.

The same effect in the ablation

We trained the same 9B backbone twice: once on answers only, once on our rewritten reasoning traces (TAR-RE). The table shows part of Table 3 of the paper, on a subset of our internal test split.

Reasoning supervision pays where answers are scored exactly. It costs 5.1 points on binary verification with explanation, a similarity-scored task. Open QA and scene description sit within two points. The answer specialist also shows the reverse risk: on items that need the video to be inspected, it can recite a likely prior (a collision implies a red light was run). BERTScore barely separates that false verdict from a true one, 0.57 against 0.61.

What we did with this

• We trained a separate answer specialist on the gold answers only, so it absorbs the reference style.
• We kept the reasoning specialist for questions where correctness decides the score.
• We used a judge that selects between whole answers and never edits them, because every paraphrase we tried moved away from the reference. The judge design is in our judge bias post.
• For multiple choice with explanation, we submit the exact wording of the predicted option, which is the strongest submission available under one reference.

Advice for evaluating video language models

• Report exact-match and similarity metrics separately. Do not average them into one number without saying so.
• Where you can, collect more than one reference per item, so that one writer's content selection does not define correctness.
• Read a sample of low-scoring answers by hand. Some of them will be right.
• Check who wrote the references. Our label audit shows why provenance matters.

In our products, a vision language model answers operator searches, and each answer returns the frame, timestamp and plate so a person can check it. See AI Search and the Research page.

Frequently asked questions

Is BERTScore a bad metric?

No. It correlates well with human judgement on many tasks. The problem here is a single machine-written reference per item, which turns style into part of the target.

Did rewriting answers to match the reference help?

No. Every post-hoc rewrite lost score, because a paraphrase made without the reference moves away from it on average.

Which answer would an operator prefer?

For a road-safety operator, the answer that reports all three vehicles is usually more useful. That is why we keep exact-match and reference-similarity results apart.