Fine-tuning a 27B video-language model on one GPU, and reproducing the run

This post documents the training and serving set-up behind our AI City Challenge 2026 entry. Everything here comes from Section 5.1 of our paper and the README of the reason-or-recite repository. Where the two differ, we say so.

Key points

• The 9B reasoning specialist trained on one RTX 5090 (32 GB). The 27B answer specialist trained on one RTX PRO 6000 Blackwell (96 GB).
• The 27B base is frozen and quantised to 4-bit NormalFloat. The vision tower is exempt and stays in bf16.
• Both models use rank-64 LoRA on twelve module types of a Gated DeltaNet hybrid architecture.
• We wrote our own training stack on transformers, TRL, PEFT and bitsandbytes, because no common framework supported video inputs for this model series at the time.
• Across cold inference runs, 5 to 10 items switch candidates. The exact submission is checked into the repository.

Why one GPU

A single-GPU recipe is easier to reproduce and cheaper to iterate on. QLoRA (Dettmers et al.) makes this possible for a 27B model: the base weights are stored in 4 bits and frozen, and only small low-rank adapters (LoRA) train in bf16.

We made one exception. The answer specialist's value depends on what it perceives, so we kept the vision tower out of quantisation. In the repository this sits on the model config:

quantization:
  load_in_4bit: true
  bnb_4bit_quant_type: nf4
  bnb_4bit_use_double_quant: true
  bnb_4bit_compute_dtype: bfloat16
  skip_modules: [visual]      # leave the vision tower in bf16

The README states that both the 27B candidate and the judge run this way.

The architecture we adapted

Qwen 3.5 and Qwen 3.6 are hybrids. They interleave Gated DeltaNet linear attention with full softmax attention in a fixed 3:1 pattern, with a SwiGLU MLP in every layer. Our LoRA adapters attach to every linear projection of all three block types:

• the full-attention q, k, v and o maps,
• the Gated DeltaNet query, key and value streams with their decay, update and output gates,
• the SwiGLU gate, up and down maps.

That is twelve module types in all. The mixer's non-matrix parameters, the vision tower and the token embeddings stay frozen.

The recipe

One difference between paper and repository: the README's 27B training command uses 24 frames, while the paper states 32 frames for both models. The README command is the one to run if you want to rebuild the weights from the repository.

FLA_TILELANG=0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python scripts/core/train.py \
  model=qwen36_27b_recite \
  data=tar_reason_enhanced \
  data.root=$AICC \
  train=lora_sft \
  train.args.num_train_epochs=2 \
  train.args.gradient_accumulation_steps=16 \
  train.args.weight_decay=0.0 \
  run.tag=recite-27b

We shortened the command above; the README also passes the list of target tasks and a run id.

Our own training stack

When we did this work, no widely used fine-tuning framework offered stable support for video inputs to the Qwen 3.5 and 3.6 series. We built our own training and evaluation pipeline on transformers, TRL, PEFT and bitsandbytes. Two choices mattered most:

• Byte-identical prompts: prompt construction is the same code path in training and inference.
• Chunked loss: the cross-entropy is computed in slices, so the full logits tensor over the large vocabulary is never held in memory at once.

The README also warns about kernels. flash-attn, flash-linear-attention and causal-conv1d are correctness requirements, not only speed. The pure-PyTorch fallbacks accept the packed-document reset arguments and then ignore them, so packed training lets one example's recurrent state bleed into the next with no error.

Data splits

We split the data 80/10/10 into training, validation and internal test sets by video. All annotations of a video share a split, and the assignment is stratified by source dataset with seed 42. Validation drove every development decision. The internal test split drove none, so its ablation numbers estimate uncontaminated generalisation.

Reproducing the leaderboard run

The repository has a six-step driver: normalise the test set, run the 9B with vLLM, run the 27B with Hugging Face generate at batch size 1, split the fused scene and summary answer, run the judged selection, and verify against the checked-in submission.

The output is not always byte-reproducible. Across cold passes, 5 to 10 items typically switch between the 27B and the 9B. The cause is the 9B's decoding. vLLM's continuous batching changes the floating-point reduction order with batch composition, so greedy decoding holds only while the top token is unambiguous. One tie-break flip changes the text, and the judge then prefers the other candidate for that item.

vLLM documents that it is not reproducible by default. Its batch-invariant mode does not cover this model:

VLLM_BATCH_INVARIANT=1
RuntimeError: VLLM batch_invariant mode is not supported for GDN_ATTN

The issue is tracked as vllm#42960. For the record, the exact submission that placed on the leaderboard is in reproduction/leaderboard-winner.csv, and step 6 of the driver checks against it. The 9B weights are public on Hugging Face.

Where this connects to our cameras

Three habits from this work apply to any constrained deployment: fit the model to the memory you have, keep perception at full precision where it matters, and record exactly what shipped. Our post on vision language models on the edge covers the on-camera side, and AI Search is the product. The method behind the training is in our paper explainer, and more work is on the Research page.

Frequently asked questions

Which libraries does the training stack use?

It is built on transformers, TRL, PEFT and bitsandbytes, with Hydra for configuration. Every setting is a command-line override, so there are no per-experiment scripts.

Why not train the vision tower too?

We tried LoRA on the last 4 and 8 vision blocks. The gains (+0.002 and +0.003) were inside the noise floor, so the vision tower stays frozen.

Can I run the 9B on a 24 GB card?

We trained it on a 32 GB RTX 5090 and have not tested smaller cards. The GPU targets listed in the README are Ampere, Ada and Blackwell.