Conventional video analytics answer a fixed list of questions: is there a helmet, is the vehicle going the wrong way, what is the plate. A VLM can answer questions nobody wrote a detector for. This post explains how that works, how it runs on a roadside camera, and where it falls short.
Key points
- A VLM joins an image encoder to a language model, so pictures and words share one representation.
- Natural-language search matches a typed description against what the model understood in each frame.
- On Recon, AI Search runs on the camera's edge processor (157 TOPS, 1024 CUDA cores), so no server-side processing is needed for the search itself.
- Each match returns the frame, the timestamp and the plate, so a person can check the result.
- Fixed detectors still handle tender line items such as helmet, seatbelt and speed.
What a vision language model is
Most current VLMs have three parts. An image encoder turns a frame into a set of feature vectors. A projector maps those vectors into the language model's input space. A language model then reads them alongside the text prompt and writes an answer. LLaVA (Liu et al., 2023) and VILA (Lin et al., 2024) follow this pattern. Recent video models such as Qwen3-VL also interleave textual timestamps with video tokens (Qwen3-VL technical report).
An older and related idea is contrastive matching of images and text in one embedding space, as in CLIP (Radford et al., 2021). It is fast for retrieval but cannot write a description or explain an answer.
| Property | Fixed detectors and classifiers | Vision language model |
|---|---|---|
| Questions it answers | A closed set fixed at training time | Open questions in plain language |
| Output | Boxes, classes, scores | Text, which can include attributes, reading and reasoning |
| Strength | Fast, stable, easy to measure per class | Rare attributes nobody labelled |
| Weakness | Cannot answer an unplanned question | Heavier compute; can state things it did not see |
How natural-language search works on a camera
Our AI Search page describes the flow in three steps:
- See: the model describes what passes, including vehicle type, colour, plate, and the text and logos on the body.
- Reason: the operator's description is matched against those understandings, so a phrase works where a dropdown cannot.
- Act: each match returns the frame, the timestamp and the plate.
Queries operators type include "orange turban", "man wearing Puma jacket", "CNG sticker on glass", "accident marks on vehicle" and "passenger using mobile phone". AI Search recognises text in English, Hindi and Marathi on vehicles, signs and other surfaces. EdgeTrack extends one query across every connected camera, so a description is checked against a whole corridor.
Why run it on the camera
- No upload: the query is answered at the gantry, and footage does not have to leave the site.
- No round trip: results come from the device that holds the video.
- Works offline: the model sits on the unit, so search does not depend on a cloud connection.
- Scales with cameras: each camera brings its own compute, so adding cameras does not load a central GPU pool.
The cost is a fixed compute budget per device. Recon's edge processor is rated at 157 TOPS with 1024 CUDA cores, and the same device also runs plate reading and violation detection. Model size, frame rate and the number of models that run together all have to fit that budget. Our post on edge and cloud architecture covers the wider trade-off.
Limits
- A VLM can state things it did not see. In our research on traffic video reasoning, the main limit on scene description was the precision of what the model claimed to see, not its capacity. The details are in our paper explainer.
- Resolution changes behaviour. In the same research, a setting with few frames at very high resolution made the model transcribe distant signage and lost score. A roadside camera sees billboards and shop signs too.
- Search results need a check. That is why every match carries the frame, timestamp and plate. A person decides what to do with it.
- Fixed detectors stay. Tender line items such as speed, helmet, seatbelt, triple riding and wrong direction need stable, per-class accuracy. On Recon, these structured events run alongside search.
Where a VLM helps most
The product page lists three main uses: law enforcement investigations that start from a description of a suspect vehicle or person, highway monitoring that looks for a specific vehicle across a corridor, and forensic search of recorded footage. In all three, the operator knows what they saw but not the plate. Traditional ANPR search starts from the plate. AI Search answers "what did you see?", not only "which plate passed?".
AI Search is an optional add-on to the Recon Global Shutter Camera and can be enabled on a deployed camera without hardware changes. For the training side of large vision language models, see our single-GPU fine-tuning post and the Research page.
Frequently asked questions
Does AI Search need the plate number?
No. You can start with a description such as a sticker, a jacket, a colour or damage. Partial plate matches are also supported.
Which languages does it support?
English, Hindi and Marathi. It also reads text in those languages on vehicles, signs and other surfaces.
Does the search run in the cloud?
No. It runs on the camera's edge processor. Results can be viewed on the EdgeTrack dashboard.
Share this article