DriveSense-VLM

DriveSense-VLM is an end-to-end multimodal ML system for detecting rare road hazards in dashcam images. It fine-tunes Qwen2.5-VL-3B-Instruct with LoRA and returns a grounded JSON report for each frame:

• hazard class;
• 2D bounding box;
• severity;
• short explanation;
• recommended driving action.

The work spans the full model lifecycle: distributed data mining, annotation, validation, fine-tuning, evaluation, regression gating, inference profiling, monitoring, and deployment.

30-second overview

This repository demonstrates more than model training:

• Data engineering: PySpark scores 34,149 nuScenes keyframes for rarity. A resumable miner streams ten image archives (about 250 GB in total) one at a time and extracts only selected CAM_FRONT frames, so the full archive never has to fit on disk.
• Label quality: nuScenes 3D ground-truth cuboids are projected into 2D with near-plane clipping. Claude supplies severity, reasoning, and driving actions, but not the boxes used to train v2/v3. Schema and box-diversity gates stop malformed or collapsed labels.
• Model and evaluation: Qwen2.5-VL-3B is fine-tuned with LoRA and measured across grounding, classification, reasoning, operating conditions, and inference behavior on a fixed test set.
• MLOps: every candidate is compared with the selected model. The larger v4 run performed worse, so the regression gate blocked it instead of promoting it on training-set size alone.

mine -> label -> validate -> train -> evaluate -> regression gate -> promote or block

Key measured outcomes

• Selected v3 model: 0.30 F1, 0.67 mean IoU, 0.94 class accuracy, and 98.7% parse rate on a fixed 1,041-frame nuScenes test set.
• Useful negative result: v4 added 1,442 targeted adverse-condition frames but fell to 0.25 F1; the automated gate correctly kept v3 in place.
• Inference optimization: prompt-lookup decoding improved T4 decode throughput by 20% and reduced median end-to-end latency by 16% with exact output match.
• Provenance experiment: GT-projected boxes outperformed foundation-model-emitted boxes in a controlled reconstruction (0.222 versus 0.128 F1), isolating localization quality as a major failure source.

All headline detection values are versioned in results/metrics_registry.json. Inference measurements and reproduction details are in docs/INFERENCE_OPTIMIZATION.md.

Results

Detection metrics come from results/metrics_registry.json. v3 and v4 use the same 1,041-frame nuScenes test set.

Detection at IoU >= 0.5

Version	Training frames	Precision	Recall	F1	Mean IoU	Class accuracy	Parse rate
v3, selected	7,228	0.40	0.24	0.30	0.67	0.94	98.7%
v4, blocked	8,670	0.37	0.19	0.25	0.656	0.946	97.4%

The model localizes matched hazards well, but recall remains the main limitation. Tiny and distant objects account for most of the missed detections.

Performance by condition

Test bucket	v3	v4
Overall	28.0%	23.0%
Tiny boxes	22.8%	17.2%
Small boxes	46.4%	42.9%
Medium boxes	52.6%	52.6%
Rain	12.5%	7.4%
Night and tiny	12.7%	10.7%
Clear and medium	69.0%	69.0%

v4 added 1,442 targeted adverse-condition frames, but the exact buckets it targeted became worse. The CI gate therefore blocked promotion. See docs/FLYWHEEL_V4_FINDINGS.md for the analysis.

Reasoning quality

The v3 reasoning evaluation used Claude Sonnet 5 as judge on 1,027 examples.

Dimension	Score
Correctness	3.03 / 5
Completeness	2.66 / 5
Action relevance	3.80 / 5
Overall	3.16 / 5
Pass rate, all dimensions >= 3.5	26%

Action recommendations were stronger than hazard coverage. This agrees with the grounding result: the model is more likely to omit a hazard than to give poor advice for one it has detected.

Box-provenance experiment

The v4 data addition changed two variables at once: scene selection and box source. Its new boxes were emitted by the foundation model instead of projected from nuScenes ground truth. A follow-up experiment trained two otherwise matched arms to isolate that difference.

Metric	FM-emitted boxes	GT-projected boxes
Recall	0.101	0.167
Precision	0.176	0.330
F1	0.128	0.222
Mean best-pair IoU	0.244	0.458
Frame detection at IoU 0.5	0.266	0.566
No-hazard accuracy	0.512	0.831

GT-projected boxes performed better on every localization measure. Matched-class accuracy was similar, which points to localization rather than class naming as the main difference.

This was a reduced-scale reconstruction because the original per-frame v3/v4 artifacts were not retained. The comparison used 2,652 base frames, 1,162 targeted frames, and 402 test frames. The single-seed result should be treated as evidence about direction, not a final estimate of effect size. Full details are in docs/TASK3_DECONFOUND.md.

Inference study

Inference was measured on one NVIDIA T4. The baseline is memory-bandwidth-bound during autoregressive decoding.

Configuration	Decode tokens/s	TTFT	End-to-end p50	Weight memory	HBM utilization	Output comparison
fp16	17.0	770 ms	11.64 s	about 6.0 GB	31.8%	reference
fp16 + prompt lookup	20.4	781 ms	9.79 s	about 6.0 GB	38.2%	exact match 1.00
NF4	12.6	820 ms	about 15.9 s	2.63 GB	10.4%	character similarity 0.36
INT8	4.6	about 1,080 ms	53.53 s	about 3.5 GB	5.0%	character similarity 0.29

Prompt lookup reduced median latency by 16% without changing the output. NF4 reduced model memory by about 2.3x but made decoding slower and changed the generated text. In this workload, quantization is useful for fitting the model in memory, not for reducing latency. fp16 throughput increased from 14.9 to 33.7 tokens/s at batch size 4.

The benchmark and measurement notes are in docs/INFERENCE_OPTIMIZATION.md. The numbers can be reproduced with scripts/inference_benchmark.py.

Pipeline

nuScenes metadata and image blobs
        |
        v
rarity scoring and bounded-storage mining
        |
        v
3D-to-2D box projection and description generation
        |
        v
schema and box-diversity validation gate
        |
        v
Qwen2.5-VL-3B LoRA training
        |
        v
grounding, reasoning, robustness, and latency evaluation
        |
        v
candidate regression gate: promote or block

Important safeguards include near-plane clipping during projection, scene-level split isolation, hard validation of box diversity, an all-zero-IoU abort, and CI checks against the selected model.

Quick start

Local development does not require a GPU:

git clone https://github.com/jayanth922/DriveSense-VLM.git
cd DriveSense-VLM
python -m venv .venv
source .venv/bin/activate
python -m pip install pyyaml pillow numpy scipy tqdm pytest
python scripts/run_sanity_check.py
python -m pytest -q tests

Training and full inference require a CUDA GPU. The notebooks under notebooks/ and the Task 3 runbook under experiments/task3_deconfound/ contain the GPU workflows.

Repository layout

Path	Purpose
src/drivesense/	Reusable data, training, inference, evaluation, and monitoring code
scripts/	Command-line entry points for each pipeline stage
configs/	Model, data, training, inference, and evaluation settings
tests/	CPU-safe unit and integration tests
notebooks/	Colab workflows for data, training, optimization, and evaluation
experiments/task3_deconfound/	Controlled FM-box versus GT-box experiment
deploy/huggingface/	Hugging Face model card and Space application
results/	Versioned metrics and compact evaluation artifacts
docs/	Design notes, runbooks, findings, and model documentation

Useful starting points:

• docs/FLYWHEEL.md: data and training loop;
• docs/DEBUGGING_POSTMORTEM.md: failures and fixes;
• docs/OBSERVABILITY.md: regression and drift monitoring;
• docs/CLOSED_LOOP.md: failure-driven data selection;
• docs/MODEL_CARD.md: model use and limitations;
• docs/TENSORRT_RUNBOOK.md: ViT export investigation.

Current status and next work

The implemented pipeline is complete. The remaining work is experimental rather than required for running the repository:

• repeat the box-provenance experiment at the original 7,228-frame scale and on the full 1,041-frame test set;
• run multiple seeds for confidence intervals;
• test a v4b training set without the 216 no_hazard examples added in v4;
• evaluate vLLM on a compatible GPU image. The T4 environment used for this study could not run it.

Limitations

• v3 recall is 24% at IoU 0.5; tiny, distant, rainy, and nighttime hazards remain difficult.
• nuScenes trainval is weighted toward daytime driving and does not represent every deployment environment.
• The reasoning judge is another language model and should not be treated as a human safety review.
• End-to-end T4 latency is measured in seconds per image, so this is not a real-time driving stack.
• The system is a research prototype. It must not be used for vehicle control or other safety-critical decisions.

Technology

• Qwen2.5-VL-3B-Instruct
• PyTorch, Transformers, PEFT, and bitsandbytes
• nuScenes v1.0-trainval and PySpark
• Anthropic Claude for description generation and reasoning evaluation
• pytest, GitHub Actions, and Weights & Biases
• Gradio and Hugging Face Spaces

Acknowledgments

This project uses Qwen2.5-VL, nuScenes, Hugging Face Transformers and PEFT, and the Anthropic API. See the linked projects and dataset terms for their licenses and usage requirements.