Detecting whether a vehicle is driven autonomously or by a human, using only video from fixed roadside infrastructure cameras. The system extracts 3D vehicle trajectories from monocular roadside footage and will classify each trajectory from its motion characteristics. The closest prior work (Maresca et al., arXiv:2403.09571) addresses this task in simulation only; this project targets the real-world setting.
This repository extends BEVHeight (Yang et al., CVPR 2023). The upstream
README is preserved at docs/prev-readme.md. Datasets,
checkpoints, and outputs are not committed: see
docs/getting-started.md for what to download,
where files go, and which script runs each stage.
roadside video
-> camera calibration (intrinsics + extrinsics, no ground truth required)
-> per-frame 3D bounding boxes (BEVHeight, CPU-ported)
-> identity-linked trajectories (AB3DMOT multi-object tracking)
-> AV vs human classification (motion-behavior analysis; not started)
| Component | Status |
|---|---|
| BEVHeight detector, Mac CPU port | Done. Matches the published DAIR-V2X-I benchmark (Car 3D AP@0.5 moderate 69.4 on a 200-frame subset vs 65.46 published full-set) |
| Self-calibration (intrinsics + extrinsics, no ground truth) | Done for the primary site. Camera height 16.2 to 16.3 m confirmed by four independent estimates within 0.1 m; per-clip rotation from a vanishing-point solver with a pose-consistency gate |
| Ground-plane convention fix | Done. The training data places the road at z = -1.73 m; feeding it at z = 0 put every box 1.6 m underground. Box bottoms now sit within 0.08 to 0.11 m of the road across all five processed clips |
| 3D detection on WI DOT camera data | Done for five clips (1,386 frames, both view directions, two seasons). Pipeline default is the DAIR 140.8 m checkpoint at score 0.45, which finds 7.0 cars per frame against 5.1 for the 102.4 m one |
| Trajectory extraction (30 Hz MOT) | Working. Tracks graded against held-out GPS using an image-only target identification (see below) |
| Trajectory accuracy vs GPS | Position RMSE 0.78 to 1.13 m on the instrumented vehicle, identified independently of GPS. The 140.8 m checkpoint holds the target 47 and 15 frames longer on the two graded clips, so the graded stretch now reaches further out where the depth bias below is larger |
| Heading (yaw) refinement | In the pipeline. Raw median axis error 5.1 degrees, 3.3 degrees after the axis consensus and the sign bit, backward-facing boxes 64% to 0.2%, on five Todd Drive clips scored against track motion. A separate self-training pilot on the detection head reached 7.9 to 1.7 degrees median heading error on a held-out clip with the detection rate unchanged |
| AV vs human classification | Not started (the research question) |
The full chain on a new clip, using the environment notes in
docs/install.md. Calibration and analysis run under
.venv; the detector and tracker run under the miniforge Python (see
docs/run_and_eval.md).
# 1. intrinsics from a single frame, no ground truth
python scripts/calibration/run_anycalib_single.py \
--image data/camera-data/<CLIP>/frames/150.jpg \
--out-dir outputs/calibration/camera-data/<CLIP>
# 2. extrinsics: per-clip rotation from the lane vanishing point, site height
# as a constant established independently (guards against PTZ re-pointing)
python scripts/calibration/site_extrinsic.py \
--frames-dir data/camera-data/<CLIP>/frames \
--anycalib-json outputs/calibration/camera-data/<CLIP>/150_anycalib_pinhole_pinhole.json \
--height 16.26 \
--reference outputs/calibration/camera-data/AV_T_WE_1/metric_extrinsic_site.json \
--out outputs/calibration/camera-data/<CLIP>/metric_extrinsic_site.json
# 3. 3D detection (defaults to the DAIR 140.8 m checkpoint at score 0.45, and
# applies the ground-plane convention automatically; --ckpt with a matching
# --config reaches the 102.4 m one)
python scripts/object_detection/run_bevheight_generic.py \
--frames-dir data/camera-data/<CLIP>/frames_all \
--anycalib-json outputs/calibration/camera-data/<CLIP>/150_anycalib_pinhole_pinhole.json \
--extrinsic-json outputs/calibration/camera-data/<CLIP>/metric_extrinsic_site.json \
--out-dir outputs/object_detection/camera-data/<CLIP>_phase1
# 4. tracking at the true frame rate
NUMBA_DISABLE_JIT=1 python scripts/tracking/run_ab3dmot.py --fps 30 --max-age 6 \
--det-dir outputs/object_detection/camera-data/<CLIP>_phase1 \
--out-dir outputs/tracking/camera-data/<CLIP>_phase1
# 4b. refine the heading in place: the axis from a score-weighted 31-frame
# consensus of the track's own detection yaws, then one travel-direction
# sign bit per track. Adds yaw_refined and yaw_det, leaves yaw alone.
python scripts/tracking/refine_yaw.py --clip <CLIP>
# 5. identify the instrumented vehicle from its hood marker (image only,
# GPS is never read), then grade that pre-selected track against GPS
python scripts/tracking/identify_target_by_marker.py --clip <CLIP>
python scripts/tracking/grade_target_vs_gps.py --clip <CLIP>Step 5 exists because selecting the track that best matches GPS and then reporting that track's error is selection on the metric being reported. The marker-based identification chooses the vehicle before GPS is consulted; on both front-view clips the choice was unanimous across every marker frame, and in both cases the best-GPS-match track was a different vehicle.
The DAIR-V2X-I benchmark evaluation is unchanged from upstream:
python experiments/dair-v2x/bev_height_lss_r50_864_1536_128x128_102.py \
--ckpt_path ./checkpoints -e -b 1 --gpus 0scripts/finetune/ implements a self-training loop: high-confidence tracks
generate 3D box labels (position from the track, heading from the track's
direction of motion, height from the road plane, dimensions from the per-track
median), a review tool renders labels beside raw predictions for human
inspection, and a trainer updates the detection head with one clip held out.
docs/server-finetune-setup.md documents the GPU-server environment for
training the full model.
Pilot result (detection head only, held-out clip): median heading error 7.9 to 1.7 degrees, boxes more than 45 degrees off reduced from 8.7% to 3.3%, detections per frame unchanged (7.30 to 7.57). Measured caveat: heading concentration on intersection scenes rises slightly (0.783 to 0.838), so preservation of lane-change behavior is tracked explicitly before classifier use.
Per-frame heading instability, which turned out to be a sign collapse. Box
orientation is predicted per frame from appearance alone; no temporal
information links frames. The earlier "roughly 10% over 45 degrees" reading was
wrong. Measured against track motion over 8,283 detections on the five Todd
Drive clips, 68% of car boxes are more than 45 degrees off the direction of
travel and 64% more than 90 degrees, while the median error on the axis alone
is 5.1 degrees. Almost all of the error is therefore one bit: at this viewpoint
the detector reports a single absolute heading for both traffic streams, so one
stream reads correct and the oncoming one reads backwards, and smoothing cannot
recover it because every detection on a track agrees with every other one. It is
a viewpoint effect, not a broken head: on DAIR-V2X-I's own frames the pretrained
weights agree with the ground-truth sign 99% of the time (measured on the 102.4 m
checkpoint), and the cameras here sit ~16 m above the road with a 59 to 66 degree
field of view, against ~6 m and 42 degrees in the training data. Heading
direction is therefore taken from the direction of travel, one bit per track,
and the axis from a temporal consensus of the track's own detections
(scripts/tracking/refine_yaw.py, step 4b above).
Range-dependent depth bias. Graded against GPS, detections read about
1.2 m too near below 40 m and up to 1 m too far beyond 60 m, with the same
signature on both graded clips. This is model depth behavior, not calibration:
swapping the entire extrinsic leaves it unchanged. Addressing it requires
training the height branch (GPU server; see docs/server-finetune-setup.md).
Duplicate detections. The circle-NMS distance test compares squared distance against the radius parameter, so the effective suppression radius for cars is the square root of the configured value, 2.0 m rather than 4 m. The bug is real, but at the 0.45 export threshold only 0.1% of car boxes have another car box within 2.5 m, so it is not currently worth patching a site-packages file.
models/, layers/, ops/ BEVHeight detector (backbone, BEV head, voxel pooling)
experiments/ training and evaluation configs (dair-v2x R50_102 is CPU-patched)
evaluators/ KITTI-format evaluation
dataset/ DAIR-format dataloader
scripts/calibration/ self-calibration (AnyCalib intrinsics, VP extrinsics, site height)
scripts/object_detection/ BEVHeight inference drivers (DAIR and generic camera data)
scripts/tracking/ AB3DMOT driver, target identification, GPS grading, visualizers
scripts/finetune/ pseudo-label generation, label review, head fine-tuning
scripts/reporting/ shareable visualization packages
scripts/data_converter/ DAIR and Rope3D to KITTI conversion
docs/ install, dataset preparation, evaluation, server setup
Datasets, model checkpoints, run outputs, and vendored third-party code are not
committed; see .gitignore.
- DAIR-V2X-I (infrastructure side): detection benchmark and the training domain of the pretrained checkpoint (7,058 frames from five intersections). Frames are sampled, not continuous video.
- WI DOT roadside clips: 10-second 1920x1080 clips at 30 Hz from Beltline highway cameras, each paired with a GPS trajectory of one instrumented vehicle used exclusively for evaluation. Not distributed with this repository. The 102.4 m checkpoint also boxes pedestrians and bicycles on this freeway footage, where there are neither: 1,605 such boxes across the five Todd Drive clips at score 0.45, against 260 for the 140.8 m default.
This repository extends BEVHeight (Yang et al., CVPR 2023); the detector architecture, configs, and pretrained weights originate there. Additions here: the Mac CPU port, the self-calibration pipeline, the ground-plane convention correction, the generic-camera inference drivers, the MOT integration with image-only target identification and GPS grading, the self-training pipeline, and the reporting tools. Tracking uses the AB3DMOT core (Weng et al., IROS 2020).
Built on: BEVHeight, BEVDepth, DAIR-V2X, AB3DMOT, AnyCalib.
@inproceedings{yang2023bevheight,
title={BEVHeight: A Robust Framework for Vision-based Roadside 3D Object Detection},
author={Yang, Lei and Yu, Kaicheng and Tang, Tao and Li, Jun and Yuan, Kun and Wang, Li and Zhang, Xinyu and Chen, Peng},
booktitle={IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
year={2023}
}