Shipping a vision system means knowing what a pixel measured, what a metric counted, and which number in a paper will survive contact with a camera. This is the reading that gets there, from a photon hitting a sensor to a model running in production, in the order that makes the next piece legible.
Scope. Image formation through deployment: sensors and optics, classical geometry, data and evaluation, the architectures and training that sit on them, the task families built from them, and what breaks in production. Not robotics control, not graphics for its own sake, and not general machine learning without a vision consequence.
Evidence. Primary sources only: the paper, the specification, the official model card, the repository, or the documentation of whoever maintains the tool. Every number behind an entry carries all the fields set out in What earns a place, or it is not used, and misc/notes/changelog.md counts what was checked and what was turned away.
Proof. Every section ends in a benchmark under misc/benchmarks/: the source, the pinned environment, the machine, the raw numbers and the analysis, all committed. Run them.
Section 1 is a path through the rest. Read it top to bottom before using the numbered sections as a reference. The rules the list is built to are in misc/notes/conventions.md.
- 1. Start here: the minimum mental model
- 2. Image formation: optics, sensors, the ISP, colour, noise
- 3. Classical foundations: filtering, features, multi-view geometry
- 4. Datasets, labels and splits: where ground truth comes from and how it lies
- 5. Evaluation: IoU, mAP, protocols, and the ways a metric misleads
- 6. Convolutional architectures
- 7. Attention, vision transformers and hybrids
- 8. Training: objectives, augmentation, schedules, regularisation
- 9. Self-supervised learning and vision foundation models
- 10. Detection and segmentation
- 11. Video: temporal modelling and tracking
- 12. 3D: depth, stereo, SfM, SLAM, radiance fields and splatting
- 13. Generative vision: diffusion, editing, control
- 14. Vision-language models
- 15. Efficiency and deployment: quantisation, distillation, compilers, edge, NPU
- 16. Robustness and failure modes: distribution shift, adversarial, bias, safety
- Watchlist
- What earns a place
- License
Each entry assumes only the ones before it, so the order is the instruction.
- Recovering High Dynamic Range Radiance Maps from Photographs - Why a stored pixel value says nothing about radiance until the camera's response curve is known.
- Camera Calibration and 3D Reconstruction - Which distortion terms a calibration routine solves for, and so which lens errors it leaves in.
- A Pixel Is Not A Little Square - Why treating a sample as a little square breaks every resampling and alignment step after it.
- A guide to convolution arithmetic for deep learning - Why a padding or stride choice moves where each output value sits on the input grid.
- ImageNet Large Scale Visual Recognition Challenge - Where the task definition and the scoring rule behind a top-one accuracy figure are written down.
- ImageNet Classification with Deep Convolutional Neural Networks - What a convolution stack trained end to end on a labelled benchmark learns in place of designed features.
- Understanding deep learning requires rethinking generalization - Why fitting the training labels and generalising to new images are separate events.
- Are we done with ImageNet? - What the single label per image assumption hides when a picture holds several objects.
- MLPerf Inference Benchmark - Why a latency figure needs a scenario, a tail percentile and a timer boundary before it can be compared.
- Hidden Technical Debt in Machine Learning Systems - Where the costs that arrive after a model ships are named, from entangled features to unseen consumers.
Reproduce it: misc/benchmarks/01-transfer-function-pixels, one scene encoded through two transfer curves becomes two different arrays.
- High-Quality Computational Imaging Through Simple Lenses - Why a real lens blurs differently across the frame and across colour channels.
- A Generic Camera Model and Calibration Method - Which polynomial fisheye model a toolbox means when it offers Kannala-Brandt.
- The Double Sphere Camera Model - What a wide-angle model has to trade between fitting the lens and inverting the projection in closed form.
- EMVA Standard 1288 - Where dynamic range, read noise and quantum efficiency get definitions a measurement can falsify.
- Practical Poissonian-Gaussian noise modeling and fitting - Why raw noise grows with the signal, and how that dependence is fitted from a single frame.
- Learning to See in the Dark - What survives in a raw frame when almost no photons arrived.
- Fast Fourier Color Constancy - Why white balance is an estimate of the illuminant with an error of its own, not a fixed gain.
- CSS Color Module Level 4 - Where the piecewise sRGB curve and its conversions to other spaces are kept in a maintained standard.
- ITU-R BT.709 - Which primaries and camera encoding curve an unlabelled video frame is assumed to use.
- ITU-R BT.2100 - Which transfer functions a high dynamic range signal may carry, and the display each one assumes.
- ICC Specifications - How a colour space travels between two pipelines that agree on pixels and disagree on colour.
- High-quality linear interpolation for demosaicing - Where the linear demosaicing baseline that later methods are scored against is specified.
- Unprocessing Images for Learned Raw Denoising - Which camera pipeline stages, in which order, a processed image is walked back through to resemble raw.
- Burst photography for high dynamic range and low-light imaging - Why a phone camera aligns and merges a burst of short frames in place of one long exposure.
Reproduce it: misc/benchmarks/02-signal-dependent-noise, noise that grows with the signal, and a denoiser tuned at one exposure failing at the others.
- Computer Vision: Algorithms and Applications - Where the resampling and pyramid machinery the later papers assume is derived in one place.
- The Laplacian Pyramid as a Compact Image Code - Why a smoothing filter and a decimation step have to be chosen together.
- On Aliased Resizing and Surprising Subtleties in GAN Evaluation - Why the same resize call returns different pixels from different libraries.
- Finding Edges and Lines in Images - Which criteria the Canny operator is derived from, and what each one trades against the others.
- Feature Detection with Automatic Scale Selection - How a detector picks the scale it responds at from the image itself.
- A Combined Corner and Edge Detector - Why the Harris corner test rejects edges and responds only where two gradients meet.
- Distinctive Image Features from Scale-Invariant Keypoints - Where the scale-invariant descriptor and the nearest neighbour ratio test are defined.
- SuperPoint - How a keypoint detector is trained from synthetic shapes and homographic warps of real images.
- SuperGlue - How correspondence is posed as an assignment between two sets of keypoints.
- Image Matching across Wide Baselines - How a matching benchmark scores features by the camera pose they recover, not by match counts.
- A Flexible New Technique for Camera Calibration - Where the planar target calibration that toolboxes implement is specified.
- Determining the epipolar geometry and its uncertainty - Why a recovered fundamental matrix depends on coordinate normalisation and on the cost minimised.
- Recognising Panoramas - Why homographies estimated pair by pair drift apart across a panorama.
- Random Sample Consensus - Where fitting a model by rejecting most of the data is first stated as a method.
- Locally Optimized RANSAC - Where the leading hypothesis is refined by local optimisation inside the sampling loop.
- MAGSAC++ - How a robust estimator scores a model when the inlier threshold is marginalised away.
- Bundle Adjustment - A Modern Synthesis - Where the cost, the sparse normal equations and the gauge freedom are set out together.
- Ceres Solver - Which linear solver and robust loss a bundle adjustment needs before it runs on real data.
Reproduce it: misc/benchmarks/03-gaussian-against-box-blur, a box blur inverting contrast at frequencies where a Gaussian stays positive.
- The PASCAL Visual Object Classes (VOC) Challenge - Where the annotation guidelines and the difficult flag that later benchmarks inherited are written down.
- Microsoft COCO - What counts as one instance, and when a region is marked as crowd and not labelled.
- LVIS - Why a long-tail vocabulary is labelled federated, with each image checked for only some categories.
- From ImageNet to Image Classification - How the crowd validation step that built the benchmark decided which images and labels survived.
- Human uncertainty makes classification more robust - Where annotator disagreement is kept as a soft label and not voted away.
- Confident Learning - How a label error estimate is produced, and the noise assumption it rests on.
- Pervasive Label Errors in Test Sets - How test set label errors are flagged by a model and then confirmed by human review.
- Do We Train on Test Data? Purging CIFAR of Near-Duplicates - Why a test image can sit in the training split as a near copy that no checksum catches.
- Do ImageNet Classifiers Generalize to ImageNet? - How a fresh test set is rebuilt from the original collection procedure, and where that procedure cannot be repeated.
- Identifying Statistical Bias in Dataset Replication - Why a replacement test set can differ from the original through its own selection step.
- Datasheets for Datasets - Which questions about collection, consent and maintenance a dataset has to answer before it is trusted.
- Model Cards for Model Reporting - What a model card has to report, results split by group included, before a checkpoint is trusted.
- Large image datasets: A pyrrhic win for computer vision? - What an audit of a shipped vision dataset finds inside it, consent included.
Reproduce it: misc/benchmarks/04-near-duplicate-leakage, the same classifier scored under a random split and a group split.
These protocols are different metrics that share a name, and a score produced under one says nothing about a score produced under another.
- Generalized Intersection over Union - Why overlap cannot rank two candidate boxes that both miss the target.
- The PASCAL Visual Object Classes Challenge 2007 Development Kit - Where the overlap threshold and the eleven point interpolated average precision are specified.
- The PASCAL Visual Object Classes Challenge 2012 Development Kit - Why sampling precision at eleven recall points and integrating it exactly give different scores.
- cocoeval.py - Where a COCO average precision is fixed in code, from recall sampling to the per-image detection cap.
- LVIS API - Why an average precision from this scorer is not comparable with one from the COCO protocol.
- Open Images evaluation protocols - What a protocol does with a label hierarchy and with boxes that cover a group of objects.
- Accounting for Variance in Machine Learning Benchmarks - Whether a gap between two reported scores survives the randomness that produced both.
- Panoptic Segmentation - Where one score covering stuff and things is defined so that segmentation and recognition errors separate.
- COCO 2018 Panoptic Segmentation Task API - How void pixels and crowd regions are handled in the code that computes panoptic quality.
- Boundary IoU - Why a mask score can rise while the boundary a viewer looks at gets worse.
- Diagnosing Error in Object Detectors - Which kinds of error one average precision folds together, from localisation to background.
- TIDE - Why two detectors with one headline score can need different fixes.
- Evaluating Large-Vocabulary Object Detectors - How average precision moves when only the calibration of scores across categories changes.
Reproduce it: misc/benchmarks/05-average-precision-protocols, one set of detections scored under four published average precision protocols.
- Very Deep Convolutional Networks - Why a stack of small kernels can replace one wide kernel, and what the parameter bill becomes.
- Going Deeper with Convolutions - Where a layer becomes parallel branches of several kernel sizes, with pointwise reductions to pay for them.
- Making Convolutional Networks Shift-Invariant Again - Why a strided network's output can change under a one-pixel shift, and where a blur before subsampling goes.
- Deep Residual Learning for Image Recognition - Where the identity shortcut around a stack of layers is introduced, with the training failure it targets.
- Identity Mappings in Deep Residual Networks - Which order of normalisation, activation and addition a residual block uses, and what reaches the shortcut.
- Wide Residual Networks - Whether depth or width carries a residual network, tested on small images by moving one at a time.
- Aggregated Residual Transformations for Deep Neural Networks - Where grouped convolution enters the residual block as a third axis beside depth and width.
- Squeeze-and-Excitation Networks - What a channel reweighting block computes, and what it costs when added to an existing backbone.
- MobileNets - Where the cost of a depthwise separable layer is derived and the width and resolution multipliers are defined.
- Xception - How the multi-branch block of the Inception family reduces to depthwise separable convolution.
- MobileNetV2 - Why an inverted bottleneck keeps activation memory inside a mobile budget.
- Bag of Tricks for Image Classification - Where the recipe changes usually credited to architecture are ablated one at a time.
- EfficientNet - Where compound scaling is proposed alongside a baseline recipe that changed at the same time.
- ResNet strikes back - Why a published baseline accuracy reports its training recipe as much as its architecture.
Reproduce it: misc/benchmarks/06-padding-stride-geometry, tensor shapes that match the formula while the feature moves.
Most comparisons in this section use unequal pretraining data or compute, so the order of entries is a reading order and not a ranking.
- Attention Is All You Need - Where the scaled dot product operator and the position encoding it needs are defined.
- Non-local Neural Networks - What a global attention block costs when it is dropped into a convolutional stack.
- On the Relationship between Self-Attention and Convolutional Layers - Whether an attention layer can express a convolution, which decides what comparing the two means.
- An Image is Worth 16x16 Words - Where an image becomes a sequence of patch tokens fed to an unmodified transformer encoder.
- Training data-efficient image transformers - How a vision transformer is trained on a public corpus with a teacher network and heavy augmentation.
- Conditional Positional Encodings for Vision Transformers - Why a fixed position embedding has to be interpolated when the input resolution changes.
- Vision Transformers Need Registers - Why a trained vision transformer stores high-norm tokens in background patches, and where extra tokens give them room.
- How to train your ViT? - How augmentation, corpus size and compute trade against each other under one protocol.
- Better plain ViT baselines for ImageNet-1k - How much of the gap between a plain vision transformer and a convolutional baseline was recipe.
- Vision Transformer and MLP-Mixer Architectures - Which checkpoint was pretrained on which corpus, stated for each released file.
- Pyramid Vision Transformer - Why dense prediction needs a feature pyramid that a single-scale token grid cannot give.
- Swin Transformer - How windowed attention makes cost grow with image area and not with its square.
- A ConvNet for the 2020s - What a convolutional network reaches when it borrows the transformer recipe and block design step by step.
- MetaFormer Is Actually What You Need for Vision - Whether the attention operator or the block around it is doing the work.
Reproduce it: misc/benchmarks/07-attention-against-convolution-scaling, attention and convolution cost fitted over a token count sweep.
- When Does Label Smoothing Help? - What label smoothing does to the clusters a classifier forms in its penultimate layer.
- Focal Loss for Dense Object Detection - What a dense grid of mostly background positions does to cross entropy, and how the loss is reweighted.
- The Lovász-Softmax loss - Why optimising an overlap measure directly differs from optimising it pixel by pixel.
- mixup - Why a training target can be a blend of two labels.
- CutMix - Why a pasted region is wasted unless the label is mixed in proportion to its area.
- AutoAugment - Where augmentation becomes a policy searched for each dataset.
- RandAugment - Whether a policy search buys anything that a single strength setting cannot.
- Fixing the train-test resolution discrepancy - Why random-resized crops make the right test resolution differ from the training crop size.
- SGDR - Where the cosine schedule that recipes reuse without its restarts is defined.
- Accurate, Large Minibatch SGD - Where the linear learning rate scaling rule and gradual warmup are stated for large batches.
- Decoupled Weight Decay Regularization - Why weight decay and an added squared penalty stop being equivalent under an adaptive optimiser.
- Mixed Precision Training - Why half-precision training keeps a single-precision copy of the weights and scales the loss.
- Image classification reference training scripts - Where the exact flags behind each reference checkpoint are written down.
- Batch Normalization - Why a layer whose output depends on the rest of the batch behaves differently in training and inference.
- Layer Normalization - Where normalisation moves from the batch axis to the feature axis.
- Group Normalization - What breaks in batch statistics when each device holds too few images to estimate them.
- Rethinking "Batch" in BatchNorm - Why a model scores differently when its test-time batch statistics are not the ones it trained with.
- Deep Networks with Stochastic Depth - Where whole residual blocks are dropped at random during training, the regulariser behind drop path.
Reproduce it: misc/benchmarks/08-augmentation-label-geometry, three ways of handling a box label under five augmentations.
Frozen feature accuracies in this section come from different probe protocols and cannot be compared across entries.
- Representation Learning with Contrastive Predictive Coding - Where the InfoNCE objective shared by the contrastive methods below is written down.
- SimCLR - Why the augmentation list, not the loss, decides what the representation keeps.
- MoCo - How the supply of negatives is decoupled from the batch that has to fit in memory.
- Alignment and Uniformity on the Hypersphere - Which two properties a contrastive loss trades, each measurable without a downstream task.
- Bootstrap your own latent - Whether a joint embedding method needs negatives to stay off a constant output.
- Exploring Simple Siamese Representation Learning - Which single operation keeps a Siamese network from collapsing to one vector.
- DINO - Where centring and sharpening stand in for negatives and the nearest neighbour probe becomes routine.
- BEiT - What a masked prediction target inherits from the tokeniser that produced it.
- Masked Autoencoders Are Scalable Vision Learners - Why reconstructing pixels teaches anything when most of the image is hidden from the encoder.
- DINOv2 - Where the scaling work moves from the objective to the curation of the pretraining corpus.
- Model Card for DINOv2 - Which checkpoints carry registers, and which probe produced each frozen feature number.
- Model Card for DINOv3 - Where the licence terms stop being permissive and the satellite-trained weights are listed apart.
- How Well Do Self-Supervised Models Transfer? - Where many-shot linear accuracy stops predicting detection and dense prediction.
- VTAB - Where the adaptation protocol is fixed so that two representations can be compared.
- Understanding Dimensional Collapse in Contrastive Self-supervised Learning - What the spectrum of an embedding exposes that a probe accuracy hides.
Reproduce it: misc/benchmarks/09-linear-probe-against-nearest-neighbour, two probes ranking the same representations in opposite orders.
Most latency figures in this section leave preprocessing and non-maximum suppression outside the timer, so they compare neither with each other nor with a deployed system.
- Faster R-CNN - Where proposals move inside the network and anchors become the assignment rule.
- SSD - How a single pass predicts boxes from default anchors on several feature maps at once.
- Feature Pyramid Networks for Object Detection - Why a box of a given size is assigned to one pyramid level and not another.
- YOLOv3 - How a grid-based single-stage detector predicts boxes at three scales from one network.
- Cascade R-CNN - Why a detector trained at one overlap threshold degrades when scored at a higher one.
- FCOS - What a detector loses and keeps when the anchor box and its hyperparameters are removed.
- Adaptive Training Sample Selection - Whether anchor-free and anchor-based detectors differ in anything but how positives are chosen.
- End-to-End Object Detection with Transformers - Where bipartite matching replaces both the anchor grid and the duplicate removal pass.
- Deformable DETR - Where attention to a few sampled points around each reference replaces dense attention in set prediction.
- U-Net - Why skip connections between encoder and decoder decide how sharp a mask boundary is.
- Mask R-CNN - Why rounding a pooled region to the feature grid misaligns its mask, and what exact-coordinate sampling changes.
- Segment Anything - Why one click returns several candidate masks with no class attached, and how they are ranked.
- NonMaxSuppression - Where a runtime keeps a box exactly at the overlap threshold, and which box formats it accepts.
- Soft-NMS - Why a neighbouring object disappears at one overlap threshold and survives at another.
- Detectron2 - Where the postprocessing defaults behind several published numbers can be read as code.
- YOLOv10 - Where duplicate removal is trained into the network and not left to a separate postprocessing step.
Reproduce it: misc/benchmarks/10-nms-threshold-operating-point, the suppression threshold moving the operating point, and what suppression costs.
- Temporal Segment Networks - Why a video accuracy depends on how many snippets were sampled and averaged at test time.
- The Kinetics Human Action Video Dataset - Why two groups training on the same named benchmark may not hold the same videos.
- Something-Something - Where class labels are defined by motion templates, so a single frame cannot carry the class.
- PySlowFast Model Zoo and Baselines - Where the crop and clip count at test time sits beside every accuracy it produced.
- Quo Vadis, Action Recognition? - Why a video backbone starts from image weights inflated along the time axis.
- A Closer Look at Spatiotemporal Convolutions - Whether a space-time filter has to be solid or can be split into space and then time.
- TSM - How shifting channels along time gives a two-dimensional network temporal reach with no extra convolution.
- SlowFast Networks for Video Recognition - Why the frame sampling rate is an architectural choice and not a property of the video.
- TimeSformer - How attention is factorised over space and time so that a video transformer fits in memory.
- MOT16 - Which public detections a tracker must consume before two tracking numbers can be compared.
- Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking - Where identity is scored over whole trajectories, in a paper whose dataset was later withdrawn.
- HOTA - Where detection accuracy and association accuracy become two numbers that can disagree.
- MOTChallenge Official Evaluation Kit - Which implementation settles a disagreement about how a tracking metric was computed.
- Simple Online and Realtime Tracking - Where a tracker reduces to a motion model and one assignment problem per frame.
- Deep SORT - What an appearance embedding adds to association when box overlap alone is ambiguous.
- Tracking without bells and whistles - How much of a tracking score belongs to the detector and not the tracker.
- ByteTrack - Where low-scoring detections are kept for a second association pass and not discarded.
Reproduce it: misc/benchmarks/11-per-frame-against-identity, per-frame accuracy rising while identity consistency falls.
- Stereo Matching by Training a Convolutional Neural Network - Where the matching cost is learned while the aggregation around it stays classical.
- RAFT - How dense flow is estimated by iterative updates over an all-pairs correlation volume.
- RAFT-Stereo - What the same iterative update becomes when correspondence is restricted to rectified rows.
- Monodepth2 - What one camera can learn about depth from its own motion, and where the scale has to come from.
- Towards Robust Monocular Depth Estimation - Where the scale-and-shift-invariant protocol for comparing monocular depth across datasets is defined.
- COLMAP - Why an incremental reconstruction registers one image at a time, and where that order lets error drift.
- Global Structure-from-Motion Revisited - Whether solving rotations and translations globally can match incremental accuracy.
- ETH3D Multi-View Evaluation Program - Why a reconstruction score is unreadable without the tolerance it was computed at.
- A Photometrically Calibrated Benchmark For Monocular Visual Odometry - Which response, vignetting and exposure calibration a capture needs before a trajectory error is trusted.
- Direct Sparse Odometry - Why photometric calibration becomes part of the algorithm when image intensities replace keypoints.
- ORB-SLAM3 - Why a feature-based system keeps several maps and merges them when tracking is lost and regained.
- DROID-SLAM - Where a learned front end meets a dense bundle adjustment back end, and what that costs in memory.
- NeRF - Where view synthesis becomes a volume rendering problem fitted one scene at a time.
- Instant Neural Graphics Primitives - How a multiresolution hash table moves an input encoding's work from network weights into lookups.
- 3D Gaussian Splatting - What replaces marching along rays when a scene is a set of rasterised primitives and not a field.
Reproduce it: misc/benchmarks/12-depth-error-against-distance, depth error growing with distance under a fixed disparity error.
Distributional scores in this section depend on the feature network, the sample count and the resize, and a small gap between two of them is not a result.
- GANs Trained by a Two Time-Scale Update Rule - Where the Fréchet Inception Distance that generative results are quoted in is defined.
- Rethinking FID - What the feature network and the sample count contribute to the number before the images do.
- GenEval - How prompt adherence is scored object by object, which a distributional score cannot see.
- Generative Modeling by Estimating Gradients of the Data Distribution - Why sampling can follow the gradient of a log density that is never normalised.
- Denoising Diffusion Probabilistic Models - Where the noise prediction objective is fixed as a reweighted form of the variational bound.
- Score-Based Generative Modeling through Stochastic Differential Equations - Why the discrete noise schedule and the score view are one continuous process.
- Denoising Diffusion Implicit Models - Why the step count is a sampler choice and not a property of the trained network.
- Elucidating the Design Space of Diffusion-Based Generative Models - Which choices of noise level, preconditioning and solver separate a sampler from its training.
- Consistency Models - How a diffusion model is distilled to sample in one or a few steps.
- High-Resolution Image Synthesis with Latent Diffusion Models - Why the process runs in an autoencoder latent, and what that autoencoder throws away.
- Classifier-Free Diffusion Guidance - Why the guidance scale in a text-conditioned interface exists and what it trades away.
- Stable Diffusion v1 Model Card - What a released text-to-image checkpoint was trained on and what its own card says it fails at.
- Scalable Diffusion Models with Transformers - Whether the convolutional backbone was ever the part doing the work.
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis - Where a straight-path objective meets separate weights for text and image tokens.
- SDEdit - How far an image can be changed by adding noise and letting the reverse process finish.
- Prompt-to-Prompt Image Editing with Cross Attention Control - Why cross-attention maps are the handle for changing one object and keeping the rest.
- ControlNet - How a spatial condition attaches to a frozen checkpoint without retraining it.
Reproduce it: misc/benchmarks/13-distributional-metric-sensitivity, a distributional distance moving under changes a viewer cannot see.
- CLIP - Where classification by text prompt is defined, and what it needs from the caption corpus.
- Sigmoid Loss for Language Image Pre-Training - Why a pairwise sigmoid loss frees the batch size from carrying the whole negative set.
- LAION-5B - Which filtering steps decided what an openly released corpus kept, and what they let through.
- DataComp - What changes when the training recipe is frozen and only the data is allowed to move.
- BLIP-2 - What a learned query bottleneck does when the image encoder and the language model both stay frozen.
- LLaVA - Where the instruction data for a conversational image model is synthesised by a text-only model.
- Prismatic VLMs - What happens when the connector, the encoder and the data are varied one at a time.
- Model Card: CLIP - What the people who trained an image-text model say it was not built to do.
- OpenCLIP - Why an overlap audit between a benchmark and a corpus needs checkpoints whose corpus is named.
- Molmo and PixMo - Where training captions are collected from spoken descriptions and not distilled from a closed model.
- Llama 3.2-Vision - Why a stated knowledge cutoff is the field an evaluation date has to be checked against.
- MMBench - Where answer options are rotated across repeated queries so a model scores only when it agrees with itself.
- MMMU - Why a multimodal exam score depends on an answer extraction rule that quoted numbers leave out.
- POPE - How object hallucination is measured with yes or no questions about objects that are absent.
- MMStar - How much of a multimodal score survives when the image is removed from the input.
- VLMEvalKit - Where the prompt template and answer extraction behind a reported score live.
Reproduce it: misc/benchmarks/14-contamination-audit, a contamination audit that needs no access to the model.
Speedups in this section are tied to their own hardware, batch size and timer boundary, and a speedup from one setting says nothing about another.
- Quantization and Training of Neural Networks - Where simulated quantisation in training and the integer-only inference pipeline are specified together.
- TensorFlow Lite 8-bit quantization specification - Why a model quantised by another tool fails on a runtime unless its zero points and ranges match.
- Up or Down? Adaptive Rounding for Post-Training Quantization - Where rounding each weight up or down is learned from calibration data.
- A White Paper on Neural Network Quantization - Where post-training quantisation and its fine-tuning fallback are set out as one decision procedure.
- SmoothQuant - Why activation outliers in transformer layers break per-tensor eight-bit quantisation, and where they are moved.
- Distilling the Knowledge in a Neural Network - Where softened outputs with a temperature become the training target for a smaller model.
- Knowledge distillation: A good teacher is patient and consistent - Which schedule length and augmentation consistency a distillation run needs before its result reproduces.
- What is the State of Neural Network Pruning? - Why two pruning results with the same headline cannot be compared without a shared baseline.
- Accelerating Sparse Deep Neural Networks - Why pruning speeds up a tensor core only when the zeros follow the pattern the hardware executes.
- Open Neural Network Exchange Intermediate Representation - Which side is wrong when one graph runs differently on two runtimes that both claim to follow it.
- Quantize ONNX Models - How a shipping runtime's post-training pass picks calibration, formats and per-channel scales.
- TVM - Why a graph compiler searches schedules for each shape and does not call one vendor library.
- ExecuTorch - Where ahead-of-time export and backend delegation replace a Python process as the deployment unit.
- XNNPACK - Which operators and instruction sets a CPU inference library covers, the list a runtime's fallback depends on.
- MLComputeUnits - Why a Core ML latency figure is unreadable until the compute units the model may use are named.
- Apple Neural Engine Transformers - Which tensor layouts and operator choices a neural engine accepts before a transformer runs on it.
- LoadGen - Where the timer boundary around each query is fixed in code.
Reproduce it: misc/benchmarks/15-int8-quantisation-accuracy-and-latency, integer quantisation accuracy and latency by layer shape.
A score on a corruption suite measures the suite as much as the model, so no score in this section transfers to another suite.
- Natural Adversarial Examples - How a natural test set is built by keeping only images a fixed classifier gets wrong.
- WILDS - Where subpopulation shift and domain generalisation are separated into benchmarks with their own splits.
- Measuring Robustness to Natural Distribution Shifts - Whether robustness to synthetic perturbation predicts robustness to natural distribution shift.
- Benchmarking Neural Network Robustness to Common Corruptions - Where the corruption suite and its error metric are defined.
- make_imagenet_c.py - What each corruption name computes before a score is attributed to a model.
- On Interaction Between Augmentations and Corruptions - Where a corruption suite is built to be perceptually unlike the one augmentations were tuned near.
- 3D Common Corruptions - Which corruptions follow scene geometry, such as depth of field and fog, and so differ from per-pixel noise.
- Explaining and Harnessing Adversarial Examples - Why a small perturbation flips a prediction and how cheaply one is found.
- Towards Deep Learning Models Resistant to Adversarial Attacks - Where the projected gradient attack and adversarial training are paired under a stated threat model.
- Obfuscated Gradients Give a False Sense of Security - Why a defence that only breaks the attacker's gradient fails when the attack is adapted to it.
- Certified Adversarial Robustness via Randomized Smoothing - What a guarantee looks like when the norm and the noise level are stated.
- Adversarial Patch - Why a printed patch placed in the scene falls outside any norm-bounded threat model.
- FairFace - What an audit set has to balance before a disparity between groups can be measured.
- On Calibration of Modern Neural Networks - Why a confidence score is not a probability until it is fitted to be one.
- Can You Trust Your Model's Uncertainty? - What happens to calibration when the test data stops matching the training data.
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples - Where the maximum softmax score becomes the baseline for rejecting inputs a model should not answer.
Reproduce it: misc/benchmarks/16-corruption-tuning-transfer, a pipeline tuned on one corruption, scored on corruptions held out from tuning.
Everything below is real but unproven: no item yet has all three of a specification or paper, a shipped implementation, and a public measurement stating every field. Each line says what would promote it. Last checked 2026-10-21.
- Deep Joint Demosaicking and Denoising - Demosaicing and denoising posed as one learned problem, with no maintained implementation. Enters the core when a maintained implementation is measured against a sequential pipeline.
- LoFTR - Detector-free matching whose pose result has been produced only by its authors. Enters the core when an outside group reruns the released configuration and checkpoint and publishes the pose result.
- ShuffleNet V2 - Design rules derived from measured latency with the precision, warmup and timer boundary unstated. Enters the core when that latency is republished with precision, framework flags, warmup and the timer boundary.
- Searching for MobileNetV3 - A searched mobile family whose on-device latency headline comes without a timer boundary. Enters the core when that latency is published with batch size, warmup and the timer boundary.
- Stand-Alone Self-Attention in Vision Models - A backbone with no convolution left, measured only by its authors. Enters the core when released weights let another group reproduce the comparison at the stated budget.
- Early Convolutions Help Transformers See Better - A stability finding about the patch stem with no released weights behind it. Enters the core when weights for the compared stems are released with the hardware stated.
- CoAtNet - Convolution stacked under attention across growing corpora, with no official code or weights. Enters the core when the authors release weights for the models trained on public data.
- Scalable Pre-training of Large Autoregressive Image Models - Autoregressive pretraining on a corpus nobody else holds. Enters the core when it is compared with contrastive and masked baselines at equal data and compute.
- DINOv3 - A finding that dense features decay over long schedules, measured only on the authors' largest private run. Enters the core when the decay is measured on a released checkpoint with the dense protocol stated.
- Mask2Former - One mask classification design claimed to serve semantic, instance and panoptic tasks alike. Enters the core when every task's result is published with its checkpoint, protocol and hardware.
- DreamBooth - Subject-driven fine-tuning whose evaluation data is public and whose implementation is not. Enters the core when the authors release the training code behind the comparison.
- FLUX - Open weights from a family with no paper, no specification and no public measurement. Enters the core when a paper or specification and a measurement stating every field are published.
- Flamingo - A frozen language model taught to read images, with no released weights or code. Enters the core when weights or an implementation are released with the evaluation protocol.
- Improved Baselines with Visual Instruction Tuning - Ablations that credit gains to the connector and the recipe, with checkpoints only for the final models. Enters the core when checkpoints for the ablation rows are released.
An entry is admitted only when every one of these holds, and the burden of proof sits with the entry.
- It is a primary source: the original paper, the authors' own repository, an official model card or weight page, framework or compiler documentation, a dataset paper or official dataset page, or a standard.
- Its URL resolves to the live canonical copy, never a mirror, an aggregator or a rehost.
- Its line says why a reader would open it, in under one hundred and twenty characters, with no number, no author name and no praise.
- Any number it rests on carries every evidence field in the tables below. A missing field sends it to the watchlist or out, and it is not softened into a caveat.
- It sits in dependency order inside its subsection, and the subsection holds three to six entries. A seventh displaces one, and the pull request names which.
- It enters the core only when a specification or paper, a shipped implementation and a public measurement stating every field all exist at once. Otherwise it waits on the watchlist with the condition that would promote it.
- Each source appears once in the list.
An accuracy claim carries eight fields:
| Accuracy claim | |
|---|---|
| 1 | model and exact checkpoint |
| 2 | dataset, split and version |
| 3 | input resolution and preprocessing |
| 4 | metric with its full definition |
| 5 | protocol, single-scale or test-time augmentation, NMS settings |
| 6 | training data used |
| 7 | hardware |
| 8 | baseline compared against |
A speed claim carries seven:
| Speed claim | |
|---|---|
| 1 | hardware and count |
| 2 | precision |
| 3 | batch size |
| 4 | input resolution |
| 5 | framework and version, with compile flags |
| 6 | warmup and the statistic reported |
| 7 | whether preprocessing and postprocessing are inside the timer |
Every URL points at the live canonical copy, and misc/scripts/check_links.py and misc/scripts/check_format.py prove it on every push and again weekly.
CONTRIBUTING.md has the rules in full.
MIT. Maintained by @usamahz.