Generalizable and Per-Scene NeRF Architectures under Sparse-View Conditions: A Systematic Benchmarking Methodology
This repository hosts the official benchmarking infrastructure, experiment pipelines, and integration wrappers presented in the study:
"Generalizable and Per-Scene NeRF Architectures under Sparse-View Conditions: A Systematic Benchmarking Methodology"
Felipe G. Silva, Leandro A. Neves, Guilherme F. Roberto, Thiago Pozati, Domingos Oliveira, Adriano B. Silva, Thaína A. A. Tosta, Marcelo Z. do Nascimento.
SIBGRAPI 2026 (Conference on Graphics, Patterns and Images).
Important
Repository Evolution & SIBGRAPI WIP Track Disclaimer:
The original publication presented at SIBGRAPI 2026 reflects the preliminary Work-In-Progress (WIP) state of this benchmark, corresponding directly to commit 426d319 (source and PDF available in docs/sibgrapi-2026/).
As an active WIP project, substantial methodological advancements, bug fixes, and quantitative breakthroughs were developed beyond that baseline:
-
5 Fine-Tuning Strategies: Formulated a formal taxonomy (
Full-FT,PFT-Decoder,PFT-Backbone,BitFit, andPEFT-LoRA) with strict gradient isolation; - Overcoming Negative Adaptation: Solved representation collapse, turning around PixelNeRF by +4.54 dB and boosting GNT by +4.65 dB (surpassing per-scene Instant-NGP);
- Step Budget Calibration: Discovered the optimal 500-step budget via systematic iteration ablation, eliminating late-stage overfitting;
-
Zero-Latency LoRA Deployment: Implemented in-place weight fusion (
$\Delta W = \frac{\alpha}{r}BA$ ) viamerge_lora_weights(); -
Statistical Rigor: Instituted multi-seed (
$42, 123, 999$ ) stratified 4-fold cross-validation with automated divergence pruning.
For an exhaustive side-by-side comparison between the preliminary SIBGRAPI WIP results and the current state-of-the-art benchmark, refer to the Evolution Section. The updated, publication-grade manuscript is maintained under docs/final-2027/.
Neural Radiance Fields (NeRFs) have redefined novel view synthesis by learning continuous 5D volumetric representations from multi-view imagery. Despite impressive visual quality under densely sampled inputs, real-world deployment in resource-constrained settings (e.g., casual mobile scanning, medical imaging, and virtual try-on) fundamentally confronts sparse-view acquisition constraints (
Under such constraints, existing neural rendering paradigms diverge:
- Per-Scene Optimization Models (e.g., Instant-NGP, Nerfacto): Optimize continuous scenes from scratch using multi-scale hash tables and lightweight MLPs. While computationally agile, they lack structural priors and are prone to geometric degradation when ray coverage is sparse.
- Generalizable Priors (e.g., PixelNeRF, Generalizable NeRF Transformer / GNT): Infer volumetric representations across unseen scenes by conditioning rendering on pre-trained 2D CNN or Vision Transformer (ViT) feature backbones, optionally adapting weights via test-time fine-tuning (FT).
Although numerous architectures exist, prior evaluations have remained fragmented—often isolated to disparate training splits, unstandardized ray budgets, or hardware configurations. This repository establishes a standardized, fully reproducible benchmarking methodology to systematically investigate the Pareto trade-offs governing visual fidelity, runtime convergence, rendering latency, and memory consumption across these paradigms.
-
Unified Benchmarking Protocol: Standardized data partitions on the forward-facing Local Light Field Fusion (LLFF) dataset, controlled ray budgets (512 training rays, 128 evaluation rays), and reproducible camera sampling regimes across
$N \in {3, 6, 10}$ views. -
Formal Fine-Tuning Taxonomy & Parameter Management: Implementation of five distinct adaptation strategies (
Full-FT,PFT-Decoder,PFT-Backbone,BitFit, andPEFT-LoRA) in modular wrappers, featuring dynamic gradient isolation and test-time in-place weight merging for zero runtime overhead. -
Stratified K-Fold Grid Search: A multi-seed (
$42, 123, 999$ ), 4-fold cross-validation protocol systematically mapping the optimal learning rate ($\eta \in {10^{-4}, 5\times 10^{-4}, 10^{-3}}$ ) and LoRA capacity ($r \in {4, 8}$ ) per strategy. -
Empirical Convergence & Overfitting Profiling: Fine-tuning step ablation (
$0$ to$2,500$ steps) demonstrating convergence bounds and validating the optimal 500-step budget. -
Modular Nerfstudio Integration Wrappers: Standalone packages extending Nerfstudio with PixelNeRF and GNT, distributed via PyPI (
nerfstudio-pixelnerf,nerfstudio-gnt) and GitHub.
The benchmark has progressed substantially from its initial Work-In-Progress (WIP) exploratory phase (snapshot at commit 426d319) to the current publication-ready framework. The preliminary benchmark established baseline evaluations under sparse views but revealed critical vulnerabilities in naive test-time fine-tuning, where unconstrained optimization led to catastrophic representation collapse.
In response, the current benchmark introduced a formal 5-strategy fine-tuning taxonomy, dynamic gradient isolation, zero-overhead LoRA weight merging, stratified 4-fold cross-validation, and empirical step calibration.
The table below contrasts the preliminary WIP results against the current calibrated benchmark across generalizable and per-scene models (
| Model | Evaluated Regime | Metric | Early WIP Baseline (Naive FT, 2500 steps) |
Current Benchmark (Best Strategy, 500 steps) |
Net Change ( (Current vs. WIP) |
Status & Breakthrough Diagnosis |
|---|---|---|---|---|---|---|
| PixelNeRF | Zero-Shot (ZS) | PSNR | 6.62 dB | 6.35 |
|
Calibrated across multi-seed stratified splits |
| PixelNeRF | Fine-Tuning | PSNR | 6.38 dB | 10.92 |
+4.54 dB (+71.2%) | Breakthrough (PFT-Backbone): Overcomes negative adaptation by isolating 2D priors and freezing coordinate MLP |
| SSIM | 0.071 | 0.162 |
+0.091 (+128%) | Resolves severe isotropic blurring and geometric collapse | ||
| LPIPS | 0.801 | 0.772 |
Significantly enhanced perceptual structure | |||
| PixelNeRF | BitFit (Bias-Only) | PSNR | Not Evaluated | 10.79 |
+4.41 dB vs. WIP | Updates |
| GNT | Zero-Shot (ZS) | PSNR | 10.73 dB | 10.51 |
|
Standardized across 6 valid LLFF scenes |
| GNT | Fine-Tuning | PSNR | 10.42 dB | 15.07 |
+4.65 dB (+44.6%) |
Breakthrough (PFT-Decoder): Overcomes |
| SSIM | 0.336 | 0.368 |
+0.032 (+9.5%) | Superior multi-view epipolar ray coherence | ||
| LPIPS | 0.724 | 0.679 |
Noticeable visual sharpness recovery | |||
| GNT | PEFT-LoRA ( |
PSNR | Not Evaluated | 14.57 |
+4.15 dB vs. WIP | Matches Full-FT quality while enabling in-place zero-latency inference merging |
| Instant-NGP | Per-Scene ( |
PSNR | 14.06 dB | 14.06 dB | — | Per-scene fast baseline (now outperformed by adapted GNT!) |
| Nerfacto | Per-Scene ( |
PSNR | 27.11 dB | 27.11 dB | — | Per-scene upper bound under sparse camera supervision |
| Dimension | Early WIP Baseline | Current Benchmark | Key Impact & Innovation |
|---|---|---|---|
| Adaptation Taxonomy | Monolithic, unconstrained full fine-tuning | 5 modular strategies: Full-FT, PFT-Decoder, PFT-Backbone, BitFit, PEFT-LoRA
|
Allows isolating spatial priors (backbone) vs. volumetric renderers (decoder) |
| Parameter Management | Unmanaged backpropagation across entire network | Dynamic parameter isolation via setup_adaptation() and strict gradient freezing |
Eliminates gradient interference and protects learned generalizable priors |
| Adaptation Budget | Arbitrary |
Calibrated |
Eliminates PixelNeRF late-stage overfitting ( |
| LoRA Runtime Cost | Standard forward pass with runtime adapter addition | In-place weight fusion (merge_lora_weights()
|
Zero inference latency overhead for LoRA adapters during rendering |
| Cross-Validation Rigor | Single ad-hoc runs without cross-validation | Stratified 4-Fold CV across 8 LLFF scenes |
Statistically reliable benchmarking; prunes diverged runs; computes standard deviation |
| Hyperparameter Grid | Fixed uncalibrated |
Grid search across |
Uncovers distinct optimal learning rate regimes for CNNs ( |
| Automation & Tooling | Manual command execution and ad-hoc log parsing | Unified CLI suite (run_stratified_kfold_grid_search.py, run_tta_steps_ablation.py) |
Automated multi-seed execution, outlier filtering, and reproducible CSV/LaTeX export |
| Scientific Manuscript | Exploratory draft in docs/sibgrapi-2026/
|
Camera-ready manuscript in docs/final-2027/ with Algorithm 1 & convergence figures |
Formalized pseudocode, empirical justification for step budget, and unified comparison tables |
-
Resolving Representation Collapse via Architectural Inversion:
- In the WIP baseline, unconstrained backpropagation caused coordinate MLPs to rapidly overfit to the few sparse rays, destroying the learned 3D geometry.
- In the current benchmark, isolating the feature backbone (PFT-Backbone) in PixelNeRF achieved 10.92 dB (+4.54 dB over WIP FT). Conversely, GNT's domain adaptation is governed strictly by its multi-view cross-attention transformer; isolating its ray decoder (PFT-Decoder) achieved 15.07 dB (+4.65 dB over WIP FT), establishing the new state-of-the-art for generalizable models on sparse LLFF.
-
Empirical Step Budget Calibration (The 500-Step Sweet Spot):
- Step ablation profiling across
$0$ to$2,500$ iterations revealed that PixelNeRF peaks at exactly 500 iterations (11.03 dB) before suffering catastrophic degradation down to 8.57 dB at 2,500 steps due to sparse-view overfitting. - GNT achieves
$>90%$ of its adaptation gains within the first 500 steps ($13.91$ dB). Standardizing the budget at 500 steps maximizes fidelity while reducing fine-tuning time by$80%$ .
- Step ablation profiling across
-
Zero-Overhead Inference via In-Place LoRA Fusion:
- Low-Rank Adaptation (PEFT-LoRA) inserts trainable decomposition matrices
$\Delta W = \frac{\alpha}{r}BA$ into linear projections. Post fine-tuning,merge_lora_weights()computes and folds$\Delta W$ directly into the base weights$W_0$ , completely eliminating runtime adapter overhead during novel view synthesis.
- Low-Rank Adaptation (PEFT-LoRA) inserts trainable decomposition matrices
-
Statistical Reliability via Multi-Seed Stratified Cross-Validation:
- Evaluating across 3 random seeds (
$42, 123, 999$ ) with 4-fold cross-validation and automated failure pruning (filtering runs with$PSNR \le 0.0$ ) ensures all reported gains are statistically robust and reproducible.
- Evaluating across 3 random seeds (
Experiments were conducted on the 8 real-world scenes of the LLFF dataset at downscale factor 8 (
The table below summarizes multi-seed aggregations (seeds
| Model | Fine-Tuning Strategy | Learning Rate ( |
LoRA Rank ( |
PSNR (dB) |
SSIM |
LPIPS |
Train / FT Time |
|---|---|---|---|---|---|---|---|
| PixelNeRF | Zero-Shot (ZS) | N/A | N/A | 6.35 |
0.085 | 0.800 | $\sim$22 s |
| PixelNeRF | Full-FT | N/A | 9.03 |
0.136 | 0.752 | 4 min 45 s | |
| PixelNeRF | PFT-Decoder | N/A | 10.58 |
0.205 | 0.773 | 4 min 12 s | |
| PixelNeRF | PFT-Backbone | N/A | 10.92 |
0.162 | 0.772 | 4 min 36 s | |
| PixelNeRF | BitFit | N/A | 10.79 |
0.119 | 0.775 | 4 min 38 s | |
| PixelNeRF | PEFT-LoRA | 4 | 8.59 |
0.175 | 0.725 | 7 min 06 s | |
| GNT | Zero-Shot (ZS) | N/A | N/A | 10.51 |
0.314 | 0.742 | $\sim$9 s |
| GNT | Full-FT | N/A | 15.06 |
0.369 | 0.681 | 5 min 57 s | |
| GNT | PFT-Decoder | N/A | 15.07 |
0.368 | 0.679 | 4 min 59 s | |
| GNT | PFT-Backbone | N/A | 10.51 |
0.314 | 0.742 | 6 min 01 s | |
| GNT | BitFit | N/A | 13.12 |
0.358 | 0.742 | 5 min 19 s | |
| GNT | PEFT-LoRA | 4 | 14.57 |
0.369 | 0.706 | 8 min 08 s | |
| Instant-NGP | Per-Scene ( |
N/A | 14.06 | 0.442 | 0.498 | 31 min 51 s | |
| Nerfacto | Per-Scene ( |
N/A | 27.11 | 0.875 | 0.171 | 8 min 57 s |
-
Architectural Divergence in Adaptation:
-
PixelNeRF derives its highest gains from adapting the 2D feature extractor (PFT-Backbone: 10.92 dB,
$+4.57$ dB over ZS) or additive biases (BitFit: 10.79 dB). Because its coordinate MLP easily overfits under sparse supervision, locking the renderer and updating only spatial priors ensures optimization stability. -
GNT achieves peak fidelity by adapting the multi-view cross-attention transformer (PFT-Decoder: 15.07 dB,
$+4.56$ dB over ZS) or low-rank adapters (PEFT-LoRA: 14.57 dB). Inverting this hierarchy (PFT-Backbone) yields no gain over zero-shot, proving that GNT's domain adaptation is governed strictly by its volumetric ray transformer.
-
PixelNeRF derives its highest gains from adapting the 2D feature extractor (PFT-Backbone: 10.92 dB,
-
Efficiency of Parameter-Efficient Fine-Tuning (PEFT):
-
BitFit tunes
$<0.15%$ of parameters (bias vectors only) yet achieves$10.79$ dB in PixelNeRF and$13.12$ dB in GNT, functioning as a built-in regularizer against floaters. -
PEFT-LoRA (
$r=4$ ) matches Full-FT visual quality in GNT while confining gradient updates to a low-dimensional manifold and supporting zero-latency inference weight merging.
-
BitFit tunes
-
Fine-Tuning Step Dynamics & Overfitting:
- PixelNeRF peaks at exactly 500 iterations (11.03 dB) before degrading to
$8.57$ dB at 2,500 iterations due to sparse-view overfitting. - GNT exhibits smooth monotonic convergence up to 2,000 steps (
$14.99$ dB), with$>90%$ of gains realized in the first 500 steps.
- PixelNeRF peaks at exactly 500 iterations (11.03 dB) before degrading to
-
Pareto Trade-offs: While per-scene Nerfacto remains the upper bound for fidelity (
$27.11$ dB), adapted GNT ($15.07$ dB) establishes the highest quality among generalizable architectures, rendering zero-shot in seconds and fine-tuning in under 5 minutes.
generalizable-nerf-benchmark/
├── assets/ # Data, artifacts, and benchmark logs
│ ├── data/baseline/nerf_llff_data/ # LLFF dataset scenes
│ └── reports/ # Consolidated CSVs, plots, and LaTeX tables
│ ├── grid_search/ # PixelNeRF grid search logs
│ ├── grid_search_gnt/ # GNT grid search logs
│ ├── tables/ # Reproducible LaTeX & CSV comparison tables
│ └── tta_steps_ablation/ # Fine-tuning step convergence logs and plots
├── docs/
│ ├── entrega-mestrado/ # Master's dissertation documentation
│ └── final-2027/ # SIBGRAPI camera-ready LaTeX sources & figures
├── scripts/ # Automated benchmark & reporting suite
│ ├── run_stratified_kfold_grid_search.py # Main stratified K-fold cross-validation grid search runner
│ ├── run_tta_steps_ablation.py # Iteration step ablation & convergence profiler
│ ├── generate_grid_search_latex_tables.py # Generates consolidated LaTeX & CSV tables
│ ├── aggregate_metrics.py # Aggregates metrics across scene logs
│ ├── generate_selected_plots.py # Generates radar charts and boxplots
│ └── generate_lr_ablation.py # Learning rate ablation formatter
├── src/ # Core benchmarking framework
│ ├── dataset/
│ │ ├── ImageDataset.py # LLFF loader, downscaling, and partitioner
│ │ └── SubsetWrapper.py # Uniform and random sparse subset sampler
│ ├── nerf/
│ │ └── NerfModel.py # Unified Nerfstudio wrapper (PixelNeRF, GNT, NGP, Nerfacto)
│ ├── utils/
│ │ ├── config.py # Global logger and execution configurations
│ │ ├── metrics.py # MetricsLogger (PSNR, SSIM, LPIPS, VRAM, FPS)
│ │ └── types.py # Enums and metric identifiers
│ └── validation/
│ └── eval_images.py # Standardized evaluation routines
├── Dockerfile # Reproducible container recipe (CUDA 12.6, COLMAP, tiny-cuda-nn)
├── docker-compose.yml # Container service configurations
├── main.py # Base execution entrypoint
├── pyproject.toml # UV package definition and dependencies
└── uv.lock # Deterministic dependency lockfile
The benchmark is designed for fully reproducible execution using Docker with NVIDIA Container Toolkit, isolating system libraries, CUDA kernels, and tiny-cuda-nn bindings.
- Linux (Ubuntu 22.04 / 24.04 recommended)
- NVIDIA GPU (Compute Capability
$\ge 7.5$ ,$\ge 6$ GB VRAM) - NVIDIA Driver
$\ge 525$ and NVIDIA Container Toolkit - Docker & Docker Compose
Run the automated installer script:
chmod +x install.sh run_docker.sh export_delivery.sh
./install.shAlternatively, download the LLFF dataset manually via the Kaggle CLI:
mkdir -p assets/data/baseline/
kaggle datasets download -d arenagrenade/llff-dataset-full -p assets/data/baseline/ --unzipLaunch the containerized experiment suite with automatic GPU architecture detection:
bash run_docker.shOr invoke Docker Compose directly:
GPU_ARCH=86 docker compose up --buildThe container mounts ./assets/data, ./assets/logs, ./src, and ./main.py, logging metrics directly to disk.
If running natively outside Docker with pre-installed CUDA 12.x and COLMAP 3.10:
# Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh
# Synchronize environment dependencies
uv sync --prerelease=allow
# Install tiny-cuda-nn bindings
uv pip install ninja "setuptools<70.0.0" wheel
uv pip install -v git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch --no-build-isolation
# Run the benchmark pipeline
python main.pyAll benchmarking, profiling, and analysis scripts are located in scripts/:
# 1. Run stratified 4-Fold cross-validation grid search across models and strategies
python scripts/run_stratified_kfold_grid_search.py --model gnt --strategy peft-lora --learning-rate 0.001 --lora-rank 4
python scripts/run_stratified_kfold_grid_search.py --model pixel-nerf --strategy pft-backbone --learning-rate 0.0001
# 2. Run fine-tuning step ablation and convergence profiling (0 to 2,500 steps)
python scripts/run_tta_steps_ablation.py --max-eval-images 10
# 3. Generate consolidated LaTeX and CSV comparison tables (e.g., table_mixed_comparison.tex)
python scripts/generate_grid_search_latex_tables.py
# 4. Consolidate and inspect global averages across all scenes and models
python scripts/aggregate_metrics.py
# 5. Generate visual trade-off radar charts and PSNR boxplots
python scripts/generate_selected_plots.py
# 6. Generate learning rate ablation analysis
python scripts/generate_lr_ablation.pyAs part of this research, custom integration wrappers for Nerfstudio were implemented and published to enable reproducible benchmarking:
- Wrapper Source Code: https://github.com/Felipe-gsilva/Few-Shot-NeRFs/
nerfstudio-gnt: PyPI | piwheelsnerfstudio-pixelnerf: PyPI | piwheels
If you utilize this benchmark, codebase, or the integration wrappers in your research, please cite our paper:
@inproceedings{silva2026generalizable,
title = {Generalizable and Per-Scene NeRF Architectures under Sparse-View Conditions: A Systematic Benchmarking Methodology},
author = {Silva, Felipe G. and Neves, Leandro A. and Roberto, Guilherme F. and Pozati, Thiago and Oliveira, Domingos and Silva, Adriano B. and Tosta, Tha{\'i}na A. A. and do Nascimento, Marcelo Z.},
booktitle = {Proceedings of the Conference on Graphics, Patterns and Images (SIBGRAPI)},
year = {2026},
publisher = {SBC},
address = {S{\~a}o Paulo, Brazil}
}This project is licensed under the MIT License. See LICENSE for details.