Skip to content
choi-yeeunPublic

About

[ECCV 2026 Oral🔥] Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

Topics

Resources

Stars

35 stars

Watchers

3 watching

Forks

Repository files navigation

🔑 Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

ECCV 2026  ·  Long Oral 🔥

Yeeun Choi1  ·  Youngbeom Yoo1  ·  Joon-Young Lee2  ·  Hyolim Kang1,†  ·  Seon Joo Kim1,†

1 Yonsei University     2 Adobe Research

arXiv   Project Page   Poster   Video   Python 3.11


MERIT = Multi-key Episodic Retrieval with Inference-time Temporal expansion

MERIT answers questions about day-scale video by keeping the memory simple and deferring semantic composition to query time. The memory is a flat key–value index built from dense captions: four complementary keys per 30-second clip. At inference, neighbor filtering expands each retrieved clip into its surrounding context and distills the evidence the question actually needs.

Stage 1 — Episodic memory construction. Each 30-second clip is densely captioned, then a multi-key extraction derives four keys per clip, forming a lightweight key–value memory.

Stage 2 — Agentic retrieval and answering. The solver regenerates a query, matches it against the keys by key-wise max similarity to retrieve clips, and neighbor filtering expands the temporal context to distill query-relevant evidence for the final answer.

⚙️ Get Started

Python 3.11, CUDA 12.x, PyTorch 2.6.0.

bash setup_env.sh             # micromamba + uv; MERIT_CUDA=cu121 for CUDA 12.1
micromamba activate merit
cp .env.example .env          # OPENAI_API_KEY

With Docker instead:

bash docker/build.sh          # merit:1.0
cp .env.example .env
bash docker/run.sh            # shell in /workspace

OPENAI_API_KEY is required for gpt-* models, GOOGLE_API_KEY / VERTEX_API_KEY only for gemini-*. Keys are read from the environment and are never written into the repo or the image.

Tip

Two checks before a long run. python tools/check_runtime.py reports GPUs, installed versions, egress and free disk on the machine you are about to run on; python tools/check_egolife_data.py verifies the data layout below, including that the clips actually resolve.

🎞️ EgoLifeQA

Download lmms-lab/EgoLife and put (or symlink) it so that this exists:

data/EgoLife/
├── EgoLifeQA/EgoLifeQA_A1_JAKE.json
├── captions/A1_JAKE/A1_JAKE_30sec_4_key.json
└── videos/A1_JAKE/DAY<n>/*.mp4

If the dataset lives elsewhere, point --qa-file and --video-root at it instead.

Preprocessing

The four retrieval keys are extracted from 30-second dense captions. We reuse the captions released by WorldMM; you can take those as well, or produce your own by following their captioning procedure. Either way, put them at data/EgoLife/captions/A1_JAKE/A1_JAKE_30sec.json and run:

bash script/extract_keys_egolife.sh \
    --subject A1_JAKE \
    --model gpt-5-mini \
    --concurrency 20

The extracted keys ship with this repo, so this step is only needed to regenerate them with a different model or prompt.

Evaluation

bash script/eval_egolife.sh \
    --exp-name merit_gpt5_top10 \
    --respond-model gpt-5 \
    --top-k 10 --max-rounds 5 --image-max-tokens 512 \
    --num-workers 4 --num-threads 1 --gpu-ids 0,1,2,3 \
    --service-tier flex --timeout 600
bash script/eval_egolife.sh \
    --exp-name merit_qwen3vl8b_top5 \
    --respond-model qwen3vl-8b \
    --top-k 5 --max-rounds 5 --image-max-tokens 512 \
    --num-workers 4 --gpu-ids 0,1,2,3
python eval/score_egolife.py    output/<exp-name>/<exp-name>_final.json
python eval/hit_rate_egolife.py output/<exp-name>/<exp-name>_final.json --margin-clips 2

Tip

On the first run the four keys of every clip are embedded on CPU and cached under .cache/multi_keys/<subject>/. Later runs reuse the cache, including across models.

The seed is fixed, but FlashAttention selects kernels per GPU architecture, and the resulting numerical differences can flip a borderline choice. Ours were run on 4× A100.

📼 Video-MME

Put the videos, test_qa.json and the subtitles from the official release under benchmarks/videomme/data/.

Preprocessing

cd benchmarks/videomme

python preprocess/sample_frames.py --num_workers 32          # -> data/frames
python preprocess/caption_clips.py --concurrency 30 --flex   # -> data/captions
python preprocess/extract_keys.py  --concurrency 30 --flex   # -> data/keys

Frame sampling needs ffmpeg; captioning and key extraction need OPENAI_API_KEY.

Evaluation

bash script/eval_videomme.sh \
    --exp-name merit_gpt5_top10 \
    --respond-model gpt-5 \
    --top-k 10 --max-rounds 5 --image-max-tokens 512 \
    --service-tier flex --timeout 600 --resume

🎬 LVBench

LVBench has no subtitles, so retrieval uses three keys (event, object, summary). Put the videos and video_info.meta.jsonl from the official repo under benchmarks/lvbench/data/.

Preprocessing

cd benchmarks/lvbench

python preprocess/sample_frames.py --num_workers 32
python preprocess/caption_clips.py --concurrency 30 --flex
python preprocess/extract_keys.py  --concurrency 30 --flex

Evaluation

bash script/eval_lvbench.sh \
    --exp-name merit_gpt5_top10 \
    --respond-model gpt-5 \
    --top-k 10 --max-rounds 5 --image-max-tokens 512 \
    --service-tier flex --timeout 600 --resume

🗂️ Layout

MERIT/
├── src/merit/                # shared library
│   ├── merit_memory.py       # MeritMemory — search / filter / answer rounds
│   ├── multikey_memory.py    # MultiKeyMemory — key embeddings + MaxSim retrieval
│   ├── embedding/  llm/      # model wrappers and the prompt templates
│   └── types.py  eval_utils.py
├── preprocess/               # retrieval-key extraction
├── eval/                     # evaluation driver, scoring, hit rate
├── script/                   # shell wrappers
├── tools/                    # setup checks
├── benchmarks/
│   ├── videomme/             # preprocess/ eval/ script/ src/ — same shape
│   └── lvbench/
├── docker/  setup_env.sh     # environment
└── data/

Runs create output/<exp-name>/, .cache/ and .log/.

📝 Citation

If you find our work valuable, please cite:

@article{choi2026keep,
  title={Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding},
  author={Choi, Yeeun and Yoo, Youngbeom and Lee, Joon-Young and Kang, Hyolim and Kim, Seon Joo},
  journal={arXiv preprint arXiv:2608.07663},
  year={2026}
}

(The ECCV 2026 proceedings BibTeX will replace this arXiv entry once available.)

🙏 Acknowledgments

MERIT builds on WorldMM, our baseline. We thank the authors for open-sourcing the EgoLife captions and the preprocessing pipeline our experiments rest on. We are also grateful to the authors of EgoLife, Video-MME and LVBench for building and releasing the datasets, benchmarks and evaluation protocols this work is measured on.

About

[ECCV 2026 Oral🔥] Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding

Topics

Resources

Stars

35 stars

Watchers

3 watching

Forks

Releases

Packages

Contributors

Languages