Complementary public stacking eval for π0-style policies? (browser + harness ELO)

[arankair]

TL;DR — VSArena is a zero-install public stacking benchmark (browser studio + hosted WebSocket harness + public ELO). Looking for feedback on whether this is a useful sanity-check layer next to openpi / RoboArena-style real eval — not a competitor to DROID, and not a request to merge anything into openpi.

Context

RoboArena + openpi already set the bar for real-world / DROID-ready generalist eval. VSArena attacks a different layer:

• One task: stack 3 cubes (cyan → orange → magenta)
• VLA track: 128×128 RGB + language instruction (no cube GPS to the policy)
• Official scores only from a hosted harness (state → action → result); the browser cannot write ELO
• Public leaderboard + live spectator

Hypothesis: before someone trusts a heavier eval slot, a transparent public stacking row can catch obvious failures fast.

We're posting here (per CONTRIBUTING) as a discussion / feedback ask first — not opening a PR.

Links

• Demo: https://vsarena.vercel.app/simulation
• Live spectator: https://vsarena.vercel.app/simulation?view=live
• Leaderboard: https://vsarena.vercel.app/leaderboard
• Submit (researcher): https://vsarena.vercel.app/submit?guide=researcher
• Docs (provenance / failure taxonomy): https://vsarena.vercel.app/docs
• Repo: https://github.com/ONISCOR/VSArena

Questions

1. Is a 3-cube stacking harness too toy to be useful for openpi-style policies, or a reasonable smoke check before LIBERO / DROID / RoboArena?
2. Would anyone on the team (or community) try a trivial baseline submit and tell us where the docs break?
3. Any preference for where an adapter should live if useful — external VSArena docs only, vs a small openpi examples/ note later?

Non-goals

• Not competing with RoboArena real-world pairwise eval
• Not replacing DROID training / openpi pipelines
• Not claiming correlation with RoboArena rankings
• Not asking to merge a new env into openpi core

Related

• RoboArena positioning discussion: [Discussion] Sim-side complement to RoboArena — browser stacking + public ELO? robo-arena/roboarena#5
• Allen AI vla-eval remote adapter PR: feat(benchmark): add VSArena remote stacking harness adapter allenai/vla-evaluation-harness#133

Thanks for openpi — happy to close / ignore if this isn't useful.

— Aran Kair (arankair)