[Testing - Not to Land] Add Metal/MLX support with local launcher#484

Closed

Jack-Khuu wants to merge 723 commits into

gpu-mode:mainfrom

Jack-Khuu:test-mlx

This pull request is big! We're only showing the most recent 250 commits

Commits on Apr 27, 2025

Commits on Apr 30, 2025

Commits on May 1, 2025

Commits on May 2, 2025

Update amd_workflow.yml (#276)
Mark Saroufim
authored
Skip GPU selection prompt if only 1 possible GPU (#277)
philipbutler
authored

Commits on May 6, 2025

Add index runs(runner, score) (#278)
b9r5
authored

Commits on May 13, 2025

pytorch-level profiler
ngc92
committed

Commits on May 25, 2025

nested db enter (#284)
ngc92
authored

Commits on May 30, 2025

Fix Timeout Bug (#280)
PaliC
authored

Commits on Jun 1, 2025

include report in API response
ngc92
committed

Commits on Jun 4, 2025

cleanup
ngc92
committed

Commits on Jun 5, 2025

compress github action payload (#290)
ngc92
authored

Commits on Jun 11, 2025

fix/failed to trigger gh action (#293)

S1ro1
and
Copilot
authored

Commits on Jun 14, 2025

Commits on Jun 24, 2025

Commits on Jun 28, 2025

Fix: correct db access (#307)
S1ro1
authored

Commits on Jul 15, 2025

filter for active leaderboards
ngc92
committed

Commits on Jul 17, 2025

Commits on Jul 23, 2025

Add health checks for nvidia and AMD (#315)
Mark Saroufim
authored
Fix badge
Mark Saroufim
committed
Update amd-health.yml
Mark Saroufim
authored
Revert "Use Modal for CI" (#314)
Mark Saroufim
authored
Libkernelbot (#316)

Mark Saroufim
and
ngc92
authored
Update modal-deploy.yml
Mark Saroufim
authored
Update modal-deploy.yml
Mark Saroufim
authored

Commits on Jul 26, 2025

refactoring: compute score in separate utility

ngc92
authored and
msaroufim
committed
update eval file

ngc92
authored and
msaroufim
committed
move definition of TypedDicts into separate file

ngc92
authored and
msaroufim
committed

Commits on Jul 27, 2025

Adding unit tests (#319)
ngc92
authored

Commits on Aug 9, 2025

Create uv.lock
Mark Saroufim
authored
Use uv.lock to stop heroku crashes (#324)
Mark Saroufim
authored
Update Procfile
Mark Saroufim
authored
Update Procfile again
Mark Saroufim
authored
Update leaderboard_db.py (#325)
Mark Saroufim
authored
rename gh token in codecov
Mark Saroufim
authored
Revert changes to coverage.yml
Mark Saroufim
authored
Update token for codecov.yml (#326)
Mark Saroufim
authored
Run coverage on PRs only
Mark Saroufim
authored
Update coverage.yml (#327)
Mark Saroufim
authored
fix db to store code with special characters correctly
ngc92
committed
working towards database tests
ngc92
committed
small leaderboard_db improvements
ngc92
committed
update migration to not delete the backed-up code column.
ngc92
committed
improved migration logging
ngc92
committed

Commits on Aug 20, 2025

Some setup fixes (#329)
Snektron
authored

Commits on Aug 23, 2025

Update README.md with OpenReview Paper (#332)

alexzhang13
and
Mark Saroufim
authored

Commits on Aug 26, 2025

fix payload in workflows (#331)
Mark Saroufim
authored
Modal tests (#333)
ngc92
authored
Add permissions for GitHub Actions workflow
Mark Saroufim
authored
Revert "fix payload in workflows (#331)" (#341)
Mark Saroufim
authored

Commits on Aug 27, 2025

Commits on Aug 28, 2025

Commits on Aug 30, 2025

remove the non-exist field updated_at (#347)
yangw-dev
authored

Commits on Aug 31, 2025

Fix: remove tzinfo (#348)
S1ro1
authored

Commits on Sep 9, 2025

Commits on Sep 10, 2025

Commits on Sep 13, 2025

Commits on Sep 14, 2025

Temporary fix, send only split log (#357)
S1ro1
authored

Commits on Sep 16, 2025

github tests (#340)
ngc92
authored

Commits on Sep 18, 2025

Pin pytorch in ROCM dockerfile
Mark Saroufim
authored

Commits on Sep 24, 2025

Enhance GPU health check with distributed testing (#360)
Mark Saroufim
authored
add rocshmem support (#359)

danielhua23
and
Mark Saroufim
authored

Commits on Sep 26, 2025

fix (#361)
Mark Saroufim
authored

Commits on Sep 27, 2025

Remove PR template
Mark Saroufim
committed
Update default timeout in modal_run_config (#363)
Mark Saroufim
authored

Commits on Nov 6, 2025

add draft_code table for code editor (#369)
yangw-dev
authored

Commits on Nov 10, 2025

Nvidia-competition setup (profiling + B200 runners) (#368)

authored

Commits on Nov 11, 2025

Commits on Nov 27, 2025

Commits on Nov 30, 2025

Commits on Dec 14, 2025

Commits on Dec 17, 2025

Feat: add longer polling (#381)
S1ro1
authored

Commits on Dec 18, 2025

Feat: cache workflow file (#382)
S1ro1
authored

Commits on Dec 19, 2025

Commits on Dec 20, 2025

Feat: round robin keys (#385)
S1ro1
authored

Commits on Jan 7, 2026

fix timezone failures (#389)
Mark Saroufim
authored
update Modal image builder version (#388)
Mark Saroufim
authored
Change runner environment for CUDA jobs (#390)
Mark Saroufim
authored
Update Python version to 3.13 for modal deploy
Mark Saroufim
authored
delete legacy leaderboard code
Mark Saroufim
committed
Fix badge links in README.md
Mark Saroufim
authored
Rename workflow files from arc health to on prem health (#391)
Mark Saroufim
authored
Skip MI300x8 tests since the machines are no longer available (#393)
Mark Saroufim
authored
Rename project to KernelBot and update description
Mark Saroufim
authored

Commits on Jan 12, 2026

hidemetadata (#394)
yangw-dev
authored

Commits on Jan 13, 2026

dont block profiling on cli (#396)
Mark Saroufim
authored
Concurrency on custom nvidia runners (#397)
Mark Saroufim
authored
Revert "Concurrency on custom nvidia runners (#397)" (#398)
Mark Saroufim
authored

Commits on Jan 14, 2026

GPU concurrency
Mark Saroufim
committed
update
Mark Saroufim
committed
GPU concurrency (#399)
Mark Saroufim
authored
Simplify NVIDIA workflow runner selection
Mark Saroufim
committed
Merge main and resolve conflicts
Mark Saroufim
committed
Merge pull request #400 from gpu-mode/per-gpu-concurrency
msaroufim
authored
Remove self-hosted label from NVIDIA job
Mark Saroufim
authored
Fix runner configuration for check-pytorch job
Mark Saroufim
authored
Remove runner configuration from check-cuda job
Mark Saroufim
authored
Change runner from nvidia-docker to custom-b200
Mark Saroufim
authored
Change runner to nvidia-docker-b200-8-x86-64
Mark Saroufim
authored

Commits on Jan 17, 2026

Fix: modal timeout
S1ro1
committed

Commits on Jan 18, 2026

Update CUDA version and dependencies in modal_runner
Mark Saroufim
authored
add symlink to python3 in modal runner
Mark Saroufim
committed

Commits on Jan 24, 2026

Increase RANKED timeout from 180 to 360 seconds
Mark Saroufim
authored
Change RANKED timeout value to 180 seconds
Mark Saroufim
authored

Commits on Jan 25, 2026

Fix: too long body (#403)
S1ro1
authored

Commits on Feb 1, 2026

Remove docs site (#410)
Mark Saroufim
authored
Fix Modal CI environment detection (#409)
Mark Saroufim
authored
Bump the uv group across 1 directory with 9 updates (#411)

dependabot[bot]
and
Mark Saroufim
authored
Add CLAUDE.md and update test_bot.md documentation

Mark Saroufim
authored and
msaroufim
committed
Add problem sync module for updating problem sets

Mark Saroufim
authored and
msaroufim
committed
Add API-only mode with admin endpoints and CLI support

Mark Saroufim
authored and
msaroufim
committed
Add tests for admin update-problems endpoint

Mark Saroufim
authored and
msaroufim
committed

Commits on Feb 5, 2026

reduce modal test time (#433)
Mark Saroufim
authored

Commits on Feb 6, 2026

streamlined readme (#434)
Mark Saroufim
authored
Add user submission management API endpoints (#412)
Mark Saroufim
authored

Commits on Feb 7, 2026

Add per-user submission rate limit (1 per hour) (#436)
Mark Saroufim
authored

Commits on Feb 8, 2026

Revert "Add per-user submission rate limit (1 per hour) (#436)" (#438)
Mark Saroufim
authored
"Claude PR Assistant workflow" (#439)
Mark Saroufim
authored
Fix workflow permissions to allow Claude to post PR reviews
Mark Saroufim
committed
Use claude_code_oauth_token instead of anthropic_api_key for OAuth token auth
Mark Saroufim
committed
Switch back to anthropic_api_key for GitHub Actions auth
Mark Saroufim
committed
Enable show_full_output to debug Claude Code Action crash
Mark Saroufim
committed

Commits on Feb 9, 2026

Add CUTLASS v4.3.5 C++ headers to Modal runner image

Mark Saroufim
authored and
msaroufim
committed
Remove test_cutlass_image.py after validating CUTLASS setup

Mark Saroufim
authored and
msaroufim
committed
Remove Claude GitHub Actions workflows

Mark Saroufim
authored and
msaroufim
committed

Commits on Feb 10, 2026

Add helion to Modal runner image
Mark Saroufim
committed

Commits on Feb 11, 2026

Bump cryptography in the uv group across 1 directory

dependabot[bot]
authored and
msaroufim
committed
Move skills from SKILLS/ to .claude/skills/ for Claude Code

Mark Saroufim
authored and
msaroufim
committed
Add nvidia model workflow for manual dispatch
Mark Saroufim
committed
Update model workflow to use uv venv
Mark Saroufim
committed
Update model workflow: persistent venv + setuptools-scm fix
Mark Saroufim
committed
Update model workflow: local venv
Mark Saroufim
committed
Pass HF_TOKEN to model workflow
Mark Saroufim
committed
Add optional leaderboard_name filter to admin show-stats (#445)
Mark Saroufim
authored

Commits on Feb 12, 2026

Add /health endpoint to API (#446)
Mark Saroufim
authored

Commits on Feb 14, 2026

Bump sqlparse from 0.5.3 to 0.5.4 in the uv group across 1 directory (#447)
dependabot[bot]
authored

Commits on Feb 24, 2026

Commits on Feb 26, 2026

Add performance indexes for leaderboard queries
Mark Saroufim
committed
Remove duplicate indexes from migration
Mark Saroufim
committed
Add Helion Docker image with auto-publish workflow (#450)
Mark Saroufim
authored
Remove unnecessary index-url for PyTorch installation
Mark Saroufim
authored
Add Northflank to acknowledgements section
Mark Saroufim
authored
Use actions-runner base image for Helion Docker image
Mark Saroufim
committed
Fix uv not found: install pip and uv as root
Mark Saroufim
committed
Fix PyGithub deprecation warning: use Auth.Token instead of login_or_token
Mark Saroufim
committed
add index for query leaderboard ranking (#451)
yangw-dev
authored

Commits on Mar 3, 2026

Add flash-linear-attention and causal-conv1d to Helion Docker image (#453)
yf225
authored
Fix CUDA version mismatch in Helion Docker image
Mark Saroufim
committed
Fix causal-conv1d build: use --no-build-isolation to avoid CUDA mismatch
Mark Saroufim
committed

Commits on Mar 4, 2026

Mi355 (#455)
Mark Saroufim
authored
Skip NVIDIA integration tests: B200 runner no longer available
Mark Saroufim
committed
Delete .github/workflows/nvidia-on-prem-health.yml
Mark Saroufim
authored

Commits on Mar 5, 2026

Add test workflow for ARC GPU isolation
Mark Saroufim
committed
Switch MI355X AMD workflow to ARC runner scale set
Mark Saroufim
committed
Update ARC skill doc: 5-node cluster, 40 max runners (#457)
Mark Saroufim
authored
Bump test workflow to 40 jobs to cover all 5 nodes
Mark Saroufim
committed
xx
Mark Saroufim
authored
Update amd-docker.Dockerfile (#458)
danielhua23
authored
Add runner input to NVIDIA workflow for Helion/Nebius B200 support
Mark Saroufim
committed
Add dedicated Helion workflow for Nebius B200 runners
Mark Saroufim
committed
Hardcode runner labels in workflows (fix GitHub 500 on dispatch)
Mark Saroufim
committed

Commits on Mar 7, 2026

Remove uvicorn concurrency limit causing 503s
Mark Saroufim
committed
Suppress C901 complexity lint on trigger method
Mark Saroufim
committed

Commits on Mar 9, 2026

Remove get-api-url command and help text (#460)
Mark Saroufim
authored

Commits on Mar 11, 2026

Commits on Mar 12, 2026

Update helion.Dockerfile
Mark Saroufim
authored

Commits on Mar 13, 2026

Add Hugging Face dataset export flow for competition submissions (#459)
SinatrasC
authored

Commits on Mar 14, 2026

Update runs (#468)
S1ro1
authored

Commits on Mar 16, 2026

Update hf export data timings (#469)
SinatrasC
authored
Add troubleshooting docs for failed ARC ephemeral runners
Mark Saroufim
committed

Commits on Mar 17, 2026

Feat: rate limiting (#470)
S1ro1
authored

Commits on Mar 18, 2026

Add user ban system for blocking submissions (#471)
Mark Saroufim
authored

Commits on Mar 23, 2026

Update ARC runner config: minRunners=40 to keep all runners online
Mark Saroufim
committed

Commits on Mar 31, 2026

more debug info for arc runners
msaroufim
committed

Commits on Apr 1, 2026

Add admin bulk submission delete endpoint (#474)
msaroufim
authored

Commits on Apr 2, 2026

Commits on Apr 4, 2026

Update Modal runtime for Princeton leaderboard
msaroufim
committed

Commits on Apr 5, 2026

Update timeout (#476)
SinatrasC
authored

Commits on Apr 7, 2026

Commits on Apr 10, 2026

Bump cryptography in the uv group across 1 directory (#478)
dependabot[bot]
authored

Commits on Apr 15, 2026

Bump pytest from 8.4.1 to 9.0.3 in the uv group across 1 directory (#479)
dependabot[bot]
authored

Commits on Apr 17, 2026

Bump python-multipart in the uv group across 1 directory (#480)
dependabot[bot]
authored

Commits on Apr 24, 2026

Bump python-dotenv in the uv group across 1 directory (#483)
dependabot[bot]
authored

Commits on May 1, 2026

Commits on May 4, 2026

Workflow file
Jack-Khuu
committed

Commits on May 5, 2026

Updated instructions for ARC GH workflow
Jack-Khuu
committed