AI Engineer in Gurugram, India · RL environments and LLM evaluation · Python, Docker, AWS
🌐 Website: abhishekcpp007.github.io
I build the environments that LLM coding agents train and get graded in, and then I try to break my own grader. My day job is turning real merged GitHub pull requests into reproducible Docker tasks with hidden tests, then checking that agents actually solve them instead of gaming the scorer. I've shipped 2,000+ of these environments and 2,000+ agent trajectories for SFT and RL post-training.
The thing I keep finding: a grader you haven't attacked is a grader you can't trust. When I red-teamed my own, a stealth cheat policy scored 7/10 before I rebuilt the canary tests and dropped it to 0/60. Details below.
I'm open to AI engineer, LLM eval and Python backend roles, remote or relocation.
My main open-source project, and the fastest way to see how I work. It mines PRs merged after a model's cutoff, rebuilds each repo at the base commit in Docker, and grades agents the way SWE-bench style benchmarks do. Then it tries hard to cheat its own grader.
| Repo | What it does |
|---|---|
| gauntlet-harness | The core. PR mining, 3-run F2P/N2P/S2P/P2P test classification, an 8-check leak scan, rubric judging, a Gymnasium env (gauntlet/SWE-v0) and exports for SWE-bench, SFT, GRPO and DPO |
| gauntlet-cheatcatcher | 11 reward-hacking rules that flag test tampering, conftest tricks and hardcoded outputs |
| gauntlet-repo2env | Turns a repo into env images and SWE-style tasks, with grading hardened against test edits |
| gauntlet-sandbox | Docker sandboxes that record every agent step as a trajectory |
| gauntlet-timemachine | Replay, branch and diff trajectories, plus an HTML viewer |
| gauntlet-arena | FastAPI app where a human and an AI agent race on the same task, with a leaderboard |
| gauntlet | Umbrella repo. One docker compose up runs the whole thing |
Gauntlet Live (Nov 2026): 53 verified tasks out of 186 PRs across 10 repos. The oracle solves 53/53 and the cheater policies get caught 53/53. Red-teaming my own grader found a "stealth" cheat that scored 7/10, so I rebuilt the canary tests until it dropped to 0/60. CI runs a nightly calibration on random tasks.
I read the code of LLM eval and interpretability libraries I use, find real bugs, and send fixes with regression tests. Every one of these started as a bug I hit or spotted myself.
| Project | ⭐ | PR | Status | What I fixed |
|---|---|---|---|---|
| virtualenv pypa |
5k | #3514 |  |
The csh activator never unset PYTHONHOME, so the activated interpreter died with No module named 'encodings' before running anything |
| tox-uv tox-dev |
356 | #418 |  |
system_site_packages was missing from python_cache, so changing it never recreated the environment and the setting had no effect in either direction |
| tox tox-dev |
4k | #4393 |  |
The same stale-environment gap upstream in VirtualEnv.python_cache, found while checking whether the tox-uv one was plugin specific |
| pyproject-api tox-dev |
25 | #354 |  |
ensure_empty_dir gated on exists(), which follows symlinks: a dangling link raised FileExistsError and a link to a directory emptied its target |
| SAELens decoderesearch |
1.6k | #776 |  |
random_firing_probabilities built a fresh torch.Generator(), which starts from a fixed seed, so unseeded calls returned identical probabilities every time |
| delphi EleutherAI |
280 | #192 |  |
split_quantiles floored the per-quantile count, so 25 samples over 10 quantiles gave 20, and fewer samples than quantiles gave 0 with no warning |
| nnsight ndif-team |
1.1k | #800 |  |
Reassembling a multi-invoke .skip() lost the container type: namedtuple outputs raised TypeError and ModelOutput/OrderedDict came back as plain dicts |
| TransformerLens TransformerLensOrg |
4k | #1965 |  |
Eval helpers hardcoded "cuda", which crashed on CPU and Apple MPS machines |
| lighteval huggingface |
2.6k | #1465 |  |
The LiveCodeBench sandbox's fake stdout had no binary buffer, so correct solutions were graded as wrong |
| sparsify EleutherAI |
740 | #146 |  |
handle_arg_string used str.isnumeric(), so -5 silently became -5.0 and Unicode numerals like ½ crashed |
| pluggy pytest-dev |
1.7k | #827 |  |
A 1.7.0 regression let subset_hook_caller call plugins it was told to exclude |
| filelock tox-dev |
986 | #908 |  |
Read-write locks took blocking=0 as blocking and waited the full timeout |
| Gymnasium-Robotics Farama |
2.0k | #362 |  |
get_joint_qvel read 4 velocities for a MuJoCo ball joint instead of 3 |
| Shimmy Farama |
932 | #169 |  |
OpenSpiel reset() dropped a game's parameters, crashing or silently changing the game |
| platformdirs tox-dev |
1.0k | #686 |  |
Media and fonts functions couldn't take use_site_for_root like their properties |
| TransformerLens TransformerLensOrg |
4k | #1976 |  |
stack_head_results silently returned the wrong residual stream for a negative layer, unlike its neuron sibling |
| Minigrid Farama |
2.5k | #551 |  |
MissionSpace rejected valid missions when two placeholders shared exactly one character |
| PettingZoo Farama |
3.5k | #1538 |  |
Random rollout helpers ignored an action mask passed in info and took illegal actions |
| MO-Gymnasium Farama |
1.2k | #172 |  |
breakable-bottles returned rewards and observations outside its declared spaces |
| more-itertools more-itertools |
4.1k | #1346 |  |
Type stubs for concurrent_tee and serialize disagreed with the runtime code |
| tox tox-dev |
4k | #4371 |  |
A re.VERBOSE regex dropped a literal space, so commented-out PEP 723 lines were read as live metadata |
| SWE-bench SWE-bench |
6.1k | #711 |  |
A bare FAIL: line crashed the Django log parser and killed grading for that instance |
| Gymnasium Farama |
12.8k | #1827 |  |
MaxAndSkipObservation returned the previous step's frames when an episode ended mid-skip |
| circuit-tracer decoderesearch |
2.9k | #124 |  |
A bare squeeze() broke single-token prompts and let real batches through unchecked |
| BrowserGym ServiceNow |
1.4k | #421 |  |
hide_bid_if_invisible crashed with an AttributeError instead of the clear error its sibling flags give |
| iniconfig pytest-dev |
64 | #89 |  |
str.splitlines() also split on form feeds, vertical tabs and U+2028, so values broke in two and lineof()/ParseError line numbers shifted |
| pytest-mock pytest-dev |
2.0k | #661 |  |
mocker.spy() on an inherited classmethod/staticmethod raised TypeError when called through an instance, a regression since 3.13.0 |
| pytest-randomly pytest-dev |
724 | #755 |  |
xdist workers re-read the cache for --randomly-seed=last, so with --cache-clear each got its own seed and collection crashed |
Bug reports
- langchain-ai/deepagents#7477:
executetimeout leaves the command's child processes running (fix ready on my fork) - langchain-ai/deepagents#7478:
FilesystemBackendwrites truncate in place, so a crash mid-write destroys the file - langchain-ai/deepagents#7479:
langchain-google-genaiis a required dependency but never imported - huggingface/lighteval#1464: LCB grader fails correct solutions that write via
sys.stdout.buffer
Daily: Python · Docker · pytest · Git · AWS (ECR, S3, Lambda, Glue) · Linux · GitHub Actions
LLM and RL: Claude, GPT and GLM APIs · LLM-as-judge · Hugging Face · PyTorch · Terminal-Bench / Harbor · Multi-SWE-bench · LangChain
Backend and data: FastAPI · PostgreSQL · Redis · SQL · Snowflake · ETL pipelines
- Ethara AI (via Quess Corp) · Software Engineer, RL Environments and Evaluation · 2026 to now
- NavAdr Technologies · Software Engineer (Data) · 2025 to 2026. Python ETL into Snowflake, 200K+ records a day
- GlobalLogic · Associate Analyst · 2024 to 2025. CI/CD with Docker and GitHub Actions on AWS
- quizbuzz-night: live multiplayer trivia buzzer. 2 to 8 players join from their phones with a room code, no login
- neetcode-submissions: my DSA practice in Python