Skip to content

DIRECTION.md

Latest commit

 

History

History
97 lines (80 loc) · 5.6 KB

DIRECTION.md

File metadata and controls

97 lines (80 loc) · 5.6 KB
Raw

Bugbix: product direction

Decision date: 2026-09-27. Status: public alpha candidate.

Current decision

Continue a focused alpha for post-fix review of Python changes when a known pre-fix Git commit exists. Local trials on merged fixes in two unrelated public projects reproduced six failure-to-pass test transitions; an existing passing test was correctly rejected. A third project's generated version module prevented test collection and exposed a setup boundary. This establishes technical utility in the tested scope, not demand. Seek feedback from developers before broadening the runner or claiming product validation.

User and job

A developer reviewing a coding agent's bug fix wants to know whether its new test actually distinguishes the broken behavior from the fixed behavior. Today that can require stashing changes, transplanting a test by hand, running the old code, restoring the fix, and collecting the results.

Bugbix performs that comparison from the current working tree in one command. It keeps the original checkout intact during snapshot operations and saves the old failure, new pass, and copied inputs for review. The name Bugbix was generated by Namelix and selected by the project owner. It gives the tool a short, playful identity connected to finding bugs.

Evidence and alternatives

  • The 2025 Stack Overflow developer survey reports that developers' leading frustration with AI tools is solutions that are almost right (66%); 45% cite time spent debugging generated code. These are broad survey signals, not Bugbix demand measurements.
  • A Codex issue about weakened regression tests describes a green test run after a coding agent changed tests in ways that weakened their intent. Bugbix checks one narrower failure mode: whether a selected test distinguishes the known pre-fix code from the current code.
  • TEBench studies breaking, stale, and missing tests during agent-driven project changes. It supports the problem area; it does not establish demand for this particular product.
  • Backcheck checks agent claims against session transcripts. Its documentation discusses stale passing test results and false positives. Bugbix directly executes selected tests against two source versions; it does not parse conversational claims.
  • ProofRun records local verification receipts and detects stale evidence. It addresses a neighboring problem.
  • RepoTrials already reconstructs historical failures and fixes to build coding-agent benchmarks. Its method overlaps this project. Bugbix's proposed advantage is a small review tool for an uncommitted change, without benchmark authoring or agent evaluation.
  • FixWitness directly checks whether a new test distinguishes a fix, and accepts generic test commands. Its documented limitation is committed revisions only.
  • Timewitness supports a dirty working tree, but requires arm before editing and supports Node's node:test runner. Its saved pre-edit state can be more accurate than using Git HEAD when the repository was already dirty before the fix.

The fail-before/pass-after technique is established. The research does not justify claiming an unprecedented invention or an empty market. Bugbix's current edge is a posthoc check on dirty Python working trees using a known Git pre-fix commit, with unittest or pytest, test-level selection, and saved traces. This is narrower than a general claim of superiority; adoption remains untested.

What to optimize

Fast local feedback, a single command, raw evidence that can be inspected, machine-readable outcomes, and careful distinctions between assertion failures and setup failures. Avoid requiring an account, model API, or a new agent framework. These are product choices informed by related tools, not claimed customer interviews.

Python fits the supported unittest and pytest test ecosystems. Filesystem snapshots, process control, Git invocation, and JSON use the standard library. Pytest must be installed in the selected test environment; Bugbix itself has no runtime package dependencies.

Decision gates

  1. Mechanical correctness: reject weak tests, errors, skips, empty suites, timeouts, changed inputs, and inconsistent repetitions. Preserve the checkout.
  2. Usability: install the built package into a separate environment and execute the public command outside its source tree. Retain reports and actual logs.
  3. Integration validation: run on real bug fixes from unrelated repositories and reject an unrelated passing test. In addition to earlier MCP proxy and fsspec trials using our own local tests, the public boltons and more-itertools PR tests passed this gate. Humanize remains inconclusive due to build setup.
  4. Product validation: ask independent developers to use it on their own fixes and compare the workflow with FixWitness, Timewitness, and manual Git worktrees. This has not happened; our own repository trials do not count as adoption.

Continue only while feedback supports this narrow job. If users already have an equally easy workflow, pivot rather than expand into another broad agent platform.

Ownership

The Bugbix source was written independently in this project. No source from the earlier mcp-retry patch, Relay, or the related projects was copied into it. The earlier derivative package remains separate and is not this product.