Repository navigation
Add PR regression-check tool for Domain-Connect/Templates - #56
Merged
Merged
Conversation
Replays the "Online Editor test results" links recorded in merged Domain-Connect/Templates PRs through this library's apply_template(), comparing the result against what was recorded at test time. Flags any PR whose recorded result no longer matches current library behavior. Decodes the same gzip+base64 token format used by that repo's check_pr_description.py, re-running apply_template() with the token's own embedded template/inputs (isolating library regressions from template edits made after the test was recorded). Caches fetched PRs in a gitignored .cache/ dir so repeated runs don't re-hit GitHub. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Some flagged discrepancies are known editor-tool quirks rather than
library regressions (e.g. PR #840, where a template mixes an APEXCNAME
record with plain host-relative CNAME records and the recorded editor
result appears to be the editor's own bug). overrides.json, committed
alongside the script and keyed by PR number + each link's saved_at
timestamp, lets these be silenced ("ignore") or have their expected
result corrected ("override_expected") without editing the check logic.
Also adds --add-to-override-ignore/--add-to-override-result to write
these entries directly from the CLI for a given --pr, rather than
hand-editing the JSON.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Key by PR number -> providerId -> serviceId -> host -> [entries with
their own params], instead of PR number -> opaque saved_at timestamp.
PR number stays the top-level key since different PRs can carry
different template versions with legitimately different expected
outcomes; providerId/serviceId/host is how a human actually identifies
a link, with params disambiguating multiple links sharing a host.
Also replace dc_apply_result's positional (new, deleted, final) tuple
with a named {new_records, deleted_records, final_records} dict, so the
override file (and code reading it) isn't relying on array position.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Collapse the dead 'fetched_new = []' branch (only ever assigned, never read) into a single guard clause with the fetch loop. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tools/pr_regression_check/check_pr_regressions.py, a tool that replays the "Online Editor test results" links recorded in merged Domain-Connect/Templates PRs through this library'sapply_template(), flagging any PR whose recorded result no longer matches current library behavior.check_pr_description.py; caches fetched PRs in a gitignored.cache/dir so repeated runs don't re-hit GitHub.overrides.json(committed, keyedPR# -> providerId -> serviceId -> host -> [entries]) to silence known editor-tool quirks or record a corrected expected result, plus--add-to-override-ignore/--add-to-override-resultCLI flags to write these from the command line.apply_template/process_records/resolve_variables, before considering the change done.Verified against ~1600 cached merged PRs (927 with usable test links): all currently pass (one genuine editor-tool discrepancy found in PR #840, recorded in
overrides.jsonwith rationale).Test plan
.venv/bin/python -m pytest test/(374 passed, unaffected).venv/bin/python tools/pr_regression_check/check_pr_regressions.pyagainst cached PR history (927/927 OK)