Repository navigation
Conversation
System UI can briefly cover the app during capture, for example the Apple Intelligence notification iOS shows shortly after boot. A covered screenshot published as the baseline from `main` fails every PR based on that commit, and three baselines on `main` were spoiled this way or by flows racing the UI. After capture, `capture.js` now finds the nearest baseline over the public bucket URL and reruns, once and after a short delay, only the flows whose screenshots differ from it or have no baseline: - A changed screenshot is replaced by its retake only if the retake matches the baseline, so a real change still fails. - A new screenshot is unstable unless both captures are identical. The manifest marks it, the report says so, and compare doesn't publish it as the baseline. Screenshots that already matched are never retaken. The step is skipped without a baseline or `VISUAL_PUBLIC_BASE_URL`. Changelog: [Internal]
meta-cla
Bot
added
the
CLA Signed
facebook-github-tools
Bot
added
the
Shared with Meta
| if (baseline == null) { | ||
| return { | ||
| unstable: [], | ||
| lines: [`No baseline for \`${environmentKey}\`, so nothing was retaken.`], |
There was a problem hiding this comment.
Shouldn't we retake pictures even if the baseline is empty? That would make sure we don't accidentally pollute the baseline with flaky screenshots
There was a problem hiding this comment.
Yeah agreed - tbh I thought this was covered already. Done now.
|
@robhogan CI is red |
I think that's unrelated. I've incorporated a fix for open-example.yml into this branch to test CI but it probably ought to be fixed separately. |
Visual regressionNo visual differences at android: no baseline for this environment ios: no capture |
yup can you send that as separate PR please? |
System UI can briefly cover the app during visual capture. On iOS 26 that's the Apple Intelligence notification shown shortly after boot, which got published as a baseline on
mainand failed #58945 with a diff it had nothing to do with. Two more baselines were spoiled by flows racing the UI. I've deleted all three by hand.This retakes, once, any screenshot that differs from the nearest baseline or has none, before upload:
Screenshots that already match aren't retaken, so a clean run costs nothing extra.
Changelog: [Internal]
Test plan:
New unit tests. Ran the retake step against the real bucket with Maestro stubbed, with one spoiled and two new screenshots, and it replaced, kept and flagged them as above. First CI run is this PR's own Test All.