Skip to content

fix(report): align recursive risk recommendations - #836

Merged
rng1995 merged 1 commit into
NVIDIA:mainfrom
chrisknvidia:feat/christopherk/recursive-risk-recommendation
Oct 20, 2026
Merged

rng1995 merged 1 commit into
NVIDIA:mainfrom
chrisknvidia:feat/christopherk/recursive-risk-recommendation

Conversation

@chrisknvidia

@chrisknvidia chrisknvidia commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Complete recursive scans could report MEDIUM/SAFE even when their child reports were MEDIUM/CAUTION. Reuse the normal risk-band mapping for the aggregate while preserving execution-failure and incomplete-analysis overrides, scoring, output schemas, and the existing greater-than-50 exit threshold.

Fixes #835.

Validation:

  • 517 focused CLI/report tests passed, including real subprocess regressions for JSON, SARIF, Markdown, and terminal.
  • 31 fresh installed-wheel CLI cases passed across risk boundaries, strict flags, file output, and incomplete scans; real MCP stdio clean/medium controls passed.
  • Full non-provider regression suite: 13,721 passed, 14 skipped, 4 expected failures (245 integration/provider tests deselected).
  • Additional TG validation: 39 actual installed-wheel CLI cases, seven base/head comparisons, and 10 actual HTTP MCP checks passed. Covered failed children, real output/count bounds, baselines, write errors, and concurrent scans.
  • Independent architecture, code, and manual security reviews completed without actionable findings. All six hosted checks passed, including the full unit workflow.
  • Follow-up review: 36 focused boundary/output checks passed (one 60-second subprocess timeout under host load passed on an unchanged isolated rerun). Five additional installed-wheel CLI checks passed for direct/recursive JSON and SARIF, strict exits and file output.

Provider-backed validation was not rerun. Native review-bot passes were unavailable; the listed reviews were manual.

Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
@chrisknvidia chrisknvidia self-assigned this Oct 14, 2026
@chrisknvidia
chrisknvidia marked this pull request as ready for review October 16, 2026 05:36
@chrisknvidia

Copy link
Copy Markdown
Collaborator Author

Ready for review at 2e5fd982. There are no outstanding review comments or known actionable findings.

All six hosted CI checks have passed.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

@chrisknvidia, thank you for tracking down #835 and fixing it at the source. Sharing one score-to-verdict helper between single-skill and recursive scans removes the duplicated band table that let the two drift apart.

Value and readiness

Before this PR, a complete --recursive scan whose top score was 21-50 reported MEDIUM/SAFE in the aggregate JSON risk_recommendation and SARIF riskAssessment.recommendation, while every child report said MEDIUM/CAUTION. The PR moves the band mapping into risk_severity_and_recommendation() in nodes/report.py. _compute_risk_score and cli._multi_skill_risk_assessment now both call it. The two recursive overrides stay: a failed execution forces DO_NOT_INSTALL, and incomplete analysis raises SAFE to CAUTION.

The report.py change is a pure extraction with the same loop and fallback, so single-skill output does not change. I checked every score band against the failed and incomplete flags. The PR is never less strict than main. Scores 0-20 and 51+ are unchanged. Scores 21-50 move from SAFE to CAUTION only for complete, non-failed scans. Exit codes are unchanged because the > 50 threshold is still checked separately. The aggregate now matches the README mapping (LOW to SAFE, MEDIUM to CAUTION, HIGH/CRITICAL to DO_NOT_INSTALL).

Verdict: correct and ready to merge.

Material findings

None blocking. One optional test note:

  1. Optional: Some of the new test cases do not guard the fix (tests/unit/test_cli_subprocess.py:149, tests/unit/test_cli.py:2629).

    • Markdown/terminal subprocess cases. A complete recursive scan in these formats prints only each child's own report. The "Recursive Risk Assessment" block is appended only if analysis_incomplete. So "MEDIUM" in result.stdout and "CAUTION" in result.stdout are satisfied by the marked child's own section on both main and this PR.
    • Score-25 rows in test_recursive_non_json_caps_preserve_aggregate_risk. These rows always hit a cap, which sets analysis_incomplete=True, and main already returned CAUTION for incomplete scans scoring 50 or less.
    • What I saw. Running the PR's tests against main's source gives 4 failed, 48 passed. The failures are medium_recommendation[json], medium_recommendation[sarif], and the bands cases for scores 21 and 50 (complete, not failed). The markdown/terminal cases and all 8 score-25 cap cases pass on main. Expected: a regression test fails on main.
    • Suggested change. Either drop markdown/terminal from that parametrization, or make those cases assert what they actually cover: no aggregate section in complete output, and each child section matching its direct scan. Keep the score-25 cap rows as non-regression coverage. The JSON/SARIF cases and the bands test are the real guards, and they work. Optionally, write the bands test's expected override values as a literal table instead of recomputing them with the same if/elif logic as the code under test.

    This finding also means the PR description overstates the Markdown/terminal coverage a little. Fix coverage is still adequate.

Verification and gaps

  • PR head 2e5fd982, merged with current main (314 commits behind, merges cleanly). Main has not changed the recommendation logic since the branch base. The nearby SC9 score floors merge in unchanged.
  • Focused suites on the merged tree: test_cli.py, test_cli_subprocess.py, nodes/test_report.py, test_multi_skill.py, test_batch_scan_reports.py gave 754 passed. test_sarif.py, test_mcp_server.py, nodes/test_analysis_completeness.py gave 104 passed. The PR's new tests pass on both the head and the merged tree (52/52).
  • Real CLI runs with --recursive --no-llm and a custom YARA marker, merged tree vs main, in all four formats, at scores 0, 10, 20, 21, 25, 50 and 67. Also covered: mixed children, a symlinked child, an executable binary that triggers the 51 floor, a failed child (LLM requested with no credentials), and a 34-skill collection past the 32-skill budget. Flags tried: --fail-on-findings, --fail-on-incomplete, --min-coverage 100, --verbose, --output. The only difference was SAFE to CAUTION for complete 21-50 scans in JSON/SARIF. Exit codes matched in every case, and no input produced SAFE when any child was HIGH/CRITICAL, failed, or incomplete.
  • A child with a degraded LLM stage is marked incomplete, so the aggregate is still raised to CAUTION.
  • CI: all 6 checks passed on 2e5fd982.
  • Not verified: provider-backed LLM runs. The "LLM partly failed but reports complete coverage" case could not be reached without a real provider. An unreachable endpoint marks the scan failed, which fails closed.

@rng1995
rng1995 merged commit 11c8b48 into NVIDIA:main Oct 20, 2026
6 checks passed
rng1995 added a commit that referenced this pull request Oct 20, 2026
Complete recursive scans reported a MEDIUM aggregate (score 21-50) as
SAFE even though each MEDIUM child report said CAUTION. The aggregate
now uses the same band mapping as single-skill reports through a new
shared report.risk_severity_and_recommendation() helper. Execution
failures still force DO_NOT_INSTALL, incomplete analysis still raises
SAFE to CAUTION, and the >50 exit threshold is unchanged. cli.py,
report.py and test_cli.py applied cleanly. The only conflict was in
tests/unit/test_cli_subprocess.py, which on this line has only the
#840 registry tests. It now also gets main's small _write_skill helper
and the new recursive MEDIUM subprocess test. The other main-only tests
in that hunk are left out.

(cherry picked from commit 11c8b48)

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rng1995 added a commit that referenced this pull request Oct 20, 2026
…1 notes

List the five analyzer, report and CLI fixes backported onto the
v2.13.0 base (#820, #823, #836, #837, #840) in the CHANGELOG and
release notes, and raise the intro count to eight backported fixes.
Document the compatibility notes: scan --mcp-registry now exits 2 with
--use-shipped-baseline or any --transitive* option, including an
explicit --transitive-depth 1; recursive multi-skill scans exit 2 with
--use-shipped-baseline; a complete recursive MEDIUM aggregate reports
CAUTION instead of SAFE in JSON and SARIF with exit codes unchanged;
and the TM1 and MP3 detection changes can change findings and scores.
main has no CHANGELOG entries for these PRs, so the text is written
from each backport.

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: align recursive aggregate risk recommendations with child reports

2 participants