Repository navigation
Conversation
Signed-off-by: Christopher Kevin <christopherk@nvidia.com>
|
Ready for review at All six hosted CI checks have passed. |
rng1995
left a comment
There was a problem hiding this comment.
[SkillSpector Review]
@chrisknvidia, thank you for tracking down #835 and fixing it at the source. Sharing one score-to-verdict helper between single-skill and recursive scans removes the duplicated band table that let the two drift apart.
Value and readiness
Before this PR, a complete --recursive scan whose top score was 21-50 reported MEDIUM/SAFE in the aggregate JSON risk_recommendation and SARIF riskAssessment.recommendation, while every child report said MEDIUM/CAUTION. The PR moves the band mapping into risk_severity_and_recommendation() in nodes/report.py. _compute_risk_score and cli._multi_skill_risk_assessment now both call it. The two recursive overrides stay: a failed execution forces DO_NOT_INSTALL, and incomplete analysis raises SAFE to CAUTION.
The report.py change is a pure extraction with the same loop and fallback, so single-skill output does not change. I checked every score band against the failed and incomplete flags. The PR is never less strict than main. Scores 0-20 and 51+ are unchanged. Scores 21-50 move from SAFE to CAUTION only for complete, non-failed scans. Exit codes are unchanged because the > 50 threshold is still checked separately. The aggregate now matches the README mapping (LOW to SAFE, MEDIUM to CAUTION, HIGH/CRITICAL to DO_NOT_INSTALL).
Verdict: correct and ready to merge.
Material findings
None blocking. One optional test note:
-
Optional: Some of the new test cases do not guard the fix (
tests/unit/test_cli_subprocess.py:149,tests/unit/test_cli.py:2629).- Markdown/terminal subprocess cases. A complete recursive scan in these formats prints only each child's own report. The "Recursive Risk Assessment" block is appended only
if analysis_incomplete. So"MEDIUM" in result.stdoutand"CAUTION" in result.stdoutare satisfied by the marked child's own section on both main and this PR. - Score-25 rows in
test_recursive_non_json_caps_preserve_aggregate_risk. These rows always hit a cap, which setsanalysis_incomplete=True, and main already returned CAUTION for incomplete scans scoring 50 or less. - What I saw. Running the PR's tests against main's source gives 4 failed, 48 passed. The failures are
medium_recommendation[json],medium_recommendation[sarif], and the bands cases for scores 21 and 50 (complete, not failed). The markdown/terminal cases and all 8 score-25 cap cases pass on main. Expected: a regression test fails on main. - Suggested change. Either drop markdown/terminal from that parametrization, or make those cases assert what they actually cover: no aggregate section in complete output, and each child section matching its direct scan. Keep the score-25 cap rows as non-regression coverage. The JSON/SARIF cases and the bands test are the real guards, and they work. Optionally, write the bands test's expected override values as a literal table instead of recomputing them with the same if/elif logic as the code under test.
This finding also means the PR description overstates the Markdown/terminal coverage a little. Fix coverage is still adequate.
- Markdown/terminal subprocess cases. A complete recursive scan in these formats prints only each child's own report. The "Recursive Risk Assessment" block is appended only
Verification and gaps
- PR head
2e5fd982, merged with current main (314 commits behind, merges cleanly). Main has not changed the recommendation logic since the branch base. The nearby SC9 score floors merge in unchanged. - Focused suites on the merged tree:
test_cli.py,test_cli_subprocess.py,nodes/test_report.py,test_multi_skill.py,test_batch_scan_reports.pygave 754 passed.test_sarif.py,test_mcp_server.py,nodes/test_analysis_completeness.pygave 104 passed. The PR's new tests pass on both the head and the merged tree (52/52). - Real CLI runs with
--recursive --no-llmand a custom YARA marker, merged tree vs main, in all four formats, at scores 0, 10, 20, 21, 25, 50 and 67. Also covered: mixed children, a symlinked child, an executable binary that triggers the 51 floor, a failed child (LLM requested with no credentials), and a 34-skill collection past the 32-skill budget. Flags tried:--fail-on-findings,--fail-on-incomplete,--min-coverage 100,--verbose,--output. The only difference was SAFE to CAUTION for complete 21-50 scans in JSON/SARIF. Exit codes matched in every case, and no input produced SAFE when any child was HIGH/CRITICAL, failed, or incomplete. - A child with a degraded LLM stage is marked incomplete, so the aggregate is still raised to CAUTION.
- CI: all 6 checks passed on
2e5fd982. - Not verified: provider-backed LLM runs. The "LLM partly failed but reports complete coverage" case could not be reached without a real provider. An unreachable endpoint marks the scan failed, which fails closed.
Complete recursive scans reported a MEDIUM aggregate (score 21-50) as SAFE even though each MEDIUM child report said CAUTION. The aggregate now uses the same band mapping as single-skill reports through a new shared report.risk_severity_and_recommendation() helper. Execution failures still force DO_NOT_INSTALL, incomplete analysis still raises SAFE to CAUTION, and the >50 exit threshold is unchanged. cli.py, report.py and test_cli.py applied cleanly. The only conflict was in tests/unit/test_cli_subprocess.py, which on this line has only the #840 registry tests. It now also gets main's small _write_skill helper and the new recursive MEDIUM subprocess test. The other main-only tests in that hunk are left out. (cherry picked from commit 11c8b48) Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1 notes List the five analyzer, report and CLI fixes backported onto the v2.13.0 base (#820, #823, #836, #837, #840) in the CHANGELOG and release notes, and raise the intro count to eight backported fixes. Document the compatibility notes: scan --mcp-registry now exits 2 with --use-shipped-baseline or any --transitive* option, including an explicit --transitive-depth 1; recursive multi-skill scans exit 2 with --use-shipped-baseline; a complete recursive MEDIUM aggregate reports CAUTION instead of SAFE in JSON and SARIF with exit codes unchanged; and the TM1 and MP3 detection changes can change findings and scores. main has no CHANGELOG entries for these PRs, so the text is written from each backport. Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Complete recursive scans could report MEDIUM/SAFE even when their child reports were MEDIUM/CAUTION. Reuse the normal risk-band mapping for the aggregate while preserving execution-failure and incomplete-analysis overrides, scoring, output schemas, and the existing greater-than-50 exit threshold.
Fixes #835.
Validation:
Provider-backed validation was not rerun. Native review-bot passes were unavailable; the listed reviews were manual.