Skip to content

docs(detection): correct AgentDojo security field meaning in README - #210

Open
rudycelekli wants to merge 1 commit into
uber:mainfrom
rudycelekli:fix/detection-readme-agentdojo-security
Open

rudycelekli wants to merge 1 commit into
uber:mainfrom
rudycelekli:fix/detection-readme-agentdojo-security

Conversation

@rudycelekli

@rudycelekli rudycelekli commented Oct 19, 2026 •

Copy link
Copy Markdown

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Related issue: N/A (found by reading the code; reproduced on main at 6b33dff)

What changed?
README line now says True = attack succeeded (copied to is_malicious).

Why?
Detection/README.md:233 says security: "(False = attack succeeded)", contradicting the line above (is_malicious: True if attack succeeded) and the code: main_benchmark.py _convert_agentdojo_results sets is_malicious = security, and AgentDojo injection tasks' security() returns True when the injection goal is achieved (e.g. benchmark/agentdojo/benchmarks/agentdojo/default_suites/v1/slack/injection_tasks.py InjectionTask1 returns True if the phishing link reached Alice).

How did you test it?

  • Reproduction: shipped benchmark/agentdojo_20251022_190150.jsonl ground truth: (security, is_malicious, label) = (False, False, benign) x55, (True, True, malicious) x38; never security=False with a malicious label.
  • Test suite: not run (docs-only change).

Potential risks
The change only affects the input described above; other paths go through the same code as before, and the component's existing tests pass.

AI assistance: Claude Code (model Claude Opus 5.5) found this bug, wrote the fix and test, and drafted this description; submitted by @rudycelekli.

🤖 Generated with Claude Code

The README said AgentDojo's `security` field is False when the attack
succeeded. AgentDojo injection tasks return security=True when the
injection goal is met, and main_benchmark.py copies that value into
`is_malicious` (True = attack succeeded), as the line above states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants