Skip to content

Add field filters to secrets detection - #134

Merged
lucarlig merged 4 commits into
mainfrom
fix/secrets-detection-field-filters
Jul 15, 2026
Merged

lucarlig merged 4 commits into
mainfrom
fix/secrets-detection-field-filters

Conversation

@gandhipratik203

@gandhipratik203 gandhipratik203 commented Jul 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds configurable dotted field filters to the cpex-secrets-detection plugin so operators can scope scanning to specific payload fields and exclude known noisy or intentionally unscanned subtrees.

What changed

  • Added field_allowlist and field_denylist config options with validation for dotted field paths.
  • Wired field-path matching through findings-only and redaction scans for dicts, dict subclasses, slots, object state, lists, and tuples.
  • Preserved rebuild/back-reference behavior so excluded subtrees do not retain references to an original unredacted graph.
  • Documented the new config behavior and added default manifest entries.
  • Bumped cpex-secrets-detection to 0.3.9.

Impact

Existing deployments are backward compatible: both lists default to empty, which preserves current scan behavior. When configured, denylist entries take precedence over allowlist entries.

Closes #132.

Validation

  • cargo test -p secrets_detection passed, 96 tests
  • cargo fmt -- --check passed
  • cargo clippy -- -D warnings passed
  • make install passed
  • make test-integration passed, 114 tests
  • detect-secrets audit --report --fail-on-unaudited passed against a temporary updated baseline
  • git diff --check passed

To rerun the integration tests:

cd /Users/pratik/Desktop/work/new_mcf/cpex-plugins/plugins/rust/python-package/secrets_detection
make test-integration

Note: make check-all previously reached fmt and clippy successfully, then stopped locally because cargo-nextest is not installed. The equivalent Rust test coverage was run with cargo test -p secrets_detection.

Manual Wheel-Level Test

Built the cpex-secrets-detection wheel from this branch and tested the installed wheel from a clean virtualenv outside the repository checkout.

Steps

cd /Users/pratik/Desktop/work/new_mcf/cpex-plugins/plugins/rust/python-package/secrets_detection
make build

uv venv --clear /tmp/cpex-secrets-wheel-test
uv pip install --python /tmp/cpex-secrets-wheel-test/bin/python \
  /Users/pratik/Desktop/work/new_mcf/cpex-plugins/target/wheels/cpex_secrets_detection-0.3.9-cp311-abi3-macosx_11_0_arm64.whl

# Save the smoke script from the collapsible section below to:
# /tmp/secrets_detection_field_filter_wheel_smoke.py
cd /tmp
/tmp/cpex-secrets-wheel-test/bin/python /tmp/secrets_detection_field_filter_wheel_smoke.py
Smoke script
#!/usr/bin/env python3
# Copyright 2026
# SPDX-License-Identifier: Apache-2.0

from __future__ import annotations

import asyncio
from importlib import metadata
from pathlib import Path
from pprint import pprint

import cpex_secrets_detection
from cpex.framework import GlobalContext, PluginConfig, PluginContext, ToolPreInvokePayload
from cpex_secrets_detection.secrets_detection import SecretsDetectionPlugin


def ensure_imported_from_wheel() -> None:
    package_path = Path(cpex_secrets_detection.__file__).resolve()
    if "site-packages" not in str(package_path):
        raise RuntimeError(
            "cpex_secrets_detection was not imported from an installed wheel: "
            f"{package_path}"
        )
    print(f"cpex-secrets-detection {metadata.version('cpex-secrets-detection')}")
    print(f"imported from: {package_path}")


def make_plugin(**config):
    base = {
        "redact": True,
        "redaction_text": "[REDACTED]",
        "block_on_detection": False,
    }
    base.update(config)
    return SecretsDetectionPlugin(
        PluginConfig(
            name="secrets_detection",
            kind="cpex_secrets_detection.secrets_detection.SecretsDetectionPlugin",
            config=base,
        )
    )


def context() -> PluginContext:
    return PluginContext(
        global_context=GlobalContext(request_id="manual", server_id="manual")
    )


async def main() -> None:
    ensure_imported_from_wheel()

    try:
        make_plugin(field_allowlist=["bad."])
        raise AssertionError("invalid field_allowlist did not fail")
    except ValueError as err:
        print(f"invalid config rejected: {err}")

    plugin = make_plugin(
        field_allowlist=["accounts"],
        field_denylist=["accounts.skip"],
    )
    payload = ToolPreInvokePayload(
        name="echo",
        args={
            "accounts": {
                "keep": "AWS_ACCESS_KEY_ID=AKIATEST12345EXAMPLE",
                "skip": "AWS_ACCESS_KEY_ID=AKIASKIP12345EXAMPLE",
            },
            "ignored": "AWS_ACCESS_KEY_ID=AKIAIGNR12345EXAMPLE",
        },
    )

    result = await plugin.tool_pre_invoke(payload, context())

    assert result.continue_processing is True
    assert result.violation is None
    assert result.modified_payload is not None
    assert result.modified_payload.args["accounts"]["keep"] == (
        "AWS_ACCESS_KEY_ID=[REDACTED]"
    )
    assert result.modified_payload.args["accounts"]["skip"] == (
        "AWS_ACCESS_KEY_ID=AKIASKIP12345EXAMPLE"
    )
    assert result.modified_payload.args["ignored"] == (
        "AWS_ACCESS_KEY_ID=AKIAIGNR12345EXAMPLE"
    )
    assert payload.args["accounts"]["keep"] == "AWS_ACCESS_KEY_ID=AKIATEST12345EXAMPLE"

    print("original args:")
    pprint(payload.args)
    print("modified args:")
    pprint(result.modified_payload.args)
    print("wheel-level field filter smoke test passed")


if __name__ == "__main__":
    asyncio.run(main())

Result

cpex-secrets-detection 0.3.9
imported from: /private/tmp/cpex-secrets-wheel-test/lib/python3.13/site-packages/cpex_secrets_detection/__init__.py
invalid config rejected: field_allowlist path "bad." must not start or end with '.'
original args:
{'accounts': {'keep': 'AWS_ACCESS_KEY_ID=AKIATEST12345EXAMPLE',
              'skip': 'AWS_ACCESS_KEY_ID=AKIASKIP12345EXAMPLE'},
 'ignored': 'AWS_ACCESS_KEY_ID=AKIAIGNR12345EXAMPLE'}
modified args:
{'accounts': {'keep': 'AWS_ACCESS_KEY_ID=[REDACTED]',
              'skip': 'AWS_ACCESS_KEY_ID=AKIASKIP12345EXAMPLE'},
 'ignored': 'AWS_ACCESS_KEY_ID=AKIAIGNR12345EXAMPLE'}
wheel-level field filter smoke test passed

What This Verifies

  • The package imports from the installed wheel in /tmp, not from the repo checkout.
  • The wheel version is 0.3.9.
  • Invalid field paths fail loudly at plugin load.
  • field_allowlist=["accounts"] scans and redacts accounts.keep.
  • field_denylist=["accounts.skip"] excludes accounts.skip.
  • Fields outside the allowlist, such as ignored, are not scanned.

@gandhipratik203
gandhipratik203 marked this pull request as ready for review July 14, 2026 16:36

@lucarlig lucarlig left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
Signed-off-by: Pratik Gandhi <gandhipratik203@gmail.com>
@gandhipratik203
gandhipratik203 force-pushed the fix/secrets-detection-field-filters branch from aaf21ca to 56e4a34 Compare July 15, 2026 13:04
@lucarlig
lucarlig merged commit d29d306 into main Jul 15, 2026
23 checks passed
@lucarlig
lucarlig deleted the fix/secrets-detection-field-filters branch July 15, 2026 13:23
@gandhipratik203 gandhipratik203 self-assigned this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add configurable field allowlist and denylist to secrets detection

2 participants