Add opt-in local PII redaction before TEE encryption#12
Conversation
Scrub high-impact PII from chat-completion prompts locally, before the request is HPKE-sealed to the TEE — defense-in-depth on top of Veil's existing end-to-end privacy. Off by default; enable with --pii-scrub / OG_VEIL_PII_SCRUB. Two layered tiers: - Regex (always on when enabled, zero new core deps): email, US SSN, and bank numbers (credit cards Luhn-checked, IBANs mod-97-checked, plus labelled routing/account numbers). DOB caught by context cues. - NER (optional [pii] extra: Presidio + spaCy en_core_web_sm): free-form addresses/locations, with an optional --pii-all-dates mode. Redaction is irreversible (no de-anonymization), so the TEE's signed output_hash covers exactly what it ran. Wired into Gateway.chat() ahead of the retry loop; covers string and multimodal text message content. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Replace the handrolled regex detectors with Microsoft Presidio so the recognizers are community-maintained rather than ours to own. Presidio detects the structured types (email, US SSN, credit cards, IBANs, US bank/routing numbers) with its own regex/checksum recognizers and uses the spaCy model for free-form addresses/locations. Because detection now depends on Presidio + a spaCy model, PII scrubbing requires the optional [pii] extra; if --pii-scrub is set without it, the server refuses to start with an actionable message instead of silently passing PII through. Notable details: - Dates of birth: a custom Presidio PatternRecognizer anchored on a birth-date cue via variable-width lookbehind (the regex module Presidio uses supports it), giving true per-occurrence gating so an unrelated invoice date is left alone. --pii-all-dates switches to redacting every date via spaCy DATE_TIME. - Pin spaCy to stable 3.8.x in the extra: the repo allows prereleases (for an SDK dep), which otherwise pulls a spaCy 4.0 dev build whose compiled extensions break against numpy 2.x. - CI/docs install the spaCy model from the release wheel rather than `spacy download`, whose CLI breaks under the click version this project pins. Added a `make install-pii` helper and a model-install step to the tests workflow; PII tests importorskip when the extra is absent. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
Dates are too entangled with legitimate prompt content to redact without mangling it, so remove all date handling: the custom DOB recognizer, the DATE_TIME path, the --pii-all-dates flag, the OG_VEIL_PII_REDACT_ALL_DATES env var, and the DOB_TAG. PII scrubbing now covers email, US SSN, bank numbers, and addresses only. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
The --pii-scrub flag / OG_VEIL_PII_SCRUB env var now set the *default*; clients can override scrubbing per request without restarting the proxy: - header `X-OpenGradient-PII-Scrub: true|false` — set once in the client/Hermes config, applies to every request (the easy path) - body field `pii_scrub` (via the OpenAI SDK's extra_body) — per call A per-request value overrides the server default; the body field wins over the header. The control field is stripped before the request reaches the TEE. The Presidio engine is now built lazily and cached, so the [pii] extra is only needed when scrubbing is actually used; if a request asks to scrub but the extra isn't installed, it fails closed with a 503 rather than forwarding PII raw. `og-veil test` gains a --scrub/--no-scrub flag. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
This reverts commit 9d5def8.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Local-only helper that loads the same Redactor the proxy uses (no login, TEE, or network) so you can try the redaction on your own text interactively or via piped input. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
The automated fix commit removed the `install-pii:` target header, leaving its recipe lines attached to `install:`. Restore the header so `make install-pii` works again. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
… address) Reframe scrubbing around Veil's actual threat model: OHTTP unlinks identity from content at the network layer, but a prompt that names you re-identifies you to the model provider through the content. So the job is de-identifying the requester. Reprioritize accordingly: - Add person names (PERSON) and phone numbers — the primary identity linkers, previously not covered at all. - Add a custom street-address recognizer: spaCy NER catches named places (cities/states) but not free-form street lines, and Presidio ships nothing for them. No transformer needed. - Upgrade the spaCy model to en_core_web_lg for materially better PERSON recall across diverse names (a ~560 MB CPU CNN model, not a transformer). Structured IDs (SSN, cards, IBAN, bank) stay as a bonus. Docs/Makefile/CI updated to the lg model wheel; tests cover names, phone, and street addresses. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
… locations
Drop the statistical-NER entities (PERSON, LOCATION). They over-redacted the
third-party names real prompts are full of ("reply to Advait about Julia") and
mislabeled uncommon names (e.g. tagging "Advait" as a location), wrecking
common workflows for little gain. Keeping names out of a prompt stays the user's
call; this layer now backstops only the concrete data.
Now redacts only deterministic pattern/checksum identifiers: email, phone, SSN,
credit cards, IBANs, US bank numbers, and street-address lines. No name or
location guessing.
Since no NER entities are redacted anymore, the spaCy model is only Presidio's
tokenizer — downgrade en_core_web_lg back to en_core_web_sm (saves ~550 MB).
Docs/Makefile/CI updated; tests assert names and cities pass through.
https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
The redactor only used pattern/checksum entities (no PERSON/LOCATION NER), so the spaCy model was dead weight — it was only there to satisfy Presidio's NLP engine. Run Presidio's predefined recognizers (email, phone, SSN, card, IBAN, US bank) plus the custom street-address recognizer directly with nlp_artifacts=None, and drop the AnalyzerEngine/NlpEngineProvider/model entirely. Result: installing PII redaction is now just `pip install 'opengradient-veil[pii]'` — no `spacy download`, no Makefile target, no pinned model wheel in docs/CI. Also faster startup (no model load) and lower RAM. A 0.4 score threshold drops Presidio's "very weak" context-dependent patterns (e.g. a bare 9-digit number as a maybe-SSN) so we don't over-redact ordinary numbers now that there's no NLP context. Dropped the now-unneeded typer pin; kept the spaCy<4 pin since presidio pulls spaCy transitively. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
Condense to the essentials — install, what's redacted vs left in, and the key caveats — cutting the verbose prose. https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a
| help="Redact high-impact PII (email, SSN, bank numbers; addresses with the [pii] extra) " | ||
| "from prompts locally before they leave this machine.", | ||
| ) |
| # Opt-in local PII redaction: scrub concrete identifiers (email, phone, SSN, | ||
| # bank numbers, street addresses) out of the agent's prompt *before* it leaves | ||
| # this process. Requires the optional ``[pii]`` extra; off by default. See | ||
| # :mod:`veil.pii`. |
| "PHONE_NUMBER": PHONE_TAG, | ||
| "US_SSN": SSN_TAG, | ||
| "CREDIT_CARD": BANK_TAG, | ||
| "IBAN_CODE": BANK_TAG, | ||
| "US_BANK_NUMBER": BANK_TAG, |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Local PII redaction (opt-in)
Adds an optional layer that redacts concrete identifiers from prompts locally, before they're encrypted to the TEE — so they never leave the machine. This backstops Veil's unlinkability guarantee: OHTTP hides who you are, but only if the prompt content doesn't name you.
Behaviour
Enable with
--pii-scrub(orOG_VEIL_PII_SCRUB=1); off by default.[REDACTED_*]tags): email, phone, US SSN, credit cards (Luhn), IBANs (mod-97), US bank/routing numbers, and street addresses.Detection uses Microsoft Presidio's pattern/checksum recognizers only — no NER model, so install is a single step (
pip install 'opengradient-veil[pii]'), with no model download.output_hashcovers exactly what it ran.https://claude.ai/code/session_014NfUqUiR5AaD3HcswJTA2a