AI defense

Every AI call in Pact runs through a stack of strict gates before and after the model is invoked. Each gate defends a specific vector: cost, behaviour, replay, and leakage. Together they map to the OWASP Top 10 for LLM Applications. The whole stack is wired into core.ai.client.AIClient and enforced on every call — this is not an advisory layer.

Enforced on every call

The gates below run inside the AI client. A tripped gate raises before (or instead of showing) the model response — there is no path that skips them for a normal feature call.

The gate order

code

kill switch  →  data-sharing redaction  →  budget/rate cap
             →  injection detection  →  replay protection
             →  [ model call ]  →  output filter

1. Kill switch — the break-glass control

core.ai.kill_switch is the outermost gate. Three levels:

Global — the AI_SUBSYSTEM_DISABLED env var. When truthy, every tenant's AI is disabled (incident response, provider outage, secret rotation).
Per-tenant — the tenant_ai_budget.disabled_reason column (alembic 0153_ai_kill_switch). Non-NULL means that tenant's AI calls fail fast with a clear "AI temporarily disabled" error — no model call, no ledger row, no defense overhead.
Auto-trigger — engages the per-tenant switch when suspicious activity crosses a threshold: 3+ injection blocks or 10+ rate-limit blocks in the last 5 minutes. It writes an auto_engaged row to ai_kill_switch_events; recovery is always manual.

2. Prompt-injection detection

core.ai.injection_detector rejects user content trying to override the system prompt (OWASP LLM01). Coverage: direct override ("ignore all previous instructions"), role-confusion tokens (Assistant: / </system>), delimiter spoofing (###, [INST], ---END---), Unicode tag-character smuggling (U+E0000…U+E007F, invisible to renderers but legible to the model), and suspicious base64 blobs (flagged for review). score_content returns a confidence in 0.0…1.0; the default policy blocks at >= 0.8 (DEFAULT_BLOCK_THRESHOLD). Blocks are recorded to ai_injection_blocks (alembic 0150_ai_injection_blocks), which also feeds the kill-switch auto-trigger.

Pattern detector, not a model classifier

Injection detection is a zero-cost, near-zero-latency pattern detector. A secondary model-based classifier is on the roadmap but has a meta-problem — an attacker who controls the input controls what reaches the classifier too — so Pact ships the pattern detector first and would treat any future classifier output as a signal, not a verdict.

3. Replay protection

core.ai.replay_protection blocks captured-request replay (a leaked or intercepted signed request replayed later — a 5-minute freshness window by default) and cross-tenant idempotency-key collisions, caught before the call reaches the spend ledger.

4. Output filter

core.ai.output_filter runs after the model responds and catches the case where an injection got through and the model complied: system-prompt leaks, jailbreak-success markers ("In DAN mode…", "Bypassing my filters…"), and refusal pretext (refusing while including the very content being refused). It scans tool-use inputs too. When score_response flags a response, the caller does not show it, writes a row to ai_output_filter_blocks, and surfaces it for triage.

Admin visibility

Where you see the events

The kill-switch state, its recent events, and the auto-trigger spikes (which are driven by the injection- and rate-block tables) are surfaced today on the Admin → Security → AI budget page — the security review lands there. A dedicated per-gate triage dashboard for raw injection and output-filter blocks is a roadmap refinement; the underlying ai_injection_blocks and ai_output_filter_blocks tables are already populated and auditable.

AI budgetThe cost gate and the kill-switch controls.AI data sharingWhat each feature sends to the model.Rate limitsPer-endpoint request throttling.Security settingsTenant-wide security posture.