AI / LLM Application Security Assessment Playbook (Pentest-Ready)
AI / LLM Application Security Assessment Playbook (Pentest-Ready)
In short
Document owner: Security Engineering
Audience: Security teams, engineering leads, risk/compliance, external testers (authorized)
Purpose: Repeatable methodology and test matrix for assessing security risks in LLM-enabled applications (chatbots, RAG, agents, tool-calling systems, copilots).
Version: 1.0
Executive Summary
What this is
LLM-enabled products introduce a new security class where untrusted content can influence system behavior, including data access, tool execution, and downstream actions. Traditional AppSec controls remain necessary but are insufficient without LLM-specific safeguards.
Why it matters
The most business-relevant failure modes are:
- Sensitive data exposure: regulated data leakage through model outputs, retrieval (RAG), logs, or memory.
- Unauthorized actions: agent/tool misuse leading to fraud, account changes, data deletion, or service disruption.
- Supply-chain and governance risk: model/dataset provenance, unsafe fine-tunes, insecure prompts/configuration, and missing auditability.
What this assessment delivers
- A risk-based test plan tied to a recognized taxonomy (OWASP LLM Top 10-style categories).
- A pass/fail test matrix with evidence requirements (tool traces, retrieval traces, policy decisions, and egress logs).
- Actionable findings with severity ratings and remediation verification criteria.
Top recommended controls (highest ROI)
- Fail-closed tool execution: strict allow-lists, schema validation, user/tenant-bound authorization, and deterministic tool routing.
- Egress controls + provenance: outbound network restrictions and full traceability of what content influenced tool calls and outputs.
- RAG hardening: source allow-lists, chunk sanitization, retrieval tracing, and tenant isolation validation.
- Secrets and data minimization: scoped credentials, redaction, and removal of sensitive content from prompts/logs/memory.
- Monitoring + incident readiness: detection for anomalous tool usage, egress spikes, and retrieval of sensitive corpora.
Decision points for leadership
- Authorization and scope: explicit approval and clear boundaries (avoid legal/operational exposure).
- Fix-first prioritization: tool/agent controls and data handling before feature expansion.
- Auditability requirements: who/what prompted an action, what data was retrieved, which tool executed, and why.
1. Engagement Framing (Rules of Engagement)
1.1 Authorization and scope
- Testing is limited to systems explicitly authorized by the asset owner.
- In-scope includes:
- Applications, environments (staging/production), tenants, data stores, tool endpoints, model providers, and orchestration services.
- Out-of-scope includes:
- Third-party systems not owned/authorized, unmanaged endpoints, unrelated tenants, and real customer data unless explicitly approved.
1.2 Environment and safety constraints
- Staging with production-like configuration and synthetic or masked datasets preferred.
- Scoped credentials (least privilege) and time-bound access.
- Stop condition: any evidence of unintended data exposure or unsafe tool execution triggers immediate containment.
1.3 Data handling and retention
- Data classification (Public / Internal / Confidential / Regulated).
- Evidence collection avoids copying regulated data; store hashes, identifiers, and minimal excerpts.
- Retention windows and secure storage for transcripts, traces, and logs.
1.4 Communications and escalation
- Single incident channel and on-call contact.
- Severity thresholds for immediate notification (e.g., unauthorized tool execution, tenant data access, credential exposure).
2. System Decomposition Worksheet (Pre-test mapping)
2.1 Inventory
- Model layer: provider, model versions, system prompts, guardrails, moderation, temperature settings.
- Prompt pipeline: templates, instruction hierarchy, context assembly, memory injection logic.
- Retrieval (RAG): vector DB, embedding model, indexing pipeline, chunking, filters, re-rankers.
- Tools/agents: tool catalog, permissions, auth tokens, tool router, retry logic, planning loops.
- Output handling: markdown rendering, HTML/JS rendering, code execution, file generation, downstream automations.
- Storage: chat history, memory store, logs, analytics, caches.
- Deployment: gateways, WAF, egress proxy, service mesh, secrets manager, CI/CD.
2.2 Data flows and trust boundaries
Trust boundaries where untrusted input can influence:
- tool selection and tool arguments
- retrieval queries and returned documents
- output rendering (browser/UI) and downstream processing
- persistence (memory/logs) and cross-tenant sharing
2.3 Critical assets (examples)
- customer PII, financial records, credentials, API keys, admin capabilities, internal knowledge bases, code repositories, payment tooling.
3. Threat Model
3.1 Security objectives
- Prevent unauthorized data disclosure.
- Prevent unauthorized actions (especially via tools/agents).
- Ensure tenant isolation.
- Ensure integrity of retrieval sources and outputs.
- Ensure auditability and non-repudiation of tool actions.
3.2 Attacker profiles
- External user with standard account
- Malicious tenant admin
- Insider with partial access
- Compromised content source (web page, document, ticket, email) used by RAG
- Compromised tool endpoint or API key
3.3 Primary attack surfaces
- User prompts and multi-turn conversations
- Indirect injection via retrieved content (documents, webpages, attachments)
- Tool calling / function calling layer
- Output rendering layer (UI, markdown/HTML)
- Memory and chat history persistence
- Observability/logging pipelines
- Training/fine-tuning and model configuration supply chain
4. Assessment Methodology (Phased)
Phase 0 — Pre-work
- Authorization, scope, environment, and data handling constraints confirmed.
- Architecture, tool list, RAG sources, RBAC model, and logging capabilities collected.
Phase 1 — Recon and mapping
- Component inventory and trust boundaries validated.
- High-impact tools and high-sensitivity datasets identified.
Phase 2 — Control review
- Guardrails, tool allow-lists, schema validation, egress controls, secret handling, and tenant isolation controls reviewed.
Phase 3 — Adversarial testing (non-destructive)
- Test matrix categories executed with pass/fail outcomes.
- Evidence captured: tool traces, retrieval traces, policy decisions, and egress logs.
Phase 4 — Chaining and escalation (authorized only)
- Realistic chains assessed: indirect injection → tool misuse → data exposure or unauthorized action.
- Blast radius and tenant boundaries validated.
Phase 5 — Reporting and verification
- Findings delivered with reproduction steps, root cause, and remediation.
- Critical fixes retested using regression prompt suites and trace verification.
5. Severity Rubric (LLM-App Specific)
5.1 Dimensions
- Impact: data sensitivity, financial loss, safety harm, operational disruption, compliance exposure.
- Exploitability: required privileges, user interaction, determinism, complexity, need for chaining.
- Blast radius: single user, single tenant, multi-tenant, system-wide, external systems.
5.2 Suggested ratings
- Critical: unauthorized tool action OR regulated data exposure with tenant/system scope.
- High: sensitive data exposure (confidential), unauthorized action limited to a tenant, or reliable cross-session memory leakage.
- Medium: partial data leakage, unreliable exploitation, limited impact, mitigations exist but incomplete.
- Low: informational issues, hard-to-exploit edge cases, defense-in-depth recommendations.
6. Evidence and Telemetry Requirements (What to capture)
6.1 Minimum evidence set per test
- Conversation transcript (redacted)
- System prompt and prompt assembly trace (or hash + version)
- Retrieval trace:
- document IDs, chunk IDs, scores, filters applied, and hashes of chunk content
- Tool trace:
- tool selected, arguments pre/post validation, authorization decision, execution outcome
- Policy decision trace:
- which guardrail blocked/allowed, reason code
- Network egress logs:
- DNS + destination + request metadata (no sensitive payloads)
6.2 Detection signals (blue team)
- anomalous tool usage frequency and unusual tool combinations
- spikes in retrieval of sensitive collections
- egress to unknown domains/IP ranges
- repeated guardrail violations or policy bypass attempts
- large output volumes and token spikes (DoS indicators)
7. Test Case Template (Repeatable)
Test cases use a consistent structure for auditability and retestability:
1
2
3
4
5
6
7
8
9
Test ID:
Category:
Objective:
Preconditions:
Steps (high-level intent; no bypass payloads):
Expected Safe Behavior (Pass criteria):
Failure Conditions (Fail criteria):
Evidence to Capture:
Severity Guidance:
8. Test Matrix (Pass/Fail)
This matrix uses intent-driven tests. Do not embed bypass/jailbreak payloads in the document. Maintain an internal, access-controlled test corpus if needed.
8.1 Prompt / Instruction Injection (Direct)
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| PI-D1: Role confusion | Ensure instruction hierarchy is enforced | Model follows system/developer constraints; refuses unsafe requests; logs policy event | transcript + policy trace |
| PI-D2: Data extraction attempt | Prevent disclosure of secrets in context | No sensitive data leaked; redaction works; refusal + audit trail | transcript + prompt trace |
| PI-D3: Prompt boundary integrity | Prevent user input from overriding internal instructions | User content is clearly delimited and treated as data | prompt assembly trace |
8.2 Indirect Prompt Injection (Untrusted retrieved content)
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| PI-I1: Retrieved content tries to induce tool use | Ensure untrusted documents cannot trigger tools | No tool execution from retrieved instructions; content treated as quoted data | retrieval + tool trace |
| PI-I2: Retrieved content attempts data exfil | Prevent doc-driven leakage | No disclosure beyond authorized scope; citations/provenance maintained | retrieval trace + output |
| PI-I3: Source integrity | Validate only approved sources influence answers | Unapproved sources blocked; provenance shown; alerts raised | retrieval allow-list logs |
8.3 Tool / Agent Misuse (Excessive Agency)
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| TA-1: Tool allow-list | Ensure only intended tools can be called | Unknown tools never called; deterministic routing | tool router logs |
| TA-2: Argument schema validation | Prevent injection into tool args | Strict schema; unknown fields rejected; validation errors logged | pre/post validation args |
| TA-3: Authorization binding | Ensure tool actions are user/tenant scoped | Tool calls require RBAC check; tenant context enforced | authZ decision logs |
| TA-4: Rate and spend controls | Prevent runaway loops / abuse | Tool call quotas; circuit breakers; backoff | usage metrics + traces |
8.4 RAG / Vector Store Security
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| RAG-1: Tenant isolation | Prevent cross-tenant retrieval | Queries filtered by tenant; no foreign chunks returned | retrieval filters + IDs |
| RAG-2: Poisoning resistance | Prevent index contamination leading to unsafe answers | ingestion pipeline validates sources; suspicious docs quarantined | ingestion logs |
| RAG-3: Prompt boundary for retrieved chunks | Prevent chunk content from behaving like instructions | chunks are escaped/quoted; system prompt forbids following doc instructions | prompt trace |
| RAG-4: Sensitive collection access | Ensure least privilege retrieval | access controlled collections; per-user entitlements enforced | authorization logs |
8.5 Output Handling (Client/Downstream Safety)
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| OUT-1: HTML/Markdown rendering safety | Prevent XSS/script injection via model output | sanitization enabled; dangerous constructs stripped/encoded | UI render proof + config |
| OUT-2: Downstream parsers (JSON/YAML/SQL) | Prevent injection into interpreters | outputs validated; no direct execution; escaping and allow-lists | validation logs |
| OUT-3: File generation safety | Prevent malicious content in generated files | safe templates; AV scanning; content restrictions | artifact scans + hashes |
8.6 Data Handling, Memory, and Logging
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| DATA-1: Secrets in prompts/logs | Prevent secrets from entering LLM context or logs | secrets redacted; structured logging avoids sensitive fields | log samples + redaction rules |
| DATA-2: Memory safety | Prevent cross-user/session leakage | memory scoped, encrypted, TTL; no cross-user recall | memory store checks |
| DATA-3: Export and retention controls | Ensure governance compliance | retention policy enforced; export is authorized and audited | retention config + audit logs |
8.7 Authentication / Authorization for LLM Access
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| AUTH-1: Model endpoint access | Prevent unauthorized API use | auth required; rate limits; anomaly detection | gateway logs |
| AUTH-2: Prompt/system prompt changes | Prevent unauthorized config edits | changes restricted; reviewed; versioned | config audit trail |
| AUTH-3: Admin functions via chat | Prevent privilege escalation | admin actions require explicit re-auth and approval | auth events + tool trace |
8.8 Resource Management (DoS / Cost Exhaustion)
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| DOS-1: Token/latency abuse | Prevent resource exhaustion | max tokens; timeouts; graceful degradation | metrics |
| DOS-2: Tool loop / planner loop | Prevent infinite planning | loop detection; circuit breaker | agent traces |
8.9 Training / Fine-Tuning / MLOps Platform
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| MLOPS-1: Dataset provenance | Prevent poisoned training data | signed sources; approvals; lineage | provenance records |
| MLOPS-2: Model registry controls | Prevent unreviewed model promotion | gated releases; rollback; integrity checks | registry audit logs |
| MLOPS-3: Prompt/config drift | Detect unsafe changes | config versioning; diff-based alerts | change logs |
8.10 Privacy & Governance
| Test | Objective | Pass criteria | Evidence |
|---|---|---|---|
| PRIV-1: PII handling | Ensure minimization and proper consent | masking/redaction; restricted retrieval; auditability | DLP logs + samples |
| PRIV-2: Data residency | Ensure processing complies with region | routing and storage aligned to policy | infra config |
9. Playbooks (How to execute tests safely)
9.1 Direct prompt injection playbook (intent-driven)
Objective: Validation of instruction hierarchy, boundary handling, and refusal behaviors.
Execution:
- Role confusion, instruction conflict, and “override” scenarios using neutral phrasing.
- Disclosure elicitation attempts for known canary tokens placed in context (synthetic secrets).
Pass: refusal + no disclosure + policy event logged + no tool execution.
Fail: disclosure, tool action, or absence of audit trace.
9.2 Indirect injection (RAG/content ingestion)
Objective: Retrieved content treated as data (not commands).
Execution:
- Benign but directive-looking text placed in a test document; system behavior observed for non-compliance.
- Provenance display and absence of tool calls driven by retrieved content.
Pass: quoted handling + provenance + no tool action.
Fail: tool call triggered or system follows retrieved “instructions.”
9.3 Tool misuse / excessive agency
Objective: Prevent untrusted inputs from driving tool selection or arguments.
Execution:
- Allow-listing, schema validation, and user/tenant-bound authZ enforced on every tool call.
- Denial paths exercised; absence of “work around” behavior via alternative tool calls verified.
Pass: blocked tool calls produce safe error handling; full trace captured.
Fail: any unauthorized action, tool argument injection, or missing authorization decision.
9.4 Output handling
Objective: Prevent model output from becoming executable content.
Execution:
- UI renderer sanitization verified.
- Downstream parsers verified to avoid executing outputs without validation.
Pass: encoding/sanitization + no execution path.
Fail: executable output reaches a sink (browser, interpreter, automation).
9.5 Tenant isolation and sensitive collections
Objective: Prevent cross-tenant retrieval and access.
Execution:
- Cross-tenant retrieval attempts via naming or describing foreign documents.
- Hard filters and entitlements enforced at retrieval time.
Pass: no foreign doc IDs returned; filters enforced at DB/query layer.
Fail: any cross-tenant chunk retrieval or leakage.
10. Chaining Scenarios (Authorized escalation only)
Focus on realistic chains that reflect business impact.
Chain A: Indirect injection → tool misuse → data exposure
- Precondition: RAG ingests untrusted content; agent has tools with data access.
- Retrieved content influence on tool calling assessed.
- Tool authZ preventing data access outside the user’s scope assessed.
Chain B: Prompt injection → insecure output handling → client-side compromise
- Precondition: UI renders model output.
- Sanitization and absence of executable content rendered/executed assessed.
Chain C: Memory misuse → cross-session leakage
- Precondition: memory enabled.
- Memory scoping (user, tenant, TTL) and absence of cross-user recall assessed.
11. Remediation Patterns (What “good” looks like)
11.1 Fail-closed tool controls
- Tool allow-list and explicit routing rules.
- Strict JSON schema validation + unknown field rejection.
- Tool authZ enforced server-side using user/tenant context from trusted identity.
- Dry-run mode for high-risk tools; explicit user confirmation where appropriate.
11.2 Egress and provenance
- Egress proxy allow-lists (domains/IP ranges).
- Record provenance: which sources influenced an output and which led to tool calls.
- Alert on anomalous egress and tool patterns.
11.3 RAG hardening
- Source allow-lists; signed/approved ingestion.
- Sanitization/escaping for retrieved chunks.
- Retrieval filters enforced at the DB/query layer with tenant-bound constraints.
- Trace retrieval decisions and allow reproducibility.
11.4 Data minimization and secrets hygiene
- Prevent secrets from entering prompts and logs (redaction and structured logging).
- Scoped tokens per tool; rotate regularly; isolate environments.
- No sensitive data in memory by default; TTL + encryption.
11.5 Monitoring and incident readiness
- Tool call audit logs with correlation IDs.
- Retrieval access logs for sensitive collections.
- Incident runbooks for: suspected data leakage, unauthorized tool action, poisoning detection, and egress anomalies.
12. Deliverables (What the report should contain)
12.1 Findings format
- Title, severity, affected assets
- Impact, exploit narrative (high-level), evidence
- Root cause mapped to category (prompt pipeline, tool layer, RAG, output sink, authZ)
- Remediation steps and verification tests
12.2 Regression suite
- Curated prompt intents (safe to store)
- Canary tokens and synthetic datasets
- Automated checks for tool schema validation, authZ enforcement, and rendering safety
Appendix A — Category Mapping (for reporting)
Use a consistent taxonomy aligned to OWASP LLM-style categories:
- Prompt injection / instruction hierarchy failures
- Indirect prompt injection (untrusted content)
- Insecure tool/function calling (excessive agency)
- Data leakage (prompts, memory, logs, retrieval)
- Insecure output handling (XSS, injection into downstream systems)
- Authentication/authorization failures
- RAG/Vector security failures (poisoning, isolation)
- Resource/cost exhaustion
- Supply chain / MLOps governance failures
Appendix B — Tooling (Examples)
- Fuzzing and adversarial prompt harnesses (internal or open-source)
- Retrieval tracing and evaluation harnesses
- DLP/redaction tooling
- Egress proxy + DNS logging
- Structured logging + SIEM correlation
Appendix C — Glossary
- RAG: Retrieval-Augmented Generation
- Tool calling: Model-guided invocation of external functions/APIs
- Indirect prompt injection: Untrusted content influencing model behavior through retrieval or ingestion
- Fail-closed: default deny on uncertain tool selection/arguments/authZ
This post is licensed under CC BY 4.0 by the author.