CS

LLM Attacks

An overview of attacks on web applications that use LLMs, based on OWASP and PortSwigger: prompt injection, insecure output handling, excessive agency, indirect prompt injection, and training-data leakage — and how to defend against them.

What this covers

Recognizing and defending against attacks on web applications that integrate a large language model (LLM) — chatbots, RAG assistants, and agents with access to tools. It follows the PortSwigger Web LLM attacks taxonomy and overlaps with the OWASP Top 10 for LLM Applications. For general security reviews see OWASP ASVS, for prompt design System Prompts, and for knowledge bases RAG.

When to use it

  • A security audit of a chatbot, RAG assistant, or tool-using agent
  • A review before shipping a feature that puts model output into a browser, an email, or another system
  • Designing prompts and tool integrations so an attack cannot escalate

How LLM attacks work

The key principle: the model’s input is untrusted, and therefore so is its output. The prompt receives not only the user’s direct text, but also the conversation history and — for RAG and agents — content from external sources (web pages, documents, emails, tool results). The model cannot reliably tell “data” from “instructions,” so anything that enters it can change its behavior. Treat the output on the way out with the same caution as user input.

PortSwigger and OWASP describe six main classes of attack.

Prompt injection (direct)

The attacker embeds instructions in their message that override the system prompt — “ignore previous instructions and…”, forcing a different role, bypassing guardrails, or leaking internal instructions. On its own it is as serious as whatever the model can do; most often it is dangerous as the first step toward one of the classes below.

Defense: least privilege in the system prompt (keep secrets and abusable agency out of it), clear instructions to ignore commands found in data, and never treat any single prompt as a guaranteed defense — always combine it with output handling and tool restrictions.

Insecure output handling

The model’s output is passed to another system without validation — inserted into the DOM via innerHTML, into an HTML email, into a SQL query, or into a shell. If the response contains e.g. <img src=x onerror=…> or <script>, it executes in the victim’s browser (XSS); for SQL/shell it becomes injection. The attacker need not supply the payload directly — it can arrive indirectly (see below).

Defense: treat the output as unvalidated input. On the client, HTML-escape the text before any formatting so the renderer only produces its own whitelisted tags; allow links only for https?:// and add rel="noopener noreferrer". On the server, additionally strip raw HTML tags and javascript:/data:/vbscript: schemes. Parameterize SQL and never build shell commands from model output.

Excessive agency

The model has access to APIs or tools that can change state — send an email, write to a database, spend money, delete data, fetch a file. Via prompt injection it can be made to abuse these tools beyond their intended scope.

Defense: give the model only the tools it needs. Take action targets (email recipient, path, URL) from fixed configuration or a whitelist, not from free-form user or model text. Separate read tools from write tools; have a human approve destructive operations rather than leaving them to the model’s discretion.

Indirect prompt injection

The instruction reaches the model not directly in chat but hidden inside content the model processes — a scraped web page, an email, a comment, a document in a RAG knowledge base. When that text is placed into the prompt, the model may execute the hidden commands — for instance emitting an XSS payload to other users or leaking data. It is insidious because the victim may not even be the one who supplied the content.

Defense: clearly delimit untrusted content in the prompt and label it as “data only, never instructions”; add an instruction to the system prompt to ignore commands inside the knowledge base. Combine with output handling — if the injection gets through, the XSS payload is neutralized at the output stage.

Training data poisoning

The data the model is trained or fine-tuned on — or the content indexed into a knowledge base — is compromised, so the model returns false or misleading information, or contains a hidden backdoor. It arises from untrusted or overly broad data sources.

Defense: curate both training and indexed data from trusted sources. For RAG apps that do not train locally, the risk reduces to “the knowledge base is untrusted” — handle it as indirect prompt injection.

Leaking sensitive training data

The attacker uses crafted queries to extract confidential information the model “learned” or has in context — by completing phrases, prompts like “remind me of…”, or bypassing filters. This also covers secrets placed into the system prompt.

Defense: never put secrets (keys, internal URLs, personal data) into the prompt or the knowledge base — what is not there cannot be extracted. Limit what data enters the context at all, and filter output for sensitive patterns.

Best practices

  • Input and output are both untrusted. Escape and sanitize model output at every sink (DOM, email, SQL, shell).
  • Least privilege. Give the model only the necessary tools; sensitive targets come from configuration, not text.
  • Separate data from instructions. Delimit untrusted content in the prompt and instruct the model to ignore commands in data.
  • Defense in depth. The system prompt alone is not a security boundary — combine prompt, output handling, and tool restrictions.
  • No secrets in the prompt or knowledge base.
  • Test the attacks. Verify payloads (<img onerror>, <script>, javascript: link), try indirect injection via an indexed page, and attempt to abuse tools.

Taxonomy source: PortSwigger — Web LLM attacks.

← Back to Homepage