AI Application Security
AI Application Security is 3.2% of the CCDV-F blueprint. It tests the whole attack surface around a model: where untrusted content enters, why generated text is hostile input downstream, how data leaves, and where authorization is enforced.
Written by Kiran Manne
3.2% of the CCDV-F blueprint, which is roughly 2 questions of 53. It sits inside Security and Safety, worth 8.1% in total.
AI Application Security is 3.2% of the paper, the biggest slice of its domain, and lands as one or two questions. The domain page treats prompt injection as the headline. This page is about the rest of the attack surface a language model creates in a production system, and about the places where an injected instruction actually cashes out.
Two ideas carry most of it. First, model output is untrusted input to everything downstream. A generated string reaching a browser, a shell, a query builder or a file path is exactly as dangerous as one typed by an anonymous stranger, and the classical defences (escaping at the sink, parameterised queries, allowlists) apply unchanged. Teams that spend their attention on what goes into the model routinely forget what comes out of it.
Second, an attack needs an exit. Instructions smuggled into a retrieved document are inert unless something in your application can carry data back out: an outbound request, a rendered link, a tool accepting an address, a message the agent may send. Closing the egress path defeats a whole class of attack that you never had to detect.
Answering well means thinking in boundaries: where untrusted data enters, where privileged actions happen, and what is allowed to leave.
What the exam tests here
- Naming the channels through which attacker-influenced content reaches an agent's context
- Treating generated text as hostile input at every downstream sink
- Recognising exfiltration paths, including rendered media, links and tools that accept an address
- Enforcing authorization inside the tool implementation against the caller's identity
- Why keyword filtering and instruction-stripping do not constitute a defence
- Logging that makes an attempted compromise reconstructable after the fact
Enumerate the entry points
Attacker-influenced content reaches a model through more channels than a threat model usually lists. The obvious ones are documents a user uploads and pages an agent fetches. The less obvious ones are the results of your own internal tools, because the data those tools return may itself have been supplied by somebody else: a support ticket body, a commit message, a filename, a profile field, a product review, a webhook payload, the description text a third-party service hands back.
Add the responses of connected servers and the outputs of other agents and it becomes clear that "we do not process untrusted content" is nearly always false. Write the list for your own system, mark which entries an outsider can influence, and design around that set rather than around the two channels that came to mind first.
Model output is an injection payload for your own stack
Whatever the model produces will be consumed by something. Rendered into a page it is a scripting vector. Handed to a shell it is command injection. Interpolated into a query it is the oldest bug in the book. Written to a path it can climb out of the directory you intended. None of this requires the model to be malicious; a faithful summary of an attacker-written document is enough.
The mitigations are the ones your stack already owns, applied at the boundary: escape on output according to the sink, parameterise queries, resolve and constrain paths, never hand generated text to an interpreter without an allowlist. This gets missed because the text feels like it came from your own system, and it did not. It came from whatever the model read.
Close the exit before you try to catch the instruction
A successful injection still needs a way to move data to the attacker, and the channels are mundane. A markdown image whose address encodes a summary of the conversation, fetched automatically when the response renders. A link the user is invited to click. A tool that accepts an arbitrary address and dutifully requests it. An email the agent is permitted to send.
Closing these is deterministic in a way that detecting malicious instructions never is. Allowlist the destinations an outbound request may reach, refuse to auto-load remote media in rendered output, forbid the agent from assembling addresses out of material drawn from its context, and require confirmation before anything leaves the system. A narrow egress policy paired with limited privilege beats a promise of better detection.
Authorization belongs in the tool, not the description
A tool description stating that a function is for administrators is a comment. It is read by a model that weighs it against everything else in context, including whatever an attacker managed to put there. The check has to exist in the implementation, executed with the identity of the human on whose behalf the request is being made, before the action happens.
That means taking the caller's identity from the session rather than from a parameter the model fills in, and authorising the specific object being touched instead of dereferencing whatever identifier arrived. An agent able to read any record because its service account can is one injected sentence away from reading the wrong one, and the audit log will show the action as legitimate because, at the level the check was made, it was.
Where candidates go wrong
Trap 1: filtering for suspicious phrases
Blocking a list of known override phrasings is the security equivalent of a spam filter matching one spelling. Instructions can appear in any language, encoded, split across a document, embedded in an image, or expressed as an innocuous convention the model infers.
Worse, the filter manufactures a sense of coverage that justifies leaving privileges wide open. Where classification helps, it helps as one signal feeding a decision to stop, not as a turnstile that content passes through on its way to a fully privileged agent.
Trap 2: assuming an internal source is a trusted source
Data does not become trustworthy by being stored in your own database. A ticket body, a profile field and a commit message were written by people outside your trust boundary and merely persisted by you, so content arriving from an internal service can carry precisely the same payload as a fetched web page.
Answers that draw the boundary at the network edge, or that exempt first-party data from provenance labelling, are the ones to discard.
Trap 3: measuring security by whether an attack was blocked
Without logging, an application that resisted an attempt and one that never faced it look identical. Record what entered the context, which tools ran with what arguments, and what left the system, with enough retention to reconstruct a session.
This is also how the problem gets found at all: the first sign of indirect injection in a real deployment is usually an odd pattern in tool arguments or an outbound request nobody can account for, rather than an alert from a control that fired.
Common questions
How many CCDV-F questions come from AI Application Security?
3.2% of 53 questions is one or two, and it is the heaviest of the four sub-skills under Security and Safety. Given the domain totals about four questions across four named areas, breadth across all of them beats depth in this one alone.
Do I really need to escape model output if the model is well behaved?
Yes, because the model's behaviour is not the variable. It reads content that other people wrote, and a faithful rendering of hostile input is still hostile input at your sink. Escape according to the destination, parameterise queries, and constrain paths exactly as you would for text submitted by an anonymous user.
How does data actually leak out of an agent?
Through anything that makes a request or shows a clickable destination. A rendered image whose address carries the payload, a link presented to the user, a fetch tool given an arbitrary address, or a message the agent is allowed to send. Allowlisting destinations and refusing to auto-load remote media closes most of it deterministically.
Can the model enforce access control if the rules are in its instructions?
No. Instructions are weighed against everything else in context, and an attacker who can place text in that context is arguing on the other side. Authorization runs in code, with the identity of the requesting human, against the specific resource, before the effect occurs.
The rest of Security and Safety
Practise this
The CCDV-F bank covers every sub-skill in the blueprint, weighted the way the real paper is. Free sample questions need no account.
Free CCDV-F questions