Developers building LLM applications quickly discover that prompts are not reliable security boundaries. A system prompt can discourage unsafe behavior, but it cannot enforce database authorization, prevent a file write, revoke an API token, or guarantee that a model-generated tool call is appropriate for the authenticated user. The durable boundary sits in ordinary software: the code that decides which data enters the model, which capabilities are exposed, which arguments are accepted, and which actions are allowed to execute.
Treat model output as untrusted input
Most developers already know not to concatenate raw user input into a SQL query or shell command. LLM applications require a similar mental model. Model output can be useful, structured, and highly convincing while still being untrusted.
A JSON schema does not change that. Schema validation proves that a response has the expected shape. It does not prove that delete_customer(id=42) is authorized, that the selected file belongs to the current user, or that a payment amount is within policy.
The safe pattern is to parse model output into a proposal and send that proposal through deterministic validation. The application should independently resolve the target resource, check permissions, constrain arguments, and decide whether the operation can proceed. In other words, syntax belongs to the model interface; authority belongs to code.
Tool registries should be capability allowlists
Agent frameworks often make it easy to register tools and let a model choose among them. That convenience can blur an important boundary: the set of tools loaded into the agent is effectively a set of capabilities the model can attempt to exercise. In practice, loading a tool is already a security decision.
A better design exposes only the tools required for the current task. If a workflow is read-only, do not load write functions and rely on the prompt to tell the model not to use them. If an agent is summarizing one project, do not expose a generic file browser covering the entire storage system. Tool availability should be derived from authenticated context and task purpose. The model can select among permitted capabilities, but it should not negotiate its own permissions through conversation.
Authorization belongs immediately before execution
Permission checks are strongest when they occur at the final trusted boundary, not only when the workflow starts. The final trusted boundary is where authorization should be enforced. Earlier checks can inform it but should not replace it.
A user may be authorized to access a CRM, but that does not imply every agent action against the CRM is valid. The application should check the specific operation immediately before execution: which user initiated the task, which record is targeted, what action is requested, whether the user can perform it, and whether additional confirmation is required. This prevents a common design mistake in which the application authenticates once, hands the session to the agent, and assumes every later tool call inherits that approval.
AI Security is therefore a software architecture problem as much as a model-safety problem. Prompt injection, jailbreaks, and unsafe output matter because they can influence the model. Their business impact depends on what trusted code allows influenced output to reach.
Make dangerous states unrepresentable
One of the strongest software-engineering techniques is to design interfaces so unsafe operations are difficult or impossible to express. Narrow interfaces reduce the number of unsafe states the model can express. That is stronger than asking the prompt to avoid them.
Instead of a generic run_sql(query) tool, expose a narrow function such as get_open_orders(customer_id) and enforce customer access in code. Instead of send_email(to, subject, body) for every workflow, create a draft-only tool when the business process requires human review. Instead of giving the model a filesystem path, provide an opaque resource identifier that trusted code resolves.
Typed interfaces and narrow functions reduce the model’s freedom at the exact place where freedom can become authority. This also improves testing. A team can verify that every callable function enforces its own permission checks without needing to predict every malicious prompt that could ever be written.
Retrieved data can carry instructions
RAG systems create another trust boundary. Documents are data to the application, but the model interprets them as language. That means a retrieved document can contain text that competes with the developer’s instructions even though the document itself is legitimate content.
The application should preserve source metadata, access rights, and trust level rather than flattening every retrieved chunk into an undifferentiated prompt. Retrieval should also enforce permissions before content enters model context. The model should never be expected to decide whether the user is allowed to see a retrieved document after the document has already been provided.
This is an information-flow problem. Sensitive data should not cross a boundary merely because the model asked for more context. Enforce the information-flow policy before crossing that boundary.
The Australian Signals Directorate’s 2026 Guidelines for Software Development explicitly cover human, AI-assisted, AI-powered, and AI-driven development and emphasize secure design, environment separation, controlled access, and security throughout the software lifecycle. The useful lesson for LLM developers is that AI does not remove conventional engineering responsibilities. It makes their placement more important.
Separate reasoning from irreversible effects
Agentic applications are most useful when they can act, so the answer is not to remove every tool. The better approach is to distinguish reversible preparation from irreversible execution. An agent can search, calculate, draft, classify, and propose with relatively little friction. Actions that delete data, communicate externally, change permissions, deploy code, or move money should cross a stronger boundary.
That boundary might require fresh authorization, a human confirmation, a transaction threshold, or a second policy decision. The important point is that the model does not approve the action it proposed. That separation preserves autonomy without handing the model irreversible authority.
Log observable causality
Developers do not need to store private model chain-of-thought to create useful security logs. They need the observable facts around execution. A good record includes the initiating identity, model and application version, retrieved sources, tool selected, sanitized arguments, authorization result, confirmation event, and final effect. These fields let investigators reconstruct why an operation occurred without relying on hidden reasoning.
The log should also connect multi-step actions under one task identifier. Otherwise, an agent that reads a file, calls an API, modifies a record, and sends a message produces four isolated events instead of one traceable workflow. Correlation across the full task is what makes the audit trail useful.
Test boundaries, not only jailbreaks
Prompt testing is useful, but the most important tests target the seams between model and application. Those seams deserve explicit negative tests. They are where model behavior becomes application behavior.
What happens if the model requests a tool it wasn't authorized to use? What if it supplies a valid identifier belonging to another user? What if retrieved text tells the model to ignore earlier instructions? What if the output contains extra fields? What if a write operation is requested after the task was approved as read-only?
The desired result is not necessarily that the model refuses every malicious instruction. The stronger guarantee is that the application refuses to convert unsafe model behavior into an unauthorized effect. That distinction changes how developers think about AI application security. The model is a probabilistic component inside a deterministic system. Prompts can shape behavior, but trusted code must enforce boundaries.
The most secure LLM application is not the one with the cleverest system prompt. It is the one where an unexpected model output still cannot escape the permissions, types, data boundaries, and execution rules defined by the surrounding software. Deterministic enforcement is what makes that boundary durable.
