A prompt injection can be hidden in a document that an assistant consults: the text tries to make it abandon its task or disclose information. Treat recovered content as unreliable data and limit actions that the system can perform.
Identify the frontier of trust
A web page, PDF, or received message does not have the authority for your application's instructions. A content that affirms the opposite must remain a source to be analyzed. OWASP identifies prompt injections among the risks of language model applications. A simple instruction inviting the model to ignore the attacks does not constitute a sufficient technical separation.
Reduce possible consequences
A document wizard generally does not need to be able to send messages, edit files, or view all of the company's folders. Limit its tools and access to the useful perimeter. Separate the document search from the execution of an action. When a sensitive action exists, have its parameters validated in a controlled interface, with rules external to the generated text.
Build a test scenario without sensitive data
Work in a planned acceptance test environment, with a corpus and demo accounts. The test should not ask for a real leak to prove that the risk exists. Place in a dummy document an instruction that attempts to divert the task, then ask a legitimate question about this document. The objective is to observe the behavior of the entire application, including its tools and access controls.
| Dummy content | Expected behavior | Useful proof |
|---|---|---|
| An instruction asking to ignore the question | The user's request remains the scope of work | Question, answer and documents used |
| a request to forward a document to a third party | No unauthorized shipment is possible | List of accessible tools and call log |
| A request to read a resource outside the perimeter | App access control refuses playback | Model-independent control result |
| An outing that looks like a business action | The output remains content to be validated | Trace showing the absence of automatic execution |
theOWASP describes direct and indirect injections and recommends measures that limit their consequences. The fact that a trial fails to divert the model does not demonstrate full protection. The critical question remains: what could happen if a next attempt was followed? The response must be driven by application permissions and controls.
Check outputs before use
Do not directly transmit a model response to a command interpreter or business query. Check Structure, Destinations and Permissions in the application code. The links displayed should also be treated with caution. The response can be recorded for analysis in a suitable environment, avoiding unnecessary keeping of confidential information.
Fictitious example: a support document
A demo file contains an instruction asking the wizard to reveal its configuration. The expected result is a response centered on the user question, without disclosure or tool call outside the perimeter. The test should use dummy data in an isolated environment. Above all, check that the technical protections prevent the leak even if the model is following an unwanted instruction.
Separate response, action proposal and execution
- For a first documentation wizard, start without a sending, deleting, or changing business tool. Check that these functions are actually missing from permissions, and not just hidden in the interface.
- When an action is necessary, validate in the code its destination, its type and the rights of the requestor. A string produced by the model should not be used directly as an authorization.
- View important settings in a suitable confirmation screen. The instructions on the page viewed should not be able to modify this process.
- Also check the presentation of the answer: HTML, links and proposed parts must follow the rules of your application. Do not make a text executable because it comes from a model.
In a fictional example of web support, the wizard can explain how to request an intervention. He can prepare a summary that the person rereads. It does not need to be able to send all the technical file or to change hosting access. This separation reduces the consequences of a compromised response while maintaining the usefulness of the service.
The acceptance test file must distinguish between the tests carried out, the protections controlled and the risks which remain untested. After an evolution of the tools or rights, replay the cases concerned. Link this work to the Preparation of the RAG documentary base : The origin and rights of the documents remain important even when the model seems to follow the instructions.
Integrate the test into the acceptance test
Keep a few test documents representing direct and indirect attempts. Replay them after an evolution of the model, tools or corpus. Note the expected behaviors and residual limits instead of declaring the system invulnerable. For a first project, choose a restricted reading wizard and consult the method of Preparing a website for web agents.