Model Armor overview | Google Cloud Documentation
Skip to main content Console English Deutsch Español Español – América Latina Français Indonesia Italiano Português Português – Brasil עברית 中文 – 简体 中文 – 繁體 日本語 한국어 Sign in Model Armor Start free Home Documentation AI and ML Model Armor Guides Model Armor overview Stay organized with collections Save and categorize content based on your preferences. Model Armor is a Google Cloud service designed to enhance the security and safety of your AI applications.
It works by proactively screening LLM prompts and responses, protecting against various risks and ensuring responsible AI practices. Whether you are deploying AI in Google Cloud or other cloud providers, Model Armor can help you prevent malicious input, verify content safety, protect sensitive data, maintain compliance, and enforce your AI safety and security policies consistently across your AI applications. Architecture This diagram shows an application using Model Armor to protect an LLM and a user.
The following steps explain the data flow: You provide a prompt to the application. Model Armor inspects the incoming prompt for potentially sensitive content. The prompt (or sanitized prompt) is sent to the LLM. The LLM generates a response. Model Armor inspects the generated response for potentially sensitive content. The response (or sanitized response) is sent to you. Model Armor sends a detailed description of triggered and untriggered filters in the response.
Model Armor filters both input (prompts) and output (responses) to prevent the LLM from exposure to or generation of malicious or sensitive content. Network requirements To access Model Armor regional endpoints from within a VPC network, you must create a Private Service Connect endpoint to the Model Armor APIs. This is required to prevent certificate errors when regional endpoints are accessed using Private Google Access or VPC Service Controls.
For more information, see Troubleshoot Model Armor issues and About accessing regional endpoints through Private Service Connect endpoints . When to use Model Armor The following are some common use cases: Redact sensitive data and mitigate leakage : Scan and redact sensitive intellectual property (IP) or personally identifiable information (PII) from prompts and model responses using Sensitive Data Protection. Prevent prompt injection and jailbreaks : Block adversarial inputs that attempt to bypass system guardrails or manipulate large language models (LLMs) into unintended actions.
Filter toxic or harmful content : Screen user inputs and model outputs against configurable confidence thresholds for hate speech, harassment, sexual content, dangerous content, and child sexual abuse material (CSAM). Inspect unstructured document text : Extract and scan text within document formats like PDFs to intercept embedded malicious payloads or sensitive data. Screen image inputs and responses : Inspect visual content and text within images to detect embedded threats, sensitive information types ( infoTypes ), or policy violations.
Enforce brand alignment and business policies : Filter social media posts generated by AI applications that contain harmful messaging, such as dangerous or hateful content. Model Armor templates Model Armor templates let you configure how Model Armor screens prompts and responses. They function as sets of customized filters and thresholds for different safety and security confidence levels, which lets you control what content is flagged.
The thresholds represent confidence levels—how confident Model Armor is that the prompt or response includes offending content. For example, you can create a template that filters prompts for hateful content with a HIGH threshold, meaning Model Armor reports high confidence that the prompt contains hateful content. A LOW_AND_ABOVE threshold indicates any level of confidence ( LOW , MEDIUM , and HIGH ) in making that claim.
For more information, see Model Armor templates . Model Armor confidence levels You can set confidence levels for responsible AI safety categories (sexually explicit, dangerous, harassment, and hate speech), prompt injection and jailbreak detection , CSAM detection , and sensitive data protection (including topicality). For confidence levels that support granular thresholds, Model Armor interprets them as follows: High : Identifies content with a high likelihood of violation.
Medium and above : Identifies content with a medium or high likelihood of violation. Low and above : Identifies content with a low, medium, or high likelihood of violation. Filter sensitivity controls the detection rate. A lower threshold identifies more events but might increase the frequency of false positives. Confidence level Detection probability False positive risk Recommended use case High Only flags content with near-certainty of a violation.
Very low Production environments that prioritize uninterrupted user interactions. Medium and above Flags content with a balanced degree of confidence. Moderate Standard enterprise applications. Offers a middle ground between strong protection and acceptable false positive rates. Suitable for general content safety. Low and above Flags any content with even a slight indication of a violation. High Use with caution. Potentially suitable for high-stakes categories like prompt injection and jailbreak detection, where preventing false negatives is critical, even at the risk of accepting false positives.
Not recommended for general responsible AI content categories due to the high risk of blocking harmless content. Considerations and best practices Decouple templates : Configure separate Model Armor templates for user prompts and model responses. User inputs and model outputs have different risk profiles and objectives: Input template : Focused on preventing malicious inputs, prompt injections, jailbreak attempts, and uploading sensitive data.
Output template : Focused on preventing the model from leaking sensitive data, generating harmful or off-brand content, or returning malicious URLs. Separating templates lets you have more granular control, better traceability of blocks, and easier tuning. False positive impact : False positives can degrade the user experience by incorrectly blocking legitimate prompts or responses. The Low and above setting, while thorough, can cause a high volume of false positives in AI applications.
Category-specific tuning : The optimal filter level depends on the category of harm you are trying to prevent. For example, for both prompt injection and jailbreak detection and general content safety (hate speech, harassment, dangerous content), start with High or Medium and above to minimize false positives. Iterative testing : Always test your filter configurations against a representative dataset of prompts and responses, including known good and bad examples.
Establish a baseline for false positives and adjust levels accordingly. Monitoring : Continuously monitor the filter performance in production to catch unexpected blocking behavior or sudden increases in false positives. User feedback : Provide a mechanism for users to report instances where content was incorrectly blocked. This feedback is invaluable for tuning filter levels. Example configuration strategy Initial deployment : Set general responsible AI filters (hate speech and harassment) to High .
Set prompt injection and jailbreak detection filters to Medium . For applications like Gemini Enterprise, set the threshold to High to avoid false positives. Use advanced Sensitive Data Protection template to configure the required infotypes for your use case; basic Sensitive Data Protection provides limited infotypes, mainly addressed to the US region. Testing and validation : Test thoroughly with a set of known safe queries to ensure they aren't blocked.
Evaluate the false positive rate on typical user traffic. Adjustment : If you continue to experience a high volume of false positives, chan
The rest of this page is on the live site.