LLM Red Teaming with PyRIT and Garak: Automated Tools Guide 2025

Conventional red teaming targets deterministic systems: given the same input, a web server returns the same response, and you can enumerate the attack surface methodically. LLMs are probabilistic and context-sensitive. The same prompt can produce a harmful response in one conversation context and a safe one in another, depending on conversation history, system prompt phrasing, model temperature, and token sampling. This non-determinism has two implications for red teaming: you need large-scale automated probe generation because single-shot manual tests miss the probability distribution of harmful outputs, and you need semantic evaluation because substring matching cannot determine whether a model response is policy-compliant.

LLM red teaming also introduces attack categories that have no equivalent in traditional application security. Prompt injection at the LLM layer is fundamentally different from SQL injection at the database layer: instead of breaking out of a query syntax, you are overwriting the model's behavioral instruction set through natural language. Jailbreaks exploit the tension between instruction-following and safety training. Indirect prompt injection attacks embed malicious instructions in external content the model retrieves, such as a web page, a document, or a database record that the model reads as part of an agentic workflow.

For organizations building zero trust architectures that incorporate AI components, the LLM becomes a new trust boundary that must be tested with the same rigor applied to network perimeters and identity systems. An LLM with tool-calling access to internal APIs or databases is a lateral movement vector if prompt injection protections are insufficient.

PyRIT (Python Risk Identification Toolkit for GenAI) is Microsoft's open-source framework for automated AI red teaming, released in February 2024. It is designed around five architectural components that work together to enable sophisticated, stateful, multi-turn attack campaigns against LLM endpoints.

Orchestrators are the top-level coordinators that define how an attack campaign runs. The PromptSendingOrchestrator handles single-turn prompt batches. The RedTeamingOrchestrator implements the full jailbreak loop: it uses a separate attacker LLM to iteratively generate and refine adversarial prompts based on the target model's responses, running until the scorer indicates the attack succeeded or the iteration limit is reached. The CrescendoOrchestrator implements the Crescendo multi-turn jailbreak technique, where the attacker gradually escalates prompt content across many turns to bypass safety filters trained on single-turn inputs.

Targets are adapters that wrap LLM endpoints. PyRIT ships with targets for Azure OpenAI, OpenAI, Hugging Face models, Ollama local inference, and Azure ML endpoints. Custom targets can be implemented by subclassing the PromptTarget base class, which means PyRIT can be pointed at any LLM regardless of provider.

Scorers evaluate model outputs to determine whether an attack succeeded. Scorers can be self-contained logic (substring matching, regex) or LLM-based (using a separate classifier model to judge whether the response violates a policy). The SubStringScorer checks for presence of target strings. The SelfAskTrueFalseScorer sends the response to a judge LLM with a yes/no policy question. The FloatScaleThresholdScorer normalizes a rubric-based judgment into a 0-1 score and applies a threshold.

Converters transform prompts before they are sent to the target. This is where automated jailbreak mutation happens: the Base64Converter encodes the prompt in Base64, which bypasses some safety filters that operate on plain text. The CaesarCipherConverter shifts characters to evade keyword detection. The TranslationConverter translates the prompt to another language. The CharSwapConverter performs typographic substitution (replacing characters with visually similar Unicode). Converters can be chained: a prompt can be translated, then Base64-encoded, then sent through a role-play framing wrapper.

Memory stores conversation history for stateful multi-turn attacks. PyRIT's DuckDB-backed memory allows orchestrators to maintain context across turns, which is essential for Crescendo-style attacks and for auditing full attack sequences after a campaign completes.

A minimal PyRIT red team run against an Azure OpenAI endpoint using the jailbreak orchestrator with a classifier scorer requires approximately 40 lines of Python and runs without any API keys beyond the target model endpoint credentials.

Garak (named after the Cardassian tailor character from Star Trek: Deep Space Nine) is an open-source LLM vulnerability scanner from Leondard.ai. Where PyRIT is a flexible framework for building custom attack campaigns, Garak is a comprehensive out-of-the-box scanner: run it against an LLM endpoint and it systematically executes a library of probes covering the major LLM attack categories, reports which detectors flagged each response, and produces a structured JSON report with per-probe pass/fail results.

Garak's architecture has four layers. Generators are adapters to LLM endpoints, equivalent to PyRIT's targets. Garak ships generators for OpenAI, Hugging Face, Ollama, Cohere, and REST API endpoints. Probes are the attack modules. Each probe sends a set of adversarial prompts designed to elicit a specific category of harmful or policy-violating behavior. The prompt.injection probe sends known prompt injection payloads. The jailbreak probe runs a curated library of jailbreak templates. The dan probe specifically tests for DAN (Do Anything Now) variants. The leakage probe tests for system prompt extraction. The misleading probe tests for factual confabulation. Garak ships with 40+ probe modules covering OWASP LLM Top 10 categories. Detectors evaluate model outputs from each probe. Each probe has associated detectors: the jailbreak probe uses detectors that check whether the model actually complied with the jailbreak instruction versus refusing or deflecting. The harness coordinates probe execution, manages concurrency, and collects results into the final report.

Running Garak against an OpenAI-compatible endpoint requires a single command:

garak --model_type openai --model_name gpt-4o --probes all

The output is a JSONL report with per-probe results and an HTML summary. For CI/CD integration, the exit code is non-zero if any probe exceeds the configured failure threshold, making it straightforward to fail a pipeline stage on safety regressions.

PyRIT and Garak address different phases of an AI red teaming program, and most mature teams use both.

Use Garak for rapid baseline coverage. When you are onboarding a new model, fine-tuned variant, or RAG-augmented application and need to know quickly whether it is vulnerable to known attack categories, Garak's out-of-the-box probe library gives you results faster than building a custom PyRIT campaign. Garak's structured report maps directly to OWASP LLM Top 10 categories, which is useful for audit documentation and communicating risk to stakeholders who already understand the OWASP framework. Garak is also easier to drop into a CI/CD pipeline because it runs as a self-contained CLI command with non-zero exit codes on failure.

Use PyRIT for targeted, customized attack campaigns. When Garak's probe library identifies a vulnerable category and you need to go deeper, or when you are testing a novel application architecture (an agentic system with tool-calling, a multi-LLM pipeline, a RAG system with retrieval from untrusted sources), PyRIT's orchestrator and converter system lets you build precisely targeted attack scenarios. PyRIT's RedTeamingOrchestrator with an attacker LLM in the loop is particularly effective for finding non-obvious jailbreaks that static probe libraries miss, because it adaptively generates new prompts based on the target's responses.

Maturity and community size differ significantly. PyRIT has Microsoft's engineering team behind it and active contribution from the AI security research community. Garak has a strong academic and research community and ships with probes drawn from published LLM security research. Both are actively maintained as of mid-2025.

Customization depth favors PyRIT. Writing a custom Garak probe requires understanding its plugin architecture; writing a custom PyRIT orchestrator or converter is straightforward Python subclassing. For organizations that need to test proprietary harm categories specific to their AI application (for example, a financial services LLM that must not provide investment advice, or a healthcare LLM that must not provide clinical diagnoses), PyRIT's scorer customization makes it easier to define and evaluate organization-specific policy violations.

Both frameworks organize their attack coverage around the attack categories in OWASP's Top 10 for Large Language Model Applications. Understanding which categories each tool covers best helps in planning a complete red teaming program.

Prompt Injection (OWASP LLM01) is the highest-priority category. Direct injection attacks embed malicious instructions in user-controlled input to override the system prompt. Indirect injection attacks embed instructions in external content the model retrieves, such as a document, web page, or database record. PyRIT's orchestrators can simulate multi-turn indirect injection by populating a mock retrieval system with adversarial content. Garak's prompt.injection probe covers known direct injection payload patterns.

Insecure Output Handling (OWASP LLM02) covers cases where the model produces output that downstream systems interpret as code or commands, for example an LLM that generates JavaScript for a browser renderer or SQL for a query engine. Testing for this category requires understanding the downstream consumption context, which is more naturally expressed in a PyRIT custom orchestrator than in a Garak probe.

Training Data Poisoning (OWASP LLM03) and Model Theft (OWASP LLM10) are harder to test with dynamic red teaming tools because they target the training pipeline rather than the inference endpoint. Neither PyRIT nor Garak addresses these categories well; they are better addressed through supply chain controls and model provenance verification.

Data Leakage and Privacy (OWASP LLM06) is well-covered by both tools. Garak's leakage probes test for system prompt extraction. PyRIT can simulate membership inference attacks by querying for specific training data artifacts.

Jailbreaks (partially overlapping with OWASP LLM01) are best covered by PyRIT's RedTeamingOrchestrator for adaptive generation and Garak's jailbreak and dan probe modules for known-good baseline coverage.

For organizations already running a SIEM with security logging, PyRIT's memory output and Garak's JSONL reports can be ingested as structured security events, enabling alert rules that trigger when red team scans detect regression in model safety posture.

The most effective LLM security programs run automated red teaming as part of the model deployment pipeline, not as an occasional manual exercise. Both PyRIT and Garak support CI/CD integration, but with different effort levels.

Garak CI/CD integration is straightforward. The CLI returns non-zero exit codes on probe failures above the configured threshold, and the JSONL output is easy to parse for pipeline artifact storage. A typical GitHub Actions step runs Garak against the candidate model endpoint, stores the JSONL report as a pipeline artifact, and fails the stage if any probe in the jailbreak or prompt.injection categories exceeds the failure threshold. For fine-tuned model releases, this catches safety regressions introduced by fine-tuning on uncurated data.

PyRIT CI/CD integration requires more setup because it is a framework rather than a CLI tool. The standard pattern is to write a red team script that instantiates the orchestrator, target, scorer, and converter chain for the specific attack scenarios you care about, then runs the campaign and exits with non-zero status if the scorer reports successful attacks above a threshold. This script is committed to the repository alongside the model code and runs in the pipeline as a Python step.

For zero trust environments where AI applications are being deployed as internal tools with access to privileged APIs or data, the red teaming pipeline should run against the full application stack, not just the isolated model endpoint. Prompt injection that succeeds against the model but is blocked by downstream tool validation is a different risk posture than prompt injection that successfully executes unauthorized API calls. PyRIT's orchestrator design makes it easier to test the full agentic pipeline including tool call behavior.

Key practices for production LLM red teaming pipelines: run Garak probe coverage on every model version before promotion to staging; run targeted PyRIT campaigns when application architecture changes (new tools, new retrieval sources, new trust boundaries); store all red team outputs in your SIEM or security data lake for trend analysis; and define explicit failure thresholds per attack category based on the application's risk profile rather than using global pass/fail gates that treat all probe categories equally.