How to Scan AI Models for Malware: Pickle Exploits, Backdoored Weights, and Poisoned Training Data

Python's pickle module was built to serialize arbitrary Python objects, not just plain data. To reconstruct an object, pickle can call that object's reduce method, which returns a callable and a set of arguments to invoke during deserialization. That is the entire mechanism an attacker needs: craft a pickle stream whose reduce output points at a callable like os.system or subprocess.Popen with attacker-chosen arguments, and the moment something calls pickle.load() (or, for PyTorch, torch.load(), which is pickle underneath for most checkpoint formats) on that file, the callable runs with the privileges of whatever process loaded it.

This is not a theoretical attack path. Researchers have documented malicious .pth files uploaded to Hugging Face under attacker-controlled accounts that, when deserialized, ran embedded shell commands to download and execute ELF binaries on the host that loaded them. The PyTorch project itself has an open, years-old issue tracking that third-party PyTorch model files can execute arbitrary code during deserialization, because the underlying format was never designed to separate trusted structure from untrusted payload.

The practical implication for a pipeline: the moment your training or inference code calls torch.load(), joblib.load(), pickle.load(), or any framework wrapper around them, on a file your organization did not produce itself, you have handed that file's author code execution on your infrastructure. That includes checkpoints pulled from a public model hub, a vendor's hosted weights, a contractor's deliverable, or even an internal artifact store if anyone with write access to it could have been compromised upstream.

"AI model security" gets used to describe three mechanically unrelated problems, and conflating them is how teams end up thinking a pickle scanner is a complete solution when it addresses exactly one of the three.

Unsafe deserialization is a file-format problem. The risk lives in how the model is packaged, not in what the model has learned. A pickle-based checkpoint can smuggle arbitrary code regardless of whether the underlying weights are a legitimate, well-trained model or garbage. This is the problem ModelScan, picklescan, and Fickling are built to catch, and it is the only one of the three that a file scanner can meaningfully address, because it is the only one where the malicious content is executable code sitting in the file rather than numeric values.

Backdoored weights are a different problem entirely. Here the file format is completely safe (it could even be a safetensors file with no code execution risk at all), but the numeric parameters themselves have been manipulated, through data poisoning during training or a targeted fine-tune, so that the model behaves normally on almost every input and produces attacker-chosen output on a specific trigger pattern. No malicious code exists anywhere in the artifact. A pickle scanner has nothing to flag, because there is nothing to flag from a deserialization standpoint. Detecting this requires behavioral or statistical analysis of the model itself (trigger-search and activation-clustering techniques are active research areas), not file scanning, and tooling here is far less mature and far less production-ready than pickle scanners are.

Poisoned training data is upstream of the model file altogether. It is a data-integrity problem at the training set, not a property of any artifact you would ever scan in a model registry. A dataset containing mislabeled, subtly manipulated, or adversarially injected examples can bias a model's behavior in ways that have nothing to do with how the resulting checkpoint is later serialized. You address this with data provenance, source vetting, and dataset auditing at ingestion time, in a pipeline stage that runs long before a model file exists to scan at all.

The rest of this guide focuses on the first category, unsafe deserialization, because it is the one with mature, deployable, open source tooling. Sections below on failure cases return to why the other two matter and what they actually require.

Before wiring a scanner into your pipeline, confirm the following.

A Python 3.8+ environment in whatever CI runner or build image handles model ingestion, since the leading open source scanners (ModelScan, picklescan, Fickling) are Python packages installed via pip.

An accurate inventory of every point in your pipeline where a model file enters your infrastructure from outside your own training runs: a scheduled pull from a model hub, a manual download by a researcher, a vendor delivery, an internal artifact store fed by multiple teams. Each of those is a separate place the scan needs to run, not a single choke point you can assume covers everything.

An inventory of the serialization formats actually in use across your model zoo (PyTorch .pt/.pth, TensorFlow SavedModel or H5, scikit-learn joblib, or safetensors). ModelScan's supported-format list currently covers Pickle and its common derivatives (including joblib, dill, and cloudpickle), H5, and SavedModel; confirm your formats fall inside that list before assuming coverage.

Write access to whatever gates deployment in your pipeline, a CI job's pass/fail status, a pre-deployment approval step, or an artifact-registry admission policy, since a scan that only logs a warning without blocking anything is not a control, it is a dashboard.

Work through these steps in order. The first few establish a working scan; the later ones turn it into an enforced gate rather than an optional check.

A scanning step that has never been proven to fail on a bad input is not validated, it is assumed. Confirm the following before trusting the pipeline gate in production.

Run the scanner against the intentionally unsafe model fixtures shipped in ModelScan's own repository (built specifically for this kind of test) and confirm the CI step reports a failure and blocks whatever it is supposed to block, not just prints a warning to a log nobody reads.

Confirm the reverse case as well: run a legitimate safetensors file and a legitimate, clean pickle-based checkpoint through the same pipeline step and confirm both pass without a false block, since a scanner tuned so aggressively that it flags ordinary custom classes will get bypassed by researchers routing around it out of frustration.

Check that the CI job's exit code, not just its log output, is what your pipeline orchestration reads to decide pass or fail. It is a common integration mistake to run a scanner, capture its output for a report, and never actually connect a non-zero exit code to the build's pass/fail status.

Verify coverage against your full format inventory from the prerequisites step. If any format in active use in your model zoo falls outside what your scanner set (ModelScan, picklescan, Fickling) actually supports, that format is currently unscanned regardless of how confident the passing CI badge looks.

Be explicit with your team about what this control covers and what it does not, because the gap is where false confidence causes the next incident.

Backdoored weights pass clean. A model whose parameters were manipulated during training or fine-tuning to misbehave on a specific trigger input contains no executable code anywhere in the file. Every scanner in this guide is built to detect unsafe deserialization, not to analyze whether a model's learned behavior has been tampered with. That is a distinct, much less mature tooling category (behavioral trigger-search and activation analysis), and it should not be assumed as covered because a pickle scan came back clean.

Poisoned training data is invisible to this entire control. Nothing in this guide's pipeline step ever looks at a training dataset, only at a serialized model artifact. A biased or adversarially poisoned dataset produces a model that scans clean by every tool here, because the resulting checkpoint's format and, in the backdoor case, even its weights, may show nothing unusual. That risk has to be addressed at data ingestion, with source vetting and dataset auditing, in a pipeline stage that runs before training, not after.

Blocklist-based scanning can be evaded. Published research on pickle-based model supply chain attacks has documented techniques specifically designed to make a malicious payload stealthy against blocklist-driven scanners, by avoiding the exact set of dangerous imports and callables the scanner's rule set checks for. A clean scan result reduces risk against known techniques; it is not a formal proof the file is safe against a motivated, scanner-aware attacker.

A scan at one ingestion point does not cover a format changed later. A file that arrives safely as safetensors can still be converted or re-wrapped into a pickle-based format somewhere downstream in a pipeline, at which point the risk this guide addresses reappears. Scanning has to run at every point a model file is loaded from an external or lower-trust source, not once at initial download.

Blocklist scanning versus format migration. Running a scanner against existing pickle-based files is fast to deploy and requires no changes to how models are produced, but it inherits the structural weakness of any blocklist: it catches known-dangerous patterns and can miss novel ones. Migrating your own exports to safetensors is a structural fix that removes the risk category entirely for files your team controls, but it does nothing for the third-party pickle files, vendor deliverables, and legacy checkpoints already in your model zoo, which still need scanning indefinitely.

Blocking builds versus tolerating false positives. Failing the build on any flagged finding is the only version of this control that actually stops something, but pickle scanners can flag legitimate, unusual-but-benign custom classes as suspicious. An exception process is necessary for real research workflows, and that exception process is itself a risk if it becomes a rubber stamp that researchers learn to request by default rather than a genuinely reviewed override.

Pipeline latency versus ingestion-point coverage. Scanning every model file at every ingestion point adds time to whatever step it runs in, more noticeably for large checkpoints. Restricting scanning to only a single, centralized intake point is faster to build and lower friction, but it only works if every model genuinely flows through that one point, and researchers loading files directly for local experimentation are a common, easy-to-miss path around it.

Centralized model intake versus researcher autonomy. The strongest version of this control routes every external model file through a single scanned, gated pipeline before anyone can load it, which is also the same centralization tradeoff covered in How to Detect Supply Chain Compromise in a Software Build Pipeline: a single controlled intake point is easier to secure and audit, but it becomes a bottleneck researchers have an incentive to route around unless the process is fast enough to not get in the way. This is a different problem from sandboxing code an AI agent generates at runtime, covered in AI Agent Code Execution Sandboxing; that guide addresses containing code the model produces after it is already running, while this one addresses a model file that should never get to run its own hidden code in the first place.