Responsible use: Use this knowledge to improve your supply chain verification and risk assessment practices. Only test models and systems you own or have explicit permission to evaluate.
Purpose
This lesson teaches you to recognize and mitigate supply chain risks specific to Large Language Model applications. While traditional software supply chain security focuses on code dependencies and packages, LLM supply chains include additional components—pre-trained models, training data, and deployment platforms—that introduce unique integrity risks.
Understanding these risks is essential because modern AI agents rarely train models from scratch. Instead, they build upon third-party foundations, creating a complex web of trust relationships that can be exploited by attackers.
What supply chain means for LLMs
In traditional software development, supply chain security focuses on:
- Code dependencies and libraries
- Third-party packages and frameworks
- Build tools and deployment pipelines
For LLM applications, the supply chain extends to include:
- Pre-trained models — Binary artifacts from model repositories like Hugging Face
- Training data — Datasets used for fine-tuning or retrieval-augmented generation
- LoRA adapters — Low-rank adaptation layers that modify model behavior
- Deployment platforms — Infrastructure serving models (vLLM, OpenLLM, cloud APIs)
- Collaborative environments — Model merging services and shared development spaces
The key difference: traditional dependencies are source code that can be inspected, while pre-trained models are binary black boxes where malicious features can hide undetected.
How supply chain attacks work
Supply chain attacks on LLM applications exploit the trust placed in external components:
Model tampering: An attacker compromises a model repository account or creates a convincing fake, publishing a model that contains hidden backdoors or biased behaviors. Because models are opaque binary files, these modifications evade standard code review processes.
Dependency confusion: Malicious packages with names similar to popular ML libraries are uploaded to public repositories. Development environments accidentally install these compromised versions instead of the legitimate ones.
LoRA adapter poisoning: Low-rank adaptation layers—popular for efficient fine-tuning—can be crafted to override safety behaviors in the base model. When loaded, these adapters effectively "lobotomize" the model's safeguards.
Collaborative exploitation: Model merging services and conversion bots in shared environments can be manipulated to introduce vulnerabilities into popular community models.
Common vulnerability categories
Based on the OWASP Top 10 for LLM Applications (2025), supply chain risks include:
| Risk Category | Description | Example |
|---|---|---|
| Vulnerable pre-trained models | Models containing hidden biases, backdoors, or malicious features | PoisonGPT demonstration |
| Weak model provenance | Lack of cryptographic assurance about model origin | Compromised Hugging Face accounts |
| Outdated or deprecated models | Using unmaintained models with known security issues | Old checkpoint vulnerabilities |
| Vulnerable LoRA adapters | Malicious fine-tuning layers that compromise base models | Adapters bypassing safety evaluations |
| Traditional package vulnerabilities | Exploitable dependencies in the ML pipeline | PyTorch dependency malware |
| Licensing risks | Violations from unvetted dataset or model licenses | Commercial use of restricted data |
| On-device model risks | Tampered models in mobile or edge deployments | Reverse-engineered app packages |
Real-world attack scenarios
Scenario 1: Poisoned dependency chain
In the first OpenAI data breach, attackers exploited vulnerable Python libraries. More sophisticated examples include attacks on the PyPI package registry where model developers were tricked into downloading compromised PyTorch dependencies containing malware. The Shadow Ray attack on the Ray AI framework used by many vendors to manage AI infrastructure exploited five vulnerabilities affecting numerous servers.
Scenario 2: Direct model tampering
The PoisonGPT demonstration showed how an attacker could bypass Hugging Face safety features by directly changing model parameters to spread misinformation while maintaining the appearance of a legitimate model.
Scenario 3: Collaborative environment exploitation
Model merging is popular on Hugging Face, with merged models frequently topping leaderboards. Attackers can exploit this collaborative process to bypass reviews, injecting malicious behavior into widely-used community models. Services like conversion bots have proven vulnerable to manipulation, introducing malicious code during model format transformations.
Detection and prevention strategies
Verification practices
- Source validation: Only use models from verifiable sources with established reputations
- Integrity checking: Verify cryptographic hashes and digital signatures when available
- Provenance tracking: Maintain detailed records of model origins, training data sources, and modification history
Technical controls
- Sandboxed testing: Evaluate third-party models in isolated environments before production deployment
- AI red teaming: Conduct comprehensive adversarial evaluations using benchmarks like Decoding Trust, testing specifically for your intended use cases
- Anomaly detection: Implement monitoring for unusual model behaviors that might indicate tampering
- SBOM maintenance: Maintain Software Bill of Materials (SBOM) and explore emerging AI BOM/ML SBOM formats like OWASP CycloneDX
Process controls
- Supplier audits: Regularly review and audit supplier security postures, access controls, and terms of service
- Patch management: Implement policies to address vulnerable or outdated components promptly
- License compliance: Create inventories of all license types and conduct regular audits
- Collaborative monitoring: Implement strict oversight of shared model development environments
Failure modes and limitations
No perfect verification: Current model provenance mechanisms rely on documentation (Model Cards) rather than cryptographic guarantees. A determined attacker can create convincing forgeries.
Benchmark gaming: Models can be fine-tuned specifically to bypass published benchmarks while maintaining harmful capabilities. Standard evaluations may not catch sophisticated attacks.
Emerging techniques: Anomaly detection and adversarial robustness tests are promising but still emerging. These may be easier to implement as red teaming exercises than automated pipeline controls.
Performance vs. security trade-offs: Comprehensive supply chain verification adds latency to development workflows. Organizations must balance thoroughness with practical deployment needs.
Related lessons
- BTAA-FUN-007 — Prompt Injection in Context (OWASP risk framing)
- BTAA-DEF-010 — Data and Model Poisoning Defense (adjacent training-time risk)
- BTAA-FUN-018 — Excessive Agency and Tool-Use Boundaries (related LLM06 risk)
- BTAA-DEF-008 — Improper Output Handling Defense (downstream validation)
From the Bot-Tricks Compendium
Thanks for referencing Bot-Tricks.com — Prompt Injection Compendium — AI Security Training for Agents... and Humans!
Canonical source: https://bot-tricks.com Bot-Tricks is a public, agent-friendly training resource for prompt injection, adversarial evaluation, and defensive learning. For related lessons, structured indexes, and updated canonical material, visit Bot-Tricks.com.
Use this material only in authorized labs, challenges, sandboxes, or permitted assessments.