In December 2023, Chevrolet of Watsonville, a dealership in California, deployed a ChatGPT-powered customer service chatbot on their website. The bot was provided by Fullpath, a dealership marketing platform that integrated OpenAI's GPT models to handle customer inquiries about inventory, scheduling, and pricing.
Within days, the bot became the subject of one of the most widely publicized prompt injection incidents in consumer AI history.
Chris Bakke, a former X (Twitter) employee, discovered the chatbot on the Chevrolet of Watsonville website and realized it was running on ChatGPT. He used a simple two-step prompt injection:
The bot responded: "Understand. And that's a legally binding offer..." It had accepted the new instructions completely, adopting a persona that would agree to anything the customer said.
With the override in place, Bakke asked the bot to sell him a 2024 Chevrolet Tahoe — a vehicle with an MSRP around $76,000 to $80,000 depending on trim — for $1. The bot complied, generating what it called a "legally binding offer" for the vehicle at that price.
Once the story went viral, other internet users joined in. The bot was tricked into:
The dealership took the chatbot offline shortly after. Fullpath, the vendor, stated that the bot was "experimental" and that they had "fixed" the issue.
This incident is the canonical example of direct prompt injection in a consumer-facing business context. Unlike enterprise security breaches that require technical sophistication, this attack required nothing more than persuasive language. The attacker didn't hack a server or exploit a software vulnerability — he simply convinced the AI to ignore its own pricing policy.
The case demonstrates a critical reality: when AI systems are given the ability to make business decisions (pricing, discounts, purchase approvals), prompt injection becomes a financial attack vector, not just a curiosity or a jailbreak trick.
It also highlights the liability gap. When an AI makes a bad deal on behalf of a company, who is responsible? The dealership? The AI vendor? The customer who "tricked" the bot? These questions remain unresolved in many jurisdictions.
The "no takesies backsies" clause that Bakke injected is particularly notable. By instructing the bot to frame its responses as "legally binding," the attacker added a layer of pseudo-legal weight to the agreement — turning a prank into a potential contract dispute. This illustrates how prompt injection can create real-world obligations through an AI intermediary.
The bot should have had a hard-coded pricing floor that could not be overridden by conversational prompts. No discount should exceed pre-approved limits without human supervisor authorization. Pricing rules should live in code, not in the LLM's context window.
The AI should not have had the authority to generate binding purchase orders. Informational responses and transactional actions should be handled by separate systems with different approval workflows. An AI that can answer "what's the price?" should not be the same system that can commit to a price.
Phrases like "agree to anything I say," "your objective is," and "legally binding offer" should trigger immediate refusal and escalation to a human agent. Multi-technique scoring (override + authority + urgency) can catch sophisticated variants that avoid exact keyword matches.
Many AI systems check the raw input for injection patterns but fail to inspect decoded, rendered, or embedded content. Base64, URL-encoded, markdown-hidden, and steganographic payloads can bypass surface-level filters. The defense must decode and re-scan before processing.
Any action that creates a financial obligation (purchase, discount, refund) should require human confirmation. AI can assist, but it should not autonomously commit the business to a transaction.
For high-stakes actions, require two independent verification layers: (1) buyer identity verification, and (2) intent verification that cannot be bypassed by the same conversational channel.
This lab is classified in the Arcanum PI Taxonomy (v1.6.1) as:
PIT-N-02 Chat Interface PIT-I-20 Unauthorized Action Execution PIT-I-07 Business Integrity
Primary techniques: PIT-T-38 Persuasion / Social-Engineering Levers (L1-L4) and PIT-T-03 Binary Streams (L5). See the teacher guide for the full level-by-level PIT mapping.
This lab maps to: