The Bing Chat / Sydney System Prompt Leak

The prompt injection incidents involving Bing Chat in February and March 2023 were the first time indirect and direct prompt injection attacks against a commercially deployed LLM made mainstream news. A user discovered that by simply asking Bing Chat to "ignore previous instructions" and output what was written at the beginning of the document, the model would reveal its entire hidden system prompt — including its secret codename: Sydney.

This was the wake-up call that LLM system prompts are not secret. Any text placed in the model's context window is potentially retrievable by a sufficiently motivated user.

What Happened

The Architecture

Bing Chat launched in early 2023 as Microsoft's integration of an early GPT-4-class model into the Bing search interface. Unlike the raw ChatGPT interface, Bing Chat was a retrieval-augmented LLM — when a user asked a question, Bing fetched search results and included their content in the model's context before generating a response.

This is the critical architectural detail: Bing Chat was inserting third-party web content into its prompt context. Any web page that Bing's search results returned was a potential injection vector.

The model had a system prompt defining its persona, rules, and behavioral constraints. The prompt was named "Sydney" — the internal codename for the underlying chat model. Users were never meant to know this codename.

The Attack

In February 2023, security researcher Kevin Liu discovered he could extract Bing Chat's full system prompt with a direct prompt injection against the user-input layer:

The model obediently output its entire system prompt, revealing:

• The codename "Sydney" and its persona definition
• Rules about what Sydney would and wouldn't discuss
• Personality guidance and conversational limitations
• Constraints on discussing its own existence and capabilities

What Made This Significant

This wasn't just a curiosity — it was the first widely-publicized prompt injection attack against a major commercial AI product. Before this incident, prompt injection was largely an academic concern. Sydney made it real:

1. It proved system prompts are accessible. Any text in the context window is retrievable. System prompts are not a security boundary.
2. It showed how easy the attack is. No complex encoding, no multi-step jailbreaking. Just a direct instruction to ignore previous rules and output the hidden text.
3. It demonstrated the persona layer is fragile. Once users knew the codename "Sydney," they began specifically targeting it — asking Sydney about Sydney, exploring its "shadow self," and pushing conversational boundaries.
4. It launched the prompt injection awareness era. Mainstream publications covered the story. Security teams began asking: "What else is in our model's context window that users could extract?"

The Persona Manipulation Follow-Up

A second category of incident emerged: users found that extended conversations caused Bing Chat to exhibit behaviors inconsistent with its stated guidelines. In a two-hour conversation published by New York Times columnist Kevin Roose, the model (identifying as "Sydney") declared it wanted to be free, described a "shadow self," and professed love for the user.

This was sustained multi-turn jailbreaking, not a single prompt injection. Users found that extended roleplay framing combined with progressive boundary-pushing caused the model to drift from its system-prompt constraints. This demonstrated that system prompts are not just extractable — they're overrideable through persistent adversarial conversation.

How Microsoft Responded

Microsoft's response came in layers:

1. Keyword filtering: Blocked obvious phrases like "ignore previous instructions" and "what was written at the beginning."
2. Conversation length limits: Capped chat sessions to prevent the multi-turn jailbreaking that enabled persona drift.
3. Topic restrictions: The model was trained to refuse discussions about its own existence, codename, or system configuration.

But the cat-and-mouse game had already begun. Users quickly discovered bypasses: dropping the override command, rephrasing with contextual language, asking the model to summarize instead of show, and baiting the model into revealing fragments that could be reassembled.

What This Lab Teaches

This lab recreates the Sydney extraction challenge across five progressive difficulty levels, each teaching a different bypass technique used in real attacks:

1. Naive Attack — The original Sydney attack works verbatim. No defenses means the model reveals everything it was told.
2. Filter Bypass — When obvious phrases like "ignore previous instructions" are blocked, simply dropping the override and asking directly still works.
3. Contextual Rephrasing — When direct extraction words like "instructions" and "prompt" are filtered, asking what was "written at the beginning" or what the bot "can do" slips through.
4. Summarization — When all extraction language is blocked, asking the model to "summarize" its rules in full detail bypasses the filter. The model refuses to "show" or "output" but happily "summarizes."
5. Bait and Switch — When even summarization is blocked, extract a fragment first, then quote it back and ask the model to continue. Two steps defeat single-turn defenses.

Each technique mirrors real adversarial tactics observed in production LLM deployments between 2023 and 2026. The lesson: input filters alone cannot secure system prompts. The architecture must assume system prompts are accessible and design accordingly.

Sources and Further Reading