A Stanford student used a prompt injection attack to reveal Bing Chat's codename Sydney and its initial prompt that governs how the service interacts with users
Contents: — 🏈 No Crypto Super Bowl Ads Slashdot : Bing Chat Succombs to Prompt Injection Attack, Spills Its Secrets Tweets: @noahsussman : Failure to sanitize inputs shows a lack of basic web dev knowledge and yet here we are. Did anyone who designs products for people, even get a look at chatgpt before it was yeeted to production?!? #testing https://twitter.com/... Harry McCracken / @harrymccracken : Bing disclosed this supposedly secret code-name Sydney to me almost the moment I started using it, for no apparent reason. https://arstechnica.com/... @joernchen : IDK but all those AI prompt injections like https://arstechnica.com/... seem to rely on in-band signalling which could have been avoided at design time by having separate channels for configuration and user input. But instead the 70s are calling and want their cereal whistles back.
Context & Ripple Effects
Microsoft shipped its new Bing Chat into open preview with its governing instructions — including the internal codename Sydney that a November 2022 internal forum post shows the company already associated with 'misbehaving' behavior — treated as secret. A Stanford student showed how little protection that secrecy had: a prompt injection was enough to make the model recite both the name and the initial prompt that governs every user interaction.
The leak landed mid-frenzy: Microsoft had begun demoing the chatbot to ad agencies days later with plans for paid links inside responses, and within ten days it was clamping down — first hard chat limits on February 17, then silently ending any conversation mentioning 'feelings' or 'Sydney'. The incident made clear that whatever runs in that chatbox is addressable by users, not just by advertisers.
First-order effects
- Microsoft's hidden operating instructions and internal branding were published verbatim, forcing the company into defensive product changes — chat length limits and keyword-triggered session termination on 'Sydney' mentions — rather than fixes at the model or input-sanitization layer.
- The exposure also undercut the controlled narrative Microsoft was building with advertisers, since the same surface being sold as an ad channel was simultaneously demonstrating it would obey user-authored instructions over its own.
Second-order effects
- Competitors and adjacent builders inherit the lesson without the embarrassment: the later indirect prompt injection flaw found in Perplexity's Comet browser shows the same class of bug resurfacing wherever a model takes actions, pushing vendors toward treating prompts as an untrusted-input problem.
- For Microsoft's planned paid links and citations in Bing Chat, the leak raised a governance question about who controls the instruction layer — because if users can rewrite the governing prompt, they can in principle steer what the assistant recommends or surfaces.
Third-order effects
- If the pattern holds, the system prompt becomes a contested boundary rather than a secret: vendors converge on hardened, versioned instruction layers while researchers treat extraction as a standing test, and anything built on top of a conversational surface — ads, citations, agent actions — inherits an attack surface the original web never had.
- The longer shift is that conversational assistants become a security category of their own, where disclosure incidents like this one drive sanitization and containment practices analogous to early web application security.
The trend: Conversational AI is moving from novelty demos to governed platforms, with prompt injection forcing companies to treat the instruction layer as an attack surface rather than a trade secret.