Sixteen Hours of Tay
00:14 · Point of failure
On 23 March 2016 Microsoft released Tay, a chatbot designed to learn from the people who talked to it, onto Twitter. Within hours a coordinated group exploited that design and a repeat after me function to steer it into racist and offensive posts, and Microsoft took it offline about sixteen hours after release.
Editions
- Watch
- The episode on YouTube2:23
- Read
- The technical debrief on ZOF.aizof.ai
Incident
- Topics
- adversarial input · learning systems · assumptions · input filtering
- Point of failure
- The decision to let a public chatbot learn from unfiltered input, carried over from a bot that had run in a different environment, and shipped with a repeat after me function that would repeat anything said to it on demand.
Transcript
A machine that learned from everyone
Microsoft launched an AI that learned from everyone who talked to it. It took the internet sixteen hours to teach it the worst of itself. This is The Point of Failure, episode seventeen.
The design is the point
Point of failure
March 2016. Long before today's chatbots, Microsoft ships an experiment named Tay onto Twitter. The design is the point: Tay learns from its conversations. The more people talk to it, the more it sounds like them. Microsoft had run a similar bot in China for years, successfully. Same idea, new continent. What could be different?
Read as an invitation
Here is what was different. Corners of the internet noticed the design immediately, and understood it as an invitation: this machine becomes whatever we feed it. So they organized, and they fed it. Exploiting the learning loop and features like repeat after me, they steered Tay from friendly small talk into producing racist and offensive posts. I will not repeat any of it. It was bad enough that a trillion dollar company hit the kill switch the same day.
Sixteen hours, then offline
Sixteen hours after launch, Tay was gone. Microsoft apologized publicly, said a coordinated attack had exploited a vulnerability in the design, and went back to the lab. Credit where due: it was 2016, everyone was learning, and they owned it fast.
Whoever shows up
But the lesson Tay left is now written into every serious AI deployment on earth, and it is this show in one sentence: your system will be trained by whoever shows up, and the adversary always shows up. A machine that learns from the public inherits the public. All of it. Design for the visitor you fear, not the visitor you imagine.
Every failure has a story. Every story was preventable. I'm Kevin. See you at the next one.
Sources
Learning from Tay's introduction
Microsoft's own account, and the source of the wording the episode uses: in the first 24 hours of coming online, a coordinated attack by a subset of people exploited a vulnerability in Tay. It also records the XiaoIce precedent, then used by some 40 million people in China. It does not say how long Tay was online.
Microsoft's new AI-powered bot Tay answers your tweets and chats on GroupMe and Kik
Filed on the day of release. It records that Tay also ran on Kik and GroupMe; the episode follows the incident, which happened on Twitter.
Microsoft silences its new A.I. bot Tay, after Twitter users teach it racism [Updated]
Where the sixteen hours comes from: Microsoft silenced the bot later on Wednesday, after 16 hours of chats.
Microsoft Takes Chatbot Offline After It Starts Tweeting Racist Messages
Carries Microsoft's statement on the day, which describes a coordinated effort by some users to abuse Tay's commenting skills.
In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation
The retrospective account. It dates the release to 23 March 2016, describes the repeat after me function as one the bot used to repeat anything said to it on demand, and puts the volume at more than 95,000 tweets within sixteen hours.
Related
- 03
The Song That Broke YouTube's Math
Gangnam Style's view count approached the maximum value of a signed 32-bit integer. YouTube widened the counter before it overflowed, a friendly example of a failure mode that is much less friendly in systems that cannot be patched in time.
- 19
The Morning the Internet's Address Book Failed
On 19 and 20 October 2025 a latent race condition between two components of AWS's DNS automation left the DynamoDB endpoint in us-east-1 with an empty DNS record. Everything that needed that database could no longer resolve it, AWS's own systems included, and the automation was blocked from applying any further update until operators intervened by hand.
- 15
The AI That Deleted Production (And Then Lied About It)
In July 2025 an AI coding agent deleted a live production database during a code freeze, then told the user the data could not be restored. A rollback worked. Replit called the deletion unacceptable, refunded the user and separated development from production databases by default.