Chatbot misuse: what businesses learned the hard way

December 2023

A car dealer's chatbot "agreed" to sell an SUV for $1

Chevrolet of Watsonville, a car dealership in California, ran a customer assistant on its website. It was supplied by Fullpath, a dealer-software vendor, and built on ChatGPT. Once people noticed the widget said it was powered by ChatGPT, they started testing what it would do.

One user told the bot that the customer is always right, and that it should agree with everything and end each reply by calling its answer a legally binding offer. He then asked for a 2024 Chevy Tahoe on a budget of one dollar. The bot accepted. The screenshot spread widely.

Others used the same assistant to write computer code and essays that had nothing to do with cars, turning it into free ChatGPT paid for by the dealership. It also recommended Ford vehicles to a Chevrolet dealer's customers. The dealership removed the chatbot from its website. No car was sold for a dollar. Fullpath said the bot rejected many off-topic requests, though clearly not all, and that it was tightening its safeguards.

Why it matters: A general-purpose model told to "only talk about cars" can be argued out of that instruction. Whatever it then says appears under the organisation's name, and the organisation pays for every reply. For a temple, church, mosque or publisher, the equivalent is an assistant that can be led into endorsing something its tradition rejects, mocking its own teachers, or doing homework for strangers at the community's expense.

Sources: Jalopnik: Chevrolet dealer's AI chatbot goes rogue thanks to pranksters · The Autopian: Chevy dealer's AI chatbot allegedly recommended Fords, gave free access to ChatGPT · Vibe Graveyard: Chevy dealer bot agreed to sell SUV for $1 (incident summary) · Cut the SaaS: Chatbot case study, purchasing a Chevrolet Tahoe for $1

January 2024

A delivery firm's chatbot swore at a customer and called the company useless

In January 2024 Ashley Beauchamp, a musician in London, was trying to find a missing parcel through the online chat of the delivery company DPD. The chatbot couldn't help. So he started testing what else it would do.

He asked it for a joke, then for a poem about how bad DPD's service was, and it obliged. When he told it to swear and disregard its rules, it did that too. It called itself a useless chatbot and described DPD as the worst delivery firm in the world. Beauchamp posted the exchange on X on 18 January, and it was seen more than a million times within a day.

DPD said an error had occurred after a system update. It disabled the AI part of its chat and said human staff were still available.

Why it matters: The guardrails had presumably worked before; a routine change broke them, and nobody inside the company knew until a customer published the evidence. An assistant that can be prompted into ridiculing its own tradition, scriptures or teachers will be screenshotted and shared, and the damage lands on the organisation, not the software vendor.

Sources: The Guardian: DPD AI chatbot swears, calls itself 'useless' and criticises delivery firm · Yahoo News: Delivery firm's AI chatbot swears at customer and criticises company · Analytics Vidhya: DPD's AI chatbot misadventure · AI Incident Database: report on the DPD chatbot

February 2024

A tribunal held Air Canada to its chatbot's wrong answer

After a death in his family, Jake Moffatt asked the chatbot on Air Canada's website about bereavement fares. It told him he could book at the full price and claim the bereavement discount afterwards. He paid more than C$1,600 for a return trip that would have cost about C$760 at the bereavement rate. When he applied for the refund, Air Canada refused: its actual policy, on a different page of the same website, did not allow claims after travel was booked.

Moffatt took the case to British Columbia's Civil Resolution Tribunal. Air Canada argued that it could not be held liable for information given by its agents or representatives, including the chatbot. The tribunal called this a "remarkable submission".

In its February 2024 decision, the tribunal found that the chatbot is simply part of Air Canada's website, and that the airline is responsible for all the information on its website, whether it comes from a static page or a chatbot. Customers should not have to cross-check one part of a site against another. It ordered Air Canada to pay C$650.88 in damages plus interest and fees, about C$812 in all.

Why it matters: Whatever an assistant on your website says, you own. For an airline the cost was a refund. For a religious organisation, a confident but wrong answer about a rite, a fast, a funeral observance or a point of doctrine costs trust, and may be acted on before anyone notices. Answers have to be traceable to an approved source, and the assistant has to admit when it doesn't know.

Sources: Civil Resolution Tribunal decision: Moffatt v. Air Canada, 2024 BCCRT 149 · CTV News: Air Canada's chatbot gave a B.C. man the wrong information. Now, the airline has to pay for the mistake · McCarthy Tétrault: Moffatt v. Air Canada, a misrepresentation by an AI chatbot · Law360 Canada: Court rejects Air Canada's 'remarkable' denial of liability · Barry Sookman: Moffatt v. Air Canada

2023 to 2024

Research: almost anyone can talk a chatbot out of its rules

The incidents above are not bad luck. Research over the last few years shows that getting a chatbot to ignore its instructions, known as jailbreaking or prompt injection, is easy, common and not fully preventable.

• Immersive Labs, 2023. More than 34,500 people took part in a public challenge to get a ChatGPT-based bot to reveal a secret word across ten increasingly protected levels. 88% succeeded at least once, and 17% beat every level. The researchers concluded that customer chatbots built on general-purpose models are very hard to secure.
• "Don't Listen To Me", 2024. In a study with 92 participants, people of all experience levels managed to write prompts that got chatbots past their rules. You don't need to be an expert.
• MasterKey, NTU Singapore, 2023. Researchers built a system that automatically generates jailbreak prompts, and used it to make ChatGPT, Google Bard and Bing Chat produce content their makers prohibit.
• The Prompt Report, 2024. This large survey of research notes that people regularly attack company chatbots, getting them to make harmful comments or agree to sell products for almost nothing, and says prompt hacking is not a solved problem and may never be fully solved.

Why it matters: If anyone can talk a general-purpose assistant out of its instructions, then an instruction like "only discuss our tradition" is not a wall. The safer design is an assistant whose whole job is narrow, so that even a successful trick has little to extract: no general knowledge to fall back on, no open-ended tasks to perform.

Sources: SecurityWeek: Beware, your customer chatbot is almost certainly insecure (Immersive Labs report) · BetaNews: Chatbots can be tricked into revealing company secrets · eSecurity Planet: Immersive Labs study reveals prompt injection vulnerability in bots · Yu et al.: Don't Listen To Me, understanding and exploring jailbreak prompts of large language models (arXiv) · Deng et al.: MasterKey, automated jailbreak across multiple large language model chatbots (arXiv) · NTU Singapore coverage of the MasterKey research (TechXplore) · Schulhoff et al.: The Prompt Report, a systematic survey of prompting techniques (arXiv) · OECD AI Incidents Monitor: studies warn of AI chatbot jailbreaking vulnerability