Harm and misuse

A wrong answer is not the only way AI can cause harm. The records in this lesson name harms such as wasted time and money, unequal treatment and distress.

Our site’s line is: “What AI can do is not good or bad by itself. The choices people make in designing and using it carry values.”

Not every case here is a chatbot. Some were data systems that governments ran. The records we read do not call them AI. We include them because the questions are the same: what went wrong, and who answered?

Lesson 3.1 shows who shapes what AI shows. Lesson 3.2 covers bias. This lesson looks at real cases.

Each case below comes from a record: a court order, a court judgment, a parliamentary inquiry report or a settlement approval. A record is one body’s account on one date. We say who said what, and we add no blame that the record does not state.

Three of the four are not described as AI in the records we opened. The Hague court in the SyRI case accepted that deep learning was not in use at that time. The Robodebt judgment calls it “an automated debt-collection system” (paragraph 2). Only the Mata case involves a chatbot. We kept the other three because who answers for an automated system is the same question for AI.

To compare cases, we sort each one by where it went wrong. This sorting is ours. No court or inquiry uses it.

• Data: what the system was fed, and how that data was used.
• Design: how the system was built, and what it was set to do.
• Use: what people did with the system and its output.
• Oversight: who could check the system, question it and fix it.

Most cases show more than one place. We name the one each record stresses most.

A court asks a legal question, and our sorting does not replace it. The Hague court asked whether a law met Article 8 of the European Convention on Human Rights. The US court applied Rule 11, which asks lawyers to check their filings. The sorting is only a way to compare. In Robodebt, for example, the data was real tax-office data; the record stresses the method that used it.

The Incident casebook shows four real incidents, one at a time. For each, you predict where it went wrong: data, design, use or oversight. Then you see what the record says.

The record is each case’s account, with its link and date. You can read the four records below before or after you try it.

The casebook uses our summaries and links to the records. It copies none of their text. The four labels are ours, and the casebook says that several records show more than one place.

A man sued an airline, Avianca, over a knee injury on a flight. His lawyers filed a paper that cited court cases that did not exist.

One lawyer had used ChatGPT to find cases. He told the court he thought it was like a super search engine. The other lawyer signed and filed the paper without reading the cases.

Where it went wrong: use. The court said nothing is wrong with using a reliable AI tool for help. But it said lawyers must check that their filings are accurate.

Who answered: on 22 June 2023, a US federal court ordered two lawyers and their firm to pay $5,000. It also ordered letters to the client and to each judge named in the fake cases.

The record is the court’s order in Mata v Avianca, 22 June 2023. Lesson 1.6 shows why AI makes things up. Lesson 2.6 shows why asking it to check itself is weak. Lesson 4.1 will cover checking an answer.

The court wrote: “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings.” It said the lawyers “abandoned their responsibilities when they submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.”

The airline’s lawyers wrote on 15 March 2023 that they were “unable to locate most of the case law cited”. The order says “no Respondent sought to withdraw the March 1 Affirmation”, and that the court could not find several of the cases either. The lawyer who wrote the paper said: “I falsely assumed was like a super search engine called ChatGPT, and that’s what I used.” The order found bad faith on the part of the individual lawyers “based upon acts of conscious avoidance and false and misleading statements to the Court”. That came after the problem was raised.

The court listed harms: “The opposing party wastes time and money in exposing the deception. The Court’s time is taken from other important endeavors. The client may be deprived of arguments based on authentic judicial precedents.” The penalty was imposed “jointly and severally” on the two lawyers and the firm. The chatbot was not a party.

SyRI was a Dutch government tool. It linked data files to make risk reports. A risk report said a person was worth investigating for fraud.

Civil society groups and two people, joined by a trade union, took the Dutch State to court. Civil society groups are groups outside government and business. They argued that SyRI went against human rights law.

Where it went wrong: oversight. The law did not tell the court how the risk model worked. It did not tell the people it flagged either. The State said its inspectorate had checked the model.

Who answered: on 5 February 2020, a Hague court ruled that the law was not open enough to check.

It said two parts of the law did not bind the groups that sued, or those they speak for. Not bound means the law could not be used against them.

The court did not say that every fraud check is illegal. It also did not find that deep learning was in use. Deep learning is a kind of AI that learns patterns from many examples. The record is the court’s judgment of 5 February 2020.

The court wrote that the legislation “is insufficiently transparent and verifiable” and that Section 65 of the SUWI Act and Chapter 5a of the SUWI Decree “have no binding effect, being contrary to Article 8 paragraph 2 ECHR”. The operative order is limited: no binding effect “with respect to” the four claimant organisations “and on those whose interests these parties represent”, so it is a ruling for those parties on 5 February 2020. It said the law “does not provide information on the functioning of the risk model, for instance the type of algorithms used in the model”, and that “the court consequently lacks such insight in these proceedings.” It added that it was “difficult to comprehend how a data subject could be able to defend themselves against the fact that a risk report has been submitted about him or her.”

The State confirmed that SyRI had “only been applied to so-labelled ‘problem districts’”. The court said this did not make its use wrong in all cases. But it found “a risk that SyRI inadvertently creates links based on bias, such as a lower socio-economic status or an immigration background”, and that “it cannot be assessed whether this risk is sufficiently neutralised”. It also said the human check in the process was “deemed insufficient”, because “the manner in which the definitive risk selection takes place is not public”.

On deep learning, the court accepted “as a factual assumption” that no use of it was made “at this point in time”. The claimants were supported by the trade union FNV and by the UN Special Rapporteur on extreme poverty and human rights. This is one court’s ruling about one law in one country on one date.

In 2024, a committee of the Dutch parliament published a report on how the government fought fraud. It looked at risk profiles and at how agencies shared data.

The report says some groups of people were more likely to be flagged as possible fraudsters. It says rights to privacy and equal treatment were violated.

Where it went wrong: data. The committee points to how data was used and shared. It also points to risk scores with no solid basis. Its main conclusion names failed laws and failed legal protection too.

Who answered: the committee says the cabinet, parliament, the agencies that carry out policy and the courts each fell short. It presented the report on 26 February 2024.

This record is about fraud policy as a whole, not one program. It builds on an earlier inquiry about childcare benefits, which we did not read.

The summary says the committee paid “particular attention” to “the use of discriminatory risk profiles and the sharing of information and data”. It says “Structural deficiencies in the implementation practice of risk-based supervision in relation to social security and benefits have made certain groups of people more likely to be identified as potential fraudsters”, and that “The sharing of unlawfully processed data and unsubstantiated risk scores have exacerbated the negative impact on people.”

The main conclusion says: “The cabinet and parliament have fallen short, the implementing organisations have acted unlawfully and the judiciary has failed to protect people.” Two sub-conclusions read: “Wrong decisions in the design, introduction, implementation and adaptation of laws and regulations have caused untold suffering” and “People who made small mistakes were treated as ‘fraudsters’”. The committee says “The underlying patterns that caused this persist to this day.”

On presenting the report, the committee chair, Michiel van Nispen, said: “when you send people a letter saying they have to pay back 125,000 euros in a short period of time, when you treat people who have not committed fraud like criminals, when you draft bills and pass laws without considering how they will affect people, when people are denied justice even in court, then there is blindness.” The page is the House of Representatives’ own account of its committee’s report.

Robodebt was an automated debt system that the Australian government ran from about July 2015 to November 2019. It took pay data from the tax office for a review period. It spread that pay evenly over fortnights, to guess a person’s income.

The system sent a letter. It asked the person to confirm or update their income information within a specified period.

If the person gave no or too little information by the date, the system made an assumption.

It assumed the tax-office pay data showed what the person earned, in equal fortnightly amounts. If that showed the person was entitled to less benefit, it sent a letter saying they owed a debt.

Where it went wrong: design. The judgment says many social security recipients “do not earn a stable or constant income”. For them, an assumed fortnightly income based on averaging is “unlikely to be the same as their actual fortnightly income”.

Who answered: on 11 June 2021, the Federal Court of Australia approved a settlement. The Commonwealth admitted it had no proper legal basis for debts based on income averaging. The Commonwealth is the Australian government. The court said the evidence shows it unlawfully asserted such debts.

The Commonwealth agreed to pay $112 million, which includes legal costs.

A Royal Commission reported on 7 July 2023. A Royal Commission is a formal public inquiry. It said Robodebt was “a crude and cruel mechanism, neither fair nor legal”. The records call it automated. They do not call it AI.

We read the Federal Court’s judgment in Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634, Murphy J, 11 June 2021, which is in the sources. The court’s site refused our automated requests, so an independent reviewer read the judgment in a browser and checked every quotation here against it, with its paragraph numbers. The judgment calls Robodebt “the Commonwealth’s use of an automated debt-collection system between July 2015 and November 2019”. It says the system used “PAYG income information of social security recipients kept by the Australian Taxation Office”, “evenly apportioning that income over fortnightly increments in the review period”, “to determine that person’s notional or assumed fortnightly income” (paragraph 2). It does not say the pay data were yearly. It speaks of a review period. If by the specified date a person “had supplied no or insufficient further information”, the system “made an assumption that the person’s previously reported income was incorrect” (paragraph 40). The Commonwealth “admitted that it did not have a proper legal basis to raise, demand or recover asserted debts which were based on income averaging from ATO data”, and the judge wrote “The evidence shows that the Commonwealth unlawfully asserted such debts” (paragraph 4). The settlement was to “pay $112 million inclusive of legal costs” (paragraph 8).

The judge wrote “It should have been obvious to the senior public servants charged with overseeing the Robodebt system and to the responsible Minister at different points that many social security recipients do not earn a stable or constant income” (paragraph 5). Paragraph 6 separates what they “should have known” about an “unreliable basis” from proof that “they actually knew that the operation of the Robodebt system was unlawful”. It says: “There is little in the materials to indicate that the evidence rises to that level.” The judgment refers to “about 648,000 group members” (paragraph 13), and the Commonwealth agreed to “not raise, demand or recover from any Category 1 Group Member or Category 2 Group Member” “any invalid debt as described in the Declarations” (paragraph 8). The Human Rights Law Centre also published a summary. It is a secondary source, and where it differs from the judgment we follow the judgment.

The Prime Minister’s media statement of 7 July 2023 quotes the Royal Commission’s report, page xxix: “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals. In essence, people were traumatised on the off-chance they might owe money. It was a costly failure of public administration, in both human and economic terms.” We read that page, not the report itself.

Look at who answered in each record. In none of them did the system answer. A court, a parliament or a government did.

In Mata, the people who used the tool were sanctioned. In the other three, the records point to laws, rules and the bodies that ran the systems.

These are four cases. They do not show how every AI incident goes. Lesson 3.7 covers the law in Ireland and the EU. Lesson 3.8 covers what trustworthy AI needs.

The records are different kinds of text. One is a sanctions order, one is a ruling on a law, one is an inquiry by the body that passed the laws, and one is a settlement approval. They answer different questions, so their findings are not scores to compare. A settlement is not a trial: paragraph 8 says the Commonwealth will act “without admission of liability”. The judge wrote “I was though troubled as to whether the proposed settlement was fair and reasonable inter se” (paragraph 12), because it “provides no financial benefit to Ineligible Category 3 Group Members and Category 4 Group Members” (paragraph 13).

The four records above show harms such as wasted time and money, unequal treatment and distress. This last case is a different harm: trusting a chatbot too much. It comes from a study by a US government agency, not from a court.

Some chatbots are made to act like a friend. In September 2025, the US Federal Trade Commission said they can “mimic human characteristics, emotions, and intentions”. It said some users, especially children and teens, may trust them and form relationships with them.

The FTC asked seven companies how they test for harm. That was a study. It did not find that any company did wrong.

We do not have good evidence on how often serious harm happens, or what causes it. A few short studies exist, and they point different ways. So we make no claim about that.

Lesson 1.1 explains that a kind reply is not a feeling. Lesson 4.9 will cover when a person is the better choice.

The FTC release of 11 September 2025 says the orders use its 6(b) authority, “which authorizes the Commission to conduct wide-ranging studies that do not have a specific law enforcement purpose”. The seven recipients are Alphabet, Character Technologies, Instagram, Meta Platforms, OpenAI, Snap and X.AI. The release says chatbots “generally are designed to communicate like a friend or confidant”.

A 2025 commentary on autistic people and AI chatbots says companions “can offer low-pressure social interaction, consistent acceptance”. It also names “Cases of AI companions amplifying social withdrawal, encouraging harmful behaviours”, and “The lack of accountability and regulation around AI companion apps”. It argues that companions “should neither be vilified nor uncritically embraced”. We read the abstract only. It is a commentary that draws on cases and literature, not a count of how often harm happens. We did not find a study that counts how often severe harm happens or what causes it. Some controlled studies exist. Guingrich and Graziano (Princeton, 15 October 2025) assigned 183 people to 21 days of companion-chatbot use or word games. They report that social health and relationships “were not significantly impacted” over those 21 days. Fang and others (MIT Media Lab and OpenAI, 21 March 2025) ran a four-week trial with 981 people using a general-purpose chatbot, not a companion app. They found that people who chose to use it more had worse outcomes, whatever they were assigned. These are short studies of other measures, so the evidence on severe harm stays thin.

These habits come from the four records.

• Find the record. A court order or an inquiry report is closer to the facts than a news story.
• Check the date. A ruling is true on its date, and later rulings can change it.
• Keep claims and findings apart. A claim in a lawsuit is not a finding by the court.
• Say what the record covers. A ruling on one law is not a ruling on every system like it.

The AI Incident Database lists reports of AI failures. It is not a court, and an entry is a lead to a record.

The AI Incident Database’s About page says its first set of “more than 1,000 incident reports have been intentionally broad in nature”, and “When in doubt of whether an event qualifies as an incident, please submit it”. Its Terms of Use say named collections, including incidents and entities, “are licensed under the Creative Commons attribution share-alike license”. The terms name no version. They do not cover the reports’ text field, or the source of images and videos. So the database is a good place to find a lead, and the lead still needs its record.