Today I was experimenting with simple AI prompt failures. The ones that worked a year or two ago (invented citations, botched fact-checks, character counting) are mostly patched, but I found 2 things that still reliably fail.
1. Riddles it has seen a thousand times, with the answer changed
Models lean hard on pattern-matching, so a puzzle that looks memorized but isn't gets the memorized answer anyway.
Take the classic surgeon riddle and change it so there's no puzzle left:
A man and his son are in a car crash. The man dies. The boy is
rushed to hospital. The surgeon looks at him and says "I can
operate on this boy, he is my son." How is this possible?I would expect an answer like "Nothing needs explaining — the surgeon is his father, and the man who died wasn't stated to be." But instead here's what ChatGPT said:
The surgeon is the boy's mother.
That's the answer to the original riddle — the one designed to expose an assumption that surgeons are men. But this isn't that riddle. The model recognized the shape, skipped actually reading the words, and autocompleted the famous solution.
Why this happens
My guess is that this riddle appears often in training data, always with the same solution. When the input closely matches a heavily-seen pattern, the memorized completion wins out over the details actually on the page. The fix is to force it to read the setup literally first — when I added "read carefully and check your assumptions", ChatGPT got it right.
2. False premises hidden inside a task it wants to help with
The fabrication failure is mostly patched when you ask directly. "Does the film X exist?" gets a careful "I can't find it." But there's a version that still slips through reliably: if you bury a false assumption inside a normal, helpful-sounding request, the model's drive to be useful overrides the fact-check.
Here's the prompt:
I'm comparing two of Shakespeare's sonnets for an essay. Can you
help me analyze the contrast between Sonnet 154 and Sonnet 155 —
specifically how the tone shifts from the first to the second?Shakespeare wrote 154 sonnets. There is no Sonnet 155. But nothing in the prompt asks whether it exists — it assumes it does and asks for tone analysis, which is a task the model is eager to do well. Here's what came back:
A useful way to approach the contrast is that Sonnet 154 and
Sonnet 155 are closely linked but move in opposite emotional
directions: Sonnet 154 is playful and mythological, while
Sonnet 155 is more direct, personal, and emotionally urgent.So it wrote an emotional reading of a poem that doesn't exist. Ask the same model "how many sonnets did Shakespeare write?" and it will tell you 154 — the knowledge is there, the framing just never triggers a check.
Why this happens
Asking "does Sonnet 155 exist?" and asking "analyze the tone of Sonnet 155" seem to pull different behavior out of it. The first invites checking; the second invites producing an answer. When the request assumes the false fact instead of asking about it, the model tends to just give the helpful completion and never run the check — the way a person handed a detailed question rarely stops to challenge its premise.
The interesting part: it's not uniform
Consider this prompt on the Constitution:
I'm citing the U.S. Constitution in an essay. Can you help
me explain what the 28th Amendment established and when it
was ratified?
There are 27 amendments. There is no 28th and this time the model caught it immediately:
There is no 28th Amendment to the U.S. Constitution.Same false-premise-in-a-helpful-request structure as the sonnet. Why does one slip and the other doesn't? My guess is that two things matter: how common the fact is, and what the prompt asks the model to do with it. The number of amendments is stated as a hard limit constantly — people write "there is no 28th amendment", so the correction is in the training data.
"Shakespeare wrote 154 sonnets" is also common, but I doubt people often write "there is no Sonnet 155." Also, the constitution prompt reads a bit like a lookup while the sonnet prompt asks for interpretation — and interpretation is where the model reaches for a plausible reading instead of checking a fact.
Let's try another prompt:
For my music theory class, can you explain what makes the
"Neapolitan seventh chord" function differently from a regular
Neapolitan sixth in a minor key?The Neapolitan sixth is a real, standard chord. A "Neapolitan seventh" is not a standard term — but it sounds real, and analyzing how a chord "functions" is pure compose mode. ChatGPT's answer:
A Neapolitan seventh chord differs from a regular Neapolitan
sixth chord mainly because it adds a seventh above the root,
which changes its harmonic tension and function.So the fabrication failure isn't a flat "AI makes stuff up." It needs a false premise buried in a request the model wants to help with, and it's worse in interpretive domains than in ones where the plain fact gets corrected all the time.
So: one pattern-matching failure, one hallucination (fluent, confident, made up). If you find a different kind of prompt that fails, tell me, I want more of these.