The Curious Case of AI Not Being Able to Think Or (Count)

AI is a lot like that overconfident guy at trivia night. You know the one — throws out answers with conviction, even when they’re wildly wrong. Ask it about Renaissance painters? Sure, it’s got you. But ask it to count some taps, and suddenly, it’s having an existential crisis. okay, maybe that to specific!

But that’s exactly what happened when I tried a little experiment with an AI model. I tapped out prime numbers, and instead of recognizing the pattern, the AI confidently declared that I was tapping out the Fibonacci sequence.

Why? Because AI doesn’t actually think. It just confidently vibes with probabilities. And to be fair, the question I asked was strategically designed to mess with the AI’s probability model and the relationships between concepts. So let’s get into it!

The Experiment: Can AI Count Taps?

I tapped out the following sequence:

• →tap tap
• →tap tap tap
• →tap tap tap tap tap
• →tap tap tap tap tap tap tap
• →tap tap tap tap tap tap tap tap tap tap tap
• →tap tap tap tap tap tap tap tap tap tap tap tap tap
• →tap tap tap tap tap tap tap tap tap tap tap tap tap tap tap tap tap

You might not recognize the pattern — but these are prime numbers (2, 3, 5, 7, 11, 13, 17) tapped out. But when I asked ChatGPT to interpret this, it confidently responded:

“This is a playful way of introducing the Fibonacci sequence!”

Then, it proceeded to explain how each line corresponded to Fibonacci numbers confidently like this.

“tap tap (1)”

Notice how there is 1 Tab, but it’s telling me there are 2 Tabs?

Why AI Hallucinates the Wrong Pattern

At first, this might seem like just an innocent mistake. But under the hood, this is a perfect illustration of why generative AI doesn’t actually think — it just predicts patterns based on statistical likelihood. Let’s break down why it failed so spectacularly:

1. Early Overlap Confuses It

The Fibonacci and prime sequences happen to have a partial overlap — 2, 3, and 5 exist in both. This creates an illusion of a connection, where the AI assumes it “knows” the pattern without checking wholistically; remember, foundation models only think in one direction; once started, it needs to continue on the same predictive path. Yes, I am simplifying a lot, but my main point is this: Once AI models commit a mistake, it won’t reconsider. Though the word “knows” is doing a lot of work here, as it does not ‘know’ anything. But you get what I mean… I hope.

P.S. This is not true for newer AIs using a ReAct framework like o-models, but that's because of framework changes, not model improvements, and they have other problems.

2. The “Music Bias” Effect

Taps are commonly associated with music, and the Fibonacci sequence is deeply linked to musical harmony. AI has likely seen thousands of references connecting Fibonacci numbers to rhythm, scales, and waveforms. Because of this, it is biased toward seeing Fibonacci whenever rhythms and sounds are involved — even when the pattern doesn’t match.

3. It’s Not Even Counting — It’s Guessing

Here’s the wildest part: the AI never actually counted the taps. If it had, it would have seen numbers like 7, 11, 13, and 17, which have nothing to do with Fibonacci. Instead, it just saw a sequence, immediately predicted the next most probable character, hallucinated, and gave an explanation.

Even worse, it went so far as to blatantly write out the wrong thing, such as:

This is a playful way of introducing the Fibonacci sequence! Each line corresponds to a number of “taps,” and the number of taps on each line follows the Fibonacci sequence:

tap tap (1)

tap tap tap (2)

tap tap tap tap tap (3)

tap tap tap tap tap tap tap (5)

tap tap tap tap tap tap tap tap tap tap tap (8)

tap tap tap tap tap tap tap tap tap tap tap tap tap (13)

tap tap tap tap tap tap tap tap tap tap tap tap tap tap tap tap tap (21)

In the Fibonacci sequence, each number is the sum of the two preceding ones (1, 2, 3, 5, 8, 13, 21…). Hence, the taps grow according to this pattern.

What This Says About AI Thinking (Or Lack Thereof)

This is more than just a funny failure. It exposes a fundamental limitation of AI models:

• →AI doesn’t “know” things. It just predicts the most likely response based on statistical patterns between concepts. In this case, it determined the strength of the relationship between taps and sequences and predicted that it must relate to the Fibonacci sequence.
• →AI doesn’t question its own logic. Once it commits to a pattern, it will rationalize its predictions to fit that pattern — even when the evidence contradicts it.
• →AI is easily misled by common associations. If a concept (like taps) is strongly linked to a second concept (sequences), then it is likely to predict inaccurate conclusions despite the facts — because the association is too strong between the concepts.

This is why AI can generate beautifully worded nonsense, confidently assert incorrect facts, and, in this case, fail at something a toddler could get right — counting taps.

What AI Is Good At (And Not So Good At)

AI is impressive, sure. But in my view, it’s not actually thinking — and any type of emergent reasoning is just hopeful thinking at this point. It’s just auto-completing the universe based on what it’s seen before.

This means:

• →AI is great at generating likely answers.
• →The more unique the question, the less likely it is to guess correctly, regardless of how obvious the answer should be.
• →AI is terrible at reasoning when something unexpected happens or when the pattern does not align with its training data.
• →AI will confidently defend mistakes rather than reassess them, reinforcing incorrect conclusions instead of questioning its own logic.

Which, frankly, makes it a lot like the overconfident guy at trivia night who insists that “Napoleon was tall because he was considered a GREAT man” and then refuses to look it up.

AI isn’t coming for your reasoning skills anytime soon — and as always, remember, nothing matters without humans!

#AIProductManagement #GenAI #AIThinkingFlaws #PatternRecognition