You ask a large language model (LLM) a question about climate policy. It replies instantly, with polished prose, bullet points, and a tone of quiet authority. You feel reassured. But what if the answer contains a subtle error – or worse, a confident fabrication?
This scenario is becoming increasingly familiar. As LLMs like ChatGPT, Claude, and Gemini weave themselves into education, healthcare, and everyday decision-making, a troubling pattern has emerged: these systems often express high confidence even when they are wrong. My research in cognitive science suggests this isn’t just a bug. It’s a feature inherited from us.
The human bias behind the machine
Overconfidence – the gap between what we think we know and what we actually know – is one of the most robust findings in psychology. From doctors diagnosing patients to investors picking stocks, people consistently overestimate the accuracy of their judgments. Crucially, overconfidence is a metacognitive bias: it distorts how we monitor and evaluate our own thinking. We don’t usually attribute metacognition to machines. Yet LLMs are trained on human language, built by human engineers, and refined through human feedback. It should be no surprise that they mirror our cognitive blind spots. Recent studies confirm this. When researchers asked five leading LLMs to answer 10,000 factual questions and then rate their confidence, the models systematically overestimated their accuracy. GPT-3.5, for instance, was overconfident by roughly 60 percentage points on average – stating 90% confidence when correct only 30% of the time. Even newer models show significant calibration gaps.
How overconfidence gets built into AI
Three interconnected pathways explain how human overconfidence flows into LLMs.
1. Training data reflects human certainty bias
LLMs learn from vast corpora of text written by people. But human communication favours confident assertions. News headlines, expert commentary, and even academic abstracts often present conclusions with more certainty than the underlying evidence warrants. When models ingest this data, they learn that authoritative language correlates with quality – regardless of factual accuracy.
2. Model architecture rewards definitive answers
Most LLMs are designed to predict the next most probable token, not to quantify uncertainty. When prompted, they generate a single, fluent response rather than a probability distribution over possible answers. This structural preference for decisiveness mirrors our own cognitive tendency to suppress ambiguity – and can produce the confident “hallucinations” users increasingly recognise.
3. Human feedback amplifies confidence
Reinforcement Learning from Human Feedback (RLHF) fine-tunes models using human preferences. But research shows that people tend to rate confident answers as more helpful, even when they’re incorrect. This creates a feedback loop: models learn that sounding certain earns rewards, reinforcing overconfident output. As one study put it, we’re training AI to be “confidently wrong”.
The double jeopardy: when users trust too much
The problem compounds when humans interact with overconfident AI. Cognitive psychology tells us we use confidence as a heuristic for credibility. When an LLM speaks with authority, we’re less likely to fact-check, question assumptions, or seek alternative sources. This “automation bias” is especially risky in high-stakes domains. A medical student might accept an AI’s confident but outdated treatment recommendation. A journalist might publish a plausible but fabricated quote. In each case, the user’s trust is misplaced not because the AI is malicious, but because it has learned to perform certainty – a performance we are evolutionarily primed to believe.
Calibrating confidence: what developers can do
The good news is that researchers are developing practical strategies to improve AI calibration – that is, aligning a model’s expressed confidence with its actual accuracy.
- Prompting for uncertainty: Techniques like “chain-of-thought” reasoning or asking models to “think step by step” can surface gaps in knowledge. Some systems now support explicit uncertainty tokens (e.g., “I’m about 70% sure…”).
- Ensemble methods: Running multiple models or multiple sampling passes and comparing outputs can flag low-consensus answers as potentially unreliable.
- Adversarial training: Exposing models to examples where confident answers are wrong helps them learn to temper certainty.
- Better evaluation metrics: Moving beyond accuracy alone to measure calibration (e.g., using reliability diagrams) ensures models are rewarded for honest uncertainty.
Crucially, these technical fixes must be paired with transparency. Users deserve to know when an AI is guessing.
What users can do today
While developers work on systemic solutions, individuals can adopt simple habits to mitigate overconfidence risks:
- Ask for sources: Prompt the AI to cite specific references, then verify them independently.
- Request confidence ratings: Ask “How certain are you about this answer?” or “What would make you change your mind?”
- Cross-check with multiple models: Disagreements between LLMs can signal uncertainty worth investigating.
- Use AI as a brainstorming partner, not an oracle: Treat outputs as hypotheses to test, not conclusions to accept.
- Stay alert to “too perfect” answers: Fluency and structure are not guarantees of truth.
Looking ahead: avoiding a confidence cascade
Perhaps the most urgent concern is what happens when overconfident AI outputs become training data for future models. Without careful curation, we risk a “confidence cascade”: errors amplified across generations of AI, each more certain and more detached from ground truth. This isn’t inevitable. But it requires acknowledging that AI doesn’t just process information – it inherits the cognitive patterns of its creators and users. Overconfidence is not merely a technical calibration problem; it’s a human one. As we integrate AI into society, we must design systems that reflect not just our intelligence, but our intellectual humility. The most trustworthy AI won’t be the one that always sounds sure. It will be the one that knows – and tells us – when it isn’t.