Confidence calibration

Rate how sure you are before checking, then compare confidence to accuracy so overconfidence stops lying to you.

When to use this: When checking yourself
Learning impact: Strong change
Learning evidence: Strong research tradition
Where you are with the topic: Checking readiness
How often to use it: Worth repeating often

When to use it

You feel sure in revision and are surprised by marks.
You want a number on “I think I know this” before feedback lands.
Some topics feel shaky and you need evidence, not vibes, for what to restudy.
You rush past items you “probably” know without ever testing that claim.

What to ask AI

Type something like: “I’ll answer each item and give a confidence from 1-5 before you mark. Then show a simple calibration: where confidence and accuracy disagree.”

Forbid marks until confidence is logged.

Ask for patterns (“high confidence wrongs”) more than item-by-item scolding.

How it helps

Calibration is how well your feeling of knowing matches reality.

Overconfidence hides weak topics; underconfidence wastes time restudying what is already solid.

A confidence number before feedback makes the mismatch visible.

The AI can tally patterns; you still must rate honestly before you peek.

How to run it

Attempt an item in writing.
Record confidence (for example 1-5) before any check.
Only then mark right/wrong.
After a set, list high-confidence errors and low-confidence corrects.
Restudy the high-confidence errors first, they are the silent killers.

Example exchange

You

Can you give me five short questions? I will answer each one and rate my confidence from 1 to 5 before you mark any of them.

Send all five answers with ratings. I will not mark until then.

You

1) Derivative of sin(3x) is cos(3x) , confidence 5. 2) NADPH carries reducing power to the Calvin cycle, confidence 2. 3) Precision means close to the true value, confidence 4. 4) Product rule needs both factors, confidence 5. 5) Water is split in the light reactions, confidence 3.

Calibration first: question 1 was high-confidence and wrong, you dropped the factor of 3. Prioritise that. Question 2 was low-confidence and correct, that caution was useful and less urgent to restudy. Question 3 was high-confidence and wrong, you used the definition of accuracy.

Copyable prompt

I will answer each item and rate confidence (1=guess, 5=certain) BEFORE you mark.
After the set, summarise calibration: high-confidence errors first, then
low-confidence corrects. Do not mark an item until its confidence is logged.

The Tell

You will know the method has flipped when you assign confidence after you see whether you were right, or you inflate ratings to look good.

Then the numbers measure theatre, not calibration.

Lock the rating in the same message as the answer, before any mark appears.

Principle evidence

Strength of the underlying learning idea, not a claim about AI products.

Metacognitive calibration, matching confidence to accuracy, predicts better study control. Rating confidence before feedback is a standard way to surface miscalibration.

AI delivery evidence

Whether an AI tutor delivers this method well is a separate question.

AI summaries of calibration are only as honest as your pre-feedback ratings. There is little evidence on AI-specific calibration tutoring beyond that bookkeeping help.