Methods · Creation, transfer & long horizon · Short guide

Methodology critique

Have your proposed method attacked before you run it, so you fix design flaws on paper instead of after weeks of sunk cost.

When to use this: When checking yourself
Learning impact: Reliable gain
Learning evidence: Credible with caveats
Where you are with the topic: Checking readiness
How often to use it: Worth repeating often

When to use it

You are about to collect data, run a build, or start a study and the method still feels hand-wavy.
A methods section looks tidy but you cannot name the top threats to validity.
You would rather find design flaws on paper than after weeks of work.

What to ask AI

Type something like: “Proposed method: Question [ ]. Method [ ]. Attack this before I run it: can it answer the question, top validity threats, measurement problems, and what to change.”

Ask for attacks, not reassurance.

Keep the hard part on your side: you revise the design; the AI does not run a fake study for you.

How it helps

Methodology critique gets your proposed method attacked before you run it, so you fix design flaws on paper.

A beautiful methods paragraph with no list of threats is theatre.

The learning is learning to see threats: sampling, measurement, confounding, feasibility, and mismatch between question and method.

How to run it

Write the question and the method in plain language (not only textbook labels).
Ask for a structured attack: fit to question, validity threats, measurement, ethics/feasibility.
For each serious threat, either redesign or accept it explicitly as a limitation.
Rewrite the method yourself after the attack.
Re-run the critique on the revised plan once.

Example exchange

You

Research question: does the new lab worksheet improve exam scores? Method: compare this year's mean to last year's mean. Attack the design before I run it.

First: can comparing year means answer whether the worksheet caused improvement? List the confounds you already see.

You

Different cohort ability, a different exam paper, and leftover pandemic effects on attendance.

Those are fatal for causal language. Propose one design change that addresses the biggest confound. You write it; I will critique the revision.

Copyable prompt

Proposed study/project method:
Question: [ ]
Method: [ ]
Attack this before I run it. Structure: (1) can this method answer the question at all?
(2) top validity threats, (3) measurement/operationalisation problems, (4) feasibility/ethics,
(5) the single highest-leverage redesign. Do not soften. I will revise.

The Tell

A beautiful methods paragraph and no list of threats you can name means you sought approval, not critique.

If the AI only compliments clarity, demand the attack, reassurance is not this method.

Principle evidence

Strength of the underlying learning idea, not a claim about AI products.

The underlying learning idea is rated moderate. Critiquing methods before execution is standard research training and design review. Anticipating validity threats improves designs; this is professional practice backed by methodology teaching more than a single trial on 'pre-critique.'

AI delivery evidence

Whether an AI tutor delivers this method well is a separate question.

AI can list textbook threats fluently and still miss field-specific ones. Treat AI critique as a first pass; verify with a human supervisor or methods text for high-stakes work.