Psychology & Behavioral Sciences: Methodology critique

What you'll be able to do: Have your own study design attacked before you run it, confounds, demand characteristics, ecological validity, sample bias, which is direct rehearsal for the exact skill coursework and exams actually mark.

The subject examines the design, not the finding

State the subject's asymmetry plainly: you spend most of your study time learning findings, and most of your marks come from evaluating methods. Coursework asks you to design a study and defend it. Exams ask you to critique a described study, name the confound, propose an alternative explanation, suggest an improvement. Neither of these is "what did the researchers find," and neither rewards fluent recall of a landmark study's headline result on its own.

Practising the critique skill on other people's finished, published studies is useful but has a ceiling: the studies you're shown in class were mostly chosen because they're good teaching examples, with a clean, nameable flaw. Real practice (the kind that transfers to an exam's unfamiliar scenario) is having your own design attacked, on a study you haven't already been told the answer to.

What to attack, systematically

Sample. Who was recruited, how, and who does that leave out? A convenience sample of psychology undergraduates answering a survey for course credit is the field's most common sampling method and its most common limitation, see the primary article's WEIRD problem, applied to your own design specifically.

Confounds. What varies alongside your independent variable that could produce the result on its own? If you're comparing two groups that weren't randomly assigned, what pre-existing difference between them might actually be doing the work?

Demand characteristics. Would a participant likely guess what you're testing for, and would that guess change how they behave? A study design that telegraphs its hypothesis risks measuring "what participants think we want" rather than the thing itself.

Ecological validity. Does the task or setting resemble the real-world behaviour you're claiming to say something about, or is it a narrow lab proxy (article 05)? A finding can be internally airtight and still not generalise past the room it was collected in.

Measurement. Is the operationalisation defensible (article 05), and is the measure reliable, would it give a similar result if repeated?

Ethics. Informed consent, deception, debriefing, right to withdraw, protection from harm, not a box-ticking afterthought here, since several of the field's most famous studies (Milgram, Zimbardo) are taught partly as ethics case studies, and exam boards examine this directly.

The drill

Here's my proposed study: [aim, design, participants, procedure, measures].
Attack it like a hostile reviewer. What's the single biggest confound? What would a participant likely guess the hypothesis is, and how might that change their behaviour? Does the sample let me generalise the way I'm claiming to?
Is the measure I'm using a reasonable operationalisation of the construct, or a narrow proxy? Don't soften this. I want the version that would come back from a real reviewer.

Ask for the attack before you run anything. A confound spotted at the design stage is a redesign; the same confound spotted after data collection is a limitations paragraph, which is worth far fewer marks and far less learning.

Escalating it

Once you have the critique, don't stop at the list, fix it and have the fix attacked too:

Here's how I'd address that confound: [your fix]. Does it actually solve the problem, or does it introduce a new one? What would a reviewer say about the fix itself?

This mirrors what a real methods section revision looks like, and it's the part that separates "I can name a flaw" from "I can design around one," which is the harder and more examined skill.

Using it on studies you're given, not just your own

Here's a described study from my course: [paste]. Before I read any commentary on it, give me the critique a hostile reviewer would give, sample, confounds, demand characteristics, ecological validity, measurement, ethics,
Then tell me which of these the original researchers themselves addressed and which they didn't.

The final question matters: some "flaws" a naive critique surfaces were already anticipated and controlled for in the original design, and knowing the difference between a real gap and one the researchers closed is itself an evaluation skill worth having.

Pitfalls

1. Asking for the critique after collecting data. The point is catching it while it's still cheap to fix.
2. Accepting the first flaw named and stopping. A real study usually has several defensible criticisms at different levels, sample, design, measurement, ethics, and marking schemes reward range.
3. Treating every critique as fatal. Some flaws limit a conclusion without destroying it; learn to say which.
4. Skipping the "did the original address this" check when critiquing a published study, which risks reinventing a criticism the researchers already handled.
5. The tell: your own study designs never come back with a confound you hadn't already spotted. You're not being attacked hard enough, or you're only proposing designs you already know are safe.

Try this today

Sketch a one-paragraph study design on a topic you're currently studying, aim, participants, procedure, measure. Ask for a hostile-reviewer critique: biggest confound, demand characteristics, sample limits, measurement validity, ethics.

Then propose one fix and ask whether it actually solves the problem or just moves it.