Better decisions start with better questions.
You do not need to become a scientist to judge a health claim. You need a small number of questions that catch most of the ways coverage goes wrong, and the willingness to ask them before you change anything.
Correlation is not causation, and you already know that
The useful version of this rule is more specific. When a study reports that people who do X are healthier, ask what else is true about people who do X. People who take vitamins also tend to exercise, sleep more, drink less, and see a doctor. A study that finds vitamin takers are healthier may have found nothing about vitamins at all.
Researchers try to adjust for this, and the adjustment is imperfect by definition — you can only adjust for what you measured. This is why a randomized trial, where a coin decides who gets the intervention, carries so much more weight.
Relative risk sounds bigger than it is
"Doubles your risk" is the most reliably misleading phrase in health coverage, because it tells you nothing about what the risk was to begin with. Doubling a very small risk produces a slightly less small risk.
Always look for the absolute numbers: how many people in each group actually had the outcome. If a report gives you only percentages of change and never the underlying counts, it has chosen the more dramatic framing, and you should treat the claim accordingly.
Ask what was actually measured
Many studies do not measure the thing you care about. They measure something faster and cheaper that is assumed to predict it — a blood marker instead of a heart attack, a lab measure instead of a fall, a questionnaire score instead of a behaviour. These are surrogate outcomes.
Surrogates are legitimate research tools and they are frequently wrong. Interventions that improve a marker sometimes fail to improve the outcome the marker was supposed to represent. When a headline says something "improves heart health", find out whether anything happened to anyone's heart.
Check who was studied
- Species. A great deal of longevity coverage rests on work in mice, worms, or cells.
- Age. A finding in twenty-two-year-old athletes may not transfer to a sixty-year-old beginner, and often does not.
- Sex. Many exercise and nutrition studies have been conducted predominantly in men.
- Health status. Results in people with a condition frequently do not apply to people without it, and the reverse is also true.
- Number of people. A dozen participants can produce a striking result that disappears in a larger group.
One study is a data point, not an answer
Individual studies disagree constantly. That is how the process works — a field converges over many studies, not within one. A single trial that contradicts a large existing body of evidence is interesting, not decisive.
This is also why systematic reviews and meta-analyses sit at the top of most evidence hierarchies. They ask what all of the available studies say together, which is a different and better question than what the newest one says.
Follow the incentive, carefully
Industry funding does not make a study wrong, and academic funding does not make one right. But funding shapes which questions get asked, which results get published, and how a finding is framed in the press release that most coverage is actually based on.
Look for a declared conflict of interest, and ask whether an independent group has reproduced the result. Replication by someone with no stake in the outcome is the strongest signal available to a non-specialist.
Six questions that catch most of it
- Was this done in humans, and in humans like me?
- Was anyone randomly assigned, or did people choose?
- What is the absolute risk or benefit, in plain numbers?
- Was the outcome the thing I care about, or a marker standing in for it?
- How many people, and for how long?
- Has anyone independent found the same thing?
If a piece of coverage will not answer these, the answer is usually unflattering.
What to do with all this
The goal is not scepticism for its own sake. Constant doubt is as unhelpful as credulity, and it tends to end in doing nothing while feeling informed.
The goal is proportion: act firmly on things that are well supported, hold the promising ones loosely, and leave the rest alone until they have earned your attention. That is exactly what the strength ratings in the evidence library are for.