Evaluating scientific claims on the Internet 

The internet is littered with websites claiming to have found the cures to many common ailments.But how many of them are true? And how do you know? With headlines being sensationalized and conspiracy theories abound, it can be difficult to know fact from fiction.

Here is a 10 step guide to understanding the likelihood that the science behind a headline is real.

  1. Check the source: Most importantly, if a website makes a claim about something, they better have a source to back up that claim. If someone says there is a cure for cancer but have no source to provide evidence for this claim, be skeptical. No source, no proof.
  2. Check your emotions: Many times our emotions get the best of us. If you are doing research into a treatment for Multiple Sclerosis and have a family member with the disease your emotions can often cloud your ability to think logically. Try to judge the strength of the information from an unbiased point of view. An article might not give you the answer you are hoping for but that doesn’t mean its wrong. Alternatively, an article might validate your beliefs and make you blind to the short comings of the research.
  3. Temper your enthusiasm: Newspapers, magazines, and websites tend to inflate the significance of results in an attempt to attract readers. Often you hear about research done in cells that will lead to a cure for some terrible disease in the next couple years. That just isn’t going to happen. Most discoveries are 5-10 years away from potentially being used in humans and most discoveries never become effective. Remember, without the readers, most websites and news outlets would not exist so it is in their best interest to attract you with exciting, world changing titles.
  4. Look for bias: If a research project is funded by the tobacco industry, would you be more or less inclined to believe the results? What if the person writing an article about the benefits of segregated schools is a member of a racist political group? Bias can cause a person to look only for facts that support their claim and ignore those that refute it. Did a scientist look at the effect of antiodixants on aging only look at young people in their study and try to use the results to make a statement on the older population?
  5. Samples size: Flip a coin 4 times and you would expect to get 2 heads and 2 tails. In reality you may get 4 heads and 0 tails. Does this mean the coin is weighted? Does it mean heads is more likely to come up? No, it means you didn’t do enough trials to minimize the error you introduce by flipping it slightly different each time, or a breeze changing the way it flips. Flip that coin 100 times and you will likely get 50 heads and 50 tails. Same thing happens in research. Look at the rates of HIV in a sample of 10 people and find no one with HIV making you think HIV is gone. Look at 1000 people and you find 8 people with it. Sample size is important, the more samples the better.
  6. Don’t assume: This can be one of the biggest problems with reporting science. A result in cells does not always apply to humans. Neither does a study in mice. Also, a result in a specific group of people doesn’t necessarily extrapolate to another group of people. For example, a trial of a new drug shows that it kills cancer cells in patients with small-cell lung carcinoma (a specific type of lung cancer). This does not mean the drug works in other types of cancer nor does it mean it works in all types of lung cancer. Or even in men and women equally. Or across of age groups. The results are true for the specific population of people studied. Find out what that population is and you’ll know how the results apply to the real world.
  7. Controls: Controls are sometimes the most important part about science research. Was there a placebo group? Did they know they were given the placebo? Placebo is a very powerful thing and can greatly impact the results of a study. If the drug came in a yellow pill was the colour important in the effect? If a drug was dissolved in alcohol (because it wasn’t soluble in water) and given to mice, was another group of mice given a similar amount of alcohol to make sure the results of the study are due to the drug and not the alcohol? Controls help us minimize variability in the results, without them science gets messy.
  8. Effect size: Is the difference small? How small is too small? If a treatment helps someone with COPD have 10% fewer trips to the emergency room is that clinically relevant? If that translates to 3 or 4 fewer trips a year then yes maybe it is important. 1 fewer trip, maybe not. Just because something is significant does not make it clinical relevant or important.
  9. Replication: Science is the foundation of knowledge. Nothing is known for certain unless it can be replicated by a number of other investigators. Look to see if this is a one off study or if others have shown the same (or different) results. Just because something has only been shown once doesn’t make it any less likely to be true. It just means you need to be careful to take it as truth without seeking out other evidence. A good example is the debate on vaccine safety. Many researchers over a number of years in different labs around the world have shown them to be safe. This information can be trusted. On the other side, only one flawed study has shown them to be potentially harmful and so this data is unlikely to be true. Replication is important.
  10. Beware regression to the mean: Extreme values have a way of normalizing after a while. This is called regression towards the mean. Think of it this way, you give 100 people a test and take the people who scored in the top 5% and give them the test again. Those 5% are likely to score worse on the new test than they did on the first test. Additionally, the people who score the worst on a test are likely to do better on the next test. This applies to medicine because with alternative therapies people tend to take them when they feel the worst. They think they help because they feel better after taking their alternative treatment but in reality they feel better because statistically speaking they were more likely to feel a little better the next day and less likely to feel worse.

Finally, remember that cells and mice are not humans and so any experiments that show promising effects in those two models need to be validated in human studies first. Cancer has been cured millions of times in a cell culture dish but it doesn’t always translate to the whole human (for many reasons).

This cartoon explains it beautifully.

Header photo credit: Frits Ahlefeldt-Laurvig 


One thought on “Evaluating scientific claims on the Internet 

  1. Pingback: The miracle cure for cancer is not being hidden from you | Science Translation

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s