Science is often misrepresented in the media, so how can you know if there is validity to the science they are reporting? Well, there are several quick strategies you can use and they don’t require you to be a scientist! These tips are certainly not meant to be exhaustive, but if everyone would follow them, they would be much more informed and much better shielded against bullshit trying to pass as fact.
1. Find out if the study in question is from a reputable journal
Hopefully, the website reporting the study actually includes a link to the actual citation. If they just say “a study” without going into details, that’s an automatic red flag and there isn’t much you can check. If they do provide the details, you can search the journal article in Google Scholar and see what journal it is published in.
Once you get the name of the journal, you can search this website to determine its impact factor. Impact factor isn’t a perfect measurement, but it essentially looks at how often the articles in the journal are cited by other articles. So sometimes you can still have a high quality journal that is not cited much because it’s highly specialized. However, generally you would want to see that a journal had an impact factor of at least 2, but anywhere from 5-10 is pretty good.
If you can’t search the journal or have trouble finding its website, that’s a bad sign. Of course bad articles can slip into high impact journals, but again, this is a quick way to see if the article in question is at least coming from a good source.
2. Find how many times the article is cited by other articles
If an article is cited often, then that means other scientists are interested in its findings. Now, of course some scientists may disagree with the findings and they are citing it contest parts of it. But if they are citing it, that means it’s good enough that it’s worth mentioning in the web of scientific knowledge. Again, Google Scholar will make this very easy as you just search the name of the article and you see how many citations it has.
Newer articles will not have a chance to be cited much and just because a study isn’t cited a lot, it doesn’t mean it’s bad. However, it’s another quick way to see the influence this study has within it’s scientific community. If a study has been out a few years and hasn’t been cited at all, that’s not good. You should at least see a few citations from other articles. Finally, you can click on those articles and see if those citations are in good journals! While Google Scholar is great, it’s not perfect and sometimes it will count citations from non-peer reviewed journals.
This is an important one. A single study can reveal an exciting finding, but we should be skeptical of it until other studies have found similar results. It could be flawed for a variety of reasons and the easiest way to check for its accuracy is to see if other studies could replicate it.
Again, newer studies might not have a chance to be replicated, but you should be able to find articles in Google Scholar that have found similar things. Sometimes scientists make this very easy for you when they do meta-analyses, which brings me to my next point
4. Find a meta-analysis on the subject (if one exists)
A meta-analysis is simply a review of many similar studies to determine if a consensus has been found among different scientists. This is one of the easiest and most explicit ways to visualize “scientific consensus.” Large issues like climate change will certainly have some meta-analyses, but more specific or newer issues may not. Luckily, you often don’t have to wait until a meta-analysis is out to get a good idea if that study is full of bunk or not. You can also sometimes find someone well-equipped to explain the subject in question.
5. Read what other experts say about the findings
This is your best bet for determining if that study is valid if there are not already studies that support it. Usually when a new article comes out, there are going to be several other articles that report on it. I would highly suggest finding an interview with someone who did not write the article, but has relevant expertise and is talking about it.
Of course people with credentials can certainly be wrong or biased, but they have been devoting their lives to studying this area and should have a good grasp on what is valid or not. In Google Scholar, not only can you check articles, you can check people too! (For example my Google Scholar isn’t huge as a grad student, but you can see I at least have some academic publications). If an article says they interviewed some scientist about a study, put that scientist’s name into Google Scholar. If they have zero publications or have only published in bad journals, that’s a red flag. Also, make sure they actually experts in an area relevant to the study in question. For example, a neurologist may not be well-versed in neuroimaging studies, but a cognitive neuroscientist would be. Googling their credentials and fields of expertise will help too if you are unsure. Too often people pass as experts in the media, but those people are not even respected among their peers!
Again, this list is not meant to be exhaustive by any means, it’s intended to give the non-scientist a set of tips they can use to improve their understanding if a random scientific article is full or shit or not. Yes, it takes a little bit of extra research, but you can greatly protect yourself from bullshit with just a few more minutes of Googling. That seems like an excellent trade-off to me!
[Featured image from Pixabay]