How to read a scientific paper on women in 7 easy steps
Posted by hannahflynn on May 2, 2009
The column inches given over to the latest research on why women should not drink during pregnancy is only the tip of the iceberg. In recent months I have seen the columns of both the mainstream media and scientific journals filled with diverse claims about the effect mothers have on their sons, daughters and society in general ranging from the tenuous to the ridiculous.
This has obviously led to a great deal of confusion for journalists, women and obviously mothers. The murky waters of peer-reviewed research can appear an impenetrable forest to the uninitiated, but by following these steps you should be able to make your mind up for yourselves.
1) Is it peer reviewed?
The recent scandal involving Merck’s payments to Elsevier to produce publications which looked like peer reviewed journals, but did in fact only publish research showing positive results for the effectiveness of the company’s product, has highlighted the importance of this. Peer review involves the paper being sent to working scientists in the relevant field who will assess the significance of the findings and decide whether the paper warrants publication. They will come to this decision based on whether the findings are original (they have not been published before), reproducible (the experiments be done again and achieve the same results) and significant (the finding is worthy of attention from other scientists in the field).
2) What questions are they asking?
This is a very significant question with regards to gender based research. Is the question the right question to be asking in the first place? For example, a lot of research has been done into the effect of drinking in pregnancy on a foetus. This is due to the existence of a syndrome known as Foetal Alcohol Syndrome which can result in low birth weight offspring in alcoholic mothers who abused alcohol during pregnancy. However, these papers rarely look at the effect of very low alcohol consumption in pregnant mothers; a far more significant question for the vast majority of mothers to be. The reason this has not been done is the effects of low alcohol consumption would be far harder to measure.
3) How big is the sample?
The MMR scare started after a paper with a sample size of 12was publishedin the Lancet. It is impossible to establish a significant result with this sample size, simply because the margin of error would be so large no trend could not be established, meaning any causal link implied would be meaningless.
A good sample size will depend on what is being looked at. Any trial regarding diet or lifestyle will require a large sample as the variables are so large. A trial looking at a rare disease may have to have a smaller sample size because of the availability of subjects. However, the bigger the sample the smaller the margin of error so the stronger the data.
4) What are the variables?
The simplest variable is gender. Are the groups, particularly in diet or lifestyle research, divided into gender? This is important as men and women have different metabolisms. It is impossible to eliminate all variables in any biological research as you are working with living organisms. However, it is possible to take all variables into account when planning the methodology and deciding conclusions. A well written experiment will explain how the groups in the trial are sampled, taking into account any variations. Variations which can not be controlled will be referred to in the conclusion as a possible source of error. Natural variation in a sample group need not result in inconclusive data, but if ignored then the whole experiment or trial is based on weak assumptions at best.
5) How quantifiable is what they are measuring?
Birth weight, genetic disease and death are all easily quantifiable results which can be used to produce meaning full conclusions. Often problems like ‘learning difficulties’, ‘problems bonding’ and ‘low satisfaction’ are not well defined in papers and can lead the conclusions which may mislead. A good paper should have quantified these in the methodology of the paper, for example learning difficulties could mean something quantifiable like a diagnosed dyslexic.
6) Are the results significant?
This is the area that is the most difficult to understand if you do not have a scientific or statistical background. I will try to explain the most commonly used statistic, the p value, to explain significance.
When looking at data from a piece of biological research it is important to calculate what the probability is that the results you have may have happened by chance. If your results are not caused by chance then you have a significant result. This probability is currently set at 5 percent, or p = 0.05. If there is less than a 5 percent chance your data was caused by chance you can claim your date is significant and conclusions can be drawn that there is a causal link. (A good explanation for journalists can be found here.)
7) Does more research need to be done?
If however p = 0.07 there is a 7 percent chance your results are caused by chance. This is not a high enough level of significance to base conclusions on but there is probably enough evidence to base further research in the area on.
Similarly if the paper has a sample size which does not seem big enough, this does not mean the findings are completely null and void, rather more research needs to be done before conclusions can be drawn.
For a more in depth analysis of statistics and other Bad Science, read Ben Goldacre’s blog and column in The Guardian. The comments left are often ripe ground for debate on how to interpret scientific papers.
For an irreverent but clear explanation of bad statistics turn to Feedback in New Scientist.