What should you be looking for when reading quantitative studies to judge their worth in the literature?
All of the issues we have discussed in previous weeks are still in play as to judging the worth of a research study, qualitative or quantitative in nature. Specifically, relatively current research studies (+/- 10 years) published by reputable authors in reputable, peer-reviewed journals. In fact, chapter 9 of our text provides reference to many questions to ask with regard to critically evaluating every aspect of quantitative studies, including title, abstract, introduction, purpose, references, subjects, research plan/design, procedures, method, figures, tables, discussion and conclusions and/or results (Locke, Silverman, & Spirduso, 2010).
What might you find while reading a quantitative study that would raise a “red flag” about the quality of the methods used or the validity of the study?
There are certain “red flags” to be aware of when reading quantitative studies. For instance, quantitative studies are essentially only valuable if the results can be replicated in future studies and/or expanded upon. Therefore, it is critical that quantitative studies include a meticulous procedural and statistical accounting from which future researchers could plausibly replicate the findings. Minus this information and the study becomes much less useful, assuming the researchers were publishing a significant finding. Missing steps from the procedures, missing data from the results and/or inaccurate reporting of statistical formulations used are all warning signs.
Another red flag with quantitative studies has to do with the number and characteristics of the subjects involved. Unfortunately, as our resources point out, quantitative studies typically adhere to a direct relationship between the number of subjects and generalizability. Hence, the more subjects participating, the more likely the results will be representative of the general population. In fact, our text references at “30-subject rule,” which is not always feasible dependent upon the type of research study involved (Locke et al., 2010, p. 162). The characteristics of the subjects are just as important as the number. For instance, the study cannot generalize to all children if the subjects were only male. Similarly, if the study only included adolescents, it would be nonsensical to allege generalizability to elementary students and/or senior citizens. There are possible huge generational differences between these age brackets which could engender confounding variables into the study and the results attained therein.
How can you screen for bias in a research study?
One of the unfortunate aspects of quantitative research is walking the line between obtaining participants for the study, while minimizing bias. For instance, as I previously mentioned, the more participants in the study the better; however, where to acquire these participants that will not engender any bias onto the study? Many studies acquire subjects from schools, pay volunteers, or provide other types of incentives. This can create bias in discerning the differences between those that chose to participate and those that did not. As an undergraduate we were required to participate in a certain number of research studies; however, what was to say that as a psychology major I did bring to the study a different bias than perhaps mathematics major?
Another consideration for bias involves the funding for the project, whether it is educational, governmental or non-profit. For instance, in doing research for the course project I came across a website that alleges forerunner research regarding usage of video gaming (neurofeedback) with attention deficit hyperactivity disorder (ADHD) children in an effort to minimize and/or cure the ADHD (http://www.smartbraintech.com). When I went to review the research, there were no actual working links and/or the links led cycled back to a blog discussing research, but with no actual verifiable citations. This site is interested in selling its neurofeedback technology to desperate parents searching for a “cure” for their child. Any research from this site would have to be viewed with a cautionary flag, if not dismissed outright, because of its affiliation with a profit product. Similar issues arise with non-profit organizations and/or even educational institutions receiving government funding for one singular type of research. Ongoing funding is often contingent on research results showing some type of positive momentum. There is likely less bias with multiple reviewers, statisticians and possibly, independent audits of results.
What ethical considerations are specifically relevant with quantitative research?
Once again quantitative research suffers from the relationship between participants and generalizability. For instance, my research project deals with children with ADHD. Therefore, the research studies I have been reviewing need to have some generalizability to children with ADHD. Any time human participants are used in a study, there are ethical considerations for deception, treatment versus non-treatment, as well as identification. When there are minors involved there are additional considerations in that the researcher needs minor participation, in addition to parental consent and/or participation. Each additional constraint further minimizes the possible participant pool.
It is easy to imagine the many types of research studies hindered by these types of dilemmas. For instance, research regarding additional social support versus non-support for children with leukemia. The researcher has to find participants who are children with leukemia. Then, they have to configure the experimental design to either include a social support group and a non-support group or alternatively, identify support differences in the overall sample in an attempt to find some type of correlation. In either instance, the research is noble in cause, but difficult to accomplish.