ENGL 128 Research Writing: Science, Engineering and Business

Evaluating a Scientific Research Article

Pyramid of Evidence 

(click on image to enlarge)

How Strong is the Evidence?

Break it Down: Evaluate a Source from Start to Finish

Now that you understand the structure of an academic research article, use the elements of an article to help you evaluate its content.

Title:
  • Does the title describe the study clearly?
  • Do the keywords express the key concepts of the study?
  • Is the title clear and concise?
  • Are the types of participants mentioned?
  • Is the nature of the research listed?
  • Does this title set you up to understand what you are about to read? 
Abstract:
  • Is the purpose of the research clearly stated?
  • Is the methodology explained in sufficient detail?
  • Is the sample mentioned?
  • Is there a brief summary of the results?
  • Does the abstract give you an overview of the central argument of the article?
  • Does the abstract provide you with the information necessary to determine if you want to read the entire article? 
Introduction:
  • Does the author clearly identify the research problem or question?
  • Is the significance of the problem discussed and supported with statistics?
  • Do the authors use theory to provide a framework to support the study and guide the analysis?
  • Does the review of the literature seem complete, current, and appropriate?
  • Is the research cited current (within 5 years of the article publication date?)
  • Are there few to no direct quotes?
  • Is the purpose of the study clearly explained?
  • Is there a clearly stated research question or hypothesis?
  • Does the review lead logically to a hypothesis or research question?
  • Are conceptual definitions for key variables given?
  • Are all the facts cited?
  • Does the introduction leave you with an understanding of the problem?
Methods:
  • Is the sample clearly defined by size, relevant characteristics, etc?
  • Are the procedures for the sample selection explained?
  • Is the exact sample size mentioned?
  • Has informed consent been secured?
  • Does the sample type and size make sense for the purpose and scope of the study?
  • If the research is experimental, was an appropriate control group used?
  • Are the materials used to conduct the study or in data collection clearly described?
  • Are the scientific procedures thoroughly described and delineated chronologically?
  • Could someone replicate the study from the information provided?
  • Is the data analysis well-described and appropriate?
  • Do the authors include a sample of their study items or questions?
  • Does the research describe how the instrument is used (settings, protocols, etc.)?
  • Is there more than one instrument for gathering data?
  • Do the authors provide the sources of existing instruments?
  • Are the reliability and validity of the instrument/s discussed?
  • Are any limitations of the instrument/s discussed?
  • Note: Some research papers do not include a description of the methods, but instead include a reference to a source where such a description can be found. This is an acceptable practice and does not affect the quality of the article.
Results:
  • Was information easily understood or were sufficient explanations given?
  • Are results statistically significant?
  • Do results answer the research question or prove/disprove hypothesis?
  • Are the important results connected directly to the hypothesis? 
  • Do results stick to the findings and refrain from theorizing why they appear as they do?
  • Is the results section clearly written and well organized? 
Discussion:
  • If there was a hypothesis, was it accepted or rejected? 
  • Does discussion tie back to Literature Review?
  • Are the findings discussed in terms of the conceptual framework, research problem, 
  • and/or hypothesis? 
  • Are limitations discussed?
  • Are suggestions for further research appropriate and are they clearly stated? 
  • If predicted results were not found, is an explanation/s offered?
  • Are specific implications for practice, policy, and future research made?
Conclusion (not always present):
  • Are the results briefly restated? 
  • Do the conclusions follow from the results?
References:
  • Is the reference list sufficiently current? 
  • Do the works cited reflect the breadth of existing literature on the topic of the study? For example, does the works cited list compare favorably with the works cited for articles written on similar topics? 
  • Are citations used appropriately in the text? 
General Impressions:
  • Is the article well written and organized? 
  • Does the study address an important problem? 
  • What are the most important things you learned from this study? 
  • What do you see as the most compelling strengths of the study? 
  • How might this study be improved?

Attribution: Professor Joy Crawford at Green River College

The information above was adapted from the following sources:

  • King's College D. Leonard Corgan Library
  • Hudson-Barr, D. (2004). “Scientific inquiry: How to read a research article.” JSPN 9 (2), 70-72. 
  • Lunsford, T. R. and Lunsford, B. R. (1996). “Research forum: How to critically read a journal 

     research article.” JPO 8 (1), 24-31. 

  • Northern Essex Community College Libraries. (2003). “Elements of a research article.” 

     Retrieved December 20, 2004, from www.necc.mass.edu/departments/library/pdf/

     researcharticle.pdf

  • Rumrill, P., Fitzgerald, S., and Ware, M. (2000). “Guidelines for evaluating research articles.” 

     Work 14 (3), 257-61. 

  • Pryczak, F. (2005) Evaluating Research in Academic Journals, Pryczak Publishing

The 4 Cs Method

Is the Research: Credible, Contributory, Communicable, and Conforming

(Click on image to enlarge)

4 Cs of Assessing Research Quality - Text below image fully describes image

The 4Cs Method ask: Is the research credible, contributory, Communicable, and Conforming?
Credible: 

Is the research Coherent, Consistent, Rigorous and Transparent?

  • Rigorous: Research that is Contextual, Internally Valid and Reliable
    • Contextual: Existing Knowledge that is relevant for the Context is used, and is presented according to Rules for Description
    • Internally Valid: A correct Scientific Method (incl. research design) is used in relation to the Question at Hand and Context, and New Knowledge is Provable
    • Reliable: The chosen Scientific Method is appropriate for the present Question at Hand and Context, and is documented in a Described Procedure that others could use to reach a similar result in the same Context
  • Consistent: New Knowledge is logically linked to Existing Knowledge and is in accordance with the Scientific Method and Question at Hand
  • Coherent: Adequate consideration is given to Existing Knowledge in the chosen Context
  • Transparent: Relevant New Knowledge in the reporting of research results is included and the process is described in relation to the Question at Hand, Scientific Method and Existing Knowledge
Contributory:

Is the research Original, Relevant and Generalizable?

  • Original Research that has an Original Idea, uses an Original Procedure and produces an Original Result
    • Original idea: The Question at Hand has not been asked before in the current Context or is interpreted in a novel way
    • Original procedure: Described Procedure is original in relation to the Question at Hand
    • Original result: New Knowledge is Provable in relation to Existing Knowledge
  • Relevant Research that has a Relevant Research Idea, Applicable Result and Current Idea 
    • Relevant research idea: Question at Hand is relevant for the current Target Group
    • Applicable result New knowledge is Beneficial for the current Target Group
    • Current idea The Question at Hand is in accordance with the current Context
  • Generalizable New Knowledge is practically or theoretically useful in Contexts other than the one studied
Communicable: 

Is the research Consumable, Accessible and Searchable?

  • Consumable Research that is Structured, Understandable and Readable
    • Structured The Research documentation follows the Rules for Description
    • Understandable The language in the Research documentation is understandable for the Target Group
    • Readable Correct language in the Research documentation for the Target Group
  • Accessible New Knowledge is easily available to the Target Group
  • Searchable The documented New Knowledge is structured according to the Rules for Description and easily found by the Target Group
Conforming:

Is the research Research Regulatory Aligned, Ethical and Sustainable?

  • Aligned with regulations The Research complies with currently applicable legal aspects of the System of Rules
  • Ethical The Research is Morally Justifiable, Open and supports Equal Opportunities
    • Morally justifiable The Research complies with currently applicable ethical standards as described in the System of Rules
    • Open The Research demonstrates Transparency with currently applicable ethical standards as described in the System of Rules
    • Equal opportunities The Research is consistent with equal treatment according to the System of Rules
  • Sustainable The Research complies with sustainable development aspects as described in the System of Rule

Sources: Credit for the 4 Cs goes to P. Mårtensson et al. / Evaluating research: A multidisciplinary approach to assessing research practice and quality / Research Policy 45 (2016) 593–603

Not All Scientific Studies are Created Equal