1.4 Evaluating Research Sources
Suzan Last; McDonald Kyte; and Jennifer Shrubsole
The importance of critically evaluating your sources for authority, relevance, timeliness, and credibility cannot be overstated. Anyone can put anything on the internet; and people with strong web and document design skills can make this information look very professional and credible—even if it isn’t. Since much research is currently done online, and many sources are available electronically, developing your critical evaluation skills is crucial to finding valid, credible evidence to support and develop your ideas. In fact, this has become such a challenging issue that there are sites like the List of Predatory Journals that subvert the peer review process and simply publish for profit.
Mark Twain, supposedly quoting British Prime Minister Benjamin Disraeli, famously said, “There are three kinds of lies: lies, damned lies, and statistics.” On the other hand, H.G. Wells has been (mis)quoted as stating, “statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write” (Quora, n.d.). The fact that the actual sources of both of these “quotations” are unverifiable makes their sentiments no less true. The effective use of statistics can play a critical role in influencing public opinion as well as persuading in the workplace. However, as the fame of the first quotation indicates, statistics can be used to mislead rather than accurately inform—whether intentionally or unintentionally.
When evaluating research sources, be careful to critically evaluate the authority, content, and purpose of the material, using questions in Table 1.4.1. You may also want to learn about Fact-Checking Information using the SIFT Method (video runtime: 6:43).
| [Skip Table] |
|
| Authority Researchers Authors Creators |
Who are the researchers/authors/creators? Who is their intended audience? What are their credentials/qualifications? What else has this author written? Is this research funded? By whom? Who benefits? Who has intellectual ownership of this idea? How do I cite it? Where is this source published? What kind of publication is it? Authoritative Sources: written by experts for a specialized audience, published in peer-reviewed journals or by respected publishers, and containing well-supported, evidence-based arguments. Popular Sources: written for a general (or possibly niche) public audience, often in an informal or journalistic style, published in newspapers, magazines, and websites with a purpose of entertaining or promoting a product; evidence is often “soft” rather than hard. |
|---|---|
|
Content |
Methodology What is the methodology of their study? Or how has evidence been collected? Is the methodology sound? Can you find obvious flaws? What is its scope? Does it apply to your project? How? How recent and relevant is it? What is the publication date or last update? |
|
Data Is there sufficient data here to support their claims or hypotheses? Do they offer quantitative and/or qualitative data? Are visual representations of the data misleading or distorted in some way? |
|
|
Purpose |
Why has this author presented this information to this audience? Why am I using this source? Will using this source bolster my credibility or undermine it? Am I “cherry-picking” – using inadequate or unrepresentative data that only supports my position, while ignoring substantial amount of data that contradicts it? Could “cognitive bias” be at work here? Have I only consulted the kinds of sources I know will support my idea? Have I failed to consider alternative kinds of sources? Am I representing the data I have collected accurately? Are the data statistically relevant or significant? |
Table 1.4.1 Evaluate the authority, content, and purpose of information (Kyte, 2024) [Used under CC BY 4.0]
If you are interested in learning more skills for evaluating sources, see Question Your Sources and Evaluate the Arguments of Others from Monash University, or watch the CTRL-F Verification Skills 2024 video playlist.
Beware of Logical Fallacies
Logic has been studied and taught for millennia. We use logic to convince others. A logical fallacy is an error in reasoning within an argument when trying to explain something or persuade someone. This differs from a factual error, which is simply being wrong about the facts. Logical fallacies are flawed statements that often sound true. The word “fallacy” derives from the Latin word fallere meaning, “to deceive, to trip, to lead into error or to trick.” The word also derives from the Greek phelos, meaning “deceitful.”
Logical fallacies are often used to falsely strengthen an argument but if the reader detects them, the argument can backfire, and damage the writer’s credibility.
Logical Fallacies
Logical fallacies can arise in various contexts, such as in safety assessments, decision-making processes, and communication. Some common logical fallacies that can occur include:
- Appeal to Authority: This fallacy occurs when an argument is accepted as true simply because an authority figure or expert says it is true. In engineering, this could manifest in decisions being made based solely on the opinion of an engineer or manager without considering other factors or evidence.
One example is the design and construction of the St. Francis Dam under civil engineer William Mulholland. While Mulholland was considered an expert, the dam failed, causing the deaths of over 400 people. Mulholland did not follow the established practices and designs of the time.
- False Cause: This fallacy assumes that because one event follows another, the first event caused the second. For example, attributing the cause of flooding to one cause without considering other possible contributing factors.
The False Cause Fallacy and Ancient Rome
The false cause fallacy is not new. In ancient Rome, sacred chickens were consulted about major undertakings. A widely known story was how Publius Claudius Pulcher (consul, 249 BC) consulted the chickens before a planned naval battle in the First Punic War (see Figure 1.4.1). The chickens were not eating their feed, which was viewed as a bad omen. Claudius, enraged, must have said something (in Latin) along the lines of “If they will not eat, let them drink!” and had them thrown into the sea. The battle went badly for Rome, resulting in the loss of the fleet. People of the time saw the correlation and assumed causation was involved: Pulcher narrowly escaped with his life. Few people today base major decisions on chickens.
Correlation does not mean causation, but it can give a nudge that the items should be examined more closely to see if there is a functional relationship (Spafford et al., 2023).
Figure 1.4.1 The Sacred Chickens of Rome, (Boese, 2021). [Used under Fair Dealing].
Hasty Generalization: Drawing a conclusion based on insufficient evidence. For instance, assuming that a particular prototype or model that functioned well in controlled lab condition will function the same in a field environment.
- False Dilemma: The false dilemma refers to presenting a situation as if there are only two possible outcomes or options when in reality there are more. For example, arguing that either safety or profitability can be prioritized, when in fact both can be achieved with proper planning and management.
- Sunk cost fallacy. The sunk cost fallacy is the tendency to continue investing in a course of action because of the resources (time, money, effort) already invested, even when it is no longer rational to do so. For example, a team may continue to develop software despite evidence that it is no longer marketable, simply because time and money were already spent on the development.
- Base Rate Fallacy: Base rate fallacy, also known as base rate neglect or base rate bias, is a cognitive bias where individuals tend to ignore general information (base rates) in favour of specific information, even when the general information is more relevant for making a decision or judgment.
Recognizing and addressing these logical fallacies is important in engineering technology to ensure that decisions are based on sound reasoning and evidence, ultimately contributing to improved safety and operational effectiveness.
We all have biases when we write or argue; however, when evaluating sources, you want to be on the lookout for bias that is unfair, one-sided, or slanted. Consider whether the author has acknowledged and addressed opposing ideas, potential gaps in the research, or limits of the data. Look at the kind of language the author uses: is it slanted, strongly connotative, or emotionally manipulative? Is the supporting evidence presented logically, credibly, and ethically? Has the author cherry-picked or misrepresented sources or ideas? Does the author rely heavily on emotional appeal? There are many logical fallacies that both writers and readers can fall prey to (see Table 1.4.2). It is important to use data ethically and accurately, and to apply logic correctly and validly to support your ideas.
| [Skip Table] | |
| Bandwagon Fallacy |
Argument from popularity – “Everyone else is using AI, so we should too!” |
|---|---|
| Hasty Generalization |
Using insufficient data to come to a general conclusion. E.g., An Australian stole my wallet; therefore, all Australians are thieves! |
| Unrepresentative Sample |
Using data from a particular subset and generalizing to a larger set that may not share similar characteristics. E.g., giving a survey to only female students under 20 and generalizing results to all students. |
| False Dilemma |
“Either/or fallacy” – presenting only two options when there are usually more possibilities to consider. E.g., You’re either with us or against us. |
| Slippery Slope |
Claiming that a single cause will lead, eventually, to exaggerated catastrophic results. |
| Slanted Language |
Using language loaded with emotional appeal and either positive or negative connotation to manipulate the reader. |
| False Analogy |
Comparing your idea to another that is familiar to the audience but which may not have sufficient similarity to make an accurate comparison. E.g., Governing a country is like running a business. |
| Post hoc, ergo prompter hoc |
“After this; therefore, because of this” E.g., A happened, then B happened; therefore, A caused B. Just because one thing happened first, does not necessarily mean that the first thing caused the second thing. |
| Circular Reasoning |
Circular argument – assuming the truth of the conclusion by its premises. E.g., I never lie; therefore, I must be telling the truth. |
| Ad hominem |
An attack on the person making an argument does not really invalidate that person’s argument. It might make them seem a bit less credible, but it does not dismantle the actual argument or invalidate the data. |
| Red Herring |
A digression from the main argument by including information that is not relevant |
| Straw Man Argument |
Restating the opposing idea in an inaccurately absurd or simplistic manner to more easily refute or undermine it. |
| Others? |
There are many more… can you think of some? For a bit of fun, check out Spurious Correlations. |
Table 1.4.2 Common Logical Fallacies (Kyte, 2024) [Used under CC BY 4.0]
For more information on logical fallacies, review the Fallacies entry at the Internet Encyclopedia of Philosophy.
Knowledge Check
Critical thinking lies at the heart of evaluating sources. You want to be rigorous in your selection of evidence because, once you use it in your report, it will either bolster your own credibility or undermine it. Review this Sage video on critical thinking. You will need to log in with your SaskPolytech credentials to access the video.
References
Beall, J. (2024, December 24). Beall’s list of potential predatory journals and publishers. https://beallslist.net/
Bennett, M. (n.d.). St. Francis Dam (California, 1928). ASDSO Dam Failures and Lessons Learned. https://damfailures.org/case-study/st-francis-dam-california-1928
Boese, A. (2021, May 5). The sacred chickens of Rome. Weird Universe. https://www.weirduniverse.net/blog/comments/sacred_chickens_of_rome
Chatfield, T. (2018). What is critical thinking? [Video]. Sage Research Methods. https://methods.sagepub.com/video/what-is-critical-thinking#_
CTRL-F: Digital Media Literacy. (2024). CTRL-F verification skills 2024 [Video playlist]. YouTube. https://www.youtube.com/playlist?list=PLsSbsdukQ8Vb_xgdQOarM7YajJbO4o8Oz
Dowden, B. (n.d.). Fallacies. In Internet encyclopedia of philosophy. https://iep.utm.edu/fallacy/
Kurland, D. (n.d.). What is critical thinking? How the Language Really Works: The Fundamentals of Critical Reading and Effective Writing. https://www.criticalreading.com/critical_thinking.htm
Kyte, M. (2024, January 27). Technical communications: Introduction to communications in aviation. Seneca. https://pressbooks.senecapolytechnic.ca/technicalcommunications/ [Used under CC BY 4.0]
Last, S. (2019). Technical writing essentials. BCcampus. https://pressbooks.bccampus.ca/technicalwriting/ [Used under CC BY 4.0]
Monash University. (n.d.). Evaluate the arguments of others. Student Academic Success. https://www.monash.edu/student-academic-success/sharpen-your-thinking/critical-thinking/evaluate-arguments-of-others
Monash University. (n.d.). Question your sources. Student Academic Success. https://www.monash.edu/student-academic-success/sharpen-your-thinking/critical-thinking/question-your-sources
Spafford, E., Metcalf, L., & Dykstra, J. (2023). Cybersecurity myths and misconceptions: Avoiding the Hazards and Pitfalls That Derail Us. Addison-Wesley Professional.
Stohr, S. (2026, January 5). Fact-checking information using the SIFT method [Video]. YouTube. https://www.youtube.com/watch?v=eZV_4sYtD40
Vigen, T. (n.d.). Spurious correlations. https://www.tylervigen.com/spurious-correlations
What is the source of the H.G. Wells quote, ‘Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write/”? (n.d.). Quora. https://www.quora.com/What-is-the-source-of-the-H-G-Wells-quote-Statistical-thinking-will-one-day-be-as-necessary-for-efficient-citizenship-as-the-ability-to-read-and-write
