The Robot Factory: Pseudoscience in Education and Its Threat to American Democracy

By Joseph Ganem

This book exposes a disturbing misuse of the scientific method to advance policies and agendas that are in fact detrimental to both science and education. The author, a physics professor, examines two related trends in education – the practice of “data-driven” reform and the disparaging of the traditional liberal arts in favor of programs with a heavy emphasis on science and technology. Many of the reforms being foisted on educators have more in common with pseudo-science than real science. The reduction of education to a commodity, and the shilling of science as a means to enhance corporate profits, lead to an impoverished and stunted understanding of science in particular, and of education in general.

How is it possible for: • schools with all students learning at grade-level to be rated as failing?• teachers to be rated as ineffective after all their students meet their learning outcomes?• rising grade-school math standards to result in more college students needing remedial math?• politicians to disparage scientists and their results but argue that more students should study science?
These bizarre outcomes have happened and are the result of an education system that misuses and misrepresents math and science in the classroom and in crafting education policies. This book exposes the flawed and fallacious thinking that is damaging education at all levels throughout the United States, and makes a compelling case for rethinking the standardized, optimized, and quantified approaches in vogue in education today to accommodate the different needs of individual teachers and students.

• Language: English
• Publisher: Springer
• Release date: Aug 27, 2018
• ISBN: 9783319778600

Book preview

Part IPseudoscience in Education

© Springer International Publishing AG, part of Springer Nature 2018

Joseph GanemThe Robot Factoryhttps://doi.org/10.1007/978-3-319-77860-0_1

1. What Science Is and Is Not

Joseph Ganem¹ 

(1)

Loyola University Maryland, Baltimore, MD, USA

All participants in the education process—teachers, admissions counselors, administrators, politicians, and policymakers—want to give their practices a scientific justification based on objective, evidence-based criteria and demand proven results based on data. This is a laudable goal, but most don’t actually understand science, the how and why of scientific measurement, data interpretation, and conclusions. As we shall see, many of the data-driven education policies and reforms thus resemble what is called pseudoscience more than real science.

Distinguishing real scientific claims from pseudoscientific ones can be difficult because it is the basis for the claims being made that needs to be evaluated, not the plausibility of the claims themselves. Both scientists and charlatans can make fantastic sounding claims and assertions. In many cases the claims of the charlatans are more believable. Consider the following sales pitches:

I have a small device that you can carry in your pocket and it will insure that you never get lost. It knows at all times exactly where you are and it will give you precise directions to any place you need to go. Just state the location out loud and follow its spoken commands.

I have a medicine that will prevent and cure migraine headaches. It uses all-natural plant-based ingredients diluted in water and has no side effects. Trained homeopathic doctors have prescribed this medication for over a hundred years and tens of thousands of patients have attested to its effectiveness.

With no prior knowledge, the latter claim is much more plausible than the former. However, the scientific understanding of nature that allows us to carry GPS enabled smartphones is legitimate; these devices really do work. But homeopathy is dismissed in the mainstream medical community as a pseudoscience. What is the difference?

After all, not all science is as reliable as a smartphone, and these devices too will make errors. Scientists, being human, also make errors, are wrong sometimes, and succumb to the same human foibles as everyone else—especially when it comes to denial and wishful thinking. In addition, the body of human knowledge collectively referred to as science evolves in time, and scientific beliefs in one era might be rejected at a later time. To further confound matters, the reliability of our scientific knowledge appears to exist along a spectrum, with some branches of science universally accepted and undisputed—e. g. Newton ’s laws of motion—and other branches considered speculative and subject to debate—e. g. string theory.

What makes knowledge scientific is the process by which it was obtained. This process has identifiable elements, some or most of which are missing in pseudoscientific works. Differentiating science from pseudoscience is an extensive topic, with a significant philosophical literature, as well as book-length treatments for general audiences [1–3]. However, we do have general consensus on a number of elements considered essential to the scientific method. After identifying and exploring the essential elements of a scientific approach to obtaining knowledge, I discuss examples of pseudoscience in education, Later, in Chapter 5, I go into more detail on the workings of the scientific process, but let’s begin with the essentials.

Doing Real Science: Essential Elements

The scientific method, as it is articulated at the K-12 level, teaches students a rigid series of steps: ask a testable question (the hypothesis ), design a controlled experiment, collect data from the experiment, analyze the data, and infer whether the data supports or refutes the hypothesis. This simplified summary of the scientific method does contain some of its essential elements: Experiments must be designed to answer specific questions, and for data to be useful it must be acquired and analyzed in a way that relates to the hypothesis. However, this is a gross oversimplification of the conduct of actual science—a topic discussed later in Chapter 5. Science isn’t that straightforward. Often major discoveries result from observations made outside of the stated hypothesis.

Moreover, this simplified view of science focuses mostly on the activities of measurement and data collection, which by themselves have almost no meaning. As a practicing scientist I am constantly faced with the measurement problem; when I walk into my lab, there are literally an infinite number of measurements I can choose to make, and I must decide what to measure and then interpret what that measurement means. Therefore I need to have a reason—a motivation—for a particular action. The reasons can be varied: testing equipment, obtaining preliminary results, collecting publishable data, or attempting to reproduce data. Acquiring lots of data for its own sake is not synonymous with doing science.

For an investigation to be scientific one must consider all of the critical elements of the scientific method, including those on the front and back ends of the measurement and data collection processes. These elements include:

1.

The question posed must be relevant.

2.

The question posed must be falsifiable.

3.

The data collection must be systematic and consistent.

4.

Causal explanations for observed effects must be provided.

5.

The explanations must offer a simplified understanding of natural processes.

6.

The explanations must avoid over-fitting the data.

7.

The experiments must be reproducible by other researchers.

8.

The understanding of nature that results must allow for the correct prediction, and if possible, control of future events.

Let’s elaborate on each of these elements:

1. A relevant question.

Before beginning an investigation, there must be a motivation—that is, a researcher must pose a question that is interesting, relevant, and whose answer will lead to further insights. Science is not testing the composition and hardness of every rock found on the ground. Science is not describing in elaborate detail every plant and animal species in a field. Science is not, as the Beatles’ song puts it, counting the four thousand holes in Blackburn, Lancashire, in order to know how many holes it takes to fill the Albert Hall. To be science, an investigation cannot just be an elaborate exercise in data collection; rather, to be science, an investigation must be organized and systematized in such a way that it serves to motivate the research and researcher by addressing a relevant question.

If we go back to the third grade class I visited, the practice of testing all the children in math is not scientific unless the tests are designed and administered in a way that addresses specific relevant questions on the teaching methods used (pedagogies) and student learning outcomes (achievements). Having a standardized test isn’t science if the only question at stake is to find out which students score higher than others, and which schools have higher overall test scores. A ranked list of test scores for students and schools with no other context provided is just trivia, because unless it relates to relevant questions, it has no inherent meaning. A scientific investigation must be motivated by scientifically valid questions of significance, not by questions of minutiae and trivialities.

2. The question posed must also be falsifiable.

This means that it must be possible to prove the hypothesis wrong, because otherwise an experimental test of the hypothesis would not be possible [4]. You cannot construct hypotheses that are tautologies —that is, statements that are always true—because this results in circular arguments. When a hypothesis is a tautology , the hypothesis is deemed true from the onset and becomes evidence for its own support. Such a hypothesis can never be falsified. For example, the statement that the Earth and all life on it are creations of God is a tautology because in Christian theology God is the creator of the universe. Creationism is not a science because it can never be falsified. This might be a valid belief system, but it is not a valid scientific hypothesis.

Likewise, saying that a good third grade math teacher has students that all do well on the state exam is a tautology. This statement cannot be falsified because by definition any class of students that does well on the state exam had a good teacher. Finding a class with high-scoring students and a bad teacher is not possible if this is the definition of a good teacher. If your hypothesis is that good teachers produce students with high test scores, then the attributes and actions of a good teacher must be defined independently of the test scores in order for it to be testable. A scientific hypothesis cannot be a tautology because it must be a statement that is either refutable or verifiable by experimentation, not true from the onset.

3. Systematic and consistent data collection.

Procedures for data collection cannot be manipulated to support the self-interest of particular groups of people. Pharmaceutical companies are not doing science when they report only the trials that show efficacy of a new drug, while they dismiss the failed trials as some kind of aberration resulting from an elaborate set of required conditions that weren’t quite right. Energy companies are not doing science when they commission environmental studies and declare the results proprietary, releasing only data that shows them in a favorable light. In both of these examples, self-selecting data is a form of manipulation that invalidates the science.

The teaching to the test I observed in third grade math, as I explained, is also a form of manipulation. Such actions by the teacher render the test data meaningless for scientific purposes. When data has been manipulated to produce the desired results, then no valid scientific conclusions can be drawn.

4. Causal explanations must be provided.

In analyzing the experimental data, the scientist must propose causal explanations for the observed effects. After conducting an experiment, simply stating that the data either supports or refutes the hypotheses is not acceptable science. An investigator might not be able to prove a chain of causes, but if unproven, there must be further investigations possible to test the proposed causes. Without causal explanations, it becomes possible to fall into the logical trap of mistaking correlations for causes. For example, a perfect (100%) correlation exists between incidents of prostate cancer and being male. Of course, being male is not a cause of the disease; simply—and obviously—it is correlated because only men have prostates [5]. In contrast: there is a strong but less than perfect correlation between being a smoker and developing lung cancer . However, smoking is a proven cause of lung cancer. Establishing the causal chain of events of how smoking results in lung cancer in some (but not all) smokers was not easy and took numerous studies and decades of scientific work. The causal pathways were not obvious even though the correlation was.

In the third grade math class I visited, the students were exhorted to show how smart they are. But, while being smart—assuming that a working definition can be made—might be correlated with student achievement, it is not causal. Simply having a class of smart students will not cause them to learn or cause teachers to be effective. Administering a test and then concluding from the results that all the students are smart is not a scientifically valid insight.

5. The causal explanations must offer a simplified understanding of natural processes.

Indeed, the point of science is to find simpler underlying principles for complex phenomena that allow for predictions of future events. Every experimental outcome cannot be assigned a unique explanation, and data that refutes a hypothesis cannot be ignored as a special case because neither practice results in a simplified understanding of nature, or makes predictions possible. If we deem the outcome of every experiment in a study a special case that needs a unique, complex, narrative in order to explain it, then we gain zero scientific insights.

Anecdotal evidence is the term used for narrative explanations of events and such evidence abounds in our everyday interactions with one another. You hear your neighbor say: My best friend felt much more energetic after taking the homeopathic medicine for migraines. But you know that your uncle tried the same medication and he felt the same afterwards. Therefore, since the homeopathic medicine cured your neighbor’s best friend it must work, and your uncle must have a different illness. This kind of reasoning from anecdotes is commonplace. However, it does not simplify our understanding of how to treat migraine headaches because these stories are unique to the individuals involved, and, therefore, no underlying principle can be articulated.

Likewise, if test scores go up in Mrs. Smith’s class after a new math curriculum is introduced, but go down in Mr. Jones’ class after introduction of the same curriculum, concluding that the new curriculum is an improvement and that there is a problem with Mr. Jones’ teaching is neither valid nor helpful for informing Mrs. Hart on whether she should adopt the new curriculum. We don’t gain from these narratives any simplified understanding of what makes a better math curriculum if every teacher has a different story to tell about their experiences.

6. Science avoids over-fitting the data.

Over-fitting is a term used in statistics that refers to using a model to make predictions that has much more complexity than the available data can support [6]. No data set is ever perfect. There is always a level of randomness present in any set of measurements—what statisticians refer to as the noise. But if a model with enough complexity is used, patterns will be found in the noise that the model can then explain.

For example, media pundits seem to have a hobby of over-fitting sports data in order to predict the outcomes of such things as election winners and stock market returns. If the Super Bowl winner is an NFC team it foreshadows a bullish year in the stock market, while if the winning team is from the AFC it portends a bearish year [7]. There is the Redskins rule that states that if the Washington Redskins football team wins their last home game the incumbent party will win the election that year, while a loss in their last home game means that the incumbent party will lose [8]. My personal favorite for this kind of analysis is the ex-Cub factor, which posits that the World Series will be won by the team with the fewest former Chicago Cubs as players [9]. At least this model uses a baseball statistic to predict a baseball outcome.

If your predictive model is so complex that it includes all available sports data, then given the literally millions of sports statistics accumulating in databases every year, it will always be possible to sift through them and find particular statistics that are perfectly correlated with low-frequency events such as World Series winners, or year-end closes of the Dow Jones Industrial Average, or outcomes of biannual elections. Of course, all of these correlations are spurious and fail to predict anything. The Redskins rule has failed to predict the last two presidential elections and the Chicago Cubs recently won a World Series. A signature of over-fitting is when a model fits past events but fails to predict future ones.

Even the pundits will admit that these correlations are spurious, and in fact their articles are usually meant to be humorous. Less humorous are the highly complex statistical models for evaluating teachers that have become widespread. In Chapters 2 and 3 we will examine models used for evaluating teaching effectiveness that use dozens of inputs to predict low-frequency outcomes such as their students’ gains in math scores at the end of the year. These models are classic examples of over-fitting; however, in contrast to the humorous articles over-fitting sports statistics to make predictions, these teaching evaluation models are taken seriously.

7. The results must be reproducible by other investigators.

Reproducibility of results in future experiments by other investigators is an essential part of the scientific method. It is how science self-corrects. If causes for an effect cannot be identified, then it will not be possible to reproduce the effect or control future outcomes.

In the late 1980s, scientists announced unexpected discoveries in physics and chemistry with profound implications for energy technology. One was cold fusion—energy production from nuclear reactions that power the sun, but at low Earth-like temperatures—and the other was high-temperature superconductivity—materials that offer no electrical resistance at temperatures much higher than previously believed possible. Investigators could not reproduce cold fusion and the work was ultimately discredited [10]. However, investigators from all over the world could reproduce high-temperature superconductors [11]. Thus, the latter was scientifically valid because others were able to reproduce the findings.

Reproducibility is a serious problem in much of the data-driven education reform movement. Part of this problem is that reproducing the exact conditions in an educational setting year after year is rarely possible. Education always takes place in a cultural context, with each new generation of students growing up in what is essentially a different culture. The American culture of the 1960s when I went to third grade was radically different in many ways from the American culture of the 2000s when my daughter went to third grade.

However, human development does not change as much as many educators think that it does, or wish that it would. In addition, the basic facts of math have also not changed. That means that it should be possible to replicate successful methods for teaching math. Yet policymakers keep altering the math curriculum , pushing more and more advanced concepts into younger grades without any reproducible evidence that the new methods are effective. As a result, ineffective, developmentally inappropriate math instruction is now a serious problem throughout the K-12 curriculum .

8. Science should have the power to predict and, when possible, control the future.

The benefit to society that comes from doing science is in obtaining knowledge of how significant events can be predicted, and if possible controlled in the future.

The National Weather Service does not collect enormous amounts of weather data from all over the world just to keep its staff busy and employed. It is using this data to build models that will predict future weather events. While weather cannot be controlled, just being able to reliably predict weather can save lives and enormous amounts of money by giving people a chance to prepare. Likewise, medical researchers are not investing enormous amounts of time and money developing imaging techniques such as MRIs, PET scans, and CAT scans because they like to look at the pictures. They want to predict if a patient is developing cancer and learn to control the spread of the disease.

In educational settings, articulating, assessing, and documenting learning outcomes isn’t inherently useful. The goal should be to use the data to build models for student learning that can predict outcomes. In fact, numerous valid research studies across many disciplines strive to use test data to predict the outcomes resulting from different teaching methods in order to identify best educational practices. However, a great deal of the test data collected by the state and the federal government is not used for this purpose. Instead it is used to harass, threaten, and intimidate schools and teachers. Using the test data to predict educational outcomes and then provide guidance on interventions to change them is rarely a priority for politicians.

These essential elements of science—relevant, falsifiable questions, systematic and consistent data collection, causal explanations, simplified understanding of natural processes, avoidance of over-fitting , reproducibility, predictability, and control—must be present in any study claiming to be scientific. Science ought to be powerfully in the service of education and yet much of the real science on education and human development is ignored in favor of evidence-based reports and data-driven practices that are missing at least one, or in many cases several, of these essential elements.

The Magical Thinking Behind Pseudoscience

The stunning success of modern science has not resulted in the extermination of supernatural explanations, or what I call magical thinking. Supernatural explanations for events such as demons, witches, astrological signs, and so on, have prevailed for most of human history. Only in the last few hundred years has scientific thinking and methodology profoundly changed how humans interact with and predict occurrences in the natural world. However, magical thinking still persists, and indeed is reincarnated as pseudoscience with all the trappings of real science.

Belief systems such as creationism , homeopathy , alien astronauts, astrology, positive thinking, and ESP are some examples. All of these belief systems come with scientific-sounding claims. All offer causal explanations of phenomena with supporting evidence. All offer statistics to back up their claims. But on closer examination, none of these belief systems stands up when subjected to true scientific scrutiny because at least one or more of the essential elements that I have listed are missing.

In each of these belief systems, the hypotheses are compelling, emotionally engaging narratives that can never be falsified because they are assumed to be true. Evidence is often in the form of anecdotes. Statistical comparisons are based on self-selected data because all evidence contrary to the hypotheses are explained as some special case and ignored. Reproducibility of results by independent investigators is non-existent. People who raise questions are vilified as enemies of the good. In fact, false choice is a defining feature of these belief systems because the response to all criticism is framed in terms of us versus them. You cannot choose to seek the truth; you can only choose