Essentials of Research Design and Methodology

By Geoffrey R. Marczyk, David DeMatteo and David Festinger

Master the essential skills for designing and conducting a successful research project

Essentials of Research Design and Methodology contains practical information on how to design and conduct scientific research in the behavioral and social sciences. This accessible guide covers basic to advanced concepts in a clear, concrete, and readable style.

The text offers students and practitioners in the behavioral sciences and related disciplines important insights into identifying research topics, variables, and methodological approaches. Data collection and assessment strategies, interpretation methods, and important ethical considerations also receive significant coverage in this user-friendly guide. Essentials of Research Design and Methodology is the only available resource to condense the wide-ranging topics of the field into a concise, accessible format for handy and quick reference.

As part of the Essentials of Behavioral Science series, this book offers a thorough review of the most relevant topics in research design and methodology. Each concise chapter features numerous callout boxes highlighting key concepts, bulleted points, and extensive illustrative material, as well as "Test Yourself" questions that help you gauge and reinforce your grasp of the information covered.

• Language: English
• Publisher: Wiley
• Release date: Jun 3, 2010
• ISBN: 9780470893531

Book preview

One

INTRODUCTION AND OVERVIEW

Progress in almost every field of science depends on the contributions made by systematic research; thus research is often viewed as the cornerstone of scientific progress. Broadly defined, the purpose of research is to answer questions and acquire new knowledge. Research is the primary tool used in virtually all areas of science to expand the frontiers of knowledge. For example, research is used in such diverse scientific fields as psychology, biology, medicine, physics, and botany, to name just a few of the areas in which research makes valuable contributions to what we know and how we think about things. Among other things, by conducting research, researchers attempt to reduce the complexity of problems, discover the relationship between seemingly unrelated events, and ultimately improve the way we live.

Although research studies are conducted in many diverse fields of science, the general goals and defining characteristics of research are typically the same across disciplines. For example, across all types of science, research is frequently used for describing a thing or event, discovering the relationship between phenomena, or making predictions about future events. In short, research can be used for the purposes of description, explanation, and prediction, all of which make important and valuable contributions to the expansion of what we know and how we live our lives. In addition to sharing similar broad goals, scientific research in virtually all fields of study shares certain defining characteristics, including testing hypotheses, careful observation and measurement, systematic evaluation of data, and drawing valid conclusions.

In recent years, the results of various research studies have taken center stage in the popular media. No longer is research the private domain of research professors and scientists wearing white lab coats. To the contrary, the results of research studies are frequently reported on the local evening news, CNN, the Internet, and various other media outlets that are accessible to both scientists and nonscientists alike. For example, in recent years, we have all become familiar with research regarding the effects of stress on our psychological well-being, the health benefits of a low-cholesterol diet, the effects of exercise in preventing certain forms of cancer, which automobiles are safest to drive, and the deleterious effects of pollution on global warming. We may have even become familiar with research studies regarding the human genome, the Mars Land Rover, the use of stem cells, and genetic cloning. Not too long ago, it was unlikely that the results of such highly scientific research studies would have been shared with the general public to such a great extent.

Despite the accessibility and prevalence of research in today’s society, many people share common misperceptions about exactly what research is, how research can be used, what research can tell us, and the limitations of research. For some people, the term research conjures up images of scientists in laboratories watching rats run through mazes or mixing chemicals in test tubes. For other people, the term research is associated with telemarketer surveys, or people approaching them at the local shopping mall to just ask you a few questions about your shopping habits. In actuality, these stereotypical examples of research are only a small part of what research comprises. It is therefore not surprising that many people are unfamiliar with the various types of research designs, the basics of how research is conducted, what research can be used for, and the limits of using research to answer questions and acquire new knowledge. Rapid Reference 1.1 discusses what we mean by research from a scientific perspective.

Before addressing these important issues, however, we should first briefly review what science is and how it goes about telling us what we know.

Rapid Reference 1.1

What Exactly is Research?

Research studies come in many different forms, and we will discuss several of these forms in more detail in Chapter 5. For now, however, we will focus on two of the most common types of research—correlational research and experimental research.

Correlational research: In correlational research, the goal is to determine whether two or more variables are related. (By the way, variables is a term with which you should be familiar. A variable is anything that can take on different values, such as weight, time, and height.) For example, a researcher may be interested in determining whether age is related to weight. In this example, a researcher may discover that age is indeed related to weight because as age increases, weight also increases. If a correlation between two variables is strong enough, knowing about one variable allows a researcher to make a prediction about the other variable. There are several different types of correlations, which will be discussed in more detail in Chapter 5. It is important to point out, however, that a correlation—or relationship—between two things does not necessarily mean that one thing caused the other. To draw a cause-and-effect conclusion, researchers must use experimental research. This point will be emphasized throughout this book.

Experimental research: In its simplest form, experimental research involves comparing two groups on one outcome measure to test some hypothesis regarding causation. For example, if a researcher is interested in the effects of a new medication on headaches, the researcher would randomly divide a group of people with headaches into two groups. One of the groups, the experimental group, would receive the new medication being tested. The other group, the control group, would receive a placebo medication (i.e., a medication containing a harmless substance, such as sugar, that has no physiological effects). Besides receiving the different medications, the groups would be treated exactly the same so that the research could isolate the effects of the medications. After receiving the medications, both groups would be compared to see whether people in the experimental group had fewer headaches than people in the control group. Assuming this study was properly designed (and properly designed studies will be discussed in detail in later chapters), if people in the experimental group had fewer headaches than people in the control group, the researcher could conclude that the new medication reduces headaches.

OVERVIEW OF SCIENCE AND THE SCIENTIFIC METHOD

In simple terms, science can be defined as a methodological and systematic approach to the acquisition of new knowledge. This definition of science highlights some of the key differences between how scientists and nonscientists go about acquiring new knowledge. Specifically, rather than relying on mere casual observations and an informal approach to learn about the world, scientists attempt to gain new knowledge by making careful observations and using systematic, controlled, and methodical approaches (Shaughnessy & Zechmeister, 1997). By doing so, scientists are able to draw valid and reliable conclusions about what they are studying. In addition, scientific knowledge is not based on the opinions, feelings, or intuition of the scientist. Instead, scientific knowledge is based on objective data that were reliably obtained in the context of a carefully designed research study. In short, scientific knowledge is based on the accumulation of empirical evidence (Kazdin, 2003a), which will be the topic of a great deal of discussion in later chapters of this book.

The defining characteristic of scientific research is the scientific method (summarized in Rapid Reference 1.2). First described by the English philosopher and scientist Roger Bacon in the 13th century, it is still generally agreed that the scientific method is the basis for all scientific investigation. The scientific method is best thought of as an approach to the acquisition of new knowledge, and this approach effectively distinguishes science from nonscience. To be clear, the scientific method is not actually a single method, as the name would erroneously lead one to believe, but rather an overarching perspective on how scientific investigations should proceed. It is a set of research principles and methods that helps researchers obtain valid results from their research studies. Because the scientific method deals with the general approach to research rather than the content of specific research studies, it is used by researchers in all different scientific disciplines. As will be seen in the following sections, the biggest benefit of the scientific method is that it provides a set of clear and agreed-upon guidelines for gathering, evaluating, and reporting information in the context of a research study (Cozby, 1993).

Rapid Reference 1.2

The Scientific Method

The development of the scientific method is usually credited to Roger Bacon, a philosopher and scientist from 13th-century England, although some argue that the Italian scientist Galileo Galilei played an important role in formulating the scientific method. Later contributions to the scientific method were made by the philosophers Francis Bacon and René Descartes. Although some disagreement exists regarding the exact characteristics of the scientific method, most agree that it is characterized by the following elements:

• Empirical approach

• Observations

• Questions

• Hypotheses

• Experiments

• Analyses

• Conclusions

• Replication

There has been some disagreement among researchers over the years regarding the elements that compose the scientific method. In fact, some researchers have even argued that it is impossible to define a universal approach to scientific investigation. Nevertheless, for over 100 years, the scientific method has been the defining feature of scientific research. Researchers generally agree that the scientific method is composed of the following key elements (which will be the focus of the remainder of this chapter): an empirical approach, observations, questions, hypotheses, experiments, analyses, conclusions, and replication.

Before proceeding any further, one word of caution is necessary. In the brief discussion of the scientific method that follows, we will be introducing several new terms and concepts that are related to research design and methodology. Do not be intimidated if you are unfamiliar with some of the content contained in this discussion. The purpose of the following is simply to set the stage for the chapters that follow, and we will be elaborating on each of the terms and concepts throughout the remainder of the book.

Empirical Approach

The scientific method is firmly based on the empirical approach. The empirical approach is an evidence-based approach that relies on direct observation and experimentation in the acquisition of new knowledge (see Kazdin, 2003a). In the empirical approach, scientific decisions are made based on the data derived from direct observation and experimentation. Contrast this approach to decision making with the way that most nonscientific decisions are made in our daily lives. For example, we have all made decisions based on feelings, hunches, or gut instinct. Additionally, we may often reach conclusions or make decisions that are not necessarily based on data, but rather on opinions, speculation, and a hope for the best. The empirical approach, with its emphasis on direct, systematic, and careful observation, is best thought of as the guiding principle behind all research conducted in accordance with the scientific method.

Observations

An important component in any scientific investigation is observation. In this sense, observation refers to two distinct concepts—being aware of the world around us and making careful measurements. Observations of the world around us often give rise to the questions that are addressed through scientific research. For example, the Newtonian observation that apples fall from trees stimulated much research into the effects of gravity. Therefore, a keen eye to your surroundings can often provide you with many ideas for research studies. We will discuss the generation of research ideas in more detail in Chapter 2.

In the context of science, observation means more than just observing the world around us to get ideas for research. Observation also refers to the process of making careful and accurate measurements, which is a distinguishing feature of well-conducted scientific investigations. When making measurements in the context of research, scientists typically take great precautions to avoid making biased observations. For example, if a researcher is observing the amount of time that passes between two events, such as the length of time that elapses between lightning and thunder, it would certainly be advisable for the researcher to use a measurement device that has a high degree of accuracy and reliability. Rather than simply trying to guesstimate the amount of time that elapsed between those two events, the researcher would be advised to use a stopwatch or similar measurement device. By doing so, the researcher ensures that the measurement is accurate and not biased by extraneous factors. Most people would likely agree that the observations that we make in our daily lives are rarely made so carefully or systematically.

An important aspect of measurement is an operational definition. Researchers define key concepts and terms in the context of their research studies by using operational definitions. By using operational definitions, researchers ensure that everyone is talking about the same phenomenon. For example, if a researcher wants to study the effects of exercise on stress levels, it would be necessary for the researcher to define what exercise is. Does exercise refer to jogging, weight lifting, swimming, jumping rope, or all of the above? By defining exercise for the purposes of the study, the researcher makes sure that everyone is referring to the same thing. Clearly, the definition of exercise can differ from one study to another, so it is crucial that the researcher define exercise in a precise manner in the context of his or her study. Having a clear definition of terms also ensures that the researcher’s study can be replicated by other researchers. The importance of operational definitions will be discussed further in Chapter 2.

Questions

After getting a research idea, perhaps from making observations of the world around us, the next step in the research process involves translating that research idea into an answerable question. The term answerable is particularly important in this respect, and it should not be overlooked. It would obviously be a frustrating and ultimately unrewarding endeavor to attempt to answer an unanswerable research question through scientific investigation. An example of an unanswerable research question is the following: Is there an exact replica of me in another universe? Although this is certainly an intriguing question that would likely yield important information, the current state of science cannot provide an answer to that question. It is therefore important to formulate a research question that can be answered through available scientific methods and procedures. One might ask, for example, whether exercising (i.e., perhaps operationally defined as running three times per week for 30 minutes each time) reduces cholesterol levels. This question could be researched and answered using established scientific methods.

Hypotheses

The next step in the scientific method is coming up with a hypothesis, which is simply an educated—and testable—guess about the answer to your research question. A hypothesis is often described as an attempt by the researcher to explain the phenomenon of interest. Hypotheses can take various forms, depending on the question being asked and the type of study being conducted (see Rapid Reference 1.3).

A key feature of all hypotheses is that each must make a prediction. Remember that hypotheses are the researcher’s attempt to explain the phenomenon being studied, and that explanation should involve a prediction about the variables being studied. These predictions are then tested by gathering and analyzing data, and the hypotheses can either be supported or refuted (falsified; see Rapid Reference 1.4) on the basis of the data.

In their simplest forms, hypotheses are typically phrased as if-then statements. For example, a researcher may hypothesize that "if people exercise for 30 minutes per day at least three days per week, then their cholesterol levels will be reduced." This hypothesis makes a prediction about the effects of exercising on levels of cholesterol, and the prediction can be tested by gathering and analyzing data.

Two types of hypotheses with which you should be familiar are the null hypothesis and the alternate (or experimental) hypothesis. The null hypothesis always predicts that there will be no differences between the groups being studied. By contrast, the alternate hypothesis predicts that there will be a difference between the groups. In our example, the null hypothesis would predict that the exercise group and the no-exercise group will not differ significantly on levels of cholesterol. The alternate hypothesis would predict that the two groups will differ significantly on cholesterol levels. Hypotheses will be discussed in more detail in Chapter 2.

Rapid Reference 1.3

Relationship Between Hypotheses and Research Design

Hypotheses can take many different forms depending on the type of research design being used. Some hypotheses may simply describe how two things may be related. For example, in correlational research (which will be discussed in Chapter 5), a researcher might hypothesize that alcohol intoxication is related to poor decision making. In other words, the researcher is hypothesizing that there is a relationship between using alcohol and decision making ability (but not necessarily a causal relationship). However, in a study using a randomized controlled design (which will also be discussed in Chapter 5), the researcher might hypothesize that using alcohol causes poor decision making. Therefore, as may be evident, the hypothesis being tested by a researcher is largely dependent on the type of research design being used. The relationship between hypotheses and research design will be discussed in more detail in later chapters.

Rapid Reference 1.4

Falsifiability of Hypotheses

According to the 20th-century philosopher Karl Popper, hypotheses must be falsifiable (Popper, 1963). In other words, the researcher must be able to demonstrate that the hypothesis is wrong. If a hypothesis is not falsifiable, then science cannot be used to test the hypothesis. For example, hypotheses based on religious beliefs are not falsifiable. Therefore, because we can never prove that faith-based hypotheses are wrong, there would be no point in conducting research to test them. Another way of saying this is that the researcher must be able to reject the proposed explanation (i.e., hypothesis) of the phenomenon being studied.

Experiments

After articulating the hypothesis, the next step involves actually conducting the experiment (or research study). For example, if the study involves investigating the effects of exercise on levels of cholesterol, the researcher would design and conduct a study that would attempt to address that question. As previously mentioned, a key aspect of conducting a research study is measuring the phenomenon of interest in an accurate and reliable manner (see Rapid Reference 1.5). In this example, the researcher would collect data on the cholesterol levels of the study participants by using an accurate and reliable measurement device. Then, the researcher would compare the cholesterol levels of the two groups to see if exercise had any effects.

Rapid Reference 1.5

Accuracy vs. Reliability

When talking about measurement in the context of research, there is an important distinction between being accurate and being reliable. Accuracy refers to whether the measurement is correct, whereas reliability refers to whether the measurement is consistent. An example may help to clarify the distinction. When throwing darts at a dart board, accuracy refers to whether the darts are hitting the bull’s eye (an accurate dart thrower will throw darts that hit the bull’s eye). Reliability, on the other hand, refers to whether the darts are hitting the same spot (a reliable dart thrower will throw darts that hit the same spot). Therefore, an accurate and reliable dart thrower will consistently throw the darts in the bull’s eye. As may be evident, however, it is possible for the dart thrower to be reliable, but not accurate. For example, the dart thrower may throw all of the darts in the same spot (which demonstrates high reliability), but that spot may not be the bull’s eye (which demonstrates low accuracy). In the context of measurement, both accuracy and reliability are equally important.

Analyses

After conducting the study and gathering the data, the next step involves analyzing the data, which generally calls for the use of statistical techniques. The type of statistical techniques used by a researcher depends on the design of the study, the type of data being gathered, and the questions being asked. Although a detailed discussion of statistics is beyond the scope of this text, it is important to be aware of the role of statistics in conducting a research study. In short, statistics help researchers minimize the likelihood of reaching an erroneous conclusion about the relationship between the variables being studied.

A key decision that researchers must make with the assistance of statistics is whether the null hypothesis should be rejected. Remember that the null hypothesis always predicts that there will be no difference between the groups. Therefore, rejecting the null hypothesis means that there is a difference between the groups. In general, most researchers seek to reject the null hypothesis because rejection means the phenomenon being studied (e.g., exercise, medication) had some effect.

It is important to note that there are only two choices with respect to the null hypothesis. Specifically, the null hypothesis can be either rejected or not rejected, but it can never be accepted. If we reject the null hypothesis, we are concluding that there is a significant difference between the groups. If, however, we do not reject the null hypothesis, then we are concluding that we were unable to detect a difference between the groups. To be clear, it does not mean that there is no difference between the two groups. There may in actuality have been a significant difference between the two groups, but we were unable to detect that difference in our study. We will talk more about this important distinction in later chapters.

The decision of whether to reject the null hypothesis is based on the results of statistical analyses, and there are two types of errors that researchers must be careful to avoid when making this decision—type I errors and Type II errors. A Type I error occurs when a researcher concludes that there is a difference between the groups being studied when, in fact, there is no difference. This is sometimes referred to as a false positive. By contrast, a Type II error occurs when the researcher concludes that there is not a difference between the two groups being studied when, in fact, there is a difference. This is sometimes referred to as a false negative. As previously noted, the conclusion regarding whether there is a difference between the groups is based on the results of statistical analyses. Specifically, with a Type I error, although there is a statistically significant result, it occurred by chance (or error) and there is not actually a difference between the two groups (Wampold, Davis, & Good, 2003). With a Type II error, there is a nonsignificant statistical result when, in fact, there actually is a difference between the two groups (Wampold et al.).

The typical convention in most fields of science allows for a 5% chance of erroneously rejecting the null hypothesis (i.e., of making a Type I error). In other words, a researcher will conclude that there is a significant difference between the groups being studied (i.e., will reject the null hypothesis) only if the chance of being incorrect is less than 5%. For obvious reasons, researchers want to reduce the likelihood of concluding that there is a significant difference between the groups being studied when, in fact, there is not a difference.

The distinction between Type I and Type II errors is very important, although somewhat complicated. An example may help to clarify these terms. In our example, a researcher conducts a study to determine whether a new medication is effective in treating depression. The new medication is given to Group 1, while a placebo medication is given to Group 2. If, at the conclusion of the study, the researcher concludes that there is a significant difference in levels of depression between Groups 1 and 2 when, in fact, there is no difference, the researcher has made a Type I error. In simpler terms, the researcher has detected a difference between the groups that in actuality does not exist; the difference between the groups occurred by chance (or error). By contrast, if the researcher concludes that there is no significant difference in levels of depression between Groups 1 and 2 when, in fact, there is a difference, the researcher has made a Type II error. In simpler terms, the researcher has failed to detect a difference that actually exists between the groups.

Which type of error is more serious—Type I or Type II? The answer to this question often depends on the context in which the errors are made. Let’s use the medical context as an example. If a doctor diagnoses a patient with cancer when, in fact, the patient does not have cancer (i.e., a false positive), the doctor has committed a Type I error. In this situation, it is likely that the erroneous diagnosis will be discovered (perhaps through a second opinion) and the patient will undoubtedly be relieved. If, however, the doctor gives the patient a clean bill of health when, in fact, the patient actually has cancer (i.e., a false negative), the doctor has committed a Type II error. Most people would likely agree that a Type II error would be more serious in this example because it would prevent the patient from getting necessary medical treatment.

You may be wondering why researchers do not simply set up their research studies so that there is even less chance of making a Type I error. For example, wouldn’t it make sense for researchers to set up their research studies so that the chance of making a Type I error is less than 1% or, better yet, 0%? The reason that researchers do not set up their studies in this manner has to do with the relationship between making Type I errors and making Type II errors. Specifically, there is an inverse relationship between Type I errors and Type II errors, which means that by decreasing the probability of making a Type I error, the researcher is increasing the probability of making a Type II error. In other words, if a researcher reduces the probability of making a Type I error from 5% to 1%, there is now an increased probability that the researcher will make a Type II error by failing to detect a difference that actually exists. The 5% level is a standard convention in most fields of research and represents a compromise between making Type I and Type II errors.

CAUTION

Type I Errors vs. Type II Errors

Type I Error (false positive): Concluding there is a difference between the groups being studied when, in fact, there is no difference.

Type II Error (false negative): Concluding there is no difference between the groups being studied when, in fact, there is a difference. Type I and Type II errors can be illustrated using the following table:

Conclusions

After analyzing the data and determining whether to reject the null hypothesis, the researcher is now in a position to draw some conclusions about the results of the study. For example, if the researcher rejected the null hypothesis, the researcher can conclude that the phenomenon being studied had an effect—a statistically significant effect, to be more precise. If the researcher rejects the null hypothesis in our exercise-cholesterol example, the researcher is concluding that exercise had an effect on levels of cholesterol.

It is important that researchers make only those conclusions that can be supported by the data analyses. Going beyond the data is a cardinal sin that researchers must be careful to avoid. For example, if a researcher conducted a correlational study and the results indicated that the two things being studied were strongly related, the researcher could not conclude that one thing caused the other. An