Investigating the Role of Test Methods in Testing Reading Comprehension: A Process-Focused Perspective

By Jufang Kong

This book explores the construct of reading comprehension by means of two main test methods. Research methods like the think aloud protocol and eye tracking are employed to tap into test-takers’ cognitive processes while engaged in input text meaning building, and in test tasks. The book is the first systematic attempt to explore test-takers’ cognitive processes through the control of test methods, and presents findings in visualized form including processing route maps and eye fixation heat maps. It offers readers essential support with “digging into” and analyzing data that has to date remained difficult to access.

• Language: English
• Publisher: Springer
• Release date: Mar 27, 2019
• ISBN: 9789811370212

Book preview

© Springer Nature Singapore Pte Ltd. 2019

Jufang KongInvestigating the Role of Test Methods in Testing Reading Comprehensionhttps://doi.org/10.1007/978-981-13-7021-2_1

1. Introduction

Jufang Kong¹  

(1)

Zhejiang Normal University, Jinhua, China

Jufang Kong

Email: [e-mail]

In this chapter, research background is presented to locate the current study in the broad context of language testing and assessment. The present study centers on the role of test methods in language testing and assessment. Major limitations of the existing relevant studies and practices from a product-focused perspective are identified to justify the necessity of conducting the current study from a process-focused perspective. The rationale of pinning on reading comprehension test response format to materialize test method investigation is put forward, followed by the presentation of the structure of this book and the summary of this chapter.

1.1 Research Background

In recent years, large-scale high-stakes language tests in China have undergone a series of reforms, especially in terms of test methods. Among them, the three most influential ones are Test for English Majors (TEM for short), College English Test (CET for short) and National Matriculation English Test (NMET for short). Since China boasts a huge population of approximately 1.3 billion, such national tests witness a considerate number of test takers, ranging from 400 thousand to 20 million each year. Reforms have been actively carried out and appropriate test methods have been explored in an unswerving effort to better evaluate test takers’ English language competence and to bring about positive consequences in educational context and society. TEM has added short answer question (SAQ for short) response format to multiple choice question (MCQ for short) response format in its reading comprehension section, banished news listening and general knowledge related to linguistics, cultures and literature, and changed the task of writing to reading-and-writing integrative task since 2016. CET has also added banked cloze and matching response formats to original MCQ response format in its reading comprehension section since 2016 although sentence translation, SAQ, True/False/Not Given, sentence completion response formats were once adopted and stepped off the stage of its history. CET has also banished vocabulary and structure section and added translation section in it. NMET has changed from writing based on a title to writing based on a text, a picture or a series of pictures since 2016. A ‘best’ test method has always been explored to balance a large population of test takers (practicality) and test fairness.

The perception of the role of test method is in nature an issue related to construct. Construct is a vital concept in testing and assessment, and the conceptualization of construct is fundamental because it can influence the rational design of a test, the interpretation of test scores, the procedures of validation and the justification of validation results. Despite the overriding importance of construct, agreement is far from being reached on the nature of construct. Theoretical models to underpin tests generally come from the theories of traits to be tested and the evidences to be collected to support test validity are commonly generated through test methods. In other words, what is measured through a test is the result of an interaction between a trait and the method used to test it and the test scores produced are the function of trait and method. Generally, test users are only interested in the trait that test takers possess rather than the function of the trait with a certain method unless the method itself is a prescribed component contained in the construct and the test scores derived from the method shall be interpreted within the context of the very method. Therefore, if test method is not an essential component of a construct and is considered as an undesirable variable, what is ideal in the eyes of test users is to minimize the effect of test method if not possibly to eliminate it completely and observed scores derived from a method can be generalized to universe scores safely regardless the factor of method. Extrapolation to other methods can be made and test scores can be interpreted regardless the factor of method. Otherwise, test method shall be prescribed in construct. Generalization from observed scores to universe scores shall be realized within the context of the given method. Extrapolation to other methods shall be refrained from and interpretations based on the scores shall be made in relation to the given method. Whether test method brings an extra variable to test performance consistency is yet to know and becomes the starting point of the current study.

A clarification of what key terms refer to is essential before discussion in any book. In this book, product-focused perspective refers specifically to the way of thinking and practice which relies mainly on collecting evidences derived from test products, i.e., observed test scores. The term opposite to product-focused perspective is process-focused perspective, which refers to, in this book, the way of thinking and practice which relies mainly on collecting evidences derived from test-taking processes, i.e., cognitive processes involved in test takers’ completion of a test.

In language testing and assessment, researchers tend to adopt a product-focused perspective since test score data are most direct to generate and most easy to observe. Test score data are usually analyzed with statistical methods such as internal or external correlations or factor analysis to establish the nature of the construct to be assessed and to validate the test. Such a comparatively convenient way from a product-focused perspective is far from being impeccable.

The first challenge facing product-focused perspective is that it is difficult to generalize the results derived solely from test score data before establishing validity of a test. The goal of psychometrics, as in a broader view of science, is to develop interpretations that can be generalized to a large context. Messick claims ‘The major concern of validity is not to explain any single isolated event, behavior or item response, because these almost certainly reflect a confounding of multiple determinants … Rather, the intent is to account for consistency in behaviors or item responses … It is important to recognize that the intrusion of context raises issues more of generalization than of interpretive validity’ (Messick, 1989, pp. 14–15). The reality is that test product (score) is the function of trait and the method used to test it and a trait may be tested differently by different methods. If a test taker is assigned different test scores in tests supposed to measure the same trait with different test methods, the difference in test scores can be attributed to the effect exerted by different methods. It is not impossible for test scores to be more of engagement with test method than with trait. In short, test product may be sensitive to the method used to generate it. Analysis based solely on test product is unable to uncover the trait that test developers intend to tap into.

The second challenge facing product-focused perspective is that it is difficult to establish validity with statistical methods. Unlike reliability, validity is difficult to achieve in a posteriori fashion through relying heavily on statistical methods to track back from the product to the process that gives rise to it. Results of statistical analyses are sensitive to the number and the type of variables involved as well as the specific analytic techniques used in studies. If two variables show substantial correlations or load on the same factor, there are chances that such a result is caused by similar stimulus characteristics rather than a similar underlying process. Hume put it in a philosophical way: ‘There are some causes, which are entirely uniform and constant in producing a particular effect; and no instance has ever yet been found of any failure or irregularity in their operation. But where different effects have been found to follow from causes, which are to appearance exactly similar, all these various effects must enter into consideration when determining the probability of the event’ (Hume, 1999, p. 132).

It has also been found in empirical studies that a correct answer to a test item can be based on a wrong hypothesis and an incorrect answer can be based on a correct hypothesis (Cohen, 1984, p. 76). A test is designed to assess what it intends to assess based on an assumption that if a test taker possesses a trait X, then he/she will achieve a test score Y. Nevertheless, if Y is achieved, the challenging thing is how confident it can be claimed to be achieved because of X. It is weak to establish a real understanding of what a specific test measures only through statistical methods.

Unless the nature is better understood of test takers’ test-taking processes under the conditions of different test methods, neither generalization nor tracking back from product to process can be examined in an appropriate manner. Only when test method is investigated from a process-focused perspective can a test claim with confidence that the intentions of the test correspond well with the functions of the test and that the test assesses definitively what it intends to assess.

A traditional product-focused perspective is inadequate to establish validity of a test. Studies from a process-focused perspective are called for and the call has been answered by a few researchers who recognize the necessity of collecting evidences derived from test-taking processes. Vollmer claimed ‘we will never really understand what correlations between tests of different skills mean, what they signify, and why some are higher than other unless we better understand and are able to model more precisely the cognitive potentials and task specific operations on which performance in the various language tests depends’ (Vollmer, 1981, p. 167). Alderson also argued ‘The validity of a test relates to the interpretation of the correct responses to items, so what matters is not what the test constructors believe an item to be testing, but which responses are considered correct, and what process underlies them’ (Alderson, 2000, p. 97).

An understanding of the trait being measured requires an insight into the cognitive processes required of test takers for completing the tasks in a test. The evidences obtained on specific mental processes involved in test takers’ arriving at test products are expected to be combined with those of test products to lead to a more definite conclusion about the construct. In addition, comparison should be made between observed or reported behaviours of test takers (including the mental processes elicited) when completing the test tasks with a test method and those with another test method to examine the possibility of generalization from observed scores to universe scores and from observed processes to universe processes, otherwise nowhere can extrapolation rest.

1.2 Research Focus

Test method effect investigation cannot be conducted abstractly with no concrete objects to target at. It must be materialized through specific trait and method. This book attempts to pin its trait and method on reading comprehension competence and response format respectively. To be more specific, the focus of the current study is to investigate the role of test method in testing reading comprehension competence through a mixed-method examination of the nature of test takers’ reading comprehension and task completion processes.

Reading comprehension competence is selected as the target trait because it is the fundamental competence for a person’s survival in his/her daily life and it has witnessed most productive research findings in the academic circle.

Reading comprehension competence is an outstanding achievement of human brain, a foundational and essential skill for academic achievement in school and success in life and career. Vast majority of the information that people obtain every day is from reading activities. Naturally, an accurate testing of such an essential competence has a great influence on all the stakeholders concerned. Indeed, how to assess reading comprehension competence has always been the concern of language testing and assessment researchers and practitioners. However, the testing of reading comprehension competence is by no means an easy task. It has been one of the most controversial issues in language testing and assessment for its lack of an overt process to be observed directly, unlike writing or speaking competence which produce a concrete performance of test takers for researchers to examine all kinds of indicators of it carefully. The traditional way to investigate validity of a reading comprehension test is to observe the response of a test taker which is elicited through an indirect test method and make an inference about the nature of his/her reading comprehension processes and whether he/she possesses the target reading comprehension competence or not. The logic of indirect inferring from product (result) to process (reason) is fallible and a conclusion drawn from a statistical analysis is not always reliable. Therefore, it is high time to visit validity issue of reading comprehension tests from a process-focused perspective to explore the possibility of discovering new and valuable findings.

Research on reading comprehension competence started much earlier than that on writing, speaking or listening competence and we believe that we are comparatively better equipped with the knowledge of the nature of reading comprehension competence. A much-studied field rather than a less explored one is preferred since it is believed to be more meaningful to compare potentially new findings obtained through a new instrument or method with the old and time-tested findings obtained through old ones. It is also more likely to resort to one among a great number of existing theories or models in a much-studied field to account for unusual phenomena in case mixed results occur unexpectedly.

Response format is identified as a representative component of test method for the concerns with potential impact(s) of response format on test takers’ test performances and the present trend of exploring new response formats in language testing and assessment practices. In a sense, developing reading comprehension tests is developing the best possible artifacts or products from which inferences are drawn concerning what must have been going on in test takers’ reading comprehension activities. The real thing—comprehension, even the smallest chunk of comprehension, cannot be observed directly, as it takes place online during the process of reading activities. The common practice is to infer from distant indices, including test methods. The challenging thing facing test developers is which test method is the best index to validate the assumption about what the test really measures.

Many modern methods adopted in reading comprehension tests are of one dimension, with relatively narrow scope of response formats, which do not vary within a given test. The drive for high reliability, especially in high-stakes national tests, often leads to significant restrictions on test methods. However, the question is whether two reading comprehension tests with the same input to test takers but in different response formats are in fact assessing the same construct. In the past, a wide range of empirical studies investigating test method effect(s) have been carried out. Up till now, a few introspective approaches have been tried out. Therefore, it is worth investigating test method effect(s) on test takers’ reading comprehension test performances from a process-focused perspective through introspective and observational approaches.

1.3 Book Structure

This book consists of two parts. The first five chapters are the first part of the book, serving to lay the theoretical foundation for the empirical investigation in the second part. The rest four chapters, the second part of the book, concern the empirical efforts made to explore the role of test methods in testing reading comprehension.

This chapter started with the recent reforms of large-scale high-stakes tests in China. Then the chapter outlined the challenges facing validity study from a product-focused perspective followed by an argument for the necessity of validating from a process-focused perspective. Based on the discussion, the chapter presented its research focus—test response format effect(s) in reading comprehension tests.

Chapter 2 approaches the key concept of reading comprehension from different perspectives. In this chapter, several influential reading theories and models are given brief discussion to. Then the chapter continues to examine the various factors that are thought to affect reading comprehension process, including reader, text and the interaction between reader and text. The chapter ends with Khalifa and Weir’s model of reading comprehension based on which the current empirical study is carried out and the summary of this chapter.

Chapter 3 discusses the concepts of construct, validity and validation. First, different views of construct are presented. Then the chapter focuses on the concept of validity and various theoretical underpinnings of test validation, including Messick’s validation framework, Kane’s interpretive argument framework, Bachman and Palmer’s assessment use argument (AUA for short) framework as well as multitrait-multimethod (MTMM for short) approach. The chapter ends with the summary of the chapter.

Chapter 4 presents and identifies the key concepts of test method and response format, also providing a theoretical foundation for the empirical study in the second part of the book. Then two specific response formats are reviewed with their respective strengths and weaknesses, i.e., MCQ response format and SAQ response format, the two most widely used response formats in reading comprehension tests. The chapter ends with the summary of the chapter.

Chapter 5 first explains and discusses three research approaches: quantitative, qualitative and mixed methods. Then the chapter put forwards the rationale of adopting mixed methods in the current study and the philosophy behind it. This chapter ends with the research design adopted in this study and the summary of the chapter.

Chapter 6 is devoted to the quantitative method in the study. In this chapter, research questions are first brought up to serve as a concrete and visible goal to reach in the current study. Then a detailed research design is discussed, including research procedures, participants and research instruments. The chapter ends with the findings tapped into through quantitative method and the summary of the chapter.

Chapter 7 operationalizes a process-focused investigation of potential effect(s) of response formats on test takers’ reading comprehension test-taking processes through think aloud protocols (TAPs for short). The tool of TAPs is justified with its advantages and limitations discussed and taken into consideration. Then a detailed research design is discussed and the findings are presented. The chapter ends with the summary of the chapter.

Chapter 8 operationalizes a process-focused investigation of potential effect(s) of response formats on test takers’ reading comprehension test-taking processes through eye tracking (ET for short). The tool of ET is justified with its strengths and weakness carefully considered. Then a detailed research design is discussed and the findings are presented. The chapter ends with the summary of the chapter.

Chapter 9 is the final and concluding chapter of the book. First, major findings of the empirical research carried out through mixed methods are presented, triangulated and discussed. Then the significance of this study and its theoretical and practical implications are highlighted. Its limitations are also presented and orientation for future studies is identified. The chapter ends with the summary of the chapter.

1.4 Summary

In this chapter, the reforms of the large-scale high-stakes language tests in China were cited as the broad context to house this study. The reforms had been mainly around the test methods employed in these national tests. Then the chapter pointed out the major limitations of the existing relevant studies and practices from a product-focused perspective and put forward the necessity of conducting studies from a process-focused perspective. The rationale of pinning on reading comprehension test response format to materialize test method investigation was put forward, followed by the presentation of the structure of this book.

References

Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.Crossref

Cohen, A. D. (1984). On taking language tests: What the students report. Language Testing,1(1), 70–81.Crossref

Hume, D. (1999). An enquiry concerning human understanding. Oxford: Oxford University Press.

Messick, S. A. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education/Macmillan.

Vollmer, H. J. (1981). Why are we interested in ‘general language proficiency’? In J. C. Alderson, & A. Hughes (Eds.), Issues in language testing (pp. 152–175). London: The British Council.

© Springer Nature Singapore Pte Ltd. 2019

Jufang KongInvestigating the Role of Test Methods in Testing Reading Comprehensionhttps://doi.org/10.1007/978-981-13-7021-2_2

2. Theories of Reading Comprehension

Jufang Kong¹  

(1)

Zhejiang Normal University, Jinhua, China

Jufang Kong

Email: [e-mail]

‘We are like sailors who on the open sea must reconstruct their ship but are never able to start afresh from the bottom. Where a beam is taken away a new one must at once be put there, and for this the rest of the ship is used as support. In this way, by using the old beams and driftwood the ship can be shaped entirely anew, but only by gradual reconstruction’ (Neurath & Cohen, 1973, p. 199). A research is by no means entirely new judged from every aspect but gradually built upon old knowledge. The following four chapters serve as a review of the important concepts of reading comprehension, validity, test method and mixed research methods in literature. This chapter mainly discusses the key theories of reading comprehension. After several influential theories and models of reading comprehension are outlined, the chapter examines the factors affecting reading comprehension process, namely, reader, text and the interaction between reader and text. The chapter ends with Khalifa and Weir’s model of reading comprehension based on which the current empirical study was carried out and the summary of this chapter.

2.1 Nature of Reading Comprehension

Reading comprehension has been studied with a long history and several influential theories and models have been put forward in the past decades. Reading comprehension involves transforming a text (or a graphic representation) into thought or meaning. It is a process of simultaneously extracting and constructing meaning through an interaction with written text. One of the most controversial discussions concerning reading comprehension is the componentiality of reading comprehension, or in other words, whether reading comprehension can be broken down into underlying skills or strategies? (Weir, Yang & Jin, 2000). Weir, Yang and Jin (2000) supported a multi-componentiality view of reading comprehension and explained that a uni-componentiality view is often arrived at through factor analyses of reading comprehension test scores. Factor analysis is a manner of reduction and tends to show apparently different reading comprehension skills behaving in a similar statistical way, insensitive to the subtle differences among related reading comprehension skills and processes. More process-oriented studies suggested that reading comprehension has more than one dimension. Coady (1979) proposed a three-component (conceptual ability, language proficiency and  background knowledge) model of reading comprehension. Bernhardt’s (1991) model also consisted of three components, but in different names, i.e., language, literacy and knowledge.

The number of theories of reading comprehension is overwhelming and an overview of the studies of the nature of reading comprehension is almost impossible (Alderson, 2000, p. 1). Koda (2005) believed that a successful identification of specific components that contribute to reading comprehension is an important paradigm in current reading literature. A componentiality view sitting squarely into a sound theory of processing can be useful in that it provides insight into the potential components in reading comprehension which require our attention. Since it is difficult to identify and exhaust the components involved in the process of reading comprehension and it is probably more difficult to make clear the stages and the structures of reading comprehension process, the review of the relevant theories in this chapter is selective rather than exhaustive. The theories of reading comprehension are outlined in line with Zou’s (2004) categorization, namely, psycholinguistic perspective, cognitive perspective and processing patterns.

2.1.1 Psycholinguistic Perspective of Reading Comprehension

The nature of reading comprehension can be investigated from a psycholinguistic perspective.

The psycholinguistic perspective of reading comprehension can date back to the late 1960s when Goodman (1967) proposed the notion of reading comprehension as a ‘psycholinguistic guessing game’. He showed much interest in reading research for decades of years and was one of the most productive scholars in the field. He argued that reading comprehension is ‘a psycholinguistic process in that it starts with a linguistic surface representation encoded by a writer and ends with meaning which reader constructs’ (Goodman, 1988, p. 12). In other words, the writer encodes thought into language while the reader decodes language into thought (Zou, 2004, p. 30). According to Goodman, in the process of reading comprehension, the reader goes through four cycles to translate linguistic representation (text) into thought (meaning), that is, optical, perceptual, syntactic and meaning cycles. The optical cycle perceives symbols; the perceptual cycle makes sense of symbols; the syntactic cycle links symbols together to build up complex information and the meaning cycle constructs meaning while reading. The four cycles are interlocked and emerge sequentially. Each of the four cycles involves five processes, namely, recognition-initiation (recognizing text and initiating reading), prediction (anticipating and predicting while seeking order and the significance of the input), confirmation (verifying or disconfirming predictions), correction (reprocessing while finding inconsistencies) and termination (ending reading activity) (Goodman, 1988, p. 16). In Goodman’s view, reading comprehension is an active, selective and interactive process. Goodman’s theory of reading comprehension emphasizes the interaction between language and thought in the process of reading comprehension.

Smith is another representative scholar supporting the psycholinguistic perspective of reading comprehension. He claimed that reading comprehension is a process of ‘getting questions answered’ (Smith, 1978, p. 102) or of attempting to reconstruct meaning through asking specific questions and getting answers from the printed text. The questions asked imply predictions made concerning what is going to be read next. The reader employs the knowledge pre-stored in