Applications of Methods of Evaluation: Four Studies of the Encampment for Citizenship

By Herbert H. Hyman, Charles R. Wright and Terence K. Hopkins

This title is part of UC Press's Voices Revived program, which commemorates University of California Press’s mission to seek out and cultivate the brightest minds and give them voice, reach, and impact. Drawing on a backlist dating to 1893, Voices Revived makes high-quality, peer-reviewed scholarship accessible once again using print-on-demand technology. This title was originally published in 1962.

• Language: English
• Publisher: University of California Press
• Release date: Nov 15, 2023
• ISBN: 9780520321908

Herbert H. Hyman

Enter the Author Bio(s) here.

Book preview

APPLICATIONS OF METHODS OF EVALUATION

Four Studies of the

Encampment for Citizenship

APPLICATIONS OF

METHODS OF EVALUATION

Four Studies of the Encampment for Citizenship

BY

HERBERT H. HYMAN, CHARLES R. WRIGHT,

and TERENCE K. HOPKINS

UNIVERSITY OF CALIFORNIA PRESS

BERKELEY AND LOS ANGELES

1962

University of California Publications in Culture and Society

Editors: W. A. LESSA, Richard Centers, R. T. Morris, R. H. Turner

VOLUME 7

University of California Press

Berkeley and LOS Angeles

California

Cambridge University Press

London, England

Cambridge University Press

London, England

© ig62 BY THE REGENTS OF THE UNIVERSITY OF CALIFORNIA

PRINTED IN THE UNITED STATES OF AMERICA

PREFACE

IN A DISCUSSION of action research, Kurt Lewin once commented that research that produces nothing but books will not suffice. Yet, here we are about to present a book on action research. It is not because we reject the good advice. Lewin’s remark was in the context of urging research that would serve the goals of rational social action, and he stressed the need to accompany planning and action with evaluation of their effectiveness. The evaluation is the objective measure of success or failure, the source of new knowledge, and the guide to improved action. This is what research should do, not merely produce books. To encourage the alliance, to join the two—evaluation and action—is not easy. But it is equally difficult to develop the principles, methods, and procedures of evaluation research. It is research that must be done under difficult conditions, and upon concepts that are generally obscure, confused, or complex. Any book that would serve to clarify the problems in such research and develop the method certainly works in the service of larger action goals, and it is in this spirit that we present it.

Our book is a case study of research conducted in the years 1955—1959 to evaluate a particular social action program, the Encampment for Citizenship. The many ventures in character education of youth may well profit from the example to be presented. Beyond this, the method of evaluation will have formal similarities, no matter what type of program is involved. We were fortunate in having extended opportunity over these years to elaborate instruments and procedures, to see whether they stood the test of time and repeated application, to compare different designs and solutions for particular problems. The presentation of all this in great detail may be of value in furthering the ideal of evaluation of planned programs of social action, and in improving the quality of evaluations in practice.

Our appreciation is expressed to the Encampment for Citizenship and its staff for their invitation to conduct the research. We must note our admiration for their courage in trusting a venture that was so dear to their hearts to appraisal by so cold and heartless a method, and we thank them for unstinting aid without interference. The special cooperation of Algernon D. Black, education director; William G. Shannon, former executive director; and Saal D. Lesser, the present executive director, is noted with thanks.

Support for the research was provided by a grant from the Schwarzhaupt Foundation, plus grants-in-aid from the Ford Foundation, and the Columbia Council for Research in the Social Sciences.

Research assistance by Roslyn Menzel, Carolyn Dexter, and Sanci and John Michael is gratefully acknowledged.

We are grateful to Dr. Henry Riecken, who contributed unpublished material from his evaluation of volunteer work camps.

All the authors were members of the Bureau of Applied Social Research, Columbia University. In 1956 Professor Wright joined the faculty of the University of California at Los Angeles, but he continued his participation as one of the codirectors of the research. The research facilities of the Bureau, technical advice by its staff, administrative help, and support in preparation of the manuscript contributed much to our work.

±1. ±1. M.

C. R. W.

T. K. H.

CONTENTS 1

CONTENTS 1

CHAPTER I

INTRODUCTION

CONCEPTUALIZATION AND MEASUREMENT OF THE OBJECTIVES OF THE PROGRAM AND OTHER UNANTICIPATED RELEVANT OUTCOMES

CONCEPTUALIZING THE OBJECTIVES OF THE PROGRAM

CONCEPTUALIZING UNANTICIPATED CONSEQUENCES

RESEARCH DESIGN AND THE PROOF OF THE EFFECTIVENESS OF A PROGRAM

CONTROLLING FOR EXTRANEOUS SOURCES OF CHANGE BY EXPERIMENTAL DESIGNS

CONTROLLING FOR EFFECT OF REPEATED TESTING

WORKABLE ALTERNATIVES TO THE CLASSIC CONTROL-GROUP DESIGN

RESEARCH PROCEDURES AND THE REDUCTION OF ERROR

QUALITY OF RESPONSE

BIASES RESULTING FROM NONRESPONSE

PROBLEMS OF INDEX CONSTRUCTION AND THE PROPER EVALUATION OF EFFECTIVENESS

WEIGHING EFFECTIVENESS IN THE LIGHT OF RESTRICTED CEILINGS FOR CHANGE

WEIGHING EFFECTIVENESS IN TERMS OF INDIVIDUAL CHANGES

WEIGHING EFFECTIVENESS BY COMBINING DISCRETE ASPECTS OF CHANGE

WEIGHING THE AMOUNT OF EFFECTIVENESS AND TESTS OF SIGNIFICANCE

UNDERSTANDING THE FINDINGS ON EFFECTIVENESS

DESCRIBING THE PROGRAM

DESCRIBING THE SUBJECTS

DIFFERENTIAL EFFECTS AMONG CONTRASTED TYPES OF SUBJECTS

THE CONTRIBUTION OF DIFFERENT ASPECTS OF THE PROGRAM

INQUIRY INTO THE PROCESSES BY WHICH THE PROGRAM PRODUCES EFFECTS

CHAPTER II

INTRODUCTION

DIVERSITY WITHIN THE TOTAL ENCAMPMENT

A PORTRAIT OF THE SUBGROUPS

CAMPERS’ SOCIAL VALUES, ATTITUDES, AND OPINIONS

SOCIAL VALUES ON THE FIRST DAY OF CAMP

CAMPERS’ CONCEPTIONS OF SOCIAL PROBLEMS

INITIAL OPINIONS AND ATTITUDES ON SOCIAL ISSUES

THE CAMPER AND THE REST OF SOCIETY

CHAPTER III

INTRODUCTION

THE NATURE OF THE ENCAMPMENT

THE EDUCATIONAL CURRICULUM

THE COMMUNAL LIFE

THE SUMMER AS SEEN BY THE CAMPERS

EXPECTATIONS, FULFILLMENTS, AND DISAPPOINTMENTS

REACTIONS TO THE FORMAL PROGRAM

REACTIONS TO THE NONDIDACTIC FEATURES OF THE ENCAMPMENT

DISRUPTIVE ELEMENTS DURING THE ENCAMPMENT AND SOME OF THEIR DETERMINANTS

SALIENT ELEMENTS IN THE ENCAMPMENT FROM THE CAMPERS’ VIEW

CAMPERS’ SELF-ESTIMATES OF CHANGE

CHAPTER IV

INTRODUCTION

REGIONS OF CHANGE

CHANGES IN SALIENT SOCIAL ATTITUDES AND OPINIONS

CHANGES IN BASIC VALUES

CHANGES IN ACTION ORIENTATION

CHANGES IN COGNITION OF SOCIAL PROBLEMS

CHANGES IN CAMPERS’ PERCEIVED RELATIONSHIP WITH THE REST OF SOCIETY

DEMOCRATIC CONDUCT DURING THE SUMMER

SUSCEPTIBILITY TO PRESTIGE SYMBOLS

AN EXPERIMENT IN RESISTANCE TO PROPAGANDA

WOULD THERE BE IMPROVEMENT WITHOUT AN ENCAMPMENT?

SUMMARY INTERPRETATION

CHAPTER V

INTRODUCTION

EXPLORING FACTORS RESPONSIBLE FOR CHANGES IN CAMPERS’ ATTITUDES DURING THE ENCAMPMENT—DIDACTIC AND COMMUNAL ASPECTS

THE CONTRIBUTION OF OTHER FACTORS DISRUPTIVE EXPERIENCES AND CHANGES

RACE AND ETHNICITY AS FACTORS GOVERNING CHANGE

THE ECOLOGY OF THE CALIFORNIA ENCAMPMENT: PRELIMINARY EXPLORATION

PROCESSES UNDERLYING CHANGES DURING ENCAMPMENT

CHAPTER VI

INTRODUCTION

THE STUDY OF THE 1955 GROUP

PERCEPTION OF THE HOME COMMUNITY

INFORMAL RELATIONSHIPS AFTER THE RETURN HOME

ATTEMPTS TO APPLY ENCAMPMENT PRINCIPLES WITHIN THE HOME SETTING

SUBJECTIVE REPORTS OF CHANGES AFTER THE ENCAMPMENT

CHAPTER VII STABILITY OF EFFECTS WITHIN THE HOME SETTING AND ITS DETERMINANTS INTRODUCTION

REGIONS OF CHANGE SALIENT SOCIAL OPINIONS AND ATTITUDES WITHIN THE HOME SETTING

ACTION ORIENTATION IN THE HOME SETTING

COGNITION OF SOCIAL PROBLEMS

PERCEIVED RELATIONSHIP WITH THE REST OF SOCIETY

DIFFERENTIAL PATTERNS OF STABILITY AS RELATED TO ETHNICITY AND SOCIAL SETTING

STABILITY OF EFFECTS AS RELATED TO SOCIAL SUPPORT

CHAPTER VIII

INTRODUCTION

COLLEGE AND THE ENCAMPMENT—THE GROUPS STUDIED

THE COLLEGE SUBGROUP

THE CONTROL GROUP

THE FINDINGS

CAMPUS POLITICAL LIFE

ACADEMIC LIFE

THE VALUE OF A COLLEGE EDUCATION

CONCLUSION

CHAPTER IX

INTRODUCTION

THE QUALITY OF THE EVIDENCE

THE OLD ENCAMPMENT—FORGOTTEN EPISODE OR LIVING MEMORY?

THE INTERVENING YEARS

REGIONS OF CHANGE SALIENT SOCIAL OPINIONS AND ATTITUDES—FOUR YEARS LATER

CHANGES IN COGNITION OF SOCIAL PROBLEMS

ALIENATION AND ANOMIE

LONG-RANGE IMPACT ON CONDUCT

CHANGE AS AN ABRUPT OR EROSIVE PHENOMENON

STABILITY AND CHANGE BY RACE, SOCIAL CONTEXT, AND HISTORY

STABILITY IN RELATION TO POST-ENCAMPMENT CONTACTS AND REFERENCE GROUPS

CHAPTER X

INTRODUCTION

PROCEDURAL DIFFICULTIES AND ERROR RESPONSE ERROR

SAMPLING PROBLEMS AND NONRESPONSE

THE FINDINGS AND THEIR AGREEMENT WITH THE 1955 FOLLOW-UP

ATTITUDE SCALES USED IN MEASURING EFFECTS OF ENCAMPMENT FOR CITIZENSHIP

QUESTIONNAIRES

THE USE OF AN INDEX OF EFFECTIVENESS TO CORRECT FOR CEILING EFFECTS

TURNOVER TABLES AND TESTS OF SIGNIFICANCE

INDEX OF NAMES

CHAPTER I

PRINCIPLES OF EVALUATION

INTRODUCTION

THROUGHOUT HISTORY, men have pooled their energies to change some state of affairs and to achieve some common social goal. Some of these efforts succeed, but many fail. Usually, only the historical record documents whether the effort was a success or a failure. Sometimes the lessons of history are swift and dramatic and clear in their meaning. We may learn the hard way, but, nevertheless, we learn that an action is ineffective. At other times, history is not a very good teacher. The verdict of success or failure may come too slowly, after much energy has been wasted and our follies have gone too far. Disaster is not a teacher to be welcomed. Even when the scale of an action program is minor, economy is to be welcomed. At still other times, the lesson of history is not so clear. The outcome of a course of action may be ambiguous; its meaning may be complicated and perhaps misleading. Is there not a more efficient and intelligent guide to evaluating social action?

The question becomes all the more compelling as the scale of efforts is enlarged. Ours is an age of action programs, where large organization and huge expenditures go into the attempted solution of every conceivable social problem. As a small assortment of these, but not in order of importance, consider programs of social casework; criminal rehabilitation; treatment of delinquents; dissemination of opinions and knowledge on a national and an international scale; training of workers, foremen, supervisors, and executives; adult education; health information and practices; technological assistance; cultural exchange; reduction of intergroup conflict; mental hygiene; general education; adjustment of the aged; training of soldiers; and treatment of alcoholism.1

There is all the more need today for rational management of such enterprises, for a spiral, to use Lewin’s figure, of steps each of which is composed of a circle of planning, action, and fact-finding about the result of the action.2 Evaluation refers to the procedures of fact finding about the results of planned social action, which in turn move the spiral of planning ever upward. It is the proper methodological accompaniment to rational action. Done well, the accompaniment is almost musical in its appeal. Men have, of course, always attempted to find out the facts about the results of their actions. The theme has always been with us; it is the performance that is different. As Kline- berg remarks, evaluation should be restricted to fact-finding methods which yield evidence that is objective, systematic and comprehensive, and should be distinguished from other forms of assessment which do not satisfy these criteria.⁸

The problem is to develop the general method which satisfies these criteria, and to abstract the principles that underlie the method. But more than this is needed. The principles and the method must take on a specialized form suited to the particular circumstances of the action program to be evaluated. Effective application of the principles to the particular case, special modifications of procedure within the general framework—this is what must be learned. It is to this end that our work is presented. It consists of two parts, a general statement of principles of evaluation, and a detailed exemplification of the principles in a series of evaluative studies conducted in one action program. Even if the program studied were a rare and exotic one, the presentation would still have the virtue of showing the translation of the method into operative research. From one point of view, however, the program is one of a large class. In their quest for a better world, many agencies put their faith in youth, and a variety of programs designed for young persons are in operation. Riecken notes that one special variety, programs of work service, accounts for more than one hundred projects annually in the United States, sponsored by more than thirty different organizations. Other types of youth programs abound, all intended to promote growth, wisdom, competence, character development. The Encampment for Citizenship—EFC—takes on its own special form, but is in the spirit of this larger general endeavor. It brings together each summer about a hundred young persons from many parts of the United States, diverse in creed and race, but united in their dedication to a better world, their strong and serious interest in social problems, and their potentialities for future leadership roles in our society. For six weeks these young persons live together in a community permeated, in principle, by an atmosphere of equality, presumably enriching their experiences, and being stimulated by a varied educational program of lectures, workshops, discussion groups, and field trips, all designed to increase skills in democratic living and inculcate values apposite to an ideal world.

Each such Encampment is regarded as a pilot project in democratic education to convey the facts that one hundred persons are but a tiny

30. Klineberg, «The Problem of Evaluation, Internat. Soc. Sci. Bull., VII (1955), 347.

*H. W. Riecken, The Volunteer Work Camp: A Psychological Evaluation (Cambridge: Addison-Wesley Press, 1952), p. 1.

army in the service of so great a cause, that the ultimate testing ground is the remote future, and, finally, that this is an ongoing endeavor— imperfect, perhaps, but changing by benefit of practice in such a difficult undertaking. Over fifteen summers have now passed since the initiation of this experiment. More than fifteen hundred young men and women have come into the ranks of the Encampment and have returned to their respective homes.

In 1955, in preparation for the tenth Encampment, the sponsors felt the time had come for a scientific evaluation to be undertaken. The responsibility for this evaluation was entrusted to the Bureau of Applied Social Research of Columbia University, and was assigned to the authors, who enjoyed complete freedom of inquiry.

The evaluation of the 1955 program was the beginning of a long sequence of studies. In 1957 and 1958 the program was again evaluated, making a total of four studies, because in 1958 two similar Encampments were in operation. The persistence of effects was subject to a number of different studies. The 1955 campers were sent questionnaires at their home communities six weeks after their departure from the Encampment and were restudied some four years later. The experiences of alumni of Encampments before 1955 were reconstructed by a special method, thus providing evidence on campers removed from exposure for periods as long as nine years. These many studies in evaluation provide the case material to be presented in subsequent chapters.

The technical features of such inquiries are exceedingly complex, and it was natural that a specialized research organization should be requested to undertake the task. Further, the requirements of objectivity and impartiality demanded that outsiders conduct the inquiry without any constraint upon their activities or conclusions. The complex nature of evaluation, however, requires the participation of the agency which is open to scrutiny. Whereas the researchers are best able to develop scientific methods for evaluating effectiveness, the ultimate goals to be achieved are defined by the action agency. The wisdom of these goals cannot be ascertained by scientific inquiry. The achievement of goals is measured indirectly by translating them into a series of measurable, component, or subsidiary objectives which in totality combine to represent the larger goals. This translation was primarily the responsibility of the research workers, after consultation with the officials of the Encampment.

All the studies address themselves to the prime problem in evaluation: to provide objective, systematic, and comprehensive evidence on the degree to which the program achieves its intended objectives plus the degree to which it produces other unanticipated consequences, which when recognized would also be regarded as relevant to the agency. It is easy to state the problem of evaluation, but it is difficult to develop the method for its solution. The method, in our formulation, has five major phases, each of which involves a body of principles. The translation of the phases and of the abstract principles into actual research will be presented in fullest form in subsequent chapters. Frequent reference to our studies and to other literature will be incorporated into the present discussion of the methodology of evaluation research. Only in this way, by its utility and applicability to a real problem, can the appropriateness of the method be demonstrated and real understanding be fostered.

1 Adapted from an unpublished memorandum by Dr. Henry Riecken on the uses of evaluation.

2 Kurt Lewin, Resolving Social Conflicts (New York: Harper, 1948), p. 206.

CONCEPTUALIZATION AND MEASUREMENT OF THE OBJECTIVES OF THE

PROGRAM AND OTHER UNANTICIPATED RELEVANT OUTCOMES

Although there are basic principles and specific rules to guide all five phases of evaluation, the obscurities in the phase of conceptualization defy the principles most of all. The design of proof is a classic problem in science; the properties of good measuring instruments are clearly stated; the corresponding procedures are laid out and the knowledge of the past has been codified. In the four other phases the evaluator, like any investigator, follows a well-marked trail. By contrast, conceptualizing any phenomenon in the prelude to research is an act of mind, of the imagination. How does one codify principles and rules for effective use of imagination? There are some useful guides, to be sure, but our ultimate hopes should not be too high. In scientific inquiry where the investigator chooses what he wants to study, the same difficulties are faced, and the counsel provided is not very encouraging: No rules can be stated for ‘hitting upon’ relevant hypotheses.1 In a recent work on the methodology of social research, which places special emphasis on the phase of concepts and indices and the appropriate formal procedures, Lazarsfeld and Rosenberg ask the basic question of how units and variables are formed, and they describe the beginning in terms that emphasize the difficulty. The first step seems to be the creation of a rather vague image. They ask, How does one ‘think up’ a number of indicators? They talk of this speculative phase of the task.⁶ The evaluator is in the same fix as any curious investigator, with one exception that makes his task both easier and harder. The ordinary investigator perhaps has to be more imaginative, as he originates his inquiry and does not have the benefit of the hints to conceptualization inherent in the intended objectives of the program. But if it turns out that he is not imaginative in conceptualizing his problem, he can always drop it and turn to something else. The evaluator is guided by the objectives set down, but at the same time his imagination is disciplined, perhaps harshly, by the requirement to conceptualize in manageable terms the preestablished objectives of the program. The cosmic way in which many social action programs state their objectives makes the task no easy one, but the evaluator cannot escape it and turn to more manageable problems. Consider the gifts of mind needed to conceptualize for evaluation a program of adult education which has the defined aim of development … of personal abilities and aptitudes, and the encouragement of social, moral and intellectual responsibility within the framework of local, national and world citizenship. Riecken, when called upon to evaluate the work-camp program of the American Friends Service Committee, experienced similar ambiguities, plus some additional ones. After studying the documents describing the program, he reported: The grave difficulty is that we have been unable to discover in these writings a simple, clearly and comprehensively stated set of aims that will meet with the universal endorsement of the directors of the program.2

The many difficulties suggested—the breadth of the thing subsumed under a particular objective, the multiple objectives encompassed by many programs, the ambiguity inherent in any or all of the objectives as stated, and the disagreement as to the objectives—are characteristic of many programs and are enough to stagger the imagination of the evaluator. Even when a program has a seemingly simple, unambiguous, unitary objective, the ease of conceptualization and subsequent measurement is often deceptive. Alternative definitions of the program’s objective come to mind. Meyer and Borgatta, in evaluating a program designed to rehabilitate the mentally ill, note that although the most general and most objective criterion of success for a person who has been hospitalized for mental illness is his permanent success in avoiding reinstitutionalization, lesser as well as greater degrees of ‘health’ or ‘adjustment’ suggest themselves as appropriate criteria if measures can be determined for them. Neglecting for the moment the snare confronting the evaluator in the innocent word permanent, they list some of the alternative definitions of success: "A second measure of recovery may be defined in terms of the ability to govern one’s emotional affairs

TE. M. Hutchinson, quoted in K, M. Miller, Evaluation in Adult Education, Internal. Soc. Sci. Bull., VII (1955), 431.

independently. … A third measure of recovery may be the inclusiveness of a person’s social relations. … A fourth measure of recovery may be economic independence, and a fifth the reality of general orientation. … A sixth measure of recovery may be one based on self-attitudes."3

These many kinds of problems confronted us. The Encampment states that its goals are to prepare young Americans for responsible citizenship and citizen leadership, to educate them in the meaning of democracy… and to train and equip them in the techniques of democratic action. Elsewhere, a sampling of goals includes training in freedom with responsibility, the provision of information and clarification on current issues, the reduction of confusion, helplessness, and apathy.4

In order to evaluate the effectiveness of the Encampment, it is a prerequisite that such objectives be subject to measurement. But to subject them to measurement we must first be able to translate these profoundly important but nevertheless broad and abstract statements into a series of simpler concepts. What is subsumed under responsible citizenship? Under techniques of democratic action and freedom with responsibility? The series of simpler concepts will then be employed to measure the original objectives.

The difficulties are still not done with. What about the unanticipated relevant consequences that might stem from the program? The camper might learn the meaning of democracy, but in the process of boiling down the creed his understanding might be reduced to a provincial or, as we subsequently labeled it, ethnocentric image of democracy, allowing for no respect of other forms of the democratic spirit. Suppose his apathy is converted not merely into activity, but rather into a hotheaded and premature activism. Suppose the sense of helplessness disappears, but is replaced by an overoptimistic sense of the ease of social change. Suppose leadership emerges at the price of the formation of a leadership elite which feel alienated from and superior to the mass of people. The problems of evaluation of character education are complicated by the fact that there are no watertight compartments within the character. A change is initiated which may know no bounds, and good and bad consequences may run side by side. Consequently, it would be most misleading if we were merely to evaluate objectives specifically sought by the Encampment without regard for other consequences, toward or untoward. But herein lies a special difficulty: How shall the evaluator anticipate the consequences of social action unanticipated by others?

Granted that certain broad and abstract objectives of the program must be translated into a much larger number of specific concepts, which are then supplemented by other concepts related to unanticipated outcomes, how shall this be done? The principles we shall advance will be of some help in harnessing imagination; but, above all, the gifted imagination is needed.

1 M. R. Cohen and E. Nagel, An Introduction to Logic and Scientific Method (New York: Harcourt, Brace, 1934), p. 202.

•P. F. Lazarsfeld and M. Rosenberg, eds., The Language of Social Research (Glencoe, Ill.: Free Press, 1955), pp. 15-16.

2 Op. cit., p. 27.

3 H. J. Meyer and E. F. Borgatta, An Experiment in Mental Patient Rehabilitation (New York: Russell Sage Foundation, 1959), pp. 26-27.

4 Encampment for Citizenship, official brochure (privately printed, 1954).

CONCEPTUALIZING THE OBJECTIVES OF THE PROGRAM

A first principle is that some attempt must be made to analyze the kinds of formal entities that are involved, to locate the regions within which the concepts are set. Do the concepts pertain to an individual or to a collective? Or do they pertain to both these regions in that the effects on the individuals are then intended to ramify to others and in turn to organizations, even whole societies? This is a first step toward clarification, for the concepts, procedures, and instruments will differ, depending on the region to be explored.1 Other coordinates may be used to distinguish major regions of concepts. What temporal reference is implied by the program? Are the concepts to be elaborated present states of the individual or of the collective, or future states, or both? In the Encampment program, the region under study is clearly the effect on the individual. By implication these young persons are supposed ultimately to ramify their influence, but in the perspective of the immediate present and the short-term future, the location of the concepts remains in the individual.

Using the first principle, the regions that are located are more properly called continents. Next, within a general region or continent, subregions must be specified. It is not difficult to implement this second principle in evaluations of effects on individuals. Psychology is rich in models of the individual which locate every subregion, even the most subterranean. An obvious and useful principle is to distinguish the subregion of overt conduct from the inner states of the individual. Thus it is immediately clear that the Encampment is engaged in a program of character education. Many of the stated objectives have no connotation of immediate overt conduct. Quite properly, the program is ultimately concerned with the way these young persons will behave when they are adults, in their future performance as citizens, but the means to this end is to develop the character structure that will govern such adult roles. Many of the concepts we started with in the study of the current group of young persons were in this internal region. Attitudes, beliefs, values, ideals, and the like are presumed to be the inner directives to the course of future conduct, and were measured by traditional, acceptable instruments involving batteries of questions which tap these inner states.

Once the subregions have been selected, they must be explored. They too are almost subcontinents. How shall the specific concepts be pinpointed? Again, models are available in the literature. Are we interested in the realm of knowledge, or in a system of beliefs, or in attitudes, or in motivations, or in values?2 3 Such classes of concepts are plentiful, and they guide the evaluator toward home base.

In conceiving of these subregions, it is useful to invoke a principle of levels or hierarchy. Some aspects of an individual’s character are more fundamental than others; some are fixed earlier in life; some are less amenable to change. In our studies, we wished to do justice to this notion of a hierarchy within the total character structure. Could the Encampment or any other brief interlude inserted in the life of a relatively mature person do more than change the peripheral? In our evaluation of the current group of subjects, we elaborated six subregions within the general region of the internal system of the individual, some regarded as more central and others as more peripheral. All six appeared to be related to one or more of the explicit objectives of the Encampment program.

The ultimate problem is to elaborate the specific concepts and variables within each subregion that will be measured. The best principle is that there is safety in numbers. We tried initially to be as comprehensive as possible. The Encampment is concerned with a wide array of changes in character. To do justice to the breadth of the program, the number of concepts had to be many. Moreover, not every measuring instrument would work effectively. A single finding might be subject to dispute, whereas many consistent findings would be strong evidence.

What leads or guides are there to the specific concepts at each level within the general region of character? A stated objective sometimes translates itself directly. When it does not, conference with the staff members of a program helps to clarify meaning and intent. For a program that has operated for many past cycles there is an additional and usually abundant resource. Many so-called evaluations make use of tes timonials from alumni, or of files of correspondence. Although these sources are dubious as evidence for a final evaluation, they provide rich leads for concepts, which then can be subjected to rigorous tests to determine the frequency of their occurrence as effects of a program. A final guide is the body of past literature, either theoretical writings about the region within which the concepts are located, or past evaluations of similar programs. In this respect we were fortunate in being able to profit from the work of Dr. Henry Riecken, who had conducted a pioneering, full-scale evaluation of the work-camp program of the American Friends Service Committee. Many of his concepts and corresponding measuring instruments were adopted in our study, with the result that comparisons can be made between two very similar ventures in character education.4

In reviewing the research there is a mental oscillation between conceptualizing and the other phases—design of the inquiry, choice of population or sample, and problems of research procedure. Sometimes these technical considerations work to restrain the beautiful flow of concepts, but sometimes they speed it. Where an instrument is ready at hand from a previous inquiry, the concept—even though less relevant— may be incorporated, with the prospect that measurement will be good, since the instrument has already stood the test of past research. More importantly, for reasons to be presented, the incorporation of such concepts and instruments yields comparative data for other groups—a very powerful tool in evaluation.

If it does not appear feasible that an instrument can be developed, a concept may be dropped. For example, in the Encampment program attention is given to increasing specialized knowledge and skills and particular intellectual capacities which are relevant to effective citizenship. The serious problems of developing and administering achievement tests led to the decision to omit the area, with one exception. Moreover, it seems very likely that such learning would be readily accomplished. The more crucial question appears to be whether such equipment in knowledge and skills is accompanied by the changes in character which would lead the campers to harness their equipment and ultimately act upon it.

One dimension relevant to the Encampment’s goals of training in skills, knowledge, and specialized intellectual capacities was included because of its special methodological interest and the challenge it posed. Many psychological variables are amenable to measurement by ques tionnaire techniques, and this is a standard and efficient apparatus for evaluations such as ours. To measure certain types of capacities—for example, memory, or motor skill—ideally one would wish to test an individual’s performance under stimulation on some appropriate experimental task. How to undertake such cumbersome procedures and incorporate them into the routine measurement process is a challenge. It can be accomplished, however, if the questionnaire is treated not merely as a set of questions for collecting information, but as a quasiexperimental vehicle, with many forms into which different levels of stimulation can be inserted and the corresponding responses observed. Chapter iv reports an experimental use of the questionnaire as a device to test whether or not the Encampment trains critical capacity or resistance to prestige symbols as propagandistic devices. In no way are the findings intended to suggest that this region of effects has been studied comprehensively. They merely serve to show that more may be done in the evaluation of capacities through ingenious and inexpensive quasi-experimental manipulation of questionnaires.

Research design and choice of population also merge with attempts at conceptualization. In our planning, it was decided to take measurements of the stability of effects after the 1955 group had returned to their homes. In the arena of the home community, the inner region of character is not the only relevant one. Conduct is no longer in the unmeasurable future. Now conduct can occur, and it becomes practical to conceptualize. It was also decided at various points in our studies to extend the research into long-range effects in this group after the passage of several years and to study even older individuals who were alumni of past Encampments. There conduct becomes even more important, and conceptualization and measurement of this seventh region of intended effects had to be incorporated into the studies.

1 See, for example, M. B. Smith, Evaluation of Exchange of Persons, Internat. Soc. Sci. Bull., VII (1955), 389.

2 The first four of these were the framework for the evaluations of orientation films conducted by Hovland and his associates. See Carl Hovland, Arthur A. Lums-

3 daine, and Fred D. Sheffield, Experiments on Mass Communication (Princeton: Princeton University Press, 1949), pp. 33-45.

4 Dr. Riecken was most helpful in making available unpublished data which aided in the planning and analysis of our evaluation.

CONCEPTUALIZING UNANTICIPATED CONSEQUENCES

The explicit objectives of a program provide a direct lead or a hint which gives the evaluator a start on his conceptualization. What does the evaluator do when he comes to conceptualization of the unanticipated consequences of a program that may well be relevant to his conclusions? Since the action agency did not anticipate them, how can the evaluator? It is not so difficult as it might appear. Four sources of hints are available. First, for programs that have been in operation before, even though some consequences were unanticipated, they have nevertheless demonstrated themselves in previous cycles. The correspondence of alumni, the records of the past, are bound to reveal the unexpected, perhaps too late to be taken into account by the planners of that earlier cycle, but not too late to be conceptualized in an evaluation of a new cycle of the program.

Second, an unanticipated consequence often is simply an extreme quantitative value of an intended effect, but at the extreme value it is transformed in its meaning. For example, the Encampment program aims to convert the sense of helplessness into the sense of competence. The evaluator must be even more farsighted about the process. One more step, and perhaps a sense of competence is transformed into overconfidence.

Third, by distinguishing four types of effects of a program, one arrives at a rather bizarre type of unanticipated consequence, about which the evaluator is given many hints. An agency has anticipated certain effects which it regards as desirable and intends to achieve. These become the explicit objectives. They justify the initiation of the program. There are other unanticipated effects of the program, some of which, if recognized, would be regarded as desirable and others as undesirable. The fourth cell includes those effects which an agency might have originally anticipated as possible effects of its activities, but which are regarded as undesirable. These are the anticipated, unintended objectives. Once these are recognized, the agency obviously tries to incorporate procedures into its program for preventing their occurrence, and may sometimes assume that it is successful. A program is planned, so to speak, to achieve certain things and avert others. The agency may often seek through evaluation to prove that it achieves the desirable goals; it may on occasion neglect to seek proof that it has averted the undesirable, simply because it assumes it is successful in this respect. It is not playing with words to say that in such instances unintended objectives, anticipated at one time, may at a later time become unanticipated consequences, as they are no longer assumed to occur. Such possibilities are revealed readily in reviewing a program with an agency. In the instance of the Encampment program, the agency had been sensitive to the possibility that in adopting a certain system of beliefs about politics, democracy, international relations, intergroup relations, and the like, the camper might enlarge this family of attitudes and adopt an ideology about domestic economic affairs. The Encampment wishes explicitly to avoid this outcome, and the program therefore omits this sphere. Yet, in the nature of psychic processes, it is perfectly possible that the economic and noneconomic somehow merge, and that the Encampment pro duces as an unanticipated consequence an increase in economic radicalism.1 2 3 Or, to take a cue from Riecken, although economic attitudes may not be communicated in the didactic parts of the program, these may be communicated informally by the staff members if their ideologies include not only liberal noneconomic attitudes, but also liberal economic attitudes.¹" Certainly this is a hypothesis worthy of testing, and therefore a scale on politico-economic conservatism was included among the instruments.

Most agencies do not reject the possibility that an unintended consequence may still occur despite their best efforts to avert it. Although it would not be correct to label such outcomes unanticipated, it would be important to conceptualize and measure changes in these variables in the course of an evaluation.

Fourth, the literature provides much guidance in conceptualizing unanticipated consequences. One thing, however, it surely proves. Thoughtful investigators have anticipated the unanticipated, and have systematically measured such consequences from various forms of stimulation and treatment. Riecken‘s evaluation proves that the task can be done, as do other examples yet to be cited. Because of the parallels in the two programs, we incorporated his concept of alienation, and, using a variety of instruments, measured whether this was an unanticipated consequence of the Encampment program. Even where an earlier investigator did not anticipate a particular outcome from some program of stimulation or treatment, its unexpected occurrence is generally reported after the fact, and the next investigator or evaluator profits. Thus, boomerang effects from programs of mass communication were not anticipated in the flush of early optimism about the potency of mass media, but as they cropped up they were reported.¹⁰ Hovland and his associates incorporated this general notion of a boomerang into their planning of the series of evaluations of orientation films produced for American soldiers, and elaborated many specific unanticipated consequences which were systematically measured.4 5

The four avenues to conceptualizing unanticipated consequences are of considerable help. They do not insure that all the relevant unanticipated consequences will be conceptualized by the evaluator in sufficient time to measure their presence before the program and after the program, in order to establish that they have in fact arisen from the program. As extra insurance, either through direct observation or through open questions, information should be obtained after the fact about additional consequences the program may have