The Social Validity Manual: A Guide to Subjective Evaluation of Behavior Interventions

By Stacy L. Carter

Applied Behavior Analysis is the science of applying experimentally derived principles of behavior to improve socially significant behavior. The research and application of ABA contributes to a wide range of practical areas, including AIDS prevention, education, gerontology, language acquisition and parenting, and ABA-based interventions have gained particular popularity in the last 20 years related to teaching students with autism spectrum disorders. Social Validity, a concept used in such behavioral intervention research, focuses on whether the goals of treatment, the intervention techniques used and the outcomes achieved are acceptable, relevant, and useful to the individual in treatment. Judgments are made (often via clinical trials) about the effects of the intervention based on statistical significance and magnitude of effect. Essentially, social validity alerts us as to whether or not the ABA-based intervention has had a palpable impact and actually helped people in ways that are evident in everyday life.

This clinical research volume offers a detailed evaluation of the extant findings on Social Validity, as well as discussion of newly emerging factors which reemphasize the need for well-developed methods of examining SV. Basic conceptualizations, measurement, research findings, applications, ethics, and future implications are discussed in full, and novel recommendations relating back to clinical treatment are provided. The volume will give readers a firm understanding of the general concept of SV, help them become familiar with the research methods and findings, and teach them how to establish and evaluate the Social Validity of individual interventions and treatment programs.

• Consolidates literature broadly distributed across journals and book chapters into single source
• Provides discussion of SV in greater depth and breadth than is found in other sources, which generally just focus on general conceptualization and broad research findings
• Describes how concept of SV can be influential in numerous areas of clinical practice

• Language: English
• Publisher: Academic Press
• Release date: Oct 1, 2009
• ISBN: 9780080954042

Stacy L. Carter

Stacy Carter is a Professor in the Special Education Program in the College of Education. Dr. Carter received his Ph.D. in Educational Psychology with an emphasis in School Psychology from Mississippi State University which was accredited by both the National Association for School Psychologists and the American Psychological Association. Dr. Carter is a Board Certified Behavior Analyst and a Nationally Certified School Psychologist. Dr. Carter is also a licensed special education teacher and a licensed specialist in school psychology. Dr. Carter has over 20 years of experience working as a Behavior Analyst in developmental centers, psychiatric hospitals, community living situations, and school settings.

Book preview

Table of Contents

Cover image

Copyright

Dedication

List of Figures

List of Tables

Preface

Chapter 1. Background of Social Validity

Chapter 2. Conceptualizations of Social Validity

Chapter 3. Instruments for Evaluating Social Validity

Chapter 4. Research on Social Validity

Chapter 5. Increasing the Social Significance of Treatment Goals

Chapter 6. Enhancing the Appropriateness of Treatment Procedures

Chapter 7. Improving the Importance of Treatment Effects

Chapter 8. Social Validity and Ethics

Chapter 9. Organization of Social Validity Data

Chapter 10. The Future of Social Validity

References

Index

Copyright

Academic Press is an imprint of Elsevier

32 Jamestown Road, London NW1 7BY, UK

30 Corporate Drive, Suite 400, Burlington, MA 01803, USA

525 B Street, Suite 1900, San Diego, CA 92101-4495, USA

Copyright © 2010 Elsevier Inc. All rights reserved

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher

Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (+44) (0) 1865 843830; fax (+44) (0) 1865 853333; email: [e-mail]. Alternatively, visit the Science and Technology Books website at www.elsevierdirect.com/rights for further information

Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verification of diagnoses and drug dosages should be made

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

ISBN: 978-0-12-374897-3

For information on all Academic Press publications visit our website at elsevierdirect.com

Typeset by Macmillan Publishing Solutions (www.macmillansolutions.com)

Printed and bound in the United States of America

10 11 12 13 14 10 9 8 7 6 5 4 3 2 1

Dedication

This book is dedicated to my parents, Tracy and Judy; my wife, Narissra; and my son, Ezra.

Thank you for everything.

List of Figures

List of Tables

Preface

Stacy L. Carter, PhD

At the heart of applied behavior analysis is social validity. Wolf (1978) referred to social validity as how applied behavior analysis was finding its heart, and in many ways social validity does appear to function like the heart of a much larger body. The field of applied behavior analysis has been mostly controlled by a brain that organizes information and responds in a logical, objective manner. Social validity appears to function in a more illogical, subjective manner that leads the field of applied behavior analysis down paths where it would typically not venture. The examination of objective, observable data has been a hallmark of applied behavior analysis, while social validity has made the field incorporate subjective, self-report data. Applied behavior analysis has been dominated by single-subject research designs, while social validity has led the field more into group research designs.

Just like in a romance novel, the heart of applied behavior analysis has taken over at times and caused it to go places, seek after things, and act in ways that sometimes don’t seem quite logical, but rather act out of a passion centered only in the heart. The passion from this heart has taken the field of applied behavior analysis into realms where new discoveries have been made and new mysteries have developed. Many of the findings from social validity research are contradictory and fleeting at times, which makes research in this area one of the most interesting and one of the most difficult to interpret. This book is designed to provide an introduction into the obscure side—the dark, passionate side, if you will—of applied behavior analysis. A history of the development of social validity within the field of applied behavior analysis is provided along with the impact that it has had within the field.

In L. Frank Baum’s The Wizard of Oz, the Tin Woodman and the Scarecrow debated the value of both a heart and a brain. The Scarecrow said, I shall ask for brains instead of a heart; for a fool would not know what to do with a heart if he had one. The Tin Woodman responded by saying, I shall take the heart, for brains do not make one happy, and happiness is the best thing in the world. Each made excellent points, and the field of applied behavior analysis has the potential to make use of both the objective, logical skills associated with the brain and the impassioned, subjective input connected to the heart. By using the brain and heart in combination, the field of applied behavior analysis can potentially make greater strides in using science to improve the lives of everyone in society and also make them happier. As stated by Wolf:

Our use of subjective measures does not relate to internal causal variables. Instead, it is an attempt to assess the dimensions of complex reinforcers in socially acceptable and practical ways. It is an evolutionary event that is occurring as a function of the contingencies of the applied research environment, contingencies that our founders would probably say they appreciate, if we had the nerve to ask them for such subjective feedback on our behavior (1978, p. 213).

Metaphorically speaking, the approach to social validity taken in this book can be compared to putting together a puzzle with a huge number of pieces. Without having a method for putting the pieces together, initial efforts toward completing the puzzle may consist of random trial-and-error responses with periodic success. This book attempts to offer some strategies for putting the pieces of the puzzle together that will hopefully reduce the randomized responses and result in faster completion of the puzzle. One strategy for completing a puzzle is to begin by sorting and organizing the border pieces, since they are typically easier to recognize because of the recognizable straight edge. Completing the border of the puzzle offers some perspective to the magnitude of completing the rest of the puzzle. The border pieces of the puzzle can be compared to the conceptualization of social validity that is used throughout this book. While several different conceptualizations are discussed, this book uses a conceptualization that is very liberal toward its inclusion of variables considered to represent and have influence on social validity. This wide-ranging view of social validity results in a large puzzle border within which a large variety of pieces can and need to be placed in order to complete the puzzle. Another strategy in completing a puzzle is to sort the pieces according to color similarities and attempting to connect pieces of like color. This book attempts to do this with social validity by organizing the chapters according to treatment goals, treatment procedures, and treatment effects. This type of organization is intended to increase possible recognition among similar aspects of treatment and allow for additional connections to occur.

In completing a puzzle, there are often sections of the puzzle which are not directly associated with the predominant picture contained within the puzzle. These pieces arrange to form a background against which the primary picture can be discriminated. The background within a puzzle is highly important although not directly a part of the primary picture within the puzzle. The background adds to the central puzzle picture by making the picture more recognizable and supports the positioning and the structural stability of the central picture. Upon putting together all the available pieces to the puzzle, there may in many cases be pieces that are missing. If enough pieces can be put together, the picture represented in the puzzle can still be determined even with some pieces missing. This book focuses on a number of variables and concepts that may not be considered direct measures of social validity, but which offer up a background upon which social validity can be more readily discriminated. By including a large number of associated variables, the evidence for social validity will be provided with a highly stable structure even if some pieces are missing and the importance of social validity will be seen as the predominant big picture that all behavior treatments are attempting to illustrate.

April 1, 2009

Lubbock, TX

Chapter 1. Background of Social Validity

Defining Social Validity

What is Social Validity?

The construct of social validity has been conceptualized in many different ways. Schwartz and Baer (1991) point out that numerous meanings have been ascribed to the construct of social validity, which have thus catapulted the construct into a mass of confusion. Even the terminology associated with social validity has been used in various ways that make determining exactly what social validity consists of extremely confusing at times. While the literature within the field of behavior analysis coined the term social validity, the term has not been used consistently even within the field where it was first used. Other fields of social science, as well as business and marketing, have used various terminology to describe similar constructs, such as social importance, social relevance, social significance, consumer satisfaction, clinical importance, clinical significance, educational relevancy, applied relevance, applied importance, ecological validity, cultural validity, and cultural significance. The sheer number of terms, which have in some instances been used interchangeably with social validity, may have resulted in some confusion about exactly what is involved in defining, measuring, and evaluating social validity. While social validity can be associated with several different disciplines and defined in many different ways, the type of social validity referred to within this book will be the social validity that was developed from the field of applied behavior analysis. There have been several proposed variations on how social validity should be defined, how it should be assessed, and how it should be used within the field of applied behavior analysis. Several of these variations will be described within this book, and each will be discussed in relationship to the original definition of social validity proposed by Wolf (1978).

For the purposes of this book, the term social validity will refer to the evaluation of the degree of acceptance for the immediate variables associated with a procedure or program designed to change behavior. This definition is derived from the definition provided by Wolf (1978). He defines social validity as (1) the social significance of the goals of treatment; (2) the social appropriateness of the treatment procedures; and (3) the social importance of the effects of treatments (Table 1.1). The most frequent method for determining the degree of acceptance for a procedure or program has been to ask those receiving, implementing, or consenting to a treatment about their opinions of the treatment. These opinions are then used to make decisions about current or future uses of the treatment.

Development of a Concept

Wolf (1978) describes, with an apology, his personal experiences with social validity that were associated with the development of a definition for the term. He describes how a last-minute deadline resulted in the following quote from Don Baer being included in the introductory issue of the Journal of Applied Behavior Analysis (JABA) to describe the purpose of the journal: It is for the publication of applications of the analysis of behavior to problems of social importance. Wolf states that he continued to think about the meaning of the term social importance and how it might be viewed by others in the field. He explains that the term social importance relied on subjective information such as opinions or judgments made by other people. Wolf feels that including this type of subjective component in the seminal issue of JABA conflicted with the objective nature of behavior analysis, which was considered to be in line with other natural sciences that avoided subjective evaluations of variables. Wolf goes on to note that the field of behavior analysis had separated itself from other social sciences (such as sociology, anthropology, psychiatry, etc.) that included subjective observations, and the board of editors for JABA wanted the journal to exemplify only strict objective measurements. The tradition of direct observation of behavior was considered necessary by Skinner (1969), as he states:

Much can be done in the study of behavior with methods of observation no more sophisticated than those available, say, to Faraday, with his magnets, wires, and cells. Eventually the investigator may move to peripheral areas where indirect methods become necessary, but until then he must forego the prestige which attaches to traditional statistical methods (p. 111).

While Wolf continued to struggle with a definition of social importance that would fit the objective nature of the journal, he found that the journal readers, article authors, and reviewers of manuscripts did not complain about the lack of a definition of social importance and they appeared to be able to readily recognize its presence or absence. This appears to be similar to the statement made by Supreme Court Justice Potter Stewart, who avoided providing a succinct definition for pornography but stated I know it when I see it…. The point is that Wolf finds that social importance is recognized, accepted, and valued even though it may not lend itself to an objective method of measurement.

Dimensions of Applied Behavior Analysis

In developing an understanding of social validity, it is important to review the development of the field of applied behavior analysis. The JABA has become the primary outlet for disseminating research in the area of applied behavior analysis. In the introductory issue of this journal, Baer, Wolf, and Risley (1968) outline the dimensions of applied behavior analysis, and these dimensions are still frequently referred to in the literature. They describe seven criteria needed for a study to be considered an example of applied behavior analysis (Table 1.2). These criteria are essential for adhering to the scientific principles established within the field of applied behavior analysis.

Criteria I: Applied

The applied criterion refers to how target behaviors are selected. With applied research, target behaviors must be selected because of their importance to society. With nonapplied research, target behaviors may be selected for reasons that provide convenience to the researcher. Applied research focuses on target behaviors that are highly relevant to society and that typically reveal immediately important relationships between the behavior and certain stimuli.

Criteria II: Behavioral

The behavioral criterion focuses on the pragmatic aspect of research. Applied research should focus on the physical events performed by individuals rather than on other factors only related to physical performance of these events, such as verbal descriptions of performance, unless independently supported by other measures.

Criteria III: Analytic

The analytic criterion refers to the reliability of the research. In order for a study to meet the criterion of being analytic, it must reasonably demonstrate some level of control over a behavior. Nonapplied research may require extensive repetition of control over behavior, but applied research has to rely on achieving as much control over behavior as can be demonstrated to make the control believable to a select audience. In other words, applied research cannot always achieve the level of demonstrated control over behavior that can be achieved in nonapplied research, due to allowances provided within social settings.

Criteria IV: Technological

The technological criterion concerns the description of the techniques used to promote behavior change. Applied research requires that all components of a treatment be completely and accurately identified and reported. This criterion does not distinguish nonapplied research, but requires that enough information be provided on the techniques used in an experiment so that others might replicate these same techniques in exactly the same manner.

Criteria V: Conceptually Systematic

The conceptually systematic criterion relates to how techniques relate to basic principles that are considered part of the field of behavior analysis. Applied research must convey how specific procedures have been derived from basic principles so that similar procedures might be derived from those same basic principles. This criterion was deemed necessary to promote the field of applied behavior analysis beyond what might be considered a series of unrelated techniques or procedures.

Criteria VI: Effective

The effective criterion relates to the relevance of outcomes to practical situations. Nonapplied research may seek to produce outcomes that have theoretical importance, while applied research must produce results that have some practical value. Applied research requires outcomes that are large enough to be considered important to those who deal closely with a behavior.

Criteria VII: Generality

The generality criterion involves the enduring qualities of a treatment over time, over settings, or across related behaviors. Since applied research must deal with socially important behaviors, the changes that take place in these behaviors may be important in ways different from the way in which the changes were initially introduced. Demonstrating that treatments are effective in more than one way increases the value of the procedure and the importance to society.

Examining these criteria in light of social validity, it seems that the applied criteria and the effective criteria play the most important roles in understanding social validity, because both of these criteria focus on aspects of social determination or judgment. In addition, the generality criterion may be highly important toward gaining a more comprehensive understanding of the influences on social validity. Although some of the dimensions of applied behavior analysis may play more important roles in the assessment of social validity, as stated by Schwartz and Baer (1991), state-of-the-art social validity assessment should address all the dimensions of applied behavior analysis.

Importance of Social Validity

Is Social Validity Important?

Social validity has been viewed as essential to the survival of the field of applied behavior analysis and as a diversionary trap that would lead to the demise of the field (Baer & Schwartz, 1991). They state, We know little about the accurate and valid assessment of what gets called social validity—and we know a great deal less about the survival of fields (p. 231). While either of these views may potentially be correct, what appears most important is to gain an understanding of the construct and what role it might play in benefiting society. If social validity has some value to society, it should be embraced and further utilized in the field of behavior analysis rather than avoided and misunderstood. Baer and Schwartz state that the primary problems facing the field of applied behavior analysis with regard to social validity were how to assess it and how to apply it to improve behavior interventions.

In some cases it may be argued that social validity is not important. An expert may develop a program that is considered highly effective at achieving a desired outcome, which may on the surface appear to be a very useful program regardless of whether nonexperts agree. An extreme hypothetical example of this might be as follows:

An expert is assigned a case in which a client is displaying aggressive acts toward others within a residential facility. The client frequently strikes, pushes, and bites others throughout the day during typical daily activities. The expert designs a program intended to reduce the frequency of aggressive acts toward others. The expert trains the staff at the facility to place the client in a time-out room following each aggressive act. The program is determined to show an immediate reduction in the frequency of aggressive acts when compared to data obtained prior to the treatment.

On the surface, this treatment may appear to have produced the desired outcome (reduction of aggression), but it actually neglects to mention a number of critical elements that would need to be considered before the treatment could truly be considered a success. First, the treatment does not clarify the amount of time the client would be required to remain in the time-out room. This problem has been highly relevant in intermediate care facilities, due to the potential for abusing this type of procedure. The staff at the facility may choose to leave the client in the time-out room for extended periods of time because the aggressive behavior that the patient displays is difficult to manage and can be avoided while he or she remains in the time-out room. In addition, the procedure may not be truly effective because the time spent in isolation reduces the opportunities for the client to engage in aggressive behavior; this lack of opportunity would certainly reduce the frequency of aggressive acts. There are numerous other concerns regarding this program, such as the potential lack of socialization that may occur, lack of training in appropriate skills, and so on.

The point of this example is not to identify all the potentially inappropriate elements, but rather to demonstrate that just because a program is considered effective does not mean that it will be considered appropriate by those closely involved in implementation of the program, or that it should even be implemented in the first place. For instance, other treatments may be available that can potentially produce similar outcomes and avoid some of the deficiencies apparent in the program described. If these other treatments are available, they might certainly be more appropriate than the treatment described, although they may not produce more effective results.

Kazdin (1980) points to three reasons why determining the acceptability of treatments was important to overall social validity. The first reason was that several techniques for treating problem behaviors may be available that have comparable efficacy, but may not be equally acceptable to the person receiving the treatment. The second reason involves ethical and legal issues, which require that procedures avoid infringing on the rights of the individuals receiving the treatments. The final reason addressed by Kazdin (1980) involves the value of identifying specific variables that could influence the acceptability of treatments. If influential variables could be identified, they could be used to encourage individuals to seek out treatment, initiate treatment, and adhere to treatment regimens.

With the increased dissemination of treatments involving less intrusive procedures and changes in educational law, a reexamination of the concept of social validity appears warranted. Michaels, Brown, and Mirabella (2005) report that experts were making fewer recommendations for the use of consequence-based treatments than they had in the past. Educational legislation such as the No Child Left Behind Act (No Child Left Behind Act of 2001, 20 U.S.C. § 6301 et seq.) and the Individuals with Disabilities Education Act Amendments (IDEA, 1997, 20 U.S.C. § 1401) has mandated the incorporation of parents, teachers, principals, and others in the development and implementation of treatments for children receiving special education services. More recent changes to IDEA have also promoted the use of empirically based treatments in the least restrictive environment. In light of these changes, the concept of social validity may benefit from further evaluation in terms of relevance to decisions made regarding treatments for behavior change programs.

Purpose of Social Validity

One purpose of social validity may be, as indicated earlier, to ensure the survival of the field of applied behavior analysis. While this may seem like an arrogant statement and has definitely been challenged by many—to the point of declaring that social validity is basically nonessential—social validity still appears to have a purpose and a place within applied behavior analysis. Schwartz and Baer (1991) state that the purpose of social validity assessment is to provide information that would enable a treatment program to survive. In order for a treatment program to survive, it must not only be effective, it must also address problems considered relevant by consumers, it must do so in a manner that consumers can tolerate or even possibly enjoy, and it must produce outcomes that are considered valued. While these may be only a few of the requirements for the survival of a treatment program, the measurement of social validity may provide a method for identifying elements of a treatment program that could lead consumers to discontinue its use, avoid it, or even seek to ban its further implementation. One purpose of social validity may be to obtain feedback from consumers regarding how well specific elements of a treatment program are liked or disliked, in order to either make changes to the treatment program or to provide a rationale to consumers regarding why a treatment contains undesirable elements. Social validity assessment can provide information on how to quickly respond to consumers in a way that will provide them with a reason to continue the treatment or a new understanding of the need for the treatment.

Schwartz and Baer (1991)