Picturing the Uncertain World: How to Understand, Communicate, and Control Uncertainty through Graphical Display

By Howard Wainer

In his entertaining and informative book Graphic Discovery, Howard Wainer unlocked the power of graphical display to make complex problems clear. Now he's back with Picturing the Uncertain World, a book that explores how graphs can serve as maps to guide us when the information we have is ambiguous or incomplete. Using a visually diverse sampling of graphical display, from heartrending autobiographical displays of genocide in the Kovno ghetto to the "Pie Chart of Mystery" in a New Yorker cartoon, Wainer illustrates the many ways graphs can be used--and misused--as we try to make sense of an uncertain world.

Picturing the Uncertain World takes readers on an extraordinary graphical adventure, revealing how the visual communication of data offers answers to vexing questions yet also highlights the measure of uncertainty in almost everything we do. Are cancer rates higher or lower in rural communities? How can you know how much money to sock away for retirement when you don't know when you'll die? And where exactly did nineteenth-century novelists get their ideas? These are some of the fascinating questions Wainer invites readers to consider. Along the way he traces the origins and development of graphical display, from William Playfair, who pioneered the use of graphs in the eighteenth century, to instances today where the public has been misled through poorly designed graphs.

We live in a world full of uncertainty, yet it is within our grasp to take its measure. Read Picturing the Uncertain World and learn how.

• Language: English
• Publisher: Princeton University Press
• Release date: Jun 8, 2021
• ISBN: 9781400832897

Howard Wainer

Howard Wainer is Distinguished Research Scientist for the National Board of Medical Examiners and Adjunct Professor of Statistics at the Wharton School of the University of Pennsylvania. He is the author of fifteen other books.

Book preview

Picturing the Uncertain World

Picturing the Uncertain World

How to Understand, Communicate,

and Control Uncertainty through

Graphical Display

Howard Wainer

PRINCETON UNIVERSITY PRESS

Princeton & Oxford

Copyright © 2009 by Princeton University Press

Published by Princeton University Press, 41 William Street,

Princeton, New Jersey 08540

In the United Kingdom: Princeton University Press, 6 Oxford Street, Woodstock, Oxfordshire OX20 1TW

All Rights Reserved

Library of Congress Cataloging-in-Publication Data

Wainer, Howard.

Picturing the uncertain world : how to understand, communicate, and control uncertainty through graphical display / Howard Wainer. p. cm.

Includes bibliographical references and index.

ISBN 978-0-691-13759-9 (cloth : alk. paper)

1. Uncertainty (Information theory)—Graphic methods. 2. Communication in science—Graphic methods. I. Title.

Q375.W35 2009

0039.54—dc22 2008053529

press.princeton.edu

eISBN: 978-1-400-83289-7

R0

To my colleagues, past and present,

who contributed their ideas

and their kind thoughts.

This book is the result of

the beauty of their minds

and the labor of mine.

Sine quibus non

Contents

Preface and Acknowledgments xv

I. Introduction and Overview

CHAPTER 1THE MOST DANGEROUS EQUATION 5

In this chapter we nominate De Moivre’s¹ description of the expected variation in the arithmetic mean for the title of the most dangerous equation. To support this conclusion we describe five separate examples where ignorance of this equation has led to enormous wastes of time, money, and human resources. These five examples span almost a thousand years and areas as diverse as monetary policy, education policy, medical practice, and the genetic basis of sex differences in intelligence.

II. Political Issues

In this section we show how five different kinds of issues that emerged from essentially political arguments could be illuminated with more careful thought and a graph or two. In chapter 6, we introduce a very simple probabilistic model that yields surprising richness of understanding, which apparently escaped the editorial writers of the New York Times.

CHAPTER 2CURBSTONING IQ AND THE 2000 PRESIDENTIAL ELECTION 23

Sometimes, when facts are hard to come by, people who are tasked to gather those facts simply substitute a guess. When this is done by census workers it is called curbstoning (as in sitting down on the curbstone in front of a house and guessing how many people live there). Curbstone estimates, although illegal and grounds for dismissal, have shown themselves to be remarkably accurate. In this chapter we look at a piece of political propaganda meant to highlight the intellectual and financial differences between red and blue states. Although it was clearly based on someone’s biases and not actual data, the conclusions we would draw from the faked data are close to actual results.

CHAPTER 3STUMBLING ON THE PATH TOWARD THE VISUAL COMMUNICATION OF COMPLEXITY 31

An op-ed piece in the New York Times written by former secretary of state George Schultz contained a statistical graph that showed the economic superiority of the two Bush administrations to the Clinton administration that was sandwiched in between. We show how this graphic distorts our perceptions by plotting rates of change instead of the actual GDP. The result is exactly the opposite of what former Secretary Schultz argues.

CHAPTER 4USING GRAPHS TO SIMPLIFY THE COMPLEX: THE MEDICARE DRUG PLAN AS AN EXAMPLE 35

The Medicare drug plan, although passed with great fanfare, quickly resolved itself into a complex puzzle. In this chapter we simplify one part of the puzzle by drawing a graph that makes clear who should sign up. The graph is not a full solution, for how the costs will be paid remains shrouded in a deep mystery indeed.

CHAPTER 5A POLITICAL STATISTIC 39

Neither graphs nor tables are guarantees of truth. Incorrect stories can be concocted with data displays just as they can with words. In this chapter we investigate a graph produced by the U.S. Department of Education that vividly shows how fourth graders’ reading scores remain stubbornly flat despite skyrocketing increases in federal expenditures for education. A more careful look indicates that there is a strong positive relationship between students’ test scores and money spent on education.

CHAPTER 6A CATCH-22 IN ASSIGNING PRIMARY DELEGATES 47

As the 2008 election loomed ever closer, states maneuvered in various ways to try to gain increased influence. The New York Times argued that New York’s citizens were not fully enfranchised because of the all-or-none delegate assignment rule used in the primaries. Using a simple mathematical model, we show that exactly the opposite is true.

III. Educational Testing

In the four thousand years since its inception in ancient China, mental testing has promised to provide an important tool toward a true meritocratic society. Replacing family connections with an individual’s ability as the key to opening the doors to economic and social success remains a principal goal of modern societies. Progress toward this goal has been impressive, but it has occurred in fits and starts. In this section we examine three proposals to aid in using test scores toward making this a more just society. The first uses a statistical method commonly employed in other circumstances to solve a vexing problem. In chapter 8 we examine a well-meaning but, at its heart, flawed scheme aimed at reducing intergroup differences. And finally, in chapter 9, we look at a recent court case involving test scoring and show that the defense’s case was based on a misunderstanding of the meaning of uncertainty.

CHAPTER 7TESTING THE DISABLED: USING STATISTICS TO NAVIGATE BETWEEN THE SCYLLA OF STANDARDS AND THE CHARYBDIS OF COURT DECISIONS 55

Test companies are in a logical bind. Standards of testing require that individual scores on tests given under nonstandard conditions (for instance, with extra time) be so labeled, while courts mandate that examinees with disabilities (who are often given accommodations like extra time) not be identified. In this chapter we show a statistical method that can provide a way to be responsive to these two seemingly contradictory requirements.

CHAPTER 8ETHNIC BIAS OR STATISTICAL ARTIFACT? FREEDLE’S FOLLY 63

Social scientist Roy Freedle startled the testing world in 2003 when he showed that black examinees outperformed matched white examinees on hard SAT items. He suggested that ethnic group differences in test performance could be reduced dramatically and tests thus made fairer by making the tests harder. In this chapter we look into the validity of this remarkable conclusion.

CHAPTER 9INSIGNIFICANT IS NOT ZERO: MUSING ON THE COLLEGE BOARD’S UNDERSTANDING OF UNCERTAINTY 74

On October 8, 2005, NCS Pearson, Inc., under contract to the College Entrance Examination Board, scored an administration of the SAT Reasoning test. Subsequently it was discovered that there was a scoring error that had affected 5,024 examinees’ scores. After rescoring it was revealed that 4,411 test scores were too low and 613 were too high. The exams that were underscored were revised upward and the revised scores were reported to the designated colleges and universities. The College Board decided that it would be unfair to re-report the scores of the 613 test takers whose scores were improperly too high and hence did not correct them. They reached this conclusion because of a misunderstanding of statistical error. In this chapter we discuss their argument and its flaws.

IV. Mostly Methodological

This section is a bit more technical than the others, focusing more explicitly on the statistical tool, with its application being secondary. In chapter 10 we look at the validity of linear extrapolation through unexpectedly consistent improvements in the world record for men running a mile that have occurred over the course of the twentieth century and speculate whether it should have been predictable, and what, if anything, it means about future improvements in the twenty-first century. The eleventh chapter looks at statistical graphics in the popular media. Chapter 12 demonstrates how a mixture of statistical tools, statistical thinking, and various graphic forms combine to provide us with a guided pathway of discovery. The last two chapters are perhaps the most narrowly focused of all, looking first at ways to show our uncertainty graphically and next at one way in which powerful computing when combined with our desire for simplicity at all costs can be used to mislead us.

CHAPTER 10HOW LONG IS SHORT? 87

All functions are well approximated by a straight line for a short part of their length. But how can we know for how long the linear approximation is suitable? Obviously, when the entire data series is in hand it is easy, but what about when it is not? What do we do when we wish to extrapolate from what appears to be linear beyond the data? For a very short extrapolation it is usually fine, but how long is short? In this chapter we look at a century’s progress in the world records in the mile run for help in answering this question.

CHAPTER 11IMPROVING DATA DISPLAYS 92

The communication media’s stock and trade is the distillation and communication of possibly complex information. To do this effectively the print media use an especially broad range of graphical formats. Sometimes they do this poorly, but sometimes they do it very well indeed. In this chapter we look at some displays devised by the media that set a standard for excellence hard to imagine given their time dead-lines, as well as others that were seriously flawed.

CHAPTER 12OLD MOTHER HUBBARD AND THE UNITED NATIONS 106

Statistical thinking and data-based graphics are two tools used together to understand the world. This chapter tells a story of how a detective might use them to track down and expose some surprising aspects of poverty.

CHAPTER 13DEPICTING ERROR 121

Communicating data without some measure of their precision can lead to misinterpretation and incorrect inferences. In this chapter, we describe and illustrate several conventions for displaying errors along with the data they modify. We also offer some alternatives that seem to provide improvements in the effective communication of error as well as increasing the ease, and hence the likelihood, of their use. These alternatives are illustrated principally with data from the National Assessment of Educational Progress.

CHAPTER 14THE MENDEL EFFECT 148

Data are often examined after being grouped into categories. For example, we might see a plot of income shown as a function of education level, in which amount of education is collapsed (binned) into specified categories like 0–8 years, 9–11, 12, 13–15, 16 or more. A typical summary plot shows the mean value of income as a function of the binned education variable, and trends are interpreted. In this chapter, I demonstrate how such trends can be epiphenomenal and are the creation of the number of bins used and their boundaries. I provide an algorithm that can take trendless data and create trends in any direction.

V. History

We understand best those things we see grow from their very beginnings.

—Aristotle, Metaphysics

The Science of Uncertainty has been under development for a long time. In this section, I pay homage to our forebears by using modern tools to investigate ancient puzzles (chapters 15 and 16), by exploring the origins of some of these modern tools (chapters 17 and 19), by defending the wisdom of the ancients from contemporary misuses (chapter 18), by communicating the wisdom of a modern master (chapter 20), and finally by a heart-rending use of graphics to paint an evocative picture of one part of what was perhaps the greatest horror in all human history.

CHAPTER 15TRUTH IS SLOWER THAN FICTION 161

Novelists often use the latest scientific findings as essential plot elements in their stories. In this chapter, we follow how some of the findings of the nineteenth-century British polymath Francis Galton were used by Arthur Conan Doyle, by Mark Twain, and by Jules Verne, and speculate on who got there first and why.

CHAPTER 16GALTON’S NORMAL 168

Francis Galton was an early adopter of the normal distribution as a means of making inferences about the frequency of occurrence of various human characteristics. In his 1869 book Hereditary Genius, he explains how to do this with a hypothetical graph showing the heights of British men. But the graph Galton made up revealed a serious misunderstanding he had about the normal distribution. In this chapter, we uncover the error and suggest its source.

CHAPTER 17NOBODY’S PERFECT 173

In 1786, the remarkable Scot William Playfair published a small book in which he invented three of the four basic graphical formats (bar charts, line charts, and pie charts). He did not invent the scatter plot. In this chapter we ask and try to answer the obvious question, why not?

CHAPTER 18WHEN FORM VIOLATES FUNCTION 179

The title of finest statistical graphic ever prepared is generally awarded to the nineteenth-century Frenchman Charles Joseph Minard’s remarkable six-dimensional plot showing the fate of the French army as it trekked between the Niemen River on the Poland-Russia border to Moscow and back during Napoleon’s ill-fated 1812–1813 campaign. In this chapter, we examine one failing attempt to usurp Minard’s famous format for another purpose.

CHAPTER 19A GRAPHICAL LEGACY OF CHARLES JOSEPH MINARD: TWO JEWELS FROM THE PAST 186

Not all of those who sought to emulate Minard’s success with data of their own failed. In this chapter, we show how followers of Minard produced treasures of their own by following in the footsteps of the master.

CHAPTER 20LA DIFFUSION DE QUELQUES IDÉES: A MASTER’S VOICE 193

Jacques Bertin (1918–) is a French semiologist, trained in Paris, whose seminal work La Semiologie Graphique (1969) laid the groundwork for modern research in graphics. Almost forty years after its publication it still provides important lessons to all those interested in the effective display of quantitative information. In 2002 he sent me a note detailing his most recent developments and asked that I continue to help him in la diffusion de quelques idées. This chapter tries to do exactly that.

CHAPTER 21NUMBERS AND THE REMEMBRANCE OF THINGS PAST 199

A single death is a tragedy; a million deaths is a statistic.

—Joseph Stalin (1879–1953)

Unquestionably cold and cruel, this epigram conveys a sentiment that sadly captures an aspect of human psychology. The mind is limited in its capacity to fathom cataclysmic events. Great numbers of deaths, particularly if they are distant in time or space, typically do not elicit the same reaction as fewer deaths nearer to us. Sponsors and designers of memorials face the challenge of stirring emotion, memory, and understanding. In this final chapter we show and discuss data displays produced by the inhabitants of the Kovno Ghetto to record their own deaths—so that they might transform what could have been only a statistic into the tragedy that it undeniably was.

VI. Epilogue

Notes 211

References 215

Source Material 225

Index 229

Preface and Acknowledgments

The French have an expression "J’étais marrié après l’âge de raison, mais avant l’âge de la connaissance. This translates as I was married after the age of reason, but before the age of knowing." L’âge de la connaissance is a deep idea, and hubris sometimes traps us into thinking that, while last year we hadn’t quite reached l’âge de la connaissance, this year we have. This has happened to me many times in the past, but each time when a year had passed I realized that the age of knowing still lurked, just out of reach, in the future.

As I sit here, in my sixty-fifth year, I am writing this book on understanding uncertainty, because I realize that many other people share with me the fact that a search for understanding, for knowing, is a constant force in their lives. This search manifests itself in many ways, for the world we live in is filled with subtle uncertainty that can be interpreted in different ways. Among optimists, it is merely an intriguing mystery to be unraveled. Others, more cynical, see evil forces combining to trick you and thence lure you into trouble. Einstein’s famous observation that God is subtle, but not malevolent clearly places him among the optimists. Among its deepest and most basic tenets, statistics, the science of uncertainty, follows Einstein’s lead.

You don’t have to be a statistician to be concerned with uncertainty; although training in statistical thinking is an invaluable aid in navigating this uncertain world. Not too long ago Sid Franks, an old college buddy and my financial planner, told me that he believes that if he did his job to perfection, the check to the undertaker would bounce. When I asked how he engineered the planning necessary to achieve this remarkably precise conclusion, he said that he plans for a life span of ninety-five years and hopes that I make it.

Why ninety-five? I thought. Since the likelihood of making it to ninety-five is so small, wouldn’t there be money to spare after the undertaker was paid? And wouldn’t I get more enjoyment out of my money by spending it myself?

Sid agreed, explaining that it was a tough prediction and said that he could do a much better job if I would just tell him well in advance exactly when it was that I was going to expire.

He understood that it is rare that such preknowledge is possible and so he had to make his plans under uncertainty. He emphasized that it is a more serious error to live beyond your money than it is to have some money left over after you die. Hence he used an upper bound for the life span.

I asked if knowing my death date was all he needed to be able to manage my account satisfactorily. He replied that, no, it would also help to know what the rate of return on my portfolio would be over the next two decades, as well as the path that inflation will take. He also pointed out that all these things change, and that the amount and direction of those changes can have a profound effect. He then emphasized the importance of having a plan as well as the flexibility to modify it as conditions change and we gain more information.

This one task contains two of the most vexing elements found in most serious decisions: uncertainty about future conditions and an asymmetric cost function (it is worse to outlive your money than to leave some over). And it is but one of thousands in all our lives that illustrate how important it is to understand and manage the uncertainty that we will inevitably encounter. But to manage and understand uncertainty, so that we can plan appropriately, we must always be gathering more data so that we test and perhaps modify our plans. This is as true of buying a house and raising a family as it is for planning our retirements. And if we do this poorly the results can be very serious indeed.

This book is about the science of uncertainty and is filled with stories that provide real life lessons on recognizing and managing uncertainty. I have accumulated these stories over twenty or so years, and in those years many debts have accumulated. It is my pleasure now to acknowledge those debts and to express my gratitude to the colleagues whose thoughts and labor are contained here.

First, my gratitude to my employer, the National Board of Medical Examiners, and its management, especially Don Melnick, Ron Nungester, and Brian Clauser, who have