Brave NUI World: Designing Natural User Interfaces for Touch and Gesture

By Daniel Wigdor and Dennis Wixon

Brave NUI World is the first practical guide for designing touch- and gesture-based user interfaces. Written by the team from Microsoft that developed the multi-touch, multi-user Surface® tabletop product, it introduces the reader to natural user interfaces (NUI). It gives readers the necessary tools and information to integrate touch and gesture practices into daily work, presenting scenarios, problem solving, metaphors, and techniques intended to avoid making mistakes.

This book considers diverse user needs and context, real world successes and failures, and the future of NUI. It presents thirty scenarios, giving practitioners a multitude of considerations for making informed design decisions and helping to ensure that missteps are never made again.

The book will be of value to game designers as well as practitioners, researchers, and students interested in learning about user experience design, user interface design, interaction design, software design, human computer interaction, human factors, information design, and information architecture.

• Provides easy-to-apply design guidance for the unique challenge of creating touch- and gesture-based user interfaces
• Considers diverse user needs and context, real world successes and failures, and a look into the future of NUI
• Presents thirty scenarios, giving practitioners a multitude of considerations for making informed design decisions and helping to ensure that missteps are never made again

• Language: English
• Publisher: Elsevier Science
• Release date: Apr 5, 2011
• ISBN: 9780123822321

Daniel Wigdor

Daniel Wigdor is an Assistant Professor of computer science at the University of Toronto. Before joining U of T, he worked at Microsoft in nearly a dozen different roles, among them serving as the User Experience Architect of the Microsoft Surface product, and as a cross company expert in the creation of Natural User Interfaces. Before joining Microsoft, he previously conducted research in advanced user interfaces and devices at Mitsubishi Electric Research Labs, and at the Initiative in Innovative Computing at Harvard University. He is also co-founder of Iota Wireless, a company dedicated to the commercialization of NUI technologies for mobile phones. Daniel’s work has been described in dozens of publications in leading international conferences, journals, and books. His is the recipient of a Wolfond Fellowship and an ACM Best Paper Award.

Book preview

Index

PART I

Introducing the NUI

1 Introduction

2 The Natural User Interface

3 Ecological Niche

Chapter 1

Introduction

Keywords

Moore’s law, GUI, WIMP, NUI, pitfalls

All things will be produced in superior quantity and quality, and with greater ease, when each man works at a single occupation, in accordance with his natural gifts, and at the right moment, without meddling with anything else.

—Plato

In the decades since the first digital computers were programmed using mechanical switches and plug boards, computing and the ways in which people interface with computers have evolved significantly. Some aspects of this evolution have both been anticipated and withstood the test of time. Moore’s law is an example. The law states that the number of transistors that can be placed inexpensively on an integrated circuit will double approximately every two years. The trend that this law describes has created opportunities for the growth of computing and its adoption into many aspects of our lives. As computers have increased in power and decreased in size and cost, new form factors have been created (e.g., smart phones, PDAs, and digital cameras), new platforms have evolved (e.g., the Internet), new infrastructures have become widely available (e.g., GPS), new industries have arisen (e.g., computer games), and new application families (e.g., spreadsheets, document processing, image creation, modification and sharing) have flourished. All of these trends have resulted in the democratization of computing as the number of people directly interacting with computers has steadily increased. This proliferation of computing has transcended national boundaries and permeated nearly all economic classes. It has changed the way people work, play, and interact with one another.

While the increase in computing power has been more or less continuous, the interfaces between human and computers have evolved more discontinuously. A widely held perspective is that interfaces have passed through phases. These phases are loosely defined but can be thought of as the phase of typing commands (the command line), followed by the graphical user interface (GUI). More specifically, most computers with which people interact regularly are based on the desktop metaphor (so called because windows are allowed to overlap, like paper atop a desk) and rely on a known set of user interface elements, commonly referred to as WIMP: windows, icons, menus, and pointers.

During that evolution some other contenders for the predominant interface, such as menu-only systems, did not attain dominance or widespread use. Instead, they were limited to niche applications, such as ATMs and televisions. A deeper analysis would show that many popular applications contain elements of each style. In effect, they are hybrids. For example, typical GUIs make use of menus (pull-down menus) and forms (dialog boxes and property sheets). In some cases, such as spreadsheets, the interaction style includes command-like elements, that is, complex formulas and sophisticated functions. Similarly, another class of successful applications, document processing, also introduces new concepts such as what you see is what you get, (WYSIWYG) that is, what is on the screen is a reasonably faithful rendition of what will be printed. In both cases, the elements of the GUI are accompanied by particularly useful and appealing aspects that propelled these applications to wide adoption. It is important to be cognizant of the fact that these approaches are neither preordained nor task neutral. WYSIWYG, for example, fundamentally shapes the way people spend their time composing text—simultaneously focusing on both form and content.

While a thoughtful interpretation of the history of human-computer interaction is complex and nuanced, certain generalizations are evident. The way in which humans interact with computers has evolved. That evolution has enabled more people to do more things with computers. It has led to a vast and rapid increase of the volume, scope, and diversity of the computer business.

There are many perspectives from which we could view this evolution. Some are merely tautological. For example, a common view is that more people use more computation because the barriers have been reduced and the functionality of the machines has increased. While true, that characterization provides precious little insight. To provide it some intellectual weight, we need to be clearer with respect to what barriers have been reduced and which functions have been provided. It is also useful to examine closely the development of certain computing niches, which thrive in limited but well-defined contexts.

Some of the early analysis of the GUI provided just this kind of deeper insight. Analysts pointed out that recognizing and choosing were easier than remembering then typing. In other words, with its menus, dialog boxes, icons, and familiar work spaces, the WIMP GUI represented a lower barrier for users than a command line interface. This difference becomes clearer if we consider specific applications. By and large, it is easier to learn and use a word processor to create simple documents than to edit in a markup language. In addition, the often-overlooked advantage of the computer, that is, that it produces revisable work products, was fully realized with a word processing system.

Functionality gains are also apparent for most users. Before the computer, a skilled typesetter could produce formatted documents, but the average citizen was confined to typing final work products in a mono-spaced font, with cumbersome correction tools. The combination of reduced thresholds for learning, easier recall, increased functionality for the average user, the reduced cost promised by Moore’s law, and the widespread capability to revise one’s work without redoing it signaled the explosive growth of a number of well-featured and -designed applications that adopted the desktop computer and the subsequent consignment of more traditional approaches to specialized niches.

When the desktop GUI was first created and made widely available, its ultimate fate was unclear. It was derided by many experts who coined its current name: the WIMP interface. Although WIMP stood for windows, icons, menus, and a pointer, it implied that the users of the GUI were not the manly men who had mastered the previous, more arcane ways of interacting with computers. Ultimately, those supposedly more manly folks were consigned to the social position of specialists or hobbyists. We can see this pattern in many other domains: personal transport, cooking, penmanship, and CB radio operators.

Now we stand at the brink of another potential evolution in computing. Natural user interfaces (NUIs) seem to be in a position similar to that occupied by the GUI in the early 1980s. Like the desktop GUIs, NUIs promise to reduce the barriers to computing still further, while simultaneously increasing the power of the user, and enabling computing to access still further niches of use. But just as GUIs did not simply make command systems easier, NUIs are not simply a natural veneer over a GUI. Instead, like GUIs, NUIs have a set of strengths based on what they make easier, how they make those things easier, how they shape the user’s interaction with technology, which niches they fit in, and whether or not these niches expand to dwarf the space occupied by traditional GUIs.

When examining this history and anticipating the future, we should not be distracted by single instances. Many of these will be failures and will not represent any overall trend. The failure of makes and models of some cars did not end the phase of personal transport. The failure of many GUI products and the inevitable consolidation of the marketplace did not impede the overall growth of personal computing or prevent GUIs from pre-eminence. It is as dangerous to generalize to the future based on a few examples as it is hard to anticipate the future when looking from our current perspective.

In this book we do not aim to provide an exhaustive overview of the NUI. We do not predict the future of human-computer interaction. We do not assume the predominance of NUI-based designs. We do not provide a complete set of rules for creating a successful NUI, because NUI is not yet at a state of evolution or standardi-zation to allow for such a definition.

We can make some relatively safe predictions. NUIs are here to stay. They either will find a successful niche, like menu system ATMs, or will come to dominate the computer landscape. If the latter comes to pass, we can still expect GUIs to persist in specialized environments. The ultimate evolution of the NUI will be determined not by the analysts and the critics, but by those who step forward and take the risk to build true NUI applications. Here we offer a way to think about NUIs that is counter to the predominant metaphor. This perspective on the NUI suggests that NUIs provide an enjoyable way for novices to move quickly and seemingly effortlessly to skilled practitioners. This approach involves more than being natural or intuitive. It means that the domain of use and the requirements of context are carefully assessed. It also means that the conventions of the GUI should be studiously ignored. It requires careful design and rigorous research. We give some guidance on how to do those things and how to re-conceptualize the NUI.

We offer a number of essays on the NUI and on methods that can be utilized to achieve it, written from the perspective of two journeyman user experience experts who have had the opportunity to immerse themselves in the nitty-gritty of designing, engineering, building, testing, researching, and shipping multiple products that have come to define the category. Overall, the essays are intended to provide a nuanced set of perspectives on NUI systems. These perspectives range from specific descriptions of the syntax and semantics of the NUI to broad analyses of the NUI in terms of the history of computing. Each essay is composed of the same essential elements. Each concludes with specific, concrete design guidelines meant to help take words into action. Those guidelines are divided into three types: must, those that we believe are necessary conditions to achieving a natural-feeling experience; should, those elements that, while nonessential, have been found to add greatly to achieving such an experience; and could, those guidelines that may apply only to certain contexts or