Human Bond Communication: The Holy Grail of Holistic Communication and Immersive Experience

By Sudhir Dixit and Ramjee Prasad

This book approaches the topic area of the Internet of Things (IoT) from the perspective of the five types of human communication. Through this perspective on the human communication types, the book aims to specifically address how IoT technologies can support humans and their endeavors. The book explores the fields of sensors, wireless, physiology, biology, wearables, and the Internet. This book is organized with five sections, each covering a central theme;

Section 1: The basics of human bond communication

Section 2: Relevance IoT, BAN and PAN

Section 3: Applications of HBC

Section 4: Security, Privacy and Regulatory Challenges

Section 5: The Big Picture (Where do we go from here?)

• Language: English
• Publisher: Wiley
• Release date: Feb 22, 2017
• ISBN: 9781119341413

Book preview

1

Introduction to Human Bond Communication

Sudhir Dixit¹,³ and Ramjee Prasad¹,²

¹ CTIF Global Capsule (CGC), Rome, Italy

² School of Business and Social Sciences, Aarhus University, Aarhus, Denmark

³ Basic Internet Foundation, Oslo, Norway

1.1 Introduction

Information and communications technologies (ICT) have progressed rapidly in this millennium for people to communicate and exchange information using multimedia (speech, video/image, text), and the same has extended to Internet of things (IoT) and machine‐to‐machine and machine‐to‐human communication. This trend is only going to accelerate in the years to come with powerful human–computer interaction technologies to deliver engaging and intuitive experiences. But these developments have remained confined to only the sensing and transmission of aural and optical information in the digital domain through the use of microphone, camera, speaker, and display devices. However, the ability to integrate the other three sensory features, namely, olfactory (smell), gustatory (taste), and tactile (touch) in information transfer and replication to deliver being there in‐person experience, are still far from reality. Human bond communication (HBC) is a novel concept that incorporates all five sensory information from sensing, to digitization, to transmission and replication at the receiver to allow more expressive, engaging, realistic, and holistic information between humans [1] and in some cases between humans and machines such as in remote sensing and robotic control. Lack of inclusion of the other three senses in the digital world of ICT limits the full exploitation of the cognitive ability of the human mind for a fuller perceptive information experience. The five senses and the environment interact in interesting ways to become complete knowledge for human species as its brain has developed and evolved naturally from the time it came into existence on this planet. The profoundness of perceiving an object depends on the incisiveness and extensity of the sense organs. Incisiveness refers to the granularity and minute details or variations an organ can detect, and extensity refers to the range of the physical property that it can detect.

In the traditional world of digital information exchange, the subject is described and presented partially via its aural and optical rendering, which gives a sense of incompleteness and dissatisfaction in fully understanding the subject. In the present era of ever increasing competition through innovation, inclusion of all five senses to deliver complete experience is the holy grail of the research community. Products have begun to appear through wearables and other embedded sensors in the body, but sensors exploiting touch, taste, and smell and embedding them into products remains a distant reality and is an area of intense research today as would become evident from the chapters included in this book.

Auditory and optical sensing is wave based. In audio sound travels through waves and can be sensed and digitized. Similarly, light shining on an object is reflected in electromagnetic radiation, and a part of this spectrum (called visible light in the range of wavelength 390–700 nm) is visible to the human eye and when rendered on the retina becomes a visual formulation of the object in the nervous system. The camera does this nicely to capture an object visually and digitize it for transmission. When rendered remotely on a display device in 2‐D or 3‐D, a person can see the object as though he or she was seeing it by being physically present at a location where the camera was located. Other human senses (tactile, olfactory, gustatory) utilize particle‐based sensing and rely on smearing the object with the sensors. Building such sensors remains a technological challenge for the research community because each type of sensor must deal with large range of parameters and their wide spectrum. Digitization of these parameters is also a major challenge, and even if some finite widely prevalent values can be captured and digitized, their replication from the digital domain to the analog domain and their sensing by a person in an unobtrusive manner is a complex human‐sensor interface issue. Figure 1.1 illustrates the HBC system and depicts what is possible today and what is not.

Flow diagram illustrating the human bond communication (HBC) concept. From subject, arrows point to the subflows olfactory, auditory, optical, gustatory, and tactile.

Figure 1.1 An illustration of human bond communication (HBC) concept. CTP, communication technology platform.

Prasad [1]. Reproduced with the permission of Springer.

HBC is about understanding the human sensory functionality and works similar to human sensory system, which includes providing a perceptually holistic understanding of an object combining all five senses while incorporating the object’s environment.

1.2 Human Bond Communication (HBC) Architecture

The HBS architecture extrapolates the contemporary communications architecture to include the missing three senses (or types of sensors): tactile, olfactory, and gustatory, not in use today along with the aural and optic sensors. Nevertheless, some limited deployments are happening in machine‐to‐machine and machine‐to‐human communication use cases where robots are being used, such as in industry, law enforcement, hazardous material handling, and surveillance. A proposed architecture is shown in Figure 1.2 [1]. It should be noted that the architecture goes beyond capturing just a person’s senses to also deploying all five types of sensors in any environment to capture smell (e.g., types of smoke, air pollutants), tactile information (e.g., surface roughness, temperature, wind speed), and taste (e.g., liquids, dirt, waste) and learning about an object or its surroundings.

Flow diagram of a proposed HBC architecture, with illustrations representing senducer colocated, sensucer distributed, human bond sensorium (HBS), and optical and aural explicator.

Figure 1.2 A proposed HBC architecture.

Prasad [1]. Reproduced with the permission of Springer.

The system consists of the three key building blocks: (i) senducers that sense the characteristic parameters through stimuli and transform those analog values to electrical and digital domain for further processing and transmission, (ii) human bond sensorium (HBS) that collects the data from the senducers, processes them to make them consumable for the human perceptive system (i.e., human consumption) by removing a large amount of nonusable and redundant data and information, transmits it to the far end to the receiver gateway, and (iii) human perceivable transposer (HPT) that transforms the received digital data to human consumable format, which includes replication of the senses to a form that one would expect if the person was physically present at the site where the sensory data were collected through senducers. Until such time the replication solutions are not available, the HPT may prefer to render the non‐audio–visual sense data through digital means (such as colors, emoticons, text, other gestures like vibration, pressure, temperature, etc.).

1.3 About the Book

Our journey into the world of intuitive and rich communication begins with the vision of extending the contemporary form of digital communication to more natural human‐to‐human communication through the novel concept of HBC. This chapter has introduced that grand vision. HBC closely embraces the advances in the fields of sensors and wireless distributed computing, physiology, biology, wearables, chemistry, medicine, analytics, Internet, and so on that will be required to bring that vision closer to reality. Therefore, this book has included invited chapters from the experts in the various fields who look at the HBC through their perspectives and delve into the technical challenges that are before the research community. They also discuss the numerous business opportunities that are unlocked due to the intersection of the innovations emanating from interdisciplinary research and entrepreneurship. Whenever appropriate the authors have looked at the historical trends to present their ideas and invoke discourse. Figure 1.3 illustrates some of the key concepts and technologies that will have a profound impact on HBC. These are discussed in the various chapters of the book.

Converging radial diagram depicting key concepts and technology enablers, pointing towards an oval in the center labeled HBC with a smiley just below it.

Figure 1.3 Key concepts and technology enablers for HBC.

Chapter 1 is an introduction of the book and lays the foundation of the grand vision for the HBC concept.

Chapter 2 presents the basic concepts behind HBC and provides an insight in the ongoing research related to the concepts of human sensory and emotional replication, physical world augmentation, and human umwelt expansion. This chapter then describes an HBC architecture and discusses its convergence with ICT. Additionally, the chapter discusses the potentials of HBC and gives a vision of possible future applications and services.

In Chapter 3, the authors postulate that the provision of enhanced augmented reality services to mobile users based on the HBC paradigm will rely on the definition of a high performance, high efficiency, and highly reconfigurable network architecture for the exchange of all the five sensory features. The objective of this chapter is to propose a novel HBC communication network architecture that is able to support the provision of such novel services incorporating all five senses. Starting from the definition of the main network, security, and quality of service requirements for HBC, a 5G network architecture based on software‐defined networking, network function virtualization, and Fog–Edge computing paradigms is presented. The main enabling technologies, including WBAN, localization techniques, and content‐oriented networking, are described together with some possible solutions to be adopted to cope with the security threats that may affect the success of HBC services.

Chapter 4 is about data mining of the human being. After describing the definition of data mining (also known as knowledge discovery in databases (KDD)) as the process of analyzing data from different perspectives and extracting hidden information and identifying patterns or relationships among the data, the author describes the various models and thereafter focuses on data mining of the human being, where the data is any fact, number, or text regarding a human being. The data can describe the human being at any level, from atoms to cells, to organs, to social level.

Chapter 5 provides an overview of ongoing research on the proposed models for IoT and summarizes their advantages and disadvantages in the context of human centric IoT. After describing potential human centric sensing (HCS) scenarios that require changes in how HCS‐based IoT should be modeled, the chapter proposes a macro‐level model and describes how it can help to achieve simplicity in the complex IoT world by understanding how to get from micro‐complexity to macro‐simplicity. It also describes HCS networks and federations and their modeling and later goes into end‐to‐end security and privacy issues. This chapter also touches upon the concept of tactile Internet as the enabler for HCS IoT.

Chapter 6 describes human body (i.e., body as a node (ByN)) as the main actor in the ICT systems, which plays an active role as a node of the ICT network, as well as part of the ICT user terminal. In addition, intrusion with technological ICT devices in the body provides to the body itself a great opportunity for the early monitoring and the daily cure of critical pathologies. After describing the ByN approach, this chapter delves into applying the underlying concept to oral cavity and presents an overview of the research in this field with its implications and perspectives for the future.

Chapter 7 explores the novel machine learning‐based approaches to cognitive radio (CR) systems developed that will lead to innovative HBC applications to serve the needs of a community. This chapter formulates novel algorithms to share spectrum through dynamic spectrum leasing methodologies and adaptive policy decision, making processes that seek to maximize the utilization of available scarce spectrum.

Chapter 8 is about the application of ICT for wildlife preservation. It is well known that various governments and nongovernmental organizations have launched diverse technology‐driven programs to arrest unprecedented decline and wherever possible successfully restore and rehabilitate wild animal species. While timely integration of technology into wildlife research, monitoring, and conservation in the last couple of decades have definitely yielded positive results, future technology solutions are likely to cater relevant information for decision making and sound management based on application of five human senses instead of just two most common human senses (seeing and hearing). This chapter describes how the sensors for all five senses can be utilized in the solutions for wildlife preservation and concludes that there is an urgent need of sharing mental models between the stakeholders, specifically between the conservationists and technologists.

Chapter 9 investigates the security and privacy issues in HBC. Three different HBC levels are defined and analyzed what these really mean. The approach is to extrapolate and speculate about future progress but to put effort into keeping the extrapolations plausible. Many different fields are involved. Therefore, this chapter serves as a survey about possible future advances in the various fields that will have an impact on HBC. The security and privacy challenges are enormous and they need to be resolved. Thus, this chapter also serves as an urgent call for research in security and privacy issues.

Chapter 10 describes how the Internet of everything (IoE) is the networked connection of people, processes, data, and things. It contains the IoT and the Internet of humans (IoH). The stream of data the IoE will produce can be turned into actionable information and will provide numerous opportunities and will be omnipresent. This chapter attempts to answer the question: Will HBC, the novel concept that incorporates smell, taste, and touch in the exchange of information, be feasible? If the technology to create an HBC ecosystem succeeds, it will bring transformational changes and a paradigm shift. This chapter fast forwards to year 2050 to envision the evolution of the IoE and to predict the anticipated impact and opportunities.

Chapter 11 focuses on the use of HBC for health applications and in particular on the ethical and legal issues that arise. For many years, the use of ICT in medicine was limited to allowing communications between remote patients and doctors (telemedicine). In the recent years, there has been a rapid evolution in the use of ICT in health. The IoT framework allows a pervasive monitoring of anything around and eventually inside us, and this could really open the way to novel diagnostic and therapeutic methods. This rapid evolution has also posed several challenges as many things are not regulated yet. This chapter attempts to address several key questions: What will happen when HBC will be a reality? Would HBC really enable novel applications in health? And if so, would that require new regulations?

Chapter 12 delves into the challenges in intellectual property (IP) and ICT law that will potentially come with the introduction of HBC. From a legal point of view, HBC means that attorneys and legal professionals should be able to conceive in short time the framework of a smart regulation, in order to provide the principles that will be governing the interaction between human beings, machines, and human umwelt expansion. The opportunities that will be unlocked with HBC will undoubtedly trigger the evolution of IP and ICT regulations in several areas. Because of the need for coherency, a multidisciplinary approach will be the key for reaching consensus among different experts and realize full implementation of the legal and general aspects of HBC.

Chapter 13 presents a historical view of the developments in wireless communication brought about by the changes in paradigm of communications from station to station to person to person and because of technology improvement that made the telephone terminal a multimedia mobile device. This chapter then delves into what is next for wireless communications? While future research could be either on technology or on applications, in reality, the success depends on several other factors such as fashion design, creating user needs, user experience, business models, and so on. These other factors require collaboration among teams in quite different areas that we call for interdisciplinary research and development. This chapter, therefore, focuses on the need for this collaborative approach for innovation and commercial success.

Chapter 14 is a broad overview of how communication among humans originated over the history of mankind and how it has evolved over time with the advances in technology. It discusses the paradox of users that while on the one side they have had choice of platforms and applications to provide enormous opportunities to exchange information in increasingly efficient ways, on the other side they chose the platforms that use only the least significant parts of the messages (i.e., text). This chapter quantifies how much information is included in text, speech, and video/image. Then it discusses technology as an enabler for improving communication over distances and differences between the various platforms, why customers seem not to choose the channel that offers optimum communication, and what are the technical characteristics of the various channels (face to face, letter, telegraph, voice, video, television, SMS/MMS, email, etc.). After presenting the data on how much data the users consume through different channels, this chapter goes into the psychological impact of the various communication channels and finally how the inclusion of the remaining three senses (touch, taste, smell) would further augment the quality of communication.

In summary, the book defines the concept of HBC, sets out its vision, and provides details on the technologies that are driving the realization of the vision and how it would transform the communication experience between humans while also significantly unlock the business opportunities between humans, machines, and their environment. This book also goes into the details of the security, privacy, IP, and regulatory challenges that must be addressed for HBC to be commercially realized.

Reference

1 Prasad, R. (2016). Human bond communication. Wireless Personal Communications, 87(3), 619–627, Springer, New York.

2

General Concepts Behind Human Bond Communications

Liljana Gavrilovska1, Valentin Rakovic1, and Sudhir Dixit2,3

¹ Ss. Cyril and Methodius University in Skopje, Skopje, Macedonia

² CTIF Global Capsule (CGC), Rome, Italy

³ Basic Internet Foundation, Oslo, Norway

2.1 Introduction

The era of aural and visual communication has been with us for quite a long time, and ICT devices and applications have made tremendous strides in past years by exploiting the exchange of the information associated with these two senses between humans and machines. With respect to aural (or speech and sound) communication, not only have humans exchanged information through vocoders, but also machine‐based speech recognition, recording, and synthesis (including through IVRs) are dominating in the present era of digital society. Similarly, visual communication has been growing by leaps and bounds between humans and machines in all forms of combinations. The machines (i.e., robots) are being equipped by voice and video/imaging sensors for machine learning and real‐time actuarial functions or for data gathering and forwarding to the destination nodes. Despite all the advances in aural and visual communication, a complete holistic presence, which mimics the face‐to‐face interaction through ICT, has evaded the technologists so far and constitutes the holy grail of major advances in communication involving human‐to‐human (H2H), human‐to‐machine (H2M), and machine‐to‐machine (M2M) communication and information exchange. Seamless integration of the remaining three senses (touch sensing and haptic feedback commonly referred to tactile system, smell sensing commonly referred to olfactory system, and taste sensing referred to gustatory system) will improve the understanding of the physical object (person, environment, anything else) in a way that we can barely fathom to imagine their impact. The interaction between the five senses enriches the information content tremendously, which is equivalent to being there. Just imagine how immersive the experience would be from looking at something while touching, hearing, smelling, and sensing its environment remotely! Gustatory experience, while requiring somewhat more active participation, can add to complete experience. A very good example when all the five senses are involved is when we go to a restaurant, to a bar, or on a picnic.

Incorporating the ability to sense all the five senses (or a subset of them) is playing a major role in robotics. Analysis of this five‐dimensional data set can provide invaluable information to a robot to learn its environment and how to react, especially in hazardous situations. Alternatively, it can simply pass that information on to its handlers who can then decide on what to do next through that robot remotely or utilize other means to react. Thus, incorporating more than the aural and optical sensory information in the messages is not only limited to H2H communication but also extends to machines and Internet of things (IoT). So far, tactile sensing and feedback (collectively known as haptic or kinesthetic communication) has made significant advances to be implemented in tactile Internet and applications that are remote and based on feel (i.e., cause) and react (control to action) principle [1–3]. Touch conveys such important information as smoothness, texture, shape, pressure, and so on. Examples of such devices are joysticks, data gloves with felt sensors, or other tactile sensors. These are increasingly being used in computer games and remote computer applications, such as remote games and telerobotics. Figure 2.1 illustrates the scope of interactions among five sensory features in H2H, H2M, and M2M communications.

Diagram of communication among five senses inside 2 circles labeled human (H2H) and machine (M2M) with double-headed arrow in the center labeled H2M/M2H.

Figure 2.1 Communication of sensory information between humans and machines.

2.2 Definition of Human Bond Communication

The concept of communication of human senses between humans or machines is not necessarily a new concept, but its development and implementation into systems through sensors, how these are packaged for transfer across distances, and how these are rendered are in rather the early stage of implementation and commercialization. It involves interdisciplinary aspects both in theory and implementation in the fields of physics, chemistry, biology, medicine, neuroscience, and engineering. The work mainly entails (i) human sensory and emotional sensing and replication, (ii) physical world augmentation (PWA), and (iii) sensory substitution and augmentation through umwelt expansion. These are explained in the following text [5].

Human senses and emotional sensing and replication. Human sensing and replication typically refers to how humans perceive the world around them through their five senses (including deriving additional intelligence from how these senses interact with each other thereby building a much richer knowledge space in the five‐dimensional space). Replication means mimicking and mirroring human senses at the receiving end through artificial means after they have been transmitted digitally. Traditional communication technologies already accomplish this for speech and video involving microphones, cameras, speakers, and display devices for many applications. Human emotional sensing and replication represent a much more advanced concept whereby the brain of a person communicates with another person’s brain directly through brain‐to‐brain (B2B) interface (BBI) comprising brain‐to‐computer interface (BCI) and computer‐to‐brain interface (CBI). In recent years, there has been significant progress in research in both the BCI and CBI.

PWA. Interaction with the physical world represented digitally by means of natural human senses represents augmenting the experience that is rich and intuitive. In short, like in the real world, a person utilizes the five natural senses to interact with an object to take appropriate actions and decisions; similarly the same is done in the digital world. The physical world is presented digitally on a device and the user interacts with it through the natural human senses. A good example of this is a wearable device, called Sixth Sense device, developed by the researchers at the Massachusetts Institute of Technology in 2009.

Sensory substitution and augmentation through umwelt expansion. The term umwelt was first introduced by Jacob von Uexküll [6] and is explained in much more detail in Section 2.7. In brief, they discovered that various organisms in the same ecosystem pick up different signals from the surroundings in the environment they are in. The same applies for the humans as well and thus represents the entire objective reality that goes much beyond the five senses. These environmental signals are fed through unusual sensory channels and together open up interesting interaction possibilities and opportunities between the digital world and the human brain.

Human bond communication (HBC) as a concept and terminology was first introduced by Prasad [7] as a holistic approach to describe and transmit the features of a subject in the way humans perceive it; therefore, all five senses need to be considered and exploited, transmitted, and recreated for complete understanding of the subject to the extent possible by a human and as mutually agreed by the interacting users of HBC. Thus, HBC’s scope is end‐to‐end communication, which includes both humans and machines as end points.

2.3 HBC Architecture and Convergence with ICT

Figure 2.2 illustrates a simplified systems view of an end‐to‐end HBC system. HBC represents the subsystem that multiplexes or demultiplexes the digital data streams corresponding to the various sensory information, network protocols, and network interface to the external world. In H2M HBC systems the communication of information is usually asymmetric with more sense information flowing from a machine to a user but only control information (such as touch, speech) flowing from user to the machine. To be successful in HBC, the system must perform a number of functions with minimum latency to deliver a real‐time immersive experience. These are shown in Figure 2.3.

Schematic of a systems level representation of the scope of human bond communication (HBC) system. It features the sensors and digital converters linked to the IP network.

Figure 2.2 A systems level representation of the scope of human bond communication (HBC) system.

Flow diagram illustrating steps to successful implementation of an HBC system, from mapping of physical subject into five senses to replicating of sensor data to five senses.

Figure 2.3 Steps to successful implementation of an HBC system.

Schematic representation of an HBC‐enabled user and/or a machine (such as a robotic platform) is depicted in Figure 2.4. A complementary stack is implemented in the receiver subsystem of an end‐to‐end HBC implementation. The communication part of the HBC system remains unchanged as in the conventional multimedia‐enabled applications. The recent advances in virtualization and cloud computing can be extended to multisensory applications involving sensing beyond just audio and visual for signal processing, analytics, measurement, and control.

Schematic of a proposed HBC platform with a multisensory user application. It features application, encryption and security controls, analytics, decision, and control engine, network layer, and the internet.

Figure 2.4 A proposed HBC platform with a multisensory user application or a machine.

2.4 Human Emotional Messaging

One of the main objectives of HBC is to facilitate the possibility to exchange information on how humans perceive the world. The HBC paradigm exploits the emotional messaging concept in order to leverage this process. This section will provide an overview of the key features and technical enablers related to the emotional messaging, highlighting their main research focus and generic issues.

Human emotional messaging represents a technological concept that provides the humans with the possibility to send words, images, and even thoughts directly to the minds of others. The common principle of human information exchange has been supported by the sensory (vision and hear) and motor facets of the human body, which are inept of facilitating the full potential of the emotional messaging. Novel and cutting edge communication concepts, such as the B2B communication, facilitate the emotional messaging. As demonstrated in Refs. [8, 9], a direct B2B communication can be achieved by exploiting a BBI. The BBI is consisted of two distinct interfaces, as illustrated in Figure 2.5:

BCI. The BCI is designed to read (or decode) useful information from neural activity.

CBI. The CBI facilitates the writing (or encoding) of digital information back into neural activity.

Flow diagram illustrating brain-to-brain communication from BCI to CBI depicted by an arrow (top) and flow from brain activity decoding/reading to brain activity encoding/writing (top).

Figure 2.5 Brain‐to‐brain communication.

Recent research activities have experienced substantial progress in both BCI and CBI communication directions. Specifically several research groups have demonstrated the possibility of decoding motor [9], conceptual [11], and visual [10, 12] information from neural activity via a range of recording techniques such as:

Implanted electrodes. The implanted electrodes are implanted directly into the grey matter of the brain during neurosurgery. Because they lie in the grey matter, invasive devices produce the highest quality of brain activity signals with respect to the BCI procedures. However, they are prone to scar tissue buildup, causing the signal to become weaker, or even nonexistent, as the body reacts to a foreign object in the brain [13].

Functional near‐infrared spectroscopy (fNIRS). The fNIRS method exploits the near‐infrared spectroscopy (NIRS) for the purpose of functional neuroimaging. By utilizing fNIRS, the brain activity is measured through hemodynamic responses associated with neuron behavior. With respect to the BCI, the fNIRS method provides the possibility for extracting brain information with high spatial resolution [14]. However, the method lacks good temporal resolution and has a bulky design.

Electroencephalography (EEG). The EEG method is a noninvasive