Tactile Internet: with Human-in-the-Loop

By Frank H. P. Fitzek

Tactile Internet with Human-in-the-Loop describes the change from the current Internet, which focuses on the democratization of information independent of location or time, to the Tactile Internet, which democratizes skills to promote equity that is independent of age, gender, sociocultural background or physical limitations. The book promotes the concept of the Tactile Internet for remote closed-loop human-machine interaction and describes the main challenges and key technologies. Current standardization activities in the field for IEEE and IETF are also described, making this book an ideal resource for researchers, graduate students, and industry R&D engineers in communications engineering, electronic engineering, and computer engineering.

• Provides a comprehensive reference that addresses all aspects of the Tactile Internet – technologies, engineering challenges, use cases and standards
• Written by leading researchers in the field
• Presents current standardizations surrounding the IETF and the IEEE
• Contains use cases that illustrate practical applications

• Language: English
• Publisher: Academic Press
• Release date: Mar 6, 2021
• ISBN: 9780128213551

Book preview

Front Cover for Tactile Internet

Tactile Internet

with Human-in-the-Loop

First edition

Frank H.P. Fitzek

Technische Universität Dresden, Dresden, Germany

Shu-Chen Li

Technische Universität Dresden, Dresden, Germany

Stefanie Speidel

National Center for Tumor Diseases, Partner Site Dresden, Division of Translational Surgical Oncology, Dresden, Germany

Thorsten Strufe

Karlsruhe Institute of Technology, Karlsruhe, Germany

Meryem Simsek

University of California, Berkeley, International Computer Science Institute, Berkeley, CA, United States

Martin Reisslein

Arizona State University, School of Electrical, Computer and Energy Engineering, Tempe, AZ, United States

publogo

Table of Contents

Cover image

Title page

Copyright

List of contributors

About the editors

Preface

Acknowledgments

Acronyms

Chapter 1: Tactile Internet with Human-in-the-Loop: New frontiers of transdisciplinary research

Abstract

1.1. Motivation and vision of TaHiL

1.2. Research objectives to meet the challenges of TaHiL

1.3. A synergistic research program

1.4. Research outreaches and societal impacts

1.5. Conclusion and outlook

Bibliography

Part 1: Domains of applications

Introduction

Outline

Chapter 2: Surgical assistance and training

Abstract

2.1. Introduction

2.2. Human-to-machine: Modeling surgical skills

2.3. Machine-to-human: Surgical training

2.4. Human–machine collaboration: Context-aware assistance

2.5. Conclusion and outlook

Bibliography

Chapter 3: Human–robot cohabitation in industry

Abstract

3.1. Introduction

3.2. Model-based cobotic cells

3.3. Tactile robots in the Tactile Internet

3.4. Embedded hardware for robotics

3.5. Synergistic links

3.6. Conclusion and outlook

Bibliography

Chapter 4: Internet of Skills

Abstract

4.1. Aims of the Internet of Skills

4.2. State-of-the-art research: Skill learning and technology

4.3. Key requirements and challenges in designing skill learning with TaHiL technology

4.4. Beyond the state-of-the-art approach: Interdisciplinary collaboration

4.5. Conclusion and outlook

Bibliography

Part 2: Key technology breakthroughs

Introduction

Outline

Chapter 5: Haptic codecs for the Tactile Internet

Abstract

5.1. Scope of haptic codecs

5.2. State-of-the-art research and technology

5.3. Key challenges

5.4. Approaches addressing challenges and beyond the state of the art

5.5. Synergistic links

5.6. Conclusion and outlook

Bibliography

Chapter 6: Intelligent networks

Abstract

6.1. Introduction and motivation

6.2. Evolution of communication networks

6.3. TaHiL communication concept

6.4. Architecture discussion

6.5. TaHiL testbeds

6.6. Synergy and collaboration

6.7. Conclusion and outlook

Bibliography

Chapter 7: Augmented perception and interaction

Abstract

7.1. Milestones for building Human-in-the-Loop systems

7.2. State-of-the-art research and technology

7.3. Identified key challenges of current research and technology

7.4. Research within TaHiL

7.5. Conclusion and outlook

Bibliography

Chapter 8: Human-inspired models for tactile computing

Abstract

8.1. Motivation and aims

8.2. Neuroscientific insights into human decision-making

8.3. Human-inspired learning

8.4. Synergetic links

8.5. Conclusion and outlook

Bibliography

Part 3: Fundamental challenges

Introduction

Outline

Chapter 9: Human perception and neurocognitive development across the lifespan

Abstract

9.1. Introduction: Multisensory perception is the gateway for interactions

9.2. State-of-the-art research on multisensory perception

9.3. Outstanding challenges in current research

9.4. Beyond the state of the art: synergistic research across disciplines

9.5. Conclusion and outlook

Bibliography

Chapter 10: Sensors and actuators

Abstract

10.1. Sensors and actuators of the future

10.2. State of the art

10.3. Key challenges

10.4. Beyond the state-of-art approaches

10.5. Synergistic links

10.6. Conclusion and outlook

Bibliography

Chapter 11: Communications and control

Abstract

11.1. Motivation

11.2. Research in the field of control

11.3. Research in the field of communications

11.4. Conclusion and outlook

Bibliography

Chapter 12: Tactile electronics

Abstract

12.1. Goals

12.2. State of the art

12.3. Research challenges

12.4. Research approaches

12.5. Collaboration

12.6. Conclusion and outlook

Bibliography

Chapter 13: Tactile computing: Essential building blocks for the Tactile Internet

Abstract

13.1. Introduction

13.2. Safe and secure infrastructure

13.3. World capturing and modeling

13.4. Scalable computation

13.5. Context-adaptive software for the Tactile Internet

13.6. Self-explanation for Tactile Internet applications

13.7. Conclusion and outlook

Bibliography

Part 4: Technological standards and the public

Introduction

Outline

Chapter 14: Traces for the Tactile Internet: Architecture, concepts, and evaluations

Abstract

14.1. Introduction

14.2. Tactile traces generic system overview

14.3. Tactile trace content examples

14.4. Application scenarios

14.5. Conclusion and outlook

Bibliography

Chapter 15: Tactile Internet standards of the IEEE P1918.1 Working Group

Abstract

15.1. Introduction

15.2. Definition of the Tactile Internet

15.3. IEEE P1918.1 Tactile Internet Standards Working Group

15.4. IEEE P1918.1 architecture

15.5. IEEE P1918.1 use cases

15.6. IEEE P1918.1 haptic codecs

15.7. Conclusion and outlook

Bibliography

Chapter 16: Public opinion and the Tactile Internet

Abstract

16.1. Introduction

16.2. Theoretical background

16.3. Survey on the public opinion regarding the Tactile Internet

16.4. Survey results

16.5. Discussion

16.6. Conclusion and outlook

Bibliography

Bibliography

Bibliography

Index

Copyright

Academic Press is an imprint of Elsevier

125 London Wall, London EC2Y 5AS, United Kingdom

525 B Street, Suite 1650, San Diego, CA 92101, United States

50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States

The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom

Copyright © 2021 Elsevier Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher's permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.

Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-12-821343-8

For information on all Academic Press publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Mara Conner

Acquisitions Editor: Tim Pitts

Editorial Project Manager: Andrea Gallego Ortiz

Production Project Manager: Nirmala Arumugam

Designer: Matthew Limbert

Typeset by VTeX

List of contributors

Gökhan Akgün     Technische Universität Dresden, Dresden, Germany

Ercan Altinsoy     Technische Universität Dresden, Dresden, Germany

Uwe Aßmann     Technische Universität Dresden, Dresden, Germany

Christel Baier     Technische Universität Dresden, Dresden, Germany

Tina Bobbe     Technische Universität Dresden, Dresden, Germany

Karlheinz Bock     Technische Universität Dresden, Dresden, Germany

Sebastian Bodenstedt     National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany

Juan A. Cabrera G.     Technische Universität Dresden, Dresden, Germany

Lingyun Chen     Technical University of Munich, Munich, Germany

Chokri Cherif     Technische Universität Dresden, Dresden, Germany

Darío Cuevas Rivera     Technische Universität Dresden, Dresden, Germany

Raimund Dachselt     Technische Universität Dresden, Dresden, Germany

Zaher Dawy     American University of Beirut, Beirut, Lebanon

Annika Dix     Technische Universität Dresden, Dresden, Germany

Clemens Dubslaff     Technische Universität Dresden, Dresden, Germany

Sebastian Ebert     Technische Universität Dresden, Dresden, Germany

Mohamad Eid     New York University Abu Dhabi, Abu Dhabi, United Arab Emirates

Frank Ellinger     Technische Universität Dresden, Dresden, Germany

Sven Engesser     Technische Universität Dresden, Dresden, Germany

Gerhard P. Fettweis     Technische Universität Dresden, Dresden, Germany

Christof W. Fetzer     Technische Universität Dresden, Dresden, Germany

Frank H.P. Fitzek     Technische Universität Dresden, Dresden, Germany

Norman Franchi     Technische Universität Dresden, Dresden, Germany

Isabel Funke     National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany

Eva Goebel     Technische Universität Dresden, Dresden, Germany

Diana Göhringer     Technische Universität Dresden, Dresden, Germany

Dominik Grzelak     Technische Universität Dresden, Dresden, Germany

Başak Güleçyüz     Technical University of Munich, Munich, Germany

Sami Haddadin     Technical University of Munich, Munich, Germany

Lutz M. Hagen     Technische Universität Dresden, Dresden, Germany

Simon Hanisch     Technische Universität Dresden, Dresden, Germany

Ardhi Putra Pratama Hartono     Technische Universität Dresden, Dresden, Germany

Rania Hassen     Technical University of Munich, Munich, Germany

Adamantini Hatzipanayioti     Technische Universität Dresden, Dresden, Germany

Jens R. Helmert     Technische Universität Dresden, Dresden, Germany

Diego Hidalgo     Technical University of Munich, Munich, Germany

Oliver Holland     Advanced Wireless Technology Group, Ltd., London, United Kingdom

Thomas Hulin     German Aerospace Center (DLR), Oberpfaffenhofen, Germany

Sebastian A.W. Itting     Technische Universität Dresden, Dresden, Germany

Lars Johannsmeier     Technical University of Munich, Munich, Germany

Stefan J. Kiebel     Technische Universität Dresden, Dresden, Germany

Konstantin Klamka     Technische Universität Dresden, Dresden, Germany

Stefan Köpsell     Technische Universität Dresden, Dresden, Germany

Jens Krzywinski     Technische Universität Dresden, Dresden, Germany

Vincent Latzko     Technische Universität Dresden, Dresden, Germany

Simone Lenk

Technische Universität Dresden, Dresden, Germany

Fraunhofer-Gesellschaft, Dresden, Germany

Shu-Chen Li     Technische Universität Dresden, Dresden, Germany

Jakub Limanowski     Technische Universität Dresden, Dresden, Germany

Tianfang Lin     Technische Universität Dresden, Dresden, Germany

Yun Lu     Technische Universität Dresden, Dresden, Germany

Lisa-Marie Lüneburg     Technische Universität Dresden, Dresden, Germany

Christian Mayr     Technische Universität Dresden, Dresden, Germany

Sebastian Merchel     Technische Universität Dresden, Dresden, Germany

Johannes Mey     Technische Universität Dresden, Dresden, Germany

Annett Mitschick     Technische Universität Dresden, Dresden, Germany

Jens Müller     Technische Universität Dresden, Dresden, Germany

Evelyn Muschter     Technische Universität Dresden, Dresden, Germany

Susanne Narciss     Technische Universität Dresden, Dresden, Germany

Krzysztof Nieweglowski     Technische Universität Dresden, Dresden, Germany

Andreas Nocke     Technische Universität Dresden, Dresden, Germany

Andreas Noll     Technical University of Munich, Munich, Germany

Luca Oppici     Technische Universität Dresden, Dresden, Germany

Sharief Oteafy     DePaul University, Chicago, IL, United States

Sebastian Pannasch     Technische Universität Dresden, Dresden, Germany

Michael Panzirsch     German Aerospace Center (DLR), Oberpfaffenhofen, Germany

Johannes Partzsch     Technische Universität Dresden, Dresden, Germany

Dirk Plettemeier     Technische Universität Dresden, Dresden, Germany

Ariel Podlubne     Technische Universität Dresden, Dresden, Germany

Martin Reisslein     Arizona State University, Tempe, AZ, United States

Dominik Rivoir     National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany

Christian Scheunert     Technische Universität Dresden, Dresden, Germany

René Schilling     Technische Universität Dresden, Dresden, Germany

Anna Schwendicke     Technische Universität Dresden, Dresden, Germany

Patrick Seeling     Central Michigan University, Mount Pleasant, MI, United States

Merve Sefunç     Technische Universität Dresden, Dresden, Germany

Meryem Simsek     International Computer Science Institute, Berkeley, CA, United States

Harsimran Singh     German Aerospace Center (DLR), Oberpfaffenhofen, Germany

Stefanie Speidel     National Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany

Eckehard Steinbach     Technical University of Munich, Munich, Germany

Thorsten Strufe     Karlsruhe Institute of Technology, Karlsruhe, Germany

Ronald Tetzlaff     Technische Universität Dresden, Dresden, Germany

Andreas Traßl     Technische Universität Dresden, Dresden, Germany

Andrés Villamil     Technische Universität Dresden, Dresden, Germany

Uwe Vogel     Fraunhofer-Gesellschaft, Dresden, Germany

Felix von Bechtolsheim     Technische Universität Dresden, Dresden, Germany

Jens Wagner     Technische Universität Dresden, Dresden, Germany

Lisa Weidmüller     Technische Universität Dresden, Dresden, Germany

Jürgen Weitz     Technische Universität Dresden, Dresden, Germany

Hans Winger     Technische Universität Dresden, Dresden, Germany

Xiao Xu     Technical University of Munich, Munich, Germany

Jiajing Zhang     Technische Universität Dresden, Dresden, Germany

Sandra Zimmermann     Technische Universität Dresden, Dresden, Germany

About the editors

Frank H.P. Fitzek is a Professor and head of the Deutsche Telekom Chair of Communication Networks at Technische Universität Dresden (TUD) coordinating the 5G Lab Germany since 2014. Since 2019 he is a speaker of the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) Cluster of Excellence Centre for Tactile Internet with Human-in-the-Loop (CeTI). He received his diploma (Dipl.-Ing.) degree in EE from RWTH Aachen, Germany, in 1997 and his Ph.D. (Dr.-Ing.) in EE from the Technical University Berlin, Germany in 2002 and became Adjunct Professor at the University of Ferrara, Italy in the same year. In 2003, he joined Aalborg University as Professor. He has visited various research institutes, including Massachusetts Institute of Technology (MIT), VTT, and Arizona State University. He cofounded several start-up companies since 1999. He received several awards, such as the NOKIA Champion Award and the Nokia Achievement Award. In 2011, he received the SAPERE AUDE research grant from the Danish government, and in 2012 the Vodafone Innovation prize. In 2015, he was awarded the honorary degree Doctor Honoris Causa from Budapest University of Technology and Economics (BUTE).

Shu-Chen Li is a Professor and head of the Chair of Lifespan Developmental Neuroscience at TUD since 2012. She is a speaker of the DFG Cluster of Excellence CeTI since 2019. She received her Ph.D. degree in cognitive psychology from the University of Oklahoma in the USA in 1994. After working as a postdoc at the McGill University in Canada, she continued her research career at the Max Planck Institute for Human Development in Germany for 16 years until she took up the professorship at TUD. From 2006 to 2008, she was also an adjunct professor of the Brain Research Center in the College of Electrical and Computer Engineering at the National Chiao-Tung University in Taiwan. A key aspect of her research focuses on understanding brain mechanisms of neuronal gain control and their implications on age-related differences in perception and cognition across the human life span. For several years she served as the associated editor of Developmental Psychology, one of the flagship journals of the American Psychological Association. She is currently a member of the editorial board of Neuroscience and Biobehavioral Reviews.

Stefanie Speidel is a Professor for Translational Surgical Oncology at the National Center for Tumor Diseases (NCT), Partner Site Dresden, since 2017 and speaker of the DFG Cluster of Excellence CeTI since 2019. She received her Ph.D. (Dr.-Ing.) from Karlsruhe Institute of Technology (KIT) with distinction in 2009 in the context of the research training group Intelligent Surgery (KIT, University of Heidelberg, DKFZ), and led a junior research group Computer-Assisted Surgery from 2012–2016 at KIT. She has been (co)-authoring more than 100 publications and regularly organizes workshops and challenges, including the Endoscopic Vision Challenge@MICCAI as well as the Surgical Data Science workshop. She has been general chair and program chair for a number of international events, including IPCAI and MICCAI conference.

Thorsten Strufe is a Professor for IT Security at Karlsruhe Institute of Technology (KIT), Adjunct Professor for Privacy and Network Security at TUD, a speaker of the DFG Cluster of Excellence CeTI, and director of the Helmholtz Security Labs KASTEL at KIT. His research interests lie in the areas of large distributed systems and social media, with a focus on privacy and resilience. More recently, he has focused on studying user behavior and security in social media, and on ways to provide privacy-friendly and secure social networking services; he is fascinated by protection through decentralization. One of the challenges that drives him is how to create competitive web services and mobile apps without extensive collection of personal information, thus respecting the privacy of their users. To this end, his group measures and analyzes behavioral data on a large scale, develops algorithms and protocols to improve privacy and security, and formally analyzes anonymization networks for making their actual protection against new attacks formally verifiable.

Meryem Simsek is a Senior Research Scientist at the International Computer Science Institute, UC Berkeley, USA. She received her Ph.D. (Dr.-Ing.) from University of Duisburg-Essen on Learning-Based Techniques for Intercell-Interference Coordination in LTE-Advanced Heterogeneous Networks in 2013. Dr. Simsek has initiated and is currently chairing the IEEE Tactile Internet Technical Committee and serves as the Vice Chair for the IEEE P1918.1 Standardization Working Group, which she co-initiated. On the basis of her roles at IEEE, she disseminates and standardizes the achievements of CeTI. She was a recipient of the IEEE Communications Society Fred W. Ellersick Prize in 2015 and the Rising Star in Computer Networking and Communications by N2Women in 2019.

Martin Reisslein is a Professor in the School of Electrical, Computer, and Energy Engineering at Arizona State University (ASU), Tempe, and an external associated investigator with the DFG Cluster of Excellence CeTI, TUD, Germany. He received the Ph.D. in systems engineering from the University of Pennsylvania, Philadelphia, in 1998. He was a post-doctoral researcher with the Fraunhofer FOKUS institute and the Technical University Berlin from 1998 to 2000, when he joined ASU as Assistant Professor.

Preface

Frank H.P. Fitzek; Shu-Chen Li; Stefanie Speidel; Thorsten Strufe; Meryem Simsek; Martin Reisslein     

This book is a result of intensive collaborations among the contributors during the period of applying for a grant from the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) to establish a Cluster of Excellence and from the ongoing research activities during the first year after the Cluster had been successfully established at Technische Universität Dresden (TUD) in 2019. Together with researchers from other participating institutions, including Technical University of Munich (TUM), the Fraunhofer Institutes and the German Aerospace Center (Deutsches Zentrum für Luft- und Raumfahrt), the National Center for Tumor Diseases (Partner Site Dresden) and several international partners, a core team of researchers from five faculties (Electrical Engineering, Mechanical Engineering, Computer Science, Psychology, and Medicine) at TUD launched the Centre for Tactile Internet with Human-in-the-Loop (CeTI) to pursue new frontiers of research to promote disruptive innovations for digitally transmitted human–machine interactions that may revolutionize many aspects of our lives. This new field of transdisciplinary research will tackle a broad spectrum of theoretical, methodological, and technological challenges. In doing so, the emerging research on Tactile Internet with Human-in-the-Loop (TaHiL) will chart new frontiers for basic and applied research in human and engineering sciences to yield breakthroughs for next-generation multimodal, quasi-real-time human–machine interactions in real, virtual, mixed, and remote environments with broad applications in medicine, industry, and digital transformation technologies for daily-life usages.

Advancing the frontiers of science and technology relies on intensive collaborations among established fields of research that in the end may yield transdisciplinary breakthroughs of much broader impacts than the sum of the outputs from the individual disciplines involved. For a large number of researchers from several disciplines to join forces in embarking on disruptive, transdisciplinary research, as in the case of the research on TaHiL, basic understandings about the key principles of the involved fields, common languages, and shared visions need to be developed across the disciplines. Furthermore, synergistic research needs to be systematically structured and interconnected. This book is conceived as a handbook for the research on TaHiL to serve exactly these purposes. The book follows the structure of a synergistic research program with twelve research building blocks that have been established in the Cluster of Excellence CeTI at TUD. The building blocks are hierarchically interconnected, such that together they form a research pyramid (see the synergistic research program introduced in Chapter 1 for details). The respective research aims, approaches, and activities of the twelve building blocks are each covered by a chapter in this book. With the aim to serve as a handbook, representative work in the relevant areas beyond the research and technologies currently pursued in CeTI are also reviewed in the respective chapters.

Following the introduction (Chapter 1), which provides an overview of the research on TaHiL, the twelve chapters are divided into three parts, proceeding from the top to the base of the pyramid of the synergistic research structure. The first part showcases three selected domains of applications, which are robotic-assisted surgery (Chapter 2), human–robot cohabitation in industrial settings (Chapter 3), and Internet of Skills for other daily applications (Chapter 4). These use-cases presented in Part 1 require the key technologies and methods—in particular haptic codecs (Chapter 5), intelligent networks (Chapter 6), augmented perception and interaction (Chapter 7), as well as human-inspired models and computing (Chapter 8)—that are presented in Part 2. The challenges combined have to be tackled by systematically organized integrative research from several disciplines. These target primary research fields are presented in Part 3, which cover basic research on human multisensory perception (Chapter 9), sensors and actuators (Chapter 10), communications and control (Chapter 11), electronics for textile integration (Chapter 12), and tactile computing (Chapter 13). The last part of the book extends to cover cross-cutting topics, such as a digital trace library (Chapter 14) and standardization (Chapter 15) as well as technology transfer and communication to the public (Chapter 16).

This volume can serve as a handbook for the research on TaHiL for students and researchers from several contributing disciplines. For readers who would like to find out what TaHiL is and what applications the research in this new frontier may have, we recommend surveying Chapter 1 and the chapters in Part 1. For researchers who are already acquainted with topics about some aspects of TaHiL, we recommend sampling chapters from Part 1 for the specific applications of interests, and reading through chapters in Part 2, which highlight key areas of technological breakthroughs that require interdisciplinary research. Furthermore, to foster interdisciplinary understanding, the chapters in Part 3 are recommended for students and researchers to gain knowledge about fundamental questions and methods that are important for the research on TaHiL from the perspectives of other disciplines. Last but not least, the chapters in Part 4 address topics on technological standards and public communication, which are also crucial for the success of developing new technologies to serve better human–machine interactions.

This book marks a beginning. We hope it will kindle more interest and attract intensive research attention for the emerging transdisciplinary field of TaHiL. Interested readers are also referred to the CeTI webpage (ceti.one) for research updates.

Dresden, Germany

2021

Acknowledgments

Frank H.P. Fitzek; Shu-Chen Li; Stefanie Speidel; Thorsten Strufe; Meryem Simsek; Martin Reisslein     

First of all, we would like to thank the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft).¹ Many results reported in this book are funded by the DFG as part of Germany's Excellence Strategy² in support of the Cluster of Excellence Centre for Tactile Internet with Human-in-the-Loop (CeTI) established at Technische Universität Dresden (TUD).

The editors would like to thank all the authors who have contributed to the different chapters collected in this book. Most of the authors are members of CeTI, including many current CeTI Ph.D. students and postdocs, who have invested significant amounts of time and effort in addition to their regular duties to make this book possible.

Many thanks also go to our international research partners and consultants, such as (i) Muriel Médard from Massachusetts Institute of Technology (MIT), Adam Gazzaley from the University of California San Francisco (UCSF), Uta Noppeney from the Radboud University, and Gene Tsudik from University of California, Irvine (UCI), who supported us while we applied for the excellence initiative funding to establish CeTI or support us in CeTI's Advisory Board.

In alphabetical order, we express deep gratitude to our industrial partners, such as Atlantic Labs, CampusGenius, Deutsche Telekom, Mimetik, and Wandelbots.

We thank our design team, Jens Krzywinski, Tina Bobbe, Lisa Lüneburg, and their associates, for the support in creating designs and graphics for several demonstrators as well as the illustrations presented in the book. Their work not only gives this book a nice touch, but has also helped us to convey CeTI's main ideas of future communication systems to the public over the last years.

We are deeply thankful to Christian Scheunert and Hrjehor Mark for their support in managing the sources and their patience over the last months. It is their achievement to have all the sources of this book pulled together.

The work presented in this book would not have been possible without the endless support of our universities, i.e., Technische Universität Dresden, Technical University of Munich, as well as the Fraunhofer Institutes, the German Aerospace Center (DLR), and the National Center for Tumor Diseases, Partner Site Dresden (NCT).

Dresden, Germany

2021

---

¹  

"Funded by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft) as part of Germany's Excellence Strategy – EXC 2050/1 – Project ID 390696704 – Cluster of Excellence Centre for Tactile Internet with Human-in-the-Loop (CeTI) of Technische Universität Dresden."

²  

A funding program of the Federal Government and the states to strengthen cutting-edge research at universities.

Acronyms

3D Three-dimensional

3GPP 3rd-Generation Partnership Project

5G Fifth Generation

ADC Analog-to-Digital Converter

AFE Analogue Frontend

AG Attribute Grammar

AI Artificial Intelligence

AM Additive Manufacturing

AOP Aspect-Oriented Programming

API Application Programmer Interface

AR Augmented Reality

ARQ Automatic Repeat Requests

ASF Acceleration Sensitivity Function

ASIC Application-Specific Integrated Circuit

ASP Application Service Provider

ASQ Action Sequence

AVB Audio-Video Bridging

BAN Body Area Network

BCH Body Computing Hub

BDD Binary Decision Diagram

BFGS Broyden-Fletcher-Goldfarb-Shanno

CACC Cooperative Adaptive Cruise Control

CATI Computer-Assisted Telephone Interviews

CBR Constant Bit Rate

CBSE Component-based Software Engineering

CELP Code-Excited Linear Prediction

CeTI Centre for Tactile Internet with Human-in-the-Loop

CEW Communication and Early Warning

CI Communication Interruption

CMD Command

CMOS Complementary Metal Oxide Semiconductor

CNN Convolutional Neural Network

CNS Central Nervous System

COM/SDB ComSoc Standards Development Board

COP Context-Oriented Programming

CORA Core Ontologies for Robotics and Automation

CPE Control Plane Entity

CPS Cyber-Physical System

CPU Central Processing Unit

CR Compression Ratio

CT Computed Tomography

DARPP-32 Dopamine- and cAMP-Regulated Neuronal Phosphoprotein

DB Deadband

DC Direct current

DCT Discrete Cosine Transform

DDoS Distributed Denial of Service

DDS Data Distribution Service

DNF Disjunctive Normal Form

DNN Deep Neural Network

DoF Degrees of Freedom

DPMC Dorsal Premotor Cortex

DPR Dynamic Partial Reconfiguration

DSL Domain-Specific Language

DT Detection Threshold

DTAG Deutsche Telekom

DTLS Datagram Transport Layer Security

DVFS Dynamic Voltage and Frequency Scaling

DWT Discrete Wavelet Transform

E2E End-to-End

ECU Electronic Control Units

EEG Electroencephalography

eMBB Enhanced Mobile Broadband

eSAP External Service Access Point

ESE Energy Storage Element

ETSI European Telecommunications Standards Institute

FB Feedback

FDX Fully Depleted Silicon-on-Insulator

fMRI Functional Magnetic Resonance Imaging

FPGA Field Programmable Gate Array

F-RAN Fog Computing based Radio Access Network

FSK Frequency Shift Keying

FSM Finite State Machine

FUNc Network Functional Compression

GN Gateway Node

GNC Gateway Node Controller

GPU Graphics Processing Unit

GRAND Guessing Random Additive Noise Decoding

HCI Human–Computer Interface

HCTG Haptic Codecs Task Group

HDL Hardware Description Language

HIC Haptic Interpersonal Communication

HLS High Level Synthesis

HO Human Operator

HPC High Performance Computing

HPD High Performance Demonstrator

HRTF Head-Related Transfer Function

HSI Human–System Interface

IAT Inter-Arrival Time

IC Integrated Circuit

ICN Information Centric Networks

IEEE Institute of Electrical and Electronics Engineers

IEEE-SA IEEE Standards Association

IMU Inertial Measurement Unit

IoS Internet of Skills

IoT Internet of Things

IP Internet Protocol

IPL Inferior Parietal Lobe

IPS Intraparietal Sulcus

ISDN Integrated Services Digital Network

ISI Inter-Stimulus Interval

ISO International Standards Organization

ISS Input-to-State Stability

ITU-T International Telecommunication Union Standardization Sector

IVR Immersive Virtual Reality

JIGSAWS JHU-ISI Gesture and Skill Assessment Working Set

JND Just Noticeable Difference

JSON Javascript Object Notation

JVM Java Virtual Machine

K Key Technologies and Methods

KPI Key Performance Indicator

LAN Local Area Network

LED Light-Emitting Diode

LGN Lateral Geniculate Nucleus

LNA Low Noise Amplifier

LTE Long Term Evolution

MAC Medium Access Control

MAD Maximally Allowable Delay

MATI Maximum Allowable Transmission Interval

MBCC Model-Based Cobotic Cell

MDE Model-Driven Engineering

MDP Markov Desicion Process

MEC Mobile Edge Cloud

MGC Medial Geniculate Complex

MGD Mini-Batch Stochastic Gradient Descent

MIMO Multiple-Input Multiple-Output

ML Machine Learning

MMT Model Mediated Teleoperation

mMTC Massive Machine Type Communication

MRI Magnetic Resonance Imaging

NAcc Nucleus Accumbens

NC Network Controller

NCS Networked Control System

NesCom New Standards Committee

NFV Network Function Virtualization

NR New Radio

NS Network Slicing

NUI Natural User Interface

OBG Observer-Based Gradient method

OFDM Orthogonal Frequency Division Multiplexing

OLED Organic Light-Emitting Diode

OOK On-Off Keying

OR Operating Room

OS Operating System

OSATS Objective Structured Assessment of Technical Skills

OSM Orthographic Software Modeling

PA Power Amplifier

PAR Project Authorization Request

PC Passivity Controller

PCTL Probabilistic Computation Tree Logic

PDMS Polydimethylsiloxane

PDU Protocol Data Unit

PE Processing Element

PHY Physical Layer

PL Programmable Logic

PLL Phase-Locked Loop

pRRH pico-Remote-Radio-Head

PSNR Peak Signal to Noise Ratio

PTP Precision Time Protocol

PU Polyurethane

QAM Quadrature Amplitude Modulation

QoC Quality-of-Control

QoE Quality-of-Experience

QoP Quality-of-Performance

QoS Quality-of-Service

QPSK Quadrature Phase-Shift Keying

RAG Reference Attribute Grammar

RAM Random-Access Memory

RAN Radio Access Network

RC Reflection Coefficient

RCS Reconfigurable Computing System

RGB Red Green Blue

RGBD Red Green Blue and Depth

RL Reinforcement Learning

RLNC Random Linear Network Coding

RO Robot Operator

ROP Role-Oriented Programming

ROS Robot Operating System

RQs Research Questions

RRI Responsible Research and Innovation

RRM Radio Resource Management

RRSI Rapid Reaction Standardization Initiative

RT Reaction Time

RTOS Real-Time Operating System

SAP Service Access Point

SAS Self-Adaptive System

SAW Spatial Audio Workstation

SDC Silent Data Corruption

SDN Software Defined Network

SE Support Engine

SFA Successive Force Augmentation

SFC Service Function Chaining

SGX Software Guard eXtension

SLP Sparse Linear Prediction

SMPTE Society of Motion Picture and Television Engineers

SMR Signal-to-Mask Ratio

SNc Substantial Nigra Parc Compacta

SNR Signal-to-Noise Ratio

SoA Service-oriented Architecture

SOC Systems-on-Chip

SPIHT Set Partitioning In Hierarchical Trees

SPL Software Product Line

SR Stimulus Response

SR-ARQ Selective Repeat ARQ

STS Superior Temporal Sulcus

SW-ARQ Stop and Wait ARQ

TA Technology Assessment

TADF Thermally Activated Delayed Fluorescence

TaHiL Tactile Internet with Human-in-the-Loop

TAM Technology Acceptance Model

TD Tactile Device

TDPA Time Domain Passivity Approach

TDPA-ER Time Domain Passivity Approach Energy Reflection

TE Tactile Edge

TEE Trusted Execution Environment

TFT Thin Film Transistor

TI Tactile Internet

TIM Tactile Internet Metadata

TLS Transport Layer Security

TNM Tactile Network Manager

ToF Time of Flight

TP Talent Pool

TPU Thermoplastic Polyurethane

TSM Tactile Service Manager

TSN Time Sensitive Network

TSX Transactional Synchronization eXtension

TT Tactile Traces

TUD Technische Universität Dresden

TUM Technical University of Munich

U Use Cases

UE User Equipment

UML Unified Modeling Language

UPE User Plane Entity

URLLC Ultra-Reliable Low-Latency Communication

UTAUT Unified Theory of Acceptance and Use of Technology

V2V/V2I Vehicle-to-Vehicle/Vehicle-to-Infrastructure

V2X Vehicle-to-Any

VCO Voltage Controlled Oscillator

VPL Ventral Posterolateral Nucleus

VPM Ventral Posteromedia Nucleus

VPMC Ventral Premotor Cortex

VR Virtual Reality

VTA Ventral Tegmental Area

WAN Wide Area Network

WFS Wave Field Synthesis

WG Working Group

WiFi Wireless Fidelity

XML eXtensible Markup Language

ZOH Zero-Order Hold

Chapter 1: Tactile Internet with Human-in-the-Loop: New frontiers of transdisciplinary research

Frank H.P. Fitzeka; Shu-Chen Lia; Stefanie Speidelb; Thorsten Strufec    aTechnische Universität Dresden, Dresden, Germany

bNational Center for Tumor Diseases, Partner Site Dresden, Dresden, Germany

cKarlsruhe Institute of Technology, Karlsruhe, Germany

What seemingly was often overlooked…is that the human brain itself…is something that is co-shaped by experience…, something that does not operate in an environmental vacuum, but at any moment is subject to environmental constraints and affordances.

– Paul B. Baltes☆

Engineering is a living branch of human activity and its frontiers are by no means exhausted.

– Igor Sikorsky

Above all things expand the frontiers of science: without this the rest counts for nothing.

– Georg C. Lichtenberg

Abstract

The emerging new field of research on Tactile Internet with Human-in-the-Loop (TaHiL) aims to achieve significant breakthroughs to enhance collaborations between humans and machines or—more generally, Cyber-Physical System (CPS)—in real, virtual, and remote environments. The vision of TaHiL is to enable humans to interact with cooperating CPS over intelligent wide-area communication networks to promote equitable access to remote work, medical, learning, social, and recreational opportunities for people of different ages, genders, cultural backgrounds, or physical limitations. Thus reaching far beyond the current state of the art in digitalization and human–machine interaction, the long-term goal of the research on TaHiL is to democratize the access to skills and expertise the same way as the current Internet has democratized the access to information. Capitalizing on recent advancements in the fields of telecommunication, electrical and material engineering, computer science, robotics, psychology, cognitive neuroscience, and medicine, researchers in this new transdisciplinary field are pursuing basic and applied research to (i) advance the understanding about complex dynamics of human goal-directed multisensory perception and action from the psychological, neurocognitive, medical, and computational perspectives; develop novel sensor and actuator technologies that augment the human mind and body; develop fast, bendable, adaptive, and reconfigurable electronics; create intelligent communication networks that connect humans and CPS by continuously adapting and learning to provide low latency, as well as high levels of resilience and security; (v) design new haptic coding schemes to cope with the deluge of information from massive numbers of body sensors; design online learning mechanisms as well as interface solutions for machines and humans to predict and augment each other's actions; and to evaluate the above technological solutions as well as to engage the general public about the potential possibilities and concerns that the new technologies will bring. The research on TaHiL will be essential for diverse applications involving human–machine interactions, including, most notably, in medicine, industry, and the Internet of Skills (IoS). This overview chapter highlights the challenges, directions, programmatic structures and application domains of the new transdisciplinary research endeavor of TaHiL. Key building blocks of TaHiL are presented in details in the 15 subsequent chapters collected in this volume.

Keywords

Age-sensitive design; haptic communication; haptic sensors; machine learning; multimodal feedback; multisensory perception and action; predictive models; robotics; Tactile Internet; user-centered design

1.1 Motivation and vision of TaHiL

In the summer of 1969 the Internet was created by coupling a small number of computer nodes to share files across different locations. Back then a small number of services was available to a small number of experts. Fifty years later the Internet is among the most important global infrastructures worldwide. It has become a key infrastructure that is used by everyone and touching almost every aspect of human daily lives. The current Internet provides the service of democratizing access to information for everybody, independent of location or time (as illustrated in Fig. 1.1). One important enabler for this function of global information access was the introduction of the World Wide Web, which allowed laypersons and computer scientists alike to create contents for distribution and to consume information through and from the Internet, respectively. Focusing on characteristics of the most popular services, such as video streaming, social networking, or web browsing, the current Internet has been optimized for high data rates to facilitate quick access and live consumption during the download. The next-generation Internet, the Tactile Internet (TI), takes these ideas one big step further. It envisions new opportunities and is faced by entirely novel challenges. The Institute of Electrical and Electronics Engineers (IEEE) P1918.1 Tactile Internet Standardization Working Group (http://ti.committees.comsoc.org/)

defines the TI communication platform as: A network or network of networks for remotely accessing, perceiving, manipulating or controlling real, or virtual objects, or processes in perceived real time by humans or machines [1,2]. To transcend the possibilities of just gaining access to information, as we use the Internet today (compare Fig. 1.1), the Human-in-the-Loop approach [3] needs to be thoroughly realized en route to further technological advancements in the TI. The resulting new field of Tactile Internet with Human-in-the-Loop (TaHiL) research aims at democratizing access to skills and expertise to promote equity for people of different age, genders, cultural backgrounds, or physical limitations (see Fig. 1.2). To reach such breakthroughs for bringing digitally transmitted human–machine interactions to a new era, it is indispensable that transdisciplinary research involving researchers from several fields—ranging from psychology, cognitive neuroscience, and medicine to the fields of computer science, electrical, mechanical, and material engineering—needs to be conducted. One such transdisciplinary research center, the Centre for Tactile Internet with Human-in-the-Loop (CeTI), has been recently established at Technische Universität Dresden (TUD).

Fig. 1.1 Nowadays Internet: Democratizing access to information for everybody regardless of location or time.

Fig. 1.2 TaHiL: Democratizing access to skills and expertise to promote equity for people of different ages, genders, cultural backgrounds, or physical limitations.

The Internet today, which is mainly optimized for increased throughput, cannot support TI applications. In this overview chapter, we will highlight several new challenges that need to be overcome to achieve the goals of TaHiL. The subsequent chapters in the book then present exemplified domains of applications that can be enabled through tackling challenges in a structured array of basic research and technological innovations. Here we highlight three frontiers in engineering and human research that still need to be explored and established: (i) a communication network that is optimized for skill (beyond information) transfer and hence supports extremely low latencies and different Quality-of-Service (QoS) support for modalities, such as video, audio, and haptics, novel human–machine interfaces that utilize a large array of sensors and actuators, and systematic understanding of goal-directed human multisensory perception and action, and the impacts of lifespan development and learning on these processes. Albeit establishing the TI as the next-generation infrastructure for global skill transfer is one important factor, the communication network alone falls short of several other challenges. Advanced wearable and adaptive sensors as well as actuators need to be developed as new types of interfaces for communications between humans and machines via the TI. Furthermore, the research on TaHiL has to consider the principles and mechanisms of human goal-oriented multisensory perception and action in people with different ages, learning experiences, and skill levels. Only then will the TI allow broad populations of human users to immerse themselves into virtual, remote, or inaccessible real environments, to exchange skills and expertise; thus create new opportunities and novel ways for people to learn, to work, and to interact (as illustrated in Fig. 1.3).

Fig. 1.3 (left) How may we learn in the future? (middle) How may our work change due to robots? (right) How may TaHiL technologies help the old and the oldest-old in the future?

1.1.1 Skill transfer from humans to machines

Different paths can be chosen to exchange skills among humans and machines. Here we consider a leading example of skill transfer from a human expert to a machine, in this case a standard industry robot (see Fig. 1.4). One way for this type of skill transfer would be to equip a human expert with any kind of human–machine interface, such as a simple remote control (e.g., a game console controller). This solution, albeit simple, has a couple of caveats. Not all human experts are able to operate such a controller, nor do they understand the relationship between the movements and accuracy of the robot.

Fig. 1.4 Skill transfer from humans to machines.

Even replacing the simplistic controller with a more advanced human–machine interface, such as wearables that track and map human behavior directly to the robot, does not solve another issue regarding the scalability of direct remote control. A single machine still needs to be controlled by one human. In scenarios with millions of consumers demand a specific robot skill, the control has to be scaled up and the skill itself has to be transferred to millions of robots to meet the demand of the consumers. Industry has long recognized this problem, and there are standard processes for conveying skills and expertise to industrial robots (as illustrated in Fig. 1.5).

Fig. 1.5 Common industrial view on skill transfer.

A domain expert in such industry use cases explains the necessary expertise and actions a robot has to perform to a computer scientist. This programming expert will then, to the best of his/her understanding, convey these descriptions into the software that is subsequently executed on the industrial robot. Once successful, the software can be deployed to several robots. While this approach is more scalable than remote control, it still has several problems. First, the communication between the human expert and the computer scientist is error prone and often only a best-effort service. Second, the cost factor of the computer scientist is two to six times higher than the cost of the industrial robot. In light of decreasing prices for robots, this ratio will become even higher. Third, completing the skill transfer, potentially in a sequence of several trial-and-error cycles, is rather time-consuming.

To overcome these limitations, a novel approach developed in the field of TaHiL research at TUD has been proposed and implemented (see Fig. 1.6). This innovation builds specifically on the idea of using wearable clothing that is instrumented by sensors, and worn by domain experts. By performing the action routine several times, the activity of the expert is used to train the robot by demonstration through natural human movements. The training sequences recorded by the sensors are evaluated by machine-learning algorithms, which then output the software for the robot automatically. In other words, by combining sensor recordings of human behavior with machine learning, a direct form of human-to-machine demonstration teaching can be established. Such an approach is one of the promising avenues for further research on TaHiL. Indeed, this solution has been spun off in 2018, leading to a start-up company, Wandelbots, that has now developed several products for industry (for example, see Fig. 1.7).

Fig. 1.6 One of the TaHiL approaches for skill transfer using machine learning and demonstration-based teaching developed at TUD and now implemented in the start-up company, Wandelbots.

Fig. 1.7 The CEO of the Volkswagen group testing the demonstration-based robot teaching developed by the TUD spin-off Wandelbots.

This natural demonstration-based teaching is promising. However, conveying the sensor data from the wearables over a communication network to remote robots in real-time is still a big challenge in the research on TaHiL. Future developments along this novel approach may meet the challenges of specific tasks that would require global exchanges of skills, for instance an expert in Europe trains a robot in Tokyo, Japan (see Fig. 1.8). In this scenario, the task of providing the expert with timely multimodal feedback, as it is required for efficient remote training and for giving the expert the feeling of virtually being right next to the remote robot, is particularly challenging. In fact, this is not possible with the Internet technology as it stands today, and derives several challenges ahead of us to realize the TI.

Fig. 1.8 Future scenarios of global skill exchange in the TI.

An overview of the manifold challenges in different research building blocks of TaHiL is shown in Fig. 1.9. The multimodal feedback needs to comprise haptic, in addition to video and audio, information. Each of these information modalities differs in its requirements of bandwidth and latency. It is already clear that video requires more bandwidth at relaxed delay constraints as compared to audio interaction. Furthermore, the properties and requirements for haptic information, so far, have not yet been extensively investigated and are far from been understood. Thus further basic research on haptic information processing in humans and applied research on haptic technologies would be necessary. With respect to latency and reliability of human sensory and perceptual processes, the latencies range from several milliseconds for video, over around 3 ms for audio, to only about 1 ms for haptic information [4] (see also Chapters 5 and 9). Such strict latency requirements create novel challenges, particularly given the laws of physics. Since the speed of light becomes a limiting factor for the possible distance between the human expert and robot for timely real-time remote interaction, as illustrated in the scenarios above. Even without considering the time needed for sensing, encoding, and processing, light travels at around 300 kilometers per millisecond, which limits the distance between the expert and robot to a range that falls substantially short of the requirements of global skill transfer and space communications.

Fig. 1.9 Challenges in global skill exchange via the TI.

Furthermore, human–machine real-time interactions in the form of coworking or training over longer distances will require local predictions of the remote behaviors, which would then also need to be corrected upon reception of the actual remote updates. While modeling a robot in a well-described physical environment without potential error sources is easy and follows the technical specifications of the machine, modeling and predicting human behaviors are significantly harder tasks that depend on many more parameters and are characterized by a huge number of degrees of freedom (see Chapters 9 and 11). These challenges require the Human-in-the-Loop approach [3] to be thoroughly realized on the foundation of human goal-directed perception and action in all aspects and steps of new technological developments.

1.1.2 Skill transfer from machines to humans

So far we have considered one direction of skill transfer, i.e., having the human to teach a machine. But the reverse direction covers scenarios that could be applicable in other use cases. Assuming that the wearables are not only equipped with sensors but also with actuators, learning signals can either be generated live or in advance, and then conveyed to the human user. Taking physical rehabilitation of the elderly as an example, the movements of a remote physiotherapist could be generated online and transmitted to either wearables equipped with actuators that the elderly person wears, or a Cyber-Physical System (CPS) to help performing physiotherapy exercises at home (see Fig. 1.10). The potential application domains for such skill transfer from machines to humans are not limited to health and nursing care, they also cover teaching new skills in schools, at work, or of personal interests, i.e., the broad domain of Internet of Skills (IoS) [5]. Fig. 1.11 depicts two specific examples of this last class of applications that involve training rowing and climbing with specific wearables for detecting and correcting inefficient or potentially harmful movements. These technologies are currently under development in CeTI at TUD.

Fig. 1.10 A scenario of reverse skill transfer from machines to humans: The case of remote health care.

Fig. 1.11 Scenarios of teaching and training humans (see Chapter 4 ) in the specific cases of (left) rowing and (right) climbing.

1.1.3 Skill transfer in holistic settings

There are several other application domains, for which digital skill transfer may also be valuable. Research on TaHiL aims to enable humans and machines to work collaboratively together in multiple learning activities in the future. Besides the aforementioned scenarios, skill transfer among robots of different manufacturers in completely different environments is a further potential application field. Furthermore, digitally mediated learning from human-to-human over long distances or between restrained environments will open disruptive new opportunities for the democratization of expertise and learning opportunities for acquiring various skills. Holistically, global digitally mediated skill exchanges can involve multiple combinations of human-to-machine, machine-to-human, and human-to-human interactions (see Fig. 1.12).

Fig. 1.12 Scenarios of the holistic human–machine skill transfer via the TI.

1.2 Research objectives to meet the challenges of TaHiL

There are several fundamental objectives for the research on TaHiL, which all revolve around the main building blocks of next generation multimodal closed-loop human–machine interactions that take place in the TI in perceived real-time (see Fig. 1.13): the human, who is augmented by large numbers of sensors and actuators that are connected through an intelligent network, cooperates with CPS (e.g., robots and other virtual- or mixed-reality entities) that are equipped with inherent sensors and actuators as well as adaptive learning mechanisms. Such quasi-real-time closed-loop interactions lead to a plethora of multisensory feedback information that has to be conveyed from the human to the machine and back over the same intelligent network, but with different communication characteristics in terms of latency and resilience. Note that the closed-loop human–machine interaction is not limited to one human, robot, or other CPS; instead, the TaHiL concept generalizes to other combinations and extensions that include an arbitrary number of these components in holistic settings of applications.

Fig. 1.13 Conceptual representation of the TaHiL.

This section describes six key research objectives of TaHiL in a logical order. As shown in Fig. 1.13, we start with the Human-in-the-Closed-Loop system reflecting the first objective on human perception and action. In particular, the first objective mainly concerns the modeling and prediction of human goal-directed multisensory perception and action. The second objective focused on human–machine coaugmentation and addresses the novel bendable electronics, sensors, and actuators required for TaHiL. The aforementioned intelligent network is reflected by the third objective, which focuses on developing human–machine networks that can assure real-time communication, storage, and computing for all involved communication elements. The fourth objective concerns learning strategies for humans and machines to learn from and adapt to each other and is therefore called human–machine learning. The fifth objective of human–machine computation targets the computing infrastructure that is necessary for human–machine interactions. The sixth objective, human–machine communication, aims to develop new information theoretical approaches for communication, compression, coding, and control. (See Fig. 1.14.)

Fig. 1.14 Logical derivation of key objectives of the research on TaHiL.

1.2.1 Objective 1: Human perception and action

Model and predict human goal-directed behavior, which entails flexible and dynamic interactions between sensation, multisensory perception, cognition, and action in contexts.

The novel technologies to be developed for human–machine interactions via the TI will create new digital environments for humans to interact with a wide range of CPS, with substantially, if not completely, changed hardware and software interfaces that require extensive multisensory information processing (see Chapter 9, Fig. 9.1). Thus innovative approaches for system and interface designs would need to be developed to optimize the new digitally transmitted closed-loop interaction between humans and machines. To establish the necessary requirements for engineering designs, computational models of flexible, human goal-directed multisensory perception and action will need to be developed (see Chapter 13). These models need to take into account relevant characteristics of individual differences, such as age and levels of expertise. In particular, the processes of human development [6] and aging [7] as well as mechanisms of skill acquisition and mastery can significantly affect the efficiency of various processes underlying goal-directed perception and action at the behavioral and brain levels. Thus models characterized by appropriate human factors are crucial for the development of new algorithms and technologies for human–machine coadaptation, in which goal awareness and action prediction are the prerequisites for smooth interactions. This requires research to go far beyond the current state of understanding. We need to characterize and understand expertise- and age-related differences in key parameters of multisensory integration and delay requirements. Methodologically, psychophysical, and neurocognitive experiments will need to be conducted with large samples of individuals covering wide age ranges and expertise levels. The experiments will need to encompass different sensory modalities (e.g., auditory, visual, and haptic) across an array of perceptual decision and sensorimotor tasks that entail flexible switching between goal sequences or task contexts. Psychophysical and Bayesian active inference models will need to be developed to model and predict expertise- and age-related differences in the complexity and constraints of human goal-directed perception and action (see Chapter 9 for details about research on these topics and experimental studies currently been pursued in CeTI).

1.2.2 Objective 2: Human–machine coaugmentation

Produce wearable peripherals for fast sensing and actuating with multimodal feedback for human perception, cognition, and action based on ultra-small, bendable, stretchable, and ultra-low-power electronic circuits that precisely localize humans and objects in real-time.

New fast and flexible sensors and actuators will need to be developed to provide plausible multimodal feedback, such as soft exoskeletons (e.g., eGloves or eBodySuits) that go beyond existing products. A high-quality multimodal feedback system should recognize and interpret the inputs from different modalities to provide multimodal outputs for human multisensory processing. Intelligent adaptive sensors are required for these purposes. Human psychophysical parameters provide the requirements for the actuators and multimodal interfaces. Psychophysical thresholds (e.g., tactile acuity or just noticeable level of difference in sound frequency) define the necessary information for developing and designing interfaces with realistic and compelling multisensory feedback that also entail tactile and kinesthetic signals. Latency sensitive haptic and visual codec applications will be required in addition to the feedback design. As a concrete scenario, novel robotic hand–arm systems will need to be developed in TaHiL that utilize haptic feedback to enable new control designs and methods for digitally transmitted remote physical manipulation and learning. The manipulation aspect of human-oriented feedback information is comparable to the approach in MP3 to avoid unnecessary information in audio. The haptic feedback will also be used to connect to smart wearables to immerse the human user in virtual or augmented reality. To achieve fast sensing and actuating for multimodal feedback, a new generation of electronics will need to be developed. It has to provide real-time operation, while consuming very low energy. Moreover, to facilitate the natural interaction between the human and the CPS, which are both equipped with high number of sensors and actuators, the electronic hardware has to be ultracompact, bendable, and stretchable. The transceiver dimensions are strongly determined by the antenna size, which massively decreases with increasing frequency. In this regard, by using very high frequencies, around 100 GHz, the antenna area can be decreased by several orders of magnitude allowing on-chip integration. To minimize the average power consumption of these wireless transceivers to below 1 mW and to allow real-time operation, aggressive duty cycling with record circuit reaction time in the nanosecond regime, allowing very efficient sleep and wake-up modes, would be necessary. The huge amount of sensor data has to be preprocessed adaptively and locally to guarantee low latency. For this, a mobile high-performance body computing platform will need to be designed with significantly reduced power consumption (20 TOPS/W). To allow mechanical flexibility, the chips will need to be thinned and integrated on stretchable substrates. Many of these new technologies are being used and developed in CeTI (see Chapters 10 and 12 for details).

1.2.3 Objective 3: Human–machine networks

Develop completely softwarized network solutions for wireless and wired communication that provide low latency, resilience, and security to enable human–machine co-operation.

To realize the vision of TaHiL, it is necessary to develop novel communication and network solutions for the body area, local area, and wide area networks that go beyond the mere relaying of information in the wireless and wired domain. New concepts of softwarization, including software defined networks, network function virtualization, and information-centric networking, will need to be exploited with respect to low latency and learning capabilities [8]. The latter is needed to allow the network to learn about different multimodal information flows (i.e., audio, video, and haptic) to adapt the network capabilities (e.g., network slicing). Moreover, the intelligent network will need to provide the means for learning from human behavior. For instance, when the connection to the human gets lost, the intelligent network should predict action courses during the period of interrupted communication (e.g., in a mobile edge cloud). Furthermore, the human's high-level but abstract mental models of behavior as well as low-level sensorimotor programs that make it possible for the human to cope with delays, ambiguities, and disruptions of the technical world will need to be better understood when designing and implementing communication networks. Such knowledge is a prerequisite for the design of efficient human–technology feedback strategies on all levels and provides an end-user-oriented direction for solving technical conflicts between latency, bandwidth, and resilience. Resolving this challenge has the potential to tremendously accelerate the progress in this field (for details see Chapters 6 and 11 for research activities on these topics that are currently being pursued in CeTI for details).

1.2.4 Objective 4: Human–machine learning

Provide an integrated framework that leverages the effects of continuous, mutual adaptive learning between humans and machines. Tune explanation facilities towards the demands and objectives of the human user. Assess boundary conditions and benefits for skill acquisition and training.

To achieve this objective, it is necessary to develop a portfolio of models, methods, and tools for automated human-inspired computing, to provide the Human-in-the-Closed-Loop interactive system with explanation and learning assistive equipment. In our view, the main innovation will be a new model-based approach that incorporates recent neuroscientific achievements on nonlinear dynamic models for human decision-making. To ensure smooth closed-loop human–machine interaction, it is important to enable human and machine to learn to predict and support each other's actions online and thereby bringing interaction to a new era of immersed closed-loop human–machine cooperation and coaugmentation. On the one hand, human actions can be augmented by machines; on the other hand, machines can learn from human behavior to represent and generate expert knowledge with various methods. The software systems for TaHiL applications have to provide self-explanation techniques to help end-users to understand the correct function and the rationale of decisions that are taken by machines. Towards an integrated framework, model-based explanations for machine behaviors, explanations for machine-learning results, and their appropriate visualization to facilitate human understanding will need to be developed. To this end, the TaHiL framework (see Figs. 1.13 and 9.1) can be applied to different domains of competency acquisition to explore the interplay of individual and situational factors to identify the conditions under which best benefits can be obtained for human–machine learning and coadaptation (see Chapter 8 for details).

1.2.5 Objective 5: Human–machine computation

Deliver a secure and scalable computing infrastructure that enables intuitive haptic interaction and automatically adapts to changes in task contexts and world models.

The softwarization of the TI leads to highly immersive software and services. The software infrastructure not only needs to perform fast, but also needs to be highly resilient to failures and attacks. Thus safety, security, privacy, and scalability are prerequisites for the tactile computing infrastructure serving TaHiL applications. The software for these applications, being naturally embedded into the physical environments, will need to rely on plausible world and context models that can also adapt to changes (see Chapter 13 for details).

1.2.6 Objective 6: Human–machine communication

Provide novel coding and compression methods, such as haptic codecs, compressed sensing, and network coding that take into consideration human factors to enable a combined control and communication system.

The TaHiL approach will produce a massive amount of sensor and actuator information to be exchanged among multiple communication and control nodes to facilitate human–machine cohabitation. Currently, control systems employ almost exclusively wired communication, because wireless solutions suffer from the traditional hard trade-off between latency, resilience, and throughput. However, the TaHiL applications need to rely on wireless communication to enable higher degrees of freedom for humans and machines. The massive amounts of sensor and actuator data that will be produced thus calls for a novel compression method—haptic codecs [9], which is akin to source compression for audio and video, but this time for all possible types of haptic information. Beside this end-to-end approach, new distributed approaches, using network coding and compressed sensing will soften the aforementioned trade-off, allowing for more flexible optimization strategies. Both approaches need to be tailored to the software-defined networks and network virtualization solutions that have to be developed. Furthermore, the amount of sensing and feedback data will need to be reduced significantly through learning strategies and optimizations of the communication network and control loop. The amount of traffic needed for closed-loop human–machine interactions in the TI will need to be reduced by at least one, but potentially up to two, orders of magnitude compared to the current standard. In addition, haptic communication requires qualitative and quantitative assessments of the user's quality-of-experience. In contrast to time-consuming subjective tests, reliable automated quality metrics for the evaluation of human-in-the-loop systems with haptic feedback will need to be developed. A range of research activities is currently being undertaken in CeTI to resolve these challenges (see Chapters 5, 6, and 11 for details).

1.3 A synergistic research program

To achieve the six research objectives and tackle the challenges presented above, we suggest a research program to facilitate synergies between the different fields for transdisciplinary research (see Fig. 1.15). Specifically, we identified five Talent Pools (TPs) that together build the foundation for the research on TaHiL. They span research activities on (i) human perception and action, novel sensors and actuators, core networking technologies, flexible electronics for on-body computation, as well as (v) computation and computing infrastructure for the TI.

Fig. 1.15 A synergistic research program involving 12 interlinked research groups for the research on TaHiL.

Through close collaborations, researchers from the disciplines of the TPs engage in developing Key Technologies and Methods (K), particularly with regards to (i) codecs for tactile and haptic communication, secure, intelligent networks, novel user interfaces, and human–machine mutual learning techniques. Outcomes from basic research conducted by the research groups at the TP-level and novel technologies developed by research groups at the K-level can, in turn, be integrated and tested in research groups at the Use Cases (U)-level (see Fig. 1.15). We have identified three broad use cases (U) involving application in (i) medicine (see left panel of Fig. 1.16), industry (see middle panel of Fig. 1.16), and the Internet of Skills (see right panel of Fig. 1.16). Other application domains can be further integrated into the research program. However, the three domains we focus on here are complete in the sense that requirements for future research in the field of TaHiL for other applications can be derived from them.

Fig. 1.16 (left) The vision for medicine and health. (middle) The vision for industry 4.0. (right) The vision for internet of skills.

An instantiation of this suggested research program is CeTI, which has been established at TUD since 2019. The structure of the research program (Fig. 1.15) illustrates the dependencies and synergistic interplays among the various groups of experts to progressively enable solutions from highly specialized to increasingly interdisciplinary teams. Outcomes of the solutions are then evaluated in different application domains, through which new assumptions and requirements are derived and transmitted back to the lower levels of the research structure for further research and technological refinements. Here we also use this structure to organize the chapters in this book. Key themes and issues indicated in the first three parts of the book and tackled by each of the twelve U, K, and TP research groups are presented in detail in corresponding chapters. Furthermore, topics on standardization, digital trace data library and communicating technological developments to the public are also treated in separate chapters in the last part of the book.

1.4 Research outreaches and societal impacts

Other than scientific and technological impacts,