Mobile Sensors and Context-Aware Computing

By Manish J. Gajjar

Mobile Sensors and Context-Aware Computing is a useful guide that explains how hardware, software, sensors, and operating systems converge to create a new generation of context-aware mobile applications. This cohesive guide to the mobile computing landscape demonstrates innovative mobile and sensor solutions for platforms that deliver enhanced, personalized user experiences, with examples including the fast-growing domains of mobile health and vehicular networking.

Users will learn how the convergence of mobile and sensors facilitates cyber-physical systems and the Internet of Things, and how applications which directly interact with the physical world are becoming more and more compatible. The authors cover both the platform components and key issues of security, privacy, power management, and wireless interaction with other systems.

• Shows how sensor validation, calibration, and integration impact application design and power management
• Explains specific implementations for pervasive and context-aware computing, such as navigation and timing
• Demonstrates how mobile applications can satisfy usability concerns, such as know me, free me, link me, and express me
• Covers a broad range of application areas, including ad-hoc networking, gaming, and photography

• Language: English
• Publisher: Elsevier Science
• Release date: Feb 22, 2017
• ISBN: 9780128017982

Manish J. Gajjar

Manish J. Gajjar is a technical program manager and early prototyping lead for sensor solutions at Intel Corporation He has 20 years of experience at Intel in chipsets and graphics products, including roles as validation architect, design/validation and emulation lead to post silicon validation program manager. Manish has also served on the Industry advisory board of California State University and as a faculty member there.

Book preview

Amani.

Chapter 1

Introduction

Abstract

This chapter provides introduction to mobile computing, its constraints and market influence, trends and future growth area examples.

Keywords

Introduction to mobile computing; constraints; market trends; growth areas

Information in This Chapter

• Definition of Mobile Computing

• Constraints and the Challenges Faced by Mobile Computing System

• Historical Perspectives and the Influences of Market

• Market Trends and Growth Areas

Definition of Mobile Computing

Mobile computing refers to the computing that happens when the user interacts while the computer or parts of the computer are in motion during the use. Hardware components (like computing silicon, various sensors, and input/output devices), software components (like programs that communicates with underlying hardware, device drivers, applications, and software stacks to support communication protocols), and communication protocols like Wi-Fi protocols and hypertext transfer protocol (HTTP) are some of the main components of mobile computing. Through these components user-to-computer or computer-to-computer communications/computing happens.

The following are the three main classes of mobile computing:

• Mobile phones: Mobile phones are primarily used for voice calls and voice communication but with the advent of smartphones these devices are now used for computing applications, games, and data access over Wi-Fi or other wireless networks. These devices are increasingly adding to computational capabilities.

• Portable computers: Portable computers are devices with only essential computing components and input/output devices. These are lighter in weight than desktops, and weight reduction is achieved through removal of nonessential input/output devices like disc drives, use of compact hard drives, and so on. Extra connecting ports like USB and Firewire are used to connect external I/O drives or other devices if needed. These compact, lightweight computers have full-character-set keyboards and host software like Windows, Android, and Mac OS. Examples are laptops, notebooks, tablets, notepads, and so on.

• Wearable computers: These are mobile computing devices that have the technology that users may want to put on their body with the dual purpose of fashion plus computation/connection to the external world through wireless or communication protocols. These devices are capable of sensing, computing, running applications/software, reporting, and connecting. Examples are watches, wristbands, necklaces, keyless implants, and so on, which can take voice commands, or sense various environmental or health parameters and communicate either with mobile phones, portable computers, or the Internet.

Let us evaluate the differences of mobile computing devices versus other computing devices, as shown in Table 1.1.

Table 1.1

Quick Comparison Between Different Forms of Mobile Computing Devices

Constraints and the Challenges Faced by Mobile Computing Systems

Mobile computing devices have a smaller form factor than traditional desktops. There is need to impose constraints on space, weight, and form factors of these devices since their users are on the move while computing or connecting. These constraints in turn impose various technological and design restrictions on the devices. Let us briefly look at those restrictions:

Resource Poor

A computer system requires various components to process, compute, or connect. Hence any device in the computer system or connected to the computer system is a resource or system resource. These resources can be physical or virtual component. Example of such resources include the CPU, RAM, hard disks, storage devices, various input/output devices like printers, and connectivity components like Wi-Fi or modem. Mobile computing devices are resource limited. For example, their screens and keyboards are small, they have reduced I/O connections, reduced RAM, power storage, and so forth. This makes them challenging to use, program, and operate.

Resource restrictions can be mitigated with the use of alternate methods for input, storage, processing, and so on. For example, alternate input methods of speech or handwriting recognition can be used instead of keyboards, alternate storage methods such as cloud storage can be used instead of hard disks, and cloud computing can be used for certain processing instead of more power hungry on device CPU. These methods however require training of the devices and efficient communication capabilities.

Less Secured/Reliable

All compute devices have important resources and store valuable data and/or programs. It is important to protect access to all of these compute resources and data through user recognition and user authentication. Appropriate gatekeeper procedures and mechanisms should be deployed to protect the underlying data, programs, and applications while enforcing appropriate privacy guidelines and protocols. Since mobile devices are mostly in transit, their security becomes increasingly more challenging since these devices may use wireless channels, public resources, or networks that can provide easy access to these mobile systems.

With the explosion in smartphone usage, a lot of personal information is now saved and stored on smartphones. Users employ smartphones for communication, planning, organizing, and accessing and processing financial transactions.

Hence smartphones and information systems supporting them carry increasingly more sensitive data, thereby introducing new security risks while posing serious privacy access and processing complexities.

Some of the sources of security risks are

• Through messaging systems like SMS, MMS

• Through connection channels like Wi-Fi networks, GSM

• Through software/OS vulnerabilities to external attacks

• Through malicious software and user ignorance about it.

Some of the mitigation options are

• Use of encryption methods (Wired Equivalent Privacy: WEP, Wi-Fi Protected Access: WPA/WPA2) encryption

• Using VPN or HTTPS to access Wi-Fi/Internet

• Allow only known MAC addresses to join or connect to known MAC addresses only.

Intermittent Connectivity [1]

Mobile computing devices may be away from various communication infrastructures like Wi-Fi or the Internet for considerable periods of time. However to access required data and programs stored at remote locations, they need to be connected even if possible only intermittently. Such intermittent connectivity needs a different kind of data transfer mechanism that can handle power management issues, package loss issues, and the like.

Mobile devices need data to be buffered in the case where only intermittent connections to the network are possible. To prevent any data loss, the data transfer mechanisms in mobile devices need to handle cases when data is generated or received more frequently than the available connectivity. There could also be interruptions, interference, downtime, and so on that cause interruptions in communication links. Power scarcity can also cause a device to throttle communication or a connection. In such cases mobile devices and their data transfer mechanisms should be able to effectively and efficiently manage all available resources and connection time while avoiding user noticeable data loss.

Mobile devices should also be able to handle and deploy additional mechanisms to support interoperability (rate, routing, and addressing methodology) among communication protocols because it may dynamically move from one protocol to another during device transit.

Energy Constrained [2]

The lack of a readily available power source, smaller and compact size and resources for power storage and complex data management, security requirements, and connectivity requirements make energy availability and battery life a key constraint for mobile devices.

In addition, mobile devices use power-hungry sensing, storage, and communication capabilities but have some very stringent power and thermal budgets. These devices are without fans, are often in close skin contact with the user, and have restricted surface area; hence they are limited by peak power consumption since the user experience is affected by the temperature of the device. This further underscores why power management and battery life are key design parameters and constraints for mobile devices.

A mitigation plan for these challenges includes power management with an emphasis on platform power optimization and user experience:

• Platform power and optimization: the power management policy should be inclusive of available hardware resources of the mobile platform and manage their operation for energy efficiency.

• User experience: the usage of mobile devices extends from CPU- or graphics-intensive usage to sensor-heavy usage. Various location-based services and applications would require sensors like accelerometers, gyrometers, and cameras. Applications using touch capabilities would require quick exit from power-managed states and gaming applications would require higher throughput with brighter display. Thus power management system should consider these use cases, and corresponding system responsiveness requirements.

Thus mobile devices need hardware resources that provide various low-power states along with energy-aware operating systems and applications. Both hardware and software should be intelligent to incorporate user interaction, sensor inputs, and computational and protocol optimizations and their dynamic behavior/loads.

Historical Perspectives and the Influences of Market

The following are some of the key factors influencing the move to mobile computing. Some of these factors also influence the form factors within mobile computing options.

Enhanced User Experience

Mobile computing changes our approach to connectivity: in how we connect to different geographical locations, different people, cultures, and processes. It changes the way we gather, interact with, and process the information based on various sensing applications and location-based services. It enhances our perception of the digital world and merges that digital world with our physical world. It increases our computing power over our traditional standalone and stationary computers while enabling us to carry this enhanced computing experiences to carry it around wherever we desire.

Improved applications have been developed to harness hardware resources: various leading providers (such as Apple and Google) are offering many applications that extend the range of functionality of smartphones and other mobile devices, such as application to measure the heart rate using the rear camera of a smartphone. Such applications use hardware for functions other than their primary functions thereby extending the capabilities of mobile devices beyond traditional usage.

Improved Technology

With improved technology, mobile devices now have improved battery life, faster processors, user-friendly and lightweight manufacturing materials, power-efficient flexible displays, and high-bandwidth networks. The devices also have numerous sensors like biometric sensors, temperature and pressure sensors, pollution sensors, and location sensors. There are also infrared keyboards, gesture and retina tracking sensors, enhanced artificial intelligence, and new context-aware user interfaces. All these features increase the clarity of videos, images, text, and the like while generating new use cases and applications, making it possible to manufacture mobile devices in various shapes, forms, and designs.

New Form Factors

The way the user interacts with and uses mobile devices will change with advancements in the underlying technology. For example, a user has to reach his/her pocket many times a day (typically 150 times a day) to access a smartphone, but as the proliferation of wearables increases over time, much of this access will become hands-free. The amount of data and content uploaded from mobile devices is also increasing at an amazing growth rate. The uploaded content includes images, video, music, and so forth, but with increasing use of wearable devices (such as Nike Fuel and Google Glass), more personal data related to fitness, financial information, location services, and so on would be uploaded.

Increased Connectivity/Computing Options

There is now an abundance of wireless connectivity through various means like cellular mobile networks, wireless LAN, Bluetooth, ZigBee, ultra-wideband networks, Wi-Fi, and satellite networks. Additionally, with the birth of cloud computing, users are now also offered shared resources along with connectivity. The user is no longer tied to a particular location or device to upload, access, or share data. Availability of such a wide variety of connectivity and shared computing options has enabled both personal and business users to add more mobile devices/computing on such networks to improve their mobility and reduce infrastructure costs for business/personal data sharing.

Market Trends and Growth Areas

There are three fundamental driving factors for the growth of mobile computing devices: new sensor technology and products, sensor fusion, and new application areas.

New Sensor Technology and Products [3–5]

Sensors are fundamental to the growth in mobile device functionality. Technological progress and enhanced functionalities of mobile MEMS (micro-electronics-mechanical systems) sensors like microphones, cameras, accelerometers, gyroscopes, and magnetometers have enabled services like navigation, context awareness, location-based services, and augmented reality in the mobile computing devices.

Cameras and display technologies are showing considerable innovation with touch sensors, flexible and energy-efficient displays that brings reality to images, videos, and even 3D perception. For example, multiple cameras are being used to track user eye movements to highlight only relevant sections of the display while masking the not needed text or images on other parts of the display.

There is also an emergence of wearable technology, particularly related to health and fitness, which rely heavily on sensors that measures temperature, pressure, humidity, physical activity, and other key aspects of the body.

The worldwide wearable computing device market [6] (wearables) is expected to reach 126.1 million units by year 2019, with higher volumes expected to be driven by increasing end-user acceptance of wearables and increasing number of vendors entering the market with more device variations.

The key takeaways from the current market analysis are:

• Basic wearables that do not run third party applications will reach 52.3 million units in 2019.

• Smart wearables which run third party applications on them will grow at a faster rate than the basic wearables and is expected to reach 73.8 million units in 2019.

• Wrist worn wearables will account for 80% of all wearable shipments by 2019.

• The worldwide wearables market revenue opportunity is expected to reach 27.9 billion by 2019.

The market analysis could be impacted by the following potential scenarios over the next 5 years:

• The mobile market plateaus and the PC regains traction, thereby cooling off the MEMS market. This might seem unlikely but 5 years ago no one would have predicted that the PC market would stall in the way that it has.

• Wearables take off in a big way, with Google Glass leading to Kindle Glass and iGlass, giving MEMS sensors an even bigger boost.

• New classes of devices emerge, fueling even greater demand for MEMS.

Fig. 1.1 shows the worldwide wearables shipment for basic and smart wearables.

Figure 1.1 Worldwide wearables shipments by product category, 2017–19 [7].

Fig. 1.2 shows the worldwide wearables average selling price for basic and smart wearables.

Figure 1.2 Worldwide wearable technology ASP by product category, 2017–19 [7].

Fig. 1.3 shows the worldwide wearables revenue for basic and smart wearables.

Figure 1.3 Worldwide wearables revenue by product category, 2017–19 [7].

Sensor Fusion

Sensor fusion [7] is the process of merging data from different sensors such that final output data conveys more information than each of the individual sensors whose data was merged.

Today’s smartphones and tablets are not exactly ideal sensing platforms, due to their inability to harness and redefine the power of inherent sensors. As manufacturers today try to keep their products compact and competitively priced, the reliability of underlying sensors is compromised, while the sensor measurements are adversely affected by increasing electrical noise as they are subjected to magnetic anomalies, temperature variations, shock, and vibrations.

Developers today may think that sensor data are not accurate or reliable and refrain from developing applications that use or enhance the usage of underlying sensor data. Following are the two dependencies that affect the accuracy and reliability the sensor data:

• Operating systems of the mobile devices with sensors: Android, IOS, and Windows Phone may not be designed to handle real-time tasks such as on-demand sensor sampling. Hence the time stamps on sensor samples are unreliable.

• Filtering/dead-banding of sensor data: Sensor data today are manipulated using low-pass filtering and dead-banding methods that can discard otherwise useful data making sensing less reliable and less responsive. (A dead band is a band where no actions occur.)

For example, a gyroscope will have numerous error sources. One of the error sources is gyroscope bias. Bias is the signal output when the gyro is not experiencing any rotation. It represents a rotational velocity. This bias error will vary with temperature, time, noise, and so on. A gyroscope with XY°/second bias may appear to be spinning when the device is at rest (Fig. 1.4).

Figure 1.4 Gyroscope basics [8].

A magnetometer measures Earth’s magnetic field and its readings are used to calculate the heading of the rigid body. But it also suffers from errors resulting in heading errors (such as those from magnetic materials in the sensor, noise from magnetic components inside the sensor, or from nearby ferrous objects).

Accelerometers measure proper/g-force acceleration. This is the acceleration an object experiences relative to freefall and is the acceleration felt by people and objects. But it also suffers from bias. This bias is the difference between the ideal 0g output and the 0g output reported by the sensor. On a perfectly horizontal surface, if there were no bias error, then the sensor output would read the ideal 0g offset voltage on the x- and y-axis, +1g output voltage on the z-axis but due to bias error, it would show values other than ideal values.

So each of the individual sensors have biases and errors. An algorithm can be developed where sensor measurements from different sensors of the same event can be processed to separate out the real data and noise/sensor errors. This process is known as sensor fusion. With appropriate implementation of such algorithms, the perception of ideal responsiveness can be maintained while offsetting errors and shortcomings of individual sensors, thereby providing useful and reliable results.

For example, if a gyroscope is used along with an accelerometer and magnetometer to determine the device’s absolute orientation (the 3D rotation angle), then the sensor fusion process can be used to generate an interpreted event by collecting data from the three different sensors, performing mathematical calculations to remove individual sensor biases and errors, converting resultant data into format suitable to the developer and representing it in the same form of interpreted events. Such interpreted events can be considered as output of virtual sensors and be represented in the same form as original sensor events. These virtual sensors provide a solution and measurements that cannot be obtained from any single sensor. Measurements from virtual sensors are in between what can actually be measured on real sensors and the ideal measurement desired by the developers. Sensor fusion algorithms can reside in low-level code, in the sensor itself, or as part of application, and they can remove the biases and errors, thus providing designers with greater flexibility in selecting and combining sensor