Ubiquitous Computing Enabled by

Abstract

As embedded systems are added to everyday electronics, an efficient human interface is needed to access this processing power. The interface must be able to control various types of systems such as computing, entertainment and electrical as well as be available on demand. A personal controller worn on the underside of the wrist that optically scans the fingers and palm of the hand and is compatible with the various types of systems mentioned above is presented herein. Pointing at an intended target and motioning with the hand or finger can perform a mouse click, play a game or turn on a light. Since the user is only touching light to effect change in their environment, this device is considered a wearable personal virtual controller.

1.0 Introduction

As consumer electronics become smaller and more feature-packed, an intuitive human interface methodology is needed to tap into the power of these embedded applications. A personal controller using optical reflectance scans the hand and fingers while leaving the hand free of obstructions (See Figure 1). With visual input from monitors, TVs, electronic displays on entertainment centers and dashboards, as well as heads up displays and glasses, the personal controller can interface with all of these platforms via a host system application using infrared or radio signals.

Utilizing a standard device driver, the personal controller can be used in tandem with current input methods. Well-structured menu driven applications can be accessed with a minimum of motion and effort. For a new interface paradigm to gain acceptance, it must be extremely easy to do and simple to set up. The device must also be comfortably worn all day [1], [2] in order to be available on demand and must not limit or constrain the customary motion of the hand. All these parameters are met with the technology presented.

Figure 1. Optical reflectance scanning

2.0 Ubiquitous computing

Whether it’s ubiquitous, pervasive or convergence, the concept has been around for quite some time. For many, the idea of being “connected” at all times is quickly becoming a reality. With cell phone PDAs, WiFi hot spots and remote control tablets, the ability to access the technology around us is becoming more fluid.

In order to be on demand, an interface device must be both cross-platform compliant and easily engaged. The personal controller is enabled and disabled by a simple hand gesture, mechanical switch or voice command. Those areas that comprise the bulk of daily interaction include the home, office and automobile.

2.1 Home

Aspects within a home that require an interface control mechanism include entertainment, lighting, various consumer electronics, environmental controls and security features. Wireless enabled displays such as the ViewSonic (See Figure 2) are capable of controlling all these necessary functions. Future systems are under development at the Microsoft Home of the Future in Redmond, Georgia Tech’s Aware Home and the GW/AOL’s Smart Home located in Virginia.

Figure 2. A/V Display w/ controller and USB base station

2.2 Office

Streamlining and cost savings associated with office operations is an ongoing task. Costs associated with carpal-tunnel syndrome alone are in the billions (US$) when lost productivity calculations are factored in. Concerns regarding germ transmission, although not pervasive in the U.S., are quite prevalent in Asian cultures such as Singapore. If a personal controller could be used to open a door, push an elevator button and interface with a computer, the amount of multi-human contact with everyday objects will be greatly reduced.

2.3 Automotive

Remote Keyless Entry (RKE) using a thumb-actuated device located on a key ring has become commonplace. With a wrist worn device, the functionality continues as a dashboard controller for turning on the sound system and changing settings. Minimal gestures and motions that do not require visual feedback will allow performance of simple tasks without distracting the driver’s gaze.

The concept of virtual control originally referred to a gloved device that controlled a virtual environment within a computer simulation. Although the onscreen environment was virtual, the hardware required to create this experience was not. Using light to sense finger and hand motion goes a step further to create a virtual image of the hand and fingers without the constraints of wearing a glove or being tethered to a wired harness. As seen in Figure 1, each digit has a range for motion sensing and light button selection.

A cursor can be controlled by hand or finger motion as selected in the accompanying personal controller software. Since the device is able to scan various hand geometries to compensate for individuals with limited mobility, many can benefit from the “no impact” device since only light is being touched.

4.0 Hardware

The basis for this technology is a wrist-worn device that uses light beams to scan the hand and fingers. The reflected profile of the hand is analyzed in order to synthesize host system inputs. This input is transmitted from the wrist to a base station that is connected to the host system via a USB connection or directly to host via infrared or radio signals. The modal features of the hardware include the ability to point and click in a hands free way that senses shapes and gestures such as moving the hand up or down in order to adjust audio volume or lighting brightness in a wireless fashion.

4.1 Point and click

The ability to perform an action with a simple gesture is a more natural way to interact with electronics than keyboard and pointer methods. Gesture inputs may be selected for optimal ergonomics, reliability of input recognition and lexicon size. Sampled sensor data must be reduced to gesture correlation with sufficient speed to provide feedback to the user with a tolerable delay. For example, a beep or typing sound confirms a keystroke onboard the personal controller.

Latency is a critical issue, particularly with real-time controls, in order to confirm actions or optimize adjustments. Typical computer or menu-driven input works well with a 15 msec delay between action and command arrival at the host. High performance simulators, musical instruments and virtual controllers require sub-millisecond response for proper operation, and typically require update rates > 2 kHz. This rate can be achieved when the personal controller specialized device driver is uploaded into the various host systems.

The idea of pointing at something from across the room and having the ability to remotely control that item while only touching light is novel. A similar point and click concept is under development at Microsoft [3] in a form factor resembling a magic wand with a physical thumb switch (Figure 2).

Figure 2. Microsoft XWand [Ref. 1] point & click device.

4.2 Hands-free

One of the problems with previous virtual reality devices is that they required a glove with various sensors attached, which was both awkward and uncomfortable. A crucial aspect of assimilating new technology is its ability to provide an unencumbered profile that allows for uninterrupted human factors.

In the example of the personal controller, it occupies an area already accustomed to an apparatus such as a watch. An additional benefit of this technology is the ability to calibrate the user’s hand and record gestures. The programmable nature of the personal controller system lends itself well to context-sensitive input and control applications, enabling a hands free experience for a multiplicity of target hosts.

Studies such as that conducted by CMU [4] indicated ergonomic challenges associated with the one handed chest mounted input device. Although a hand may be unobstructed, if the scanning source is fixed, the hand also needs to remain in a particular inclination or be able to return to the exact sensing spot easily, which degrades ergonomic comfort.

4.3 Infrared/radio

Wireless communication protocol presently includes WiFi 802.11b, Bluetooth and Infrared Remote Control (TV remote). These wireless methods allow input directly to the host system by means of a device driver, where needed. Future protocols that look very promising are 802.15.4 (Zigbee) and Ultra Wideband.

Low power consumption is crucial for any wearable device. Optical data is processed in power-efficient algorithms and transmitted short range to the target host(s). Variable sample rates and sleep cycles conserve power during inactivity. Communications power demands must be minimized, and transmit-only configurations are preferable for many applications since the controller can inherently send and receive optical communications for bi-directional exchanges. For example, a radio transmitter complemented by IRDA provides configuration upload from the host. Additional information concerning power consumption can be garnered by referring to a thesis by T. Sheikh [5] at VA Tech.

The scanning mechanism adjusts to various hand and finger shapes creating an efficient ergonomic human interface device. Optical personal controllers detect and process real-time hand shape information directly. This greatly reduces the magnitude of signal processing required to detail the hand position accurately. Shape sensing may utilize “bug vision” [6] by employing a relatively few emitter-sensor pairs to individually sense fingers and hand-wrist angle however this simple configuration forces the user to properly locate the hand with the integrated scanning light beams.

Alternatively, many smaller-beam sources may be employed to enhance resolution at the cost of greater processing demands to resolve individual fingers. Accurately tracking the hand-wrist angle allows determination of finger paths across the optical plane. Typically, the wrist-hand angle measurements of 12 bits or more are realized due to the close proximity of the heel of the hand to the wrist-worn device. Additionally, tilt sensors may be used to allow tracking of hand attitude and rotation, or gyroscopes to track hand motion.

Optical reflectance measurements record the intensity of the reflected light-beam, which varies with the angle and position of the reflecting element (e.g. finger). The wrist X angle is measured and used to calculate column offset, enabling finger tracking. The wrist Y angle is used to adjust button threshold level, to create a uniform keystroke gesture anywhere within the active optical field. A cone shaped fan array of optical emitter/receiver pairs exploits the cupped geometry of a hand at rest. Proper location of the cone apex underneath the wrist is crucial to avoid saturating the optical receivers with near-field reflections on the heel of the hand.

Hands have traditionally been a focal point for communicating intentions. Either by a wave or pointing in a direction, simple hand motions can speak volumes. The personal controller can be used for recognizing these basic hand and finger gestures.

All machine inputs are synthesized by emulating hand gestures. A keystroke is a simple gesture, for example. Keyboards, joysticks and mice may be emulated, as they are universally understood; however, the power of gesture recognition is better exemplified by sign language. Real-time conversations in sign language pose processing challenges to hardware designers pursuing electronic capture of the conversation for translation or transcription purposes. Both hands are typically used for signing, and the benefits of wrist-worn sensor arrays are realized. Neural processing of sensory data to resolve gesture inputs is demanding, and is greatly enhanced by optical sensor accuracy.

Simple, one-handed gestures may also be used to control the input device itself. For example, the utility of a wrist-worn device is extended to daily life when simple, unique gestures are employed to engage and disengage the device. Emulation modes may be similarly engaged: one moment emulating an aware-building light switch, the next emulating a television remote control.

Although the personal controller itself is a hardware platform, the coding required to operate, communicate, integrate and proliferate is essential. Circuits have been programmed with the most efficient lower level languages. Wherever possible, standardized protocols were used such as the device driver. Specialized applications have been constructed using Windows OS. The controller supports platform independent embedded and stand alone applications as well.

7.1 Embedded

Software embedded into the wrist-worn device is written in ANSI C, and gate-array Boolean, state machine and Digital Signal Processing implementations are written in VHDL. Standardized code facilitates optimal portability, maintenance and continuing engineering. Ballistics calculations include standardized algorithms referenced from development systems (e.g. Linux, Microsoft and Apple, among others). Standardized ports support System Development Kits for developers and toolkits for users.

The initial device driver for the personal controller at initialization of the USB interface is acknowledged as a Human Interface Device (HID) mouse, a standard offered on most operating systems. The USB HID base station delivers packets to the target device that default to standard mouse packets:

Additionally, standard HID VR-Glove packets are supported (finger angles). Special applications requiring raw vectors will be disclosed under license, and will facilitate “bug-vision” shape image scanning for environmental capture.

Device drivers are written in C Language, but are typically host-system dependent. Device drivers may be simple relay devices or may include additional input processing if the host system permits. For example, a device driver that tracks cursor position on the screen may be used to change the context of the virtual input: pushbuttons in one region may become slider adjustments in another area of the screen.

In order to be assimilated into the various systems required for ubiquitous coverage, device drivers must be limited to a thin layer of code that is cross-platform compliant; the thinner the layer the better.

As people go about their daily computing activities, a new interface device will only be adopted if it can be incorporated into their favorite applications. Without additional software using the HID driver, the personal controller will perform mouse functions. More advanced functions such as typing, require an onscreen keyboard (Figure 4) and personal controller software. As the keys are first highlighted then actuated, the data appears in the application, just as if it was being done on a physical keyboard.

8.0 Specialty Applications

As a controller that can be used on varied platforms and systems, the applications that interface with this type of controller may already be compatible due to the selection in the initial construction as a HID. Specialty applications such as gaming, sterile medical interface and intrinsically safe “no spark” control, may require specialized application software to take full advantage of the versatility of the personal controller technology.

8.1 Gaming/Simulation

Current gaming controllers consist of two-handed controller pads, joysticks or computer keyboard designated keys. The personal controller can function as an on-demand joystick with multiple buttons available. As early adopters of new technology, gamers will take great pleasure in a cool new way to interact with their systems. Eventually games will be created incorporating the controller’s full capabilities.

8.2        Medical/Sterile

The medical applications are an important aspect of this new technology. For medical professionals and support staff, it will provide a sterile interface control function that will allow for data input and retrieval while remaining sterile. Since many functions can be performed without actually touching the various pieces of equipment, there will be a marked reduction in germ transmission. In addition, expensive equipment will require less maintenance due to deteriorating mechanical switches.

Patients will benefit from this technology since they will be able to interface with onscreen menus with a limited amount of effort and motion. As our population ages, the topic of limited mobility is going to become a prominent issue.

8.3 ALS and Parkinson’s

In examples where mobility is severely restricted such as ALS, paralysis or stroke, the personal controller’s ability to sense a fraction of an inch of motion from a single digit, will allow them to have a switching mechanism that will restore a quality of life that they felt was lost forever. Many times a solitary twitch is all these individuals have to communicate and the controller can take advantage of this aspect.

Another benefit of a virtual controller is what occurs on one end, does not need to be reflected on the other. A person with Parkinson’s, for example, may appear to have a steady hand on their monitor. The tremors may be compensated onscreen via software and the individual will be able to regain a “sense” of steadiness that had been lost.

8.4 American Sign Language Translation

Since the personal controller can sense shape and motion, a natural evolution of the technology is to translate American Sign Language (ASL). Eventually the device and specialized software will be able to translate sign language and emit the spoken word via the speaker located in the device. This will assist those who are hard of hearing and allow them to assimilate into a hearing society. For additional information regarding gesture recognition and ASL, consult the work of Thad Starner at Georgia Tech [7].

Mechanical switches risk spark ignition in combustible environments due to the physics of “switch bounce” during state changes. A properly designed optical switch may be used in such hazardous environments. Non-contact input further reduces electro-static discharge possible with hardware controls.

9.0 Market Viability

In the extremely crowded and competitive arena of consumer electronics, products are desperate for new features that will provide meaningful differentiation from the competition. This technology offers such an edge.

This input device could give a manufacturer in almost any consumer electronics product category a meaningful leap ahead of the competition -- and for the first few products to adopt this technology there will be great "wow" factor. Gaming will be a natural.

10.0 Conclusion

The personal controller technology presented herein will transform the way individuals interact with their electronics. Whether able bodied or dealing with a limitation, human interactions with electronics will become more intuitive and efficient. The need for all those remote controls (Figure 5) will become a thing of the past. One personal controller interacting with various devices will open up innovative technologies to those who previously felt intimidated. Onscreen menus and tutorials will be available to offer guidance to using these new advances. With easy to use personal controller and easy to follow software, the adoption rate of new innovations will hopefully increase and along with it, the probability of ubiquitous computing a reality.

Figure 5. Remote controls vs. wrist controller.

More information concerning applications and technical aspects of the personal controller can be found at www.lightglove.com.

References

[1] A. Toney, B. Mulley, B. Thomas, W. Pierkarski. “Minimum Social Weight User Interactions for Wearable Computers in Business Suits,” IEEE Sixth International Symposium on Wearable Computers (ISWC) 2002
[2] J. Knight, C. Baber, A. Schwirtz and H. Bristow. “The Comfort Assessment of Wearable Computers,” IEEE Sixth International Symposium on Wearable Computers (ISWC) 2002
[3] Microsoft, article “the speed of thought” nwaworldtraveler.com/0404/feature02/index.html

[4] E. Ayoob, B. Gollum, J. Siegel. “Design Principles for Wearable Systems Interfaces and Interaction” Carnegie Mellon University, 2003

[5] T. Sheikh, thesis “Modeling of Power Consumption and Fault Tolerance for Electronic ” VA Tech, 2003 scholar.lib.vt.edu/theses/available/etd-10162003-005726/ unrestricted/tsheikh_2003_thesis.pdf
[6] A Mioni, A. Bouzerdoum, A. Yakovleff and K. Eshraghian. “A Two Dimensional Motion Detector Based on the Insect Vision,” Presented in the Focal Plane Arrays and Electronic Cameras, Oct 1996
[7] T. Starner and A. Pentland. “Real-time American Sign Language” recognition from video using hidden markov models. In SCV95, page 5B Systems and Applications, 1995.

Four 8-bit bytes representing:
Buttons,	X-vector,	Y-vector,	Scroll.

Bruce Howard and M.G. Howard