Augmented Reality Fingers
Two concepts for precise number input using the hands in AR
This Augmented Reality prototype was a response to the earlier project that used the HoloLens. Rather than accept the given “Gaze, Gesture, Voice” paradigm, I thought about alternative approaches to the same problem (inputting a precise number, e.g. date of birth or phone number).
I decided early on that I didn’t want to use any kind of input device. Ultimately, the fastest method of inputting a series of digits would be to use an additional, ideally tactile device. Simple existing solutions are a wireless keyboard, or a smartphone keyboard.
I wanted to design an interaction for the ‘idealistic AR scenario’, i.e. when headsets are so integrated into our lives that they are ubiquitous. In this case, not occupying your hands with a device is ideal.
Rather than taking the previous ‘head controlled mouse’ approach, I wanted to try using my hands themselves as the input device. Unfortunately, Microsoft don’t make the depth camera data available. I considered trying to build it using 2D computer vision, using an additional sensor (like the Leap Motion), or using background keying.
Considering that this hardware already exists on the device, and would likely be present on future AR headsets, I decided that building for this technology made sense, but creating a technical workaround would have too much time consuming busywork. I decided the better use of time would be to communicate the design concepts through video. After so I fired up After Effects and got to work.
AR Concept Sketches including hand gestures, GGV, sliders, line crossing, a Morphic inspired "Halo" and a smartphone as controller.
As I was avoiding using speech, this would be a person that was deaf, far away, or in a noisy room
The two approaches I made videos for used the hands in different ways. The first followed a design principle I’m fond of: “If I were interacting with a person (rather than a computer), how would I do this?”. The solution that first came to mind was just to show the digits, using the fingers on both hands. This has the added benefit of being a very concrete expression of the digit.
As I was avoiding using speech, this would be a person that was deaf, far away, or in a noisy room
The second approach was inspired by a method that I associated my time studying music in India, but have since found it difficult to find a reference for. The joints and tips of one hand are used to count up to 16, using the thumb (this is also useful for counting in hexadecimal). This has the advantage of only using one hand, or could be combined with the second hand for much larger numbers. I also adapted the mapping to the more familiar keypad layout, which seems like a suitable adaptation as we only need 10 of the 16 positions for digits.
Other finger counting methods
I later learned about Chinese number gestures as another method of counting to ten on one hand. It turns out there are a number of finger counting methods, including a base 60 (sexagesimal) one used in ancient Babylonia source.