Goodbye mouse, hello fish

Toby Howard

This article first appeared in Personal Computer World magazine, June 1999.

LOVE IT OR LOATHE IT, the WIMP -- Windows, Icons, Mouse and Pointer -- interface is universal, and has changed little since its birth at the Xerox research labs in the early 70s. But soon you may be putting your mouse to one side, and reaching for a fish instead.

The ideal human-computer interface, according to a group of researchers at the Massachusetts Institute of Technology, is to have no apparent interface at all. The conventional mouse, with its buttons, desk-bound wheel and restrictive cabling, is really an encumbrance. It's easy to deal with the cabling -- and switching to a wireless mouse which uses digital radio to communicate with the PC can really be a liberating experience. But what about doing away with the mouse altogether?

Various strategies have been tried for creating a "device-less" interface, based on interpreting the user's hand positions and gestures. Approaches like infra-red beams and ultrasound have the drawback of suffering interference, and cannot reliably provide accurate measurements. Multi-camera video systems can capture 3D information, but the images require an enormous amount of computation to extract the data, making full real-time interaction difficult to achieve. The problem is that there's simply too much information in video. To track the movement of a hand, for example, many systems use edge detection algorithms to find the outline of the hand in the video image. Compared to the whole picture, this is a tiny amount of information, but extracting it is a real headache.

One technology, however, looks promising. It's called "electric field sensing". Leon Theremin used it in 1919 to control one of the first electronic musical instruments -- the theremin. Everything about this instrument is strange, from its eerie plaintive tone to its "user interface". It's possibly the only musical instrument you never actually touch. There are two antennae: one controls the pitch of the sound, the other its volume. As the player's hands move in front of the antennae, the capacitance of the system changes, which in turn alters the pitch and volume of a beat-frequency oscillator. In the hands of a novice the device can sound like a pack of howling dogs, but skilled theremin players, such as the Russian musician Lydia Kavina can use tiny movements of the hands and fingers for exquisite control of the instrument. You've probably heard the theremin on the Beach Boys classic "Good Vibrations". You can still buy one today.

Now, the MIT team has taken the hands-off idea and applied it to the human-computer interface. They call their system a "fish", since many species of electric fish use electric fields for sensing their environment, and, unlike a mouse, a fish can move around effortlessly in three dimensions. The user sits in a chair which conceals a low-power, low-frequency radio transmitter. Around the monitor are small receivers, which with some simple processing can track the user's hand gestures in the space around and above the desktop. The beauty of the system is that the accuracy of the gesture recognition can be increased by simply adding more sensors, as needed, to form what the researchers call (predictably) a "school of fish".

As well as being successfully prototyped in the lab, the fish has already been successfully used in a touring multimedia art exhibit called the Brain Opera. One part of the exhibit is the "Gesture Wall". The participant stands on a metal transmitter pad, in front of a large back-projection screen. The graphics on the screen and accompanying music change in response to the way the player's body and hands move.

For conventional screen-based applications like word-processing and email, the WIMP probably has a guaranteed future, but for sophisticated 3D applications like navigating virtual environments and VR gaming, the future might well be fishy. Watch this plaice.

Toby Howard teaches at the University of Manchester.