Gesture-based user interfaces

Operating a computer hasn’t changed much since the introduction of the mouse-windows-menu idea 25 years ago. Consumer goodies like the iPhone and the Wii show there’s another way. Simon Eccles looks forward to a change. Not being a complete techno-nerd, I don’t yet have an iPhone. When my mobile provider offered me a ‘free replacement phone’ a few months ago, I decided to pay a little extra and go for the Nokia N95, the iPhone’s hottest rival, on the grounds that it had real 3G abilities, GPS (never get lost again) and a 5 million pixel camera with a decent lens. The iPhone has none of those, and its camera is a pathetic 2 mp.

And yet. Even though it’s expensive to buy and run and doesn’t use European-standard 3G, the iPhone is a very important bit of kit. It introduces most significant new user interface since the original Apple Macintosh of 1984. If you haven’t bought an iPhone either, look out for it on Apple’s current TV campaign.

It uses gesture-based navigation commands, taking the touch-screen idea but turbo-charging it into a completely new way of operating a computer.

The whole front surface is both a screen and a touch-surface. You stroke your finger across its surface to select menu lists, and you can zoom in or out of any area by using pinching motions with your finger and thumb. Move our finger quickly in a stroking motion and the menus will accelerate and continue to move, and then gradually slow down. Touch again and the motion stops. It’s a work of interfacing genius, though you literally need to get your hands on it before you can fully appreciate it.

Less impressively, the touch-screen based typing of letters is slow and painful, so you wouldn’t want to compose long e-mails that way.

Cascading documents
iPhone’s image and files are displayed in a cascade effect that displays a stack of documents in a way that lets you flick through them at high speed until you see the one you want, when you can stop the cascade and open it.

If you fancy the fabulous user interface but don’t actually need it in a phone, Apple also offers it in the iPod touch, which likewise is far more than just another music player: it can handle on-the-move e-mail and internet access, and play videos as well as music. Unfortunately it’s not cheap at present: from about £229 all the way up to £399, which would buy you a basic-spec Windows laptop.

We can also look to another piece of consumer gadgetry for user interface inspiration. Nintendo’s Wii is a cheap and cheerful games player that’s currently taking the market by storm, to the particular discomfiture of Sony, which thought that it was going to clean up with its more powerful but expensive Playstation 3.

Wii is pretty sophisticated electronically, but what users love is its Remote, which uses natural physical movements and gestures to control things on-screen. It looks like a TV remote controller, but inside is a combination of built-in accelerometers and infrared detection that it uses to sense its position in 3D space and transmit this back to the control unit via Bluetooth. There’s also a more streamlined ergonomic controller called a Nanchuck, which is better for some games. Depending on what game you play, it becomes a sword, a golf club, a gun, and so on, where you wave it around appropriately and the computer senses the motion and translates that to activities on screen, such as the motion of a ball.

So why rave about consumer gadgetry in a print magazine? For one thing, the iPhone and iPod Touch are in reality small hand-held computers that just happen to do mobile telephony, or music playing. Lurking under the smooth case and gorgeous icons is Mobile X, a cut-down version of OS X Leopard, the very same operating system that runs on real Macintosh computers.

Will it work on grown-up computers?
If you can do that on a little hand-held doohickey today, how about on a proper computer in future? Well, it’s starting to happen. The gesture control is built into the latest Macintosh laptops, using the trackpad control surface. But in future, maybe it will be more widely used – an obvious extension would be to the touch-sensitive Wacom tablets that are widely used by graphic and CAD designers – with much larger sensitive surfaces, these have more gesturing potential than an iPhone or MacBook.

The biggest and priciest Wacom is actually a touch/pressure-sensitive LCD monitor called Cintiq, which you can lay on your desk and draw onto directly with a stylus. Wacom has been making these for a few years in sizes between 16 and 21 inches and while they haven’t taken the world by storm, they’re potentially an interesting route back to the centuries-old method of working directly onto the media, whether it’s paper, papyrus or clay tablets.

Now imagine running the iPhone gesture-based user interface on a desk-mounted A3 screen. Would that be an interesting way to sort documents, make up pages and retouch photos? It’s hard to say without trying it, but it’s worth exploring.

Already the new user interface concepts are starting to break out into the wider world. Users of Apple’s iTunes will probably be familiar with its way of displaying album covers in a cascade effect that scroll through the window, alternately enlarging and receding into the background. Other computer and video designers are also starting to use this effect.

It’s about reach the graphic arts world. As revealed in Image Reports February, the next generation of DALiM’s MiSTRAL Virtual Library digital asset management software will use an iTunes-style cascade to quickly display and navigate through a stack of images in a database.

Which brings us to another idea that the HCI people are playing with. Oh no, another acronym! HCI stands for Human-Computer Interface, and explores ways of making it easier for we mere primates to control the wretched things.

For the past few years, nerd-fests like Siggraph have seen developers showing off prototype user interfaces of the future. And a lot of them look like giant iPhones. Gesture-based commands are the next wave, if you believe what they’ve been saying.

Microsoft chairman Bill Gates, patron saint of geeks, is bowing out of daily operations. He made a final keynote speech at the Consumer Electronic Show in January. He was criticised for saying little about the future, but actually, he did. He showed a prototype Microsoft Surface, a ‘screen’ built into a tabletop, where objects are moved and controlled by touch – finger gestures and taps.

Gates’ demo harks back a year ago to the TED (Technology Entertainment Design) conference in Monterey, California. Blaise Aguera y Arcas (that’s the name of just one guy, not a company), showed up with a barnstorming display of a new user interface technology he’d developed, called Seadragon. Microsoft liked it so much that it had bought the concept and hired Arcas in 2006. Seadragon is a way to display vast amounts of image or text data on a screen as small visual blocks and instantly zoom into any area in high resolution for reading or printing.

Seadragon gets ready to roar
Seadragon isn’t yet ready for full-time appearance in an operating system like Vista, but it has been built into a potential end-user program called Photosynth, which is under development by Microsoft Live Labs, a sort of hothouse for new ideas.

Still only at the downloadable demo stage, Photosynth is a way of combining any number photographs into a virtual 3D display, to navigate through them and choose one or more to view. Say you have a whole collection of pics of a specific location – Microsoft uses an example of the area around Notre Dame cathedral in Paris. Photosynth automatically analyses the image content for similar items within photographs taken from different locations or angles (say the corners of windows or doors, points on spires, roof tops etc). It then starts to match them up. By calculating the difference between matched points, it can work out their spatial relationship. It then displays all the points as a sort of ‘cloud’ that have a 3D structure – it builts up a ghostly 3D model of an area, with the individual pics appearing as you run a cursor over the cloud. You can then navigate and zoom in seamlessly using Seadragon technology.

It’s very impressive but hard to imagine without seeing it yourself. Http://labs.live.com/photosynth has an animated preview, or you can download a demo set onto your own computer (Windows only and you’ll need a fairly powerful PC and graphics card).

Last month I was asked to help a fellow editor to learn to use QuarkXPress, to do live editing on-screen rather than send e-mails back and forth to the designer. It turned out that actually she had used XPress 15 years ago, so she only needed a quick refresher.

This drove home the point that the user interfaces of Quark XPress, and most other design and image editing packages, haven’t changed substantially since they were all invented in the late 1980s.

Yes, they have more bells and whistles and they need vastly more power and memory from the computers they run on.

But if you can come back to a program after an absence of 15 years and still sit down and use it, that suggests one of two things. Either the programmers got it right all those years ago, or they aren’t thinking too hard about ease of use any more.

Back to top