Put us on record saying it: iPhone apps that use augmented reality technology will soon be the
focus of an Apple TV commercial, and when this
commercial appears, minds will be blown.
Augmented reality is essentially any piece of
media that pulls elements from the real world
(say, the video feed from your iPhone’s camera)
and combines them with digitized information
(say, landmark coordinates pulled from a mapping
database). You’ve probably already seen a form of
the technology in football broadcasts, where yellow
first-down markers are laid on top of live video of the
playing field.
The iPhone is shaping up to become the new
augmented reality epicenter, combining real-world
and technology-driven perspectives to enhance
navigation, gaming, and more. In a nutshell, instead
of looking at your iPhone for information and
mentally linking its screen display to what’s going
on around you, certain apps let you hold up the
phone, look at the screen, and see contextual data
overlaid on real-world images.
Several apps, including Wikitude, Nearest Wiki,
and Robot Vision, let you point at a landmark to see
its Wikipedia entry onscreen. Similarly, DishPointer,
Sun Seeker, and Pocket Universe superimpose the
positions of stars and satellites to help you locate
unobstructed TV reception, estimate sun exposure,
and view star charts, respectively. Peak.ar and Peaks
superimpose the names and statistics of your local
mountains. The list goes on—and will inevitably
become more science-fictiony and amazing.
The new apps are made possible by iPhone OS 3. 1,
which lets developers access the iPhone camera’s
video feed. But it’s more than that, says Freeverse
president Ian Lynch Smith: “It’s not just the live
video. When combined with the compass
and accelerometer, there’s something
unique about seeing someone interact with
their mobile device by panning it around,
turning themselves around, and actually
looking at the world, only with new stuff
digitally added.”
Freeverse’s Fairy Trails game takes this
approach. Sprites hover around you, and
you have to move the iPhone to see them
floating over your real-world backgrounds.
The effect works best when leveraging the
compass in the 3GS, and, in fact, many of
these apps require Apple’s latest iPhone to work
at all.
Blair MacIntyre, associate professor at Georgia
Tech’s College of Computing, says some apps are
disappointing because of rough GPS and other
sensor data.
“The compass and accelerometers are not as
accurate as you’d like for doing this,” he says. “You
have to hold the phone relatively still, which isn’t
bad when you’re looking at, say, a bunch of photos.
But if you’re trying to look at anything that’s
moving, or anything where [the data] needs to be
accurately overlaid, that’s a problem.”
Yelp’s augmented reality mode, for example,
looks great if you point it at restaurants that are
about 40 yards away; the Yelp ratings align with
your real-world view. But if you’re too close to a
restaurant, data inaccuracies can scatter ratings
away from their proper locations.
Better visual data is one of the keys to better
augmented reality apps. The current software
development kit (SDK) only lets developers
superimpose graphics over the live, real-time
picture feed, which MacIntyre says is a problem.
“You really need for each frame of video to get back
that bitmap, that image,” he says. “That lets you
look at the pixels and analyze them.” Developers
can burrow beyond the iPhone SDK to approximate
some of this camera data, but they risk Apple’s ire.
With full camera data, developers could program
the iPhone to recognize objects and scenes in the
real world, all in real time. For example, Xsights
can recognize pictures of certain theater posters
after analyzing a still picture. With better data
access, that sort of application could superimpose
real-time details over a poster, such as offering
rush ticket sales for that night. Similarly,
museums could animate 3D tour guides
that appear on your iPhone when you
enter a certain exhibit.
The current flood of GPS-, compass-,
and accelerometer-based apps show the
iPhone’s augmented reality potential—and
its limitations. But once developers can
analyze video data, expect augmented
reality games, ads, maps, and educational
tools to redefine our real-world
perspectives. It’s beyond amazing, and it’s
coming to a screen near you.—Zack Stern
The background: real. The butterfly
and dragonfly: Not so real.
Sun Seaker provides info on solar
paths, benefitting photographers,
gardeners, and more.
Point Peak.ar at towering mountains (or even humble
hills) for a display of names and elevations.
