Age of Experience

This is an excerpt from an upcoming book on the discipline of Creative Technology.

I have a confession to make. I love watching ghost hunting shows. Since Ghost Hunters started on the SciFi channel almost a decade ago, it’s been fascinating to see people go into locations around the world to attempt to find concrete examples of paranormal activity. Ghost hunting has blown up since then. We have bros with t-shirts two sizes too small yelling into empty rooms. There are teams of pseudo-scientific engineers creating Tesla coil powered pyramids to capture a spirit in a “focus crystal.” Beyond the historical aspect of learning about asylums, castles and abandoned mansions, it’s also interesting to hear why people believe in ghosts. Not in the sense of mediums and ghost whisperers, but what are the environmental effects that people feel that cause cold spots? Why certain rooms in buildings cause a sense of dread in some people?

In the early 1980s, Vic Tandy, an engineer and IT lecturer at Coventry University had an encounter with the paranormal. Tandy was working in a medical device research lab when he, “was sweating but cold, and the feeling of depression was noticeable – but there was also something else. It was as though something was in the room with me.” As the feeling grew more intense he started to see an apparition appear in his peripheral vision. When Tandy turned to look at the apparition, it disappeared.

The next day Tandy returned to the same laboratory. An avid fencer, Tandy had his foil clamped in a vice so he could polish it. When he walked away from the blade it started to vibrate wildly. Another case of the phantom of the research lab? Quite by accident Vic Tandy stumbled upon the theory that infrasound was present in the lab.

Sound is an incredibly powerful thing for animals, from vocal communications that allow us to convey and share complex concepts, to music which allows us to convey a range of emotions. There are also other sounds that we can experience that triggers things that are more primal. Things in our reptile brains that we know are instinctually dangerous. Have you ever heard a lion roar? Not on TV. In person. Where you hear that majestic roar that has a deep bass that shakes you to your bones. That triggers your reptile brain to know that there’s something dangerous around.

Sound waves are physical pressure waves that vibrate in a medium like air or water. If we step back into high school physics class, waves consist of a few key properties: the frequency of a wave is defined by the number of repeating occurrences per period of time, measured in hertz (Hz); the inverse of frequency is the wavelength, the distance over which a wave repeats once, typically measured in meters for sound; and amplitude is the measure of change in the wave over a period of time, also measured in meters.

In terms of human hearing, we generally say that humans can hear sound from 20Hz to 20,000Hz. Frequencies under 20Hz are called infrasound while frequencies above 20,000Hz are called ultrasound.

Armed with the theory that infrasound was present in the lab, Tandy ran a series of experiments. Tandy’s theory was correct. Infrasound was present in the lab, caused by the presence of a newly installed extractor fan that generated sound waves at 18.9Hz. The infrasonic sound waves were strongest at Tandy’s desk, the location where he experienced his ghost sighting and where his fencing foil mysteriously shook on its own. When the extractor fan was turned off, the sense of dread lifted and the foil stopped dancing.

Able to reproduce the experiment, Tandy published his findings in the Journal of the Society for Psychical Research. While it had been known for some time that infrasound can trigger feelings of fear and shivering, Vic Tandy was the first researcher who linked that experience to ghostly sightings. Tandy followed up by testing his theory at other haunted locations like Edinburgh Castle and the Tourist Information Bureau next to Coventry Cathedral, finding the presence of infrasound at each location.

Modern day ghost hunters use a wide variety of technology tools to search for the source of apparitions. Electromagnetic field meters, white noise generators, thermal cameras. Perhaps ghost hunters should add an infrasonic microphone to their standard load out for the next ghost adventure.

So infrasound can instill a sense of fear and dread in people. Unless you are in the business of building haunted houses, which would be very cool, how does this help build engaging experiences? One of the interesting things about human emotions, especially physiological reactions, is that emotions lie on a spectrum. Sometimes two different emotions can trigger a similar physiological response.

Pipe organs trace their origins back to the 3rd century BC where the Greeks built a hydraulis, a water powered organ. A hydraulis would use the force of water, from either a waterfall or a pump, to generate air pressure that was pushed through pipes of various lengths. To see an amazing and modern version of a water organ, look at Nikola Bašić’s Sea Organ in Zadar, Croatia.

Around the 2nd century AD people started replacing the water mechanisms with inflated leather bags. Around the 6-7th century AD bellows were being used to drive the air pressure through the pipes. Pipe organs have grown more mechanically complex over the centuries and still show up in cathedrals and theaters around the world.

In One God Clapping: The Spiritual Path of a Zen Rabbi, Rabbi Alan Lew explains there are several different words for fear in biblical Hebrew. Pachad, or dread, is “projected or imagined fear,” the “fear whose objects are imagined.” This can be thought of in terms of instinctual fears as well, triggered in our reptile brain. Like hearing the roar of a lion.

Lew also describe a second type of fear described in Hebrew. Yirah is described as, “the fear that overcomes us when we suddenly find ourselves in possession of considerably more energy than we are used to, inhabiting a larger space than we are used to inhabiting.” That sounds a bit more like awe.

If you think of fear and awe in terms of physiological reaction, they are often quite similar. Goose bumps on your skin, hackles on your neck raise as your heart rate accelerates. If you experience those physical symptoms in a cathedral, your mind will probably wander to a different place than if you experience them in the basement of an abandoned sanitarium. And could those 64’ organ pipes that generate an 8Hz infrasonic tone be reinforcing another feeling in that cathedral?

While it’s impractical to install a pipe organ in a modern physical-digital experience, learnings from a 2000 year old technology can still help us add layers of depth of emotion to our experiences. By understanding traditional technology and the visceral reactions they can cause on human physiology and emotion, we can provide more compelling experiences that shape and focus users’ emotional responses.
It Doesn't Have to be Emerging to be Amazing

Lateral Thinking with Withered Technology

This is an excerpt from an upcoming book on the discipline of Creative Technology.

In the early ‘80s, the nascent video game industry became a victim of its own success, peaking in 1983 and crashing a few years later. While the industry sorted itself out, technology marched on, producing more powerful processors, lower powered chips, multi-color screens and longer battery life. At the leading edge of the Fourth Generation of video game consoles, Nintendo released the Game Boy in April of 1989, a handheld video game system that kicked off the handheld console wars. Some industry experts criticized the Game Boy for it’s small, monochrome screen and its dated 8-bit processor.

Not to be outdone, Atari quickly followed up by launching the Lynx in October of 1989, the first handheld with a full color screen, sport a 4906 color palette, and a backlit screen. The Lynx received rave reviews, with some critics saying it, “throws the Gameboy [sic] into the prehistoric age.”

Sega released the Game Gear the following year in October of 1990. The Game Gear upped the technology ante by increasing resolution and displaying up to 32 colors at once, doubling the 16 colors the Lynx could display.

In December of 1990, NEC released the TurboExpress. The TurboExpress featured a 16-bit processor, versus the 8-bit processors used by Nintendo, Sega and Atari. The screen resolution of the TurboExpress was over double of its competitors and could display 481 colors at the same time. From a technical specification standpoint, the TurboExpress was light years ahead of its competitors. But it was not widely adopted by gamers.

As the handheld wars continued throughout the 1990s, Nintendo pulled ahead as a clear leader, with the Game Boy selling 6 times as many units as its nearest competitor, the Sega Game Gear. Nintendo would continue to dominate the handheld gaming market throughout the 2000s, taking 4 of the top 5 selling console spots and selling 412 million units vs. Sony’s 86 million, its next nearest competitor.

How did a clearly technically inferior product gain and maintain market domination among handheld gaming consoles? How did outdated hardware make more compelling experiences than the bleeding edge of the industry?

In 1965 Gunpei Yokoi was an electrical engineer maintaining assembly line machines in a Nintendo factory. Hiroshi Yamuchi visited that factory in 1966 and took notice of a toy that Yokoi had made, a simple grab and extend arm. Yamuchi liked the toy and assigned Yokoi to develop it into a proper product, the final edition being the Nintendo Ultra Hand. After enjoying huge success with the Ultra Hand, Yokoi was reassigned to work on other Nintendo toys, moving to digital devices in the 1970s, like the Nintendo Game & Watch a line of handheld electronic games. With numerous successful products under his belt, Yokoi switched to developing software, overseeing Donkey Kong and Mario Bros. As the video game industry began to recover in the mid-to-late 1980s, Gunpei Yokoi led the creation of the Game Boy, Nintendo’s runaway success in the fourth generation handheld console wars.

To understand the success of the Game Boy, it’s important to understand Yokoi’s philosophy of product development. Yokoi suggested that expensive, cutting edge technology often gets in the way of developing great products. Novel, fun gameplay are the things that product and game designers should focus on. This philosophy, embraced to this day by Nintendo, is called “Lateral Thinking with Withered Technology.”

Lateral Thinking with Withered Technology takes cheap and well understood technology and looks to apply a radically different use for it. In the development of the Game Boy, by eschewing a backlit, color screen, the Game Boy’s battery life was over twice as long as competitors. This philosophy extends to modern day Nintendo products like the Wii. Instead of trying to keep up with the Xbox or Playstation in processing power and higher resolution graphics, the Wii changed the conversation by developing new video game controls with internal motion sensors.

How do pull this thinking into the discipline of Creative Technology? What valuable lessons can we learn? One emerging technology that embraces this philosophy is probably sitting in your pocket right now.

In the early 1800s Hans Christian Ørsted, Carl Friedrich Gauss and Michael Faraday had discovered the basic principles of electromagnetic induction. Ørsted and André-Marie Ampère built on these discoveries and uncovered that an electric current produces a magnetic field. As the 1800s progressed, William Sturgeon developed the first rotary electric motor, a design that has all of the essential elements of our modern direct current (DC) motors.

The full scope of how rotary DC motors work is outside the scope of this work, but it essentially involves a positive and negative current, some permanent magnets and an assembly that spins in the center. You can find DC motors on robots, RC cars and other electronics that require something that spins.

How does this relate to your pocket? Most of the time you mount something on a rotary motor you want the object to have evenly distributed weight. If the weight isn’t evenly distributed the rotation will wobble and cause vibration. But sometimes you want vibration. Like when your cell phone rings. Inside your cell phone is a small DC motor with an unbalanced load. When current is applied to the motor the assembly spins and vibrates. A 200 year old invention is the basis for the field of haptic technology.

Haptic technology is a field of research and development that recreates the sensation of touch. Haptics are ubiquitous today, from your cell phone to your Fitbit, that invokes the Pavlovian response when you hit your daily goal of steps. Both of those haptic technologies utilize DC motors to achieve force feedback. But there are a number of other technologies being explored that fit well into Yokoi’s Lateral Thinking with Withered Technology.

In 1965 the Wham-O toy company introduced the Air Blaster toy. The Air Blaster was an example of an air vortex cannon, a pistol shaped object that sent a blast of air that could blow out a candle from 6.1m away. Pulled of the market after some children found dangerously creative uses for it, DIY plans for air vortex cannons have been found as early as the 1970s. The air vortex cannon was reintroduced in 2004 as a toy called the Airzooka, a safer version that is still for sale today. Instructions to build DIY versions of an air vortex cannon are easy to find on the Internet.

This 40 year old technology is now under research as a type of non-contact haptic technology. Using the same basic principles of an air vortex cannon, Disney Research explored a touchless haptic system dubbed AIREAL. AIREAL used a series of audio speaker drivers in a 3D printed cube to generate puffs of air with intensities and velocities. Microsoft Research also looked into air vortex rings, focusing on vortex formation theory. Disney’s use of speaker drivers to create a variable electromechanical mechanism for generating vortex rings is a perfect example of lateral thinking using withered technology.

On the other end of the audio spectrum from infrasound is ultrasound, audio waves that have frequencies above 20,000Hz. In the 1930s researchers developed methods for using ultrasonic to detect flaws in solid objects. These days you can pick up ultrasonic speakers that act as range finders for robots, hyperdirectional speakers that use an array of ultrasonic speakers to direct a narrow beam of sound in one direction and ultrasonic speakers that are being used for haptics.

In 2013 the company Ultrahaptics was founded to create non-contact haptic technology based on research done at the University of Bristol. Ultrahaptics uses ultrasound to create virtual objects, invisible interfaces and other haptic feedback by changing the air pressure of the environment. Paired with VR or AR, this interface provides an extremely convincing experience for users. Ultrahaptics is another example of using a well understood technology in new ways to help make physical-digital experiences more immersive and engaging.
Age of Experience

A Westworld Experience Is Closer Than We Think
Last Saturday, I got into a rather excited argument with my four year-old over what music we were going to listen to. The argument started with, “Alexa, play Stevie Ray Vaughn,” to which my son responded, “Alexa, play the Chainsmokers.” That sequence repeated itself until my four year-old won out, because we ain’t ever getting older. Put Closer on a postmodern player-piano and we can transport ourselves to a saloon in Westworld, an anachronistic phenom of a show that outlines some of the coming ethical and moral questions surrounding artificial intelligence.
 
Westworld exists at an unidentified point in time, but at a time that is likely closer than most of us think. In this world, Hosts, human replicants driven by Artificial Intelligence, are programmed with complex storylines in a theme park where human Guests can come and live out their Wild West fantasies. Although our ability to fabricate Hosts to the level of reality outlined in the show is far off, machine learning and AI is on the cusp of the exponential growth that will make deep conversational interfaces possible. 
 
One of the challenges of creating conversational interfaces is the geometric growth of the decision trees that power the experience. Combining machine learning with novel approaches like Computer Assisted Authoring of Interactive Narratives could provide us with tools that greatly accelerate the rate at which we can develop these interfaces, while dramatically reducing the complexity. This method of collaboration, human/AI symbiosis, is one likely way forward in a world where machines continue to dominate humans in an increasing number of tasks. By amplifying human creativity with machine intelligence, we can create chatbots that provide semi-scripted experiences, so users will have a unique experience with each visit.
 
In Westworld, the Host Maeve has been programmed to read and manipulate human emotions. Machines being able to read and evoke emotions in people may seem like a futuristic concept, but the field of Affective Computing has been around for decades now. Advances in machine learning and image recognition have allowed facial recognition services, like Google and Microsoft, to have emotion detection baked into their systems. Even more impressive is the announcement that we can now detect emotional responses using WiFi signals. While WiFi routers can’t approach Maeve’s cunning, their presence is more ubiquitous. Interlacing conversational experiences with emotional feedback will create more immersive and engaging experiences. And the use of emotion detection can extend to out-of-home displays to create playful experiences with passers-by.
 
As we enter the Age of Experience, it’s critical to understand and leverage the amazing array of tools that AI is providing us. As neural networks dream in order to learn faster, teach themselves how to remember and learn to read lips better than we can, how can we weave these technical marvels into more engaging brand experiences? As experiences become the product, can we use these tools to create nonlinear narratives, replacing traditional digital experiences?
 
Some of the most interesting, and potentially creepy, applications of these technologies will happen when we merge conversational UIs with adaptive learning based on data from vendors like Acxiom and El Toro. Instead of the sledgehammer of retargeting, we can shape a user’s experience with a palette knife, using a combination of AI and segmented behavior analysis. Even better, with the advent of group predictive sentiment analysis, we can custom tailor experiences for users before they know they want them. An objectively formulated, data derived experience, born of machine-driven insights, but tweaked, tuned and tailored by the creativity of humans.
 
Chatbots are but one tool in the oncoming stampede of machine learning technologies that marketers can use to drive engagement and create compelling experiences. The best marketers will embrace this rapid onslaught of change and emerge from the maze (so to speak) with the knowledge of how to use these technologies to amplify simple brand truths. And the truly brave will question the nature of their reality, creating anticipatory experiences instead of relying on the same old safe bet.