Vineel Shah is a software engineer, writer, and speaker with three decades of imagining, experimenting, building, and living a tech life. He has worked for startups like Red Rover Labs and Salido, and Fortune 500 companies like Yahoo and MTV/Viacom. A long time ago, he wrote Lingo and Shockwave Sourcebook, about an early interactive media authoring system called Director. He hails from the early days of New York University's Media Research Lab under Professor Ken Perlin. He is currently transforming the way the world's top professionals share expertise and learn at GLG.
In the war for America’s spending dollar, brick and mortar stores fight for survival with e-commerce. Brick and Mortar stores are losing — in fact they’re doing even worse than you might guess.
Part of this loss is natural. As more goods and services become digital, there’s no need for a physical location to purchase them. Also for the purchase of many goods, the convenience of shopping at home or on the go just can’t be beat.
However, brick and mortar is failing as badly as it is because of a pathetic, industry-wide lack of imagination, investment, and courage.
In the face of competing with Amazon, one of the most innovative companies in the world, most Brick and Mortar stores have stood still. Some have piloted 1 incremental idea at a time — and watched it fail — and thrown up their hands in despair while sitting down in a deck chair on the Titanic.
Seriously, how hard should it be for a shopping experience crafted by multimillion dollar companies in large, fully controlled spaces to compete with the shopping experience on a 5-inch touch screen? These phone-apps usually lack sound, smell, taste, excitement, group interaction, and more. This contest should be Disney World vs the Disney website. Instead it’s like Warehouse vs Warehouse Catalog.
There is nothing sacred about the standard design of a Mall or Department Store.
The design of the modern Department Store dates back to the mid 1800s and a store called Le Bon Marche. The store was “an entire wonderland of clothes, fabrics, furniture, trinkets, jewelry, and countless other goods — all collected together in one extravagant space.” It was an echo of the “seductive power of sensory overload” as demonstrated by the Worlds Fair of 1855. The idea was to invite shoppers in with fancy windows and trap them by confusing their senses — and sell them things they didn’t need as they wandered around in a daze.
The “Modern Mall” was designed in the 1950s by an avant-garde European socialist named Victor Gruen. He was attempting to recreate the vibrancy of a small European village in weather-protected, managed space. He had planned a beautiful town around the mall — instead the developers surrounded the market space with a parking lot. The corrupted design did spectacularly well — it suited the lifestyles of the 1950s suburbanites. Access to all the same goods as their city counterparts, some excitement and social interaction during the cold winters and hot summers, and a place to spend lots of time. (For more, read Wonderland by Steven Johnson. http://amzn.to/2jknWR8 )
Neither the Department Store or the Modern Mall address the needs and desires of modern shoppers. Modern shoppers do not react well to being treated as a mass market, ripe to being fooled by shiny things. Online competitors have shown us that we deserve more — personalized experiences, which are time-efficient and fun. The Mall/Store should be doing all the hard work, while we are treated like royalty. Satisfying the modern shopper requires re-inventing the shopping experience, nuts-to-bolts.
Below is an example of what we think is possible — using technologies that are available in beta or production today (January 2017.) We think this is the very least that must happen to save modern retail.
Shopping at the Mall in the Age of Experience
Priya is a typical suburban mom in a typical large suburb.
She has an automated setup with Amazon.com and various other websites that purchases her consumables when they are needed and delivers them to her door. She goes shopping for things she finds more fun — clothes, gifts, technology, and the like. Today she needs a new dress for a party that she’s attending and new jeans for her son.
Priya’s self-driving SUV pulls up to the Mall entrance. Her phone registers her presence with the Mall, which accesses her account. The Mall knows many facts about her and her family members:
Family Members: She, her husband, her 6 year old son, her 4 year old daughter
interests and hobbies
The Mall assigns the car a parking spot and it goes and parks itself there. She walks into the Mall with her kids.
She knows that her son had a recent growth spurt, so she first stops in the fitting booth. He is 3D scanned and his new measurements are stored with the Mall and sent to Priya by email.
She exits the booth and whispers into her phone, requesting a Children’s Concierge. A young woman soon pulls up in a small electric vehicle. She greets Priya and the children by name, and wristbands each child. Priya can now track her kids wherever they are in the mall, and the kids can call her or pay for a snack. After kissing them both, the concierge whisks the kids off to the child center. The Child Center is a Mall facility which is part playground, part video game arcade, and part heathy snack bar. Every Mall has one.
Now Priya is free to shop without having to worry about her kids going crazy with boredom, and driving her crazy in response.
Priya has already been to the gym today, so she doesn’t feel the need to walk for exercise. She grabs a Segway from the stand, and activates it using her phone. Her vehicle session is registered to her account, and she’ll be charged a small rental fee unless she purchases something during her time at the Mall. Directed by the vehicle, she rolls up a ramp, through a couple of corridors, and emerges into the women’s formalwear department of Macy’s.
The store is not arranged as the Macy’s of today. Gone are the racks of the same model of garment in different sizes. Each type of garment is shown on its own, or in an ensemble, or on a manikin. Each display is beautifully laid out, showing off the product in unique ways. The store is constantly trying new layouts and sensors track how many customers visit the displays, how long they browse, how happy they seem, and how much they purchase.
Priya parks the vehicle and walks through the displays, enjoying the clothes. She finds a dress that she likes. She tells the display the she likes it, and it indicates that there is a dress in her size. She says that she will try it on. She wanders around and finds 3 more dresses that she likes. Finally she glances at her phone, which shows pictures of the 4 dresses she has on her list. She tells the phone “Let’s try them on!” The phone displays a map to the nearest dressing room, which is not too far away.
She walks over to the dressing room, and it greets her by name. Lights on the floor guide her to dressing room 4, which has her dresses in them. On the wall is a large video-mirror. Priya tries the dresses on, taking a few pictures by asking the wall to “take a photo” at various times. Each one is stamped with Priya’s account, the time of day, and the model and size of the dress. She even takes some video of her favorite dress. She tells the dressing room that she’ll take her favorite, but she’s not sure if the others are worth buying. She sends the whole package of photos and videos to a couple of girlfriends for second and third opinions, then she leaves Macy’s on her Segway and rolls over to a smaller store called “The Children’s Place.”
Her phone asks her who she is shopping for, and she answers with her sons name. Now the Children’s Place knows the measurements and gender of the target, and it reacts the same way that Macy’s did. Some of the clothes will be a little long for him — and the clothes displays show her that by fitting the jeans to a 3D model of him, from his scan when they walked in. He’s growing quickly, so she decides to buy them anyway. She buys 3 pairs of jeans in different styles.
She calls her kids on their bands — they’re having fun and don’t want to be disturbed. She smiles and rides her Segway over to the food court, to grab a Froyo and just enjoys the solo time.
Finally she rolls over to the Child Center to pick up the children. She could have asked a concierge to bring them to the exit, but her kids typically hate leaving the center, and can get surly. She knows she’s better off being there to pull them away from the fun.
She leaves her Segway (which drives itself back to its stand) and is driven back to the exit by another electric car. The phone has ordered the SUV to come meet them, and has delivered her purchases to the exit. All arrive at the same time, and a robot concierge loads the purchases into the back of the car. As the SUV drives them home, Priya gets responses from her friends about the dresses. She’s convinced that the green one looks good on her, so she makes the purchase on her phone. An hour later, the dress is delivered to her house by drone.
We already know that, just because we read that “The President said blah blah blah” — that the President might not have said that. It’s easy to fake text, all it takes is a keyboard and a fibber with an agenda. BAM! Fake News, in your face. Nobody automatically trusts that text is evidence of anything.
Photography, once worth a thousand words, is now worth nothing, when it comes to evidence. Pyramids are rearranged, women become inhumanly thin and long, and people’s faces often appear atop the wrong bodies. As John Stewart once said, “Yeah, we have Photoshop.” Bzzzz — fake news!
We have however, as a society, come to trust video. If we can see it happen, we believe that it happened. Sure, there’s a little blurring around the edges — the camera angle might hide some of the action, the quality might obscure some detail. But in general, seeing is believing,
That trust is about to die.
So far, creating convincing fake video is difficult, expensive, and slow. It takes a Hollywood specifical-effects house and millions of dollars to make a scene that looks, feels, and sounds real. As humans, we’re good at reading human facial expressions — we have low tolerance for RobotFace. We’re good at recognizing voices — we know an impostor when we hear one. When we watch an approximation of a real-life scene in a movie, we marvel when it gets close.
In a year or two, creating fake video is will become quick, cheap, and easy. How?
Here’s one way — Adobe is working on a project called VoCo. It can take 20 minutes of an audio track of a person speaking, and create a voice model based on it. Then you can type in arbitrary text, and hear the person speak your text. In a believable fashion. It sounds good — and will undoubtedly get much better before it’s near-future launch.
Here’s another — a group of university researchers have demonstrated a system that maps your face — from a webcam — to another face. Any facial expression, tick, or emotion that crosses your face is reflected on the other. In this demonstration of Face2Face, they show an everyday joe driving The Donald’s face with his own. And it looks really, really good.
These technologies will be finding their way into consumer products, for fun, gaming, ebook-reading, live-mediated acting, and thousands of other applications. As humans, we’ve always enjoyed mimicry and identity-bending, and these technologies will be all the rage at a Christmas not too far from now.
But what does this mean for recordings of news? The next time you see one man killing another in a video, will you believe what you’re seeing? If you see a politician make a speech and say something controversial, will you wonder if it’s her, or will you assume that it is some kid with Adobe’s Creative Cloud?
As a society, we’ve developed ways of dealing with easy-to-fake evidence. We’ve evolved “authority” — voices that we trust like the Police, News organizations, the Government. At least — our parents used to trust them. We’ve been raised to question everything we’re told, whether it’s news about the President or science about the weather. Many people find it easier to believe completely falsified “news” if it confirms their existing biases and worldview, regardless of the truth. We’ve left “Truth” behind and become a “Truthy” culture, in which nobody can convince anybody of the truth of anything.
Even our major tool in convincing others — Evidence — is about to become “Evidency.”
The soul of Steve Jobs is being channeled by Jony Ives, Tim Cook, and others. Steve’s ghost is killing the company.
A living Steve often changed his mind. He was against local area networking, until he was for it. He was for Adobe, until he was against it. He was against Microsoft, before he was fine with it.
Apple has enough great people and great potential– it’s just stuck in a rut. It’s DNA has picked up some evolutionary dead ends that must be snipped before Apple can continue on to new heights. We need a business-level CRISPR.
Obsolete Idea 1: Tablets And Computers Are Different Things
Somehow, Apple has decided that people don’t want real computers that have screens that they can touch with fingers or stylus or other form. Instead, it has focused on making ever larger touchpads. If it can drop this idea, it can catch up to Microsoft with it’s Surface line, 2 in 1s, take advantage of Leap Motion and other innovations. Once Apple starts moving on, who knows where they could go?
A long time ago, it made sense for Apple’s phones and tablets to have a simpler, less resource-hungry operating system — thus, the split between iOS and OS X. Today, though, we can fit a whole supercomputer into a phone or tablet — Microsoft has proven this and paved the way. I believe that there is a huge, pent-up demand for Mac 2-in-1s, pen-based Macs, etc. Some have reluctantly switched to Windows to get this form factor — but many are still waiting.
Obsolete Idea 2: Battery Life Is Better Than Functionality
If what you’re holding is a real computer — sometimes you ask it to do hard work, even in the background. Hard work sometimes wears down the battery quickly — which has become anathema to Apple’s SDKs. App developers beg the OS for scraps of time to do specific tasks that Apple allows — ensuring that iOS stays out of hard-core applications. Gamers know that great power comes at the cost of… well… power. The rest of us can deal with that — stop curtailing the most powerful SDKs in the world to match this silly constraint!
Obsolete Idea 3: Thinner is Better
There’s an old saying that “You can’t be too rich or too thin, and if you don’t think so, you’ve never been poor or fat.” It’s nice to have thinner and lighter gear — but ENOUGH ALREADY! No super-thin iPhone is complete without a case that bulks it up gives it a chance to survive real life. I’d so much rather have USB and headphone ports and thicker glass and serious waterproofing in a slightly thicker body than a stupid lightning port in a thin body.
Obsolete Idea 4: GUIs Are 2D
In the late seventies and early eighties, researchers at SRI and Xerox Parc (and at Apple, kind of) invented the graphical user interface. It was raster/pixel-based, with Windows, icons, menus, and a mouse. Some prototypes even had touch and pen screens. Fast-forward to that late twenty-teens, and GUIs are basically the same. Sure there’s more color, more pixels, and we’re starting to experiment with voice, but not much has changed.
Heads up — the third dimension is coming to town. Virtual Reality, Augmented Reality, and 3D printed everything is here and growing. The third dimension is poking at us, and we’re almost ready for it. It awaits that final spark — somebody to visualize it so well and implement it well enough that 3D user experience will explode in a mainstream way. That’s Apple’s traditional market entrance. But the world is getting rounder, it’s a plane with a new vector, and Apple is stuck in Flatland.
Obsolete Idea 5: Media is Linear
It seems hard to believe now, but at one point in history, Apple was a pioneer in creating new forms of media. Everybody who became anybody in the 90s, in the fields of gaming, non-linear storytelling, hyper/linked media, had worked for Apple. Back in the day, it looked like there would be a new world of new types of media, with Apple technologies leading the charge. Instead, we got convergence — linear video, linear audio, books, and news all playing through the same device. Much more convenient, more pervasive, and more scalable than what we had before, but certainly not revolutionary in form. There are new methods of telling stories to be birthed — new media industries to be enabled — and Apple is still uniquely positioned to research and invent our way there.
What could Apple be if it threw off the chains of its malaise? It could contribute significantly to an improved world. A really usable 3D user interface, with integrated voice, gesture, language, GPS, data-mining, and situation-specific domain knowledge could unlock a new level of empowerment for all of us — and Apple could double or triple in size, supplying the tools and the delivery mechanisms for the Age of Experience. The world is willing to give Apple it’s crazy margins, it’s haughty attitudes, it’s closed ecosystems. Just, please, make it worthwhile again!
For Making Realities’ inaugural post, we thought a quick review of the history of Virtual Reality was in order. We always like to begin at the beginning.
NASA Ames and Inspiration
Although the idea of computational realties had been explored for a long time, most notably in the field of Flight Simulation, the first glimmer of “VR” tech is usually thought to have been born at the NASA Ames Research Lab at Moffet Field, CA in the 1980s.
This research project had all the basic parts of a modern VR system — a graphical, stereoscopic, head-tracking, head-mounted display, and spatially tracked hand sensors. The graphics were primitive by today’s standards, but the whole shebang was there in 1980s form.
Science Fact Becomes Science Fiction
Perhaps the most important legacy of VIEW was that William Gibson got to see it. It inspired him to imagine a future that combined a Virtual Reality interface with a vast network of computers that connected society (today we’d call it the “internet”) — and he coined the combination cyberspace in his 1984 cyberpunk novel, Neuromancer. The cyberpunk movement invented many versions of cyberspace — notably in True Names (1981) by Vernor Vinge and as the metaverse in Neal Stephenson’s Snow Crash (1992.)
Science Fiction Becomes Crappy Products (VR’s First Wave)
A generation of technologists grew up yearning for cyberspace and the metaverse — which resulted in the first great wave of Virtual Reality in the early 1990s. A company called VPL Research was created to bring VR to the masses, by a very weird guy named Jaron Lanier. (VPL stood for “Visual Programming Language.”) VPL’s major “products” were the Data Glove, based on work by Thomas G. Zimmerman when he worked at NASA Ames, and the EyePhone, the Head Mounted Display.
VPL tried popularizing the spacy/artsy/trippy side of the Virtual Reality experience. Lanier was trying to promote enlightenment through avatars — something he called “homuncular flexibility”. While there were more serious projects at VPL Research (like this VR brain explorer,) the high price, research-oriented system, and lack of the “killer app” drove the company to bankruptcy in 1990. In 1999, Sun Microsystems bought all their patents, which was in turn bought by Oracle in 2010.
Another company tried a more proven application genre for VR — arcade games. A company called Virtuality tried launching VR arcade games from 1991-1994. The machines were expensive and the games were primitive and not very well designed for the arcade playing environment. Virtuality, too, went bust.
Even Nintendo dipped its toe into VR during this period with the Nintendo Power Glove. It was a 3-D spatially-tracked glove that was supposed to let you control the 2D 16-bit sprites on the day’s gaming consoles. About as much fun as it sounds.
During this period, there was also a druggy/trippy aspect to the hopes for this tech. None other than the famous Timothy Leary, a Harvard professor famous for exploring the LSD and philosophy in the 1960s, was fascinated by the possibilities of VR. He sought to reach beyond the “Absolute Reality” of the “Real World” and access higher levels of consciousness through computational simulation, instead of direct chemical stimulation. To be honest, today’s VR is much better set up to accomplish this than the tech of his time. Unfortunately, his trip ended in 1996 at the age of 75.
The Rise of 3D Is Doomed
Even while the first wave was crashing, the seeds of VR’s second wave were being planted in the early 1990s in Austin, TX, at the offices of a small game company called Id. John Carmack, Adrian Carmack (no relation) and John Romero were pioneering a new genre of gaming, called the first-person shooter — as embodied by the game series Castle Wolfenstein, Doom, and Quake. Over the next decade, these games (and hundreds more in the genre) became so popular that they drove the evolution of 3D hardware and software at the consumer level, in consoles and desktop computers (and eventually phones and tablets.) 3D Gaming became such a big business that 3D gaming engines became licensable, some were even free and opensource. 3D tools are everywhere, game programming has become more like authoring than hard-core technology development. As of the early 2010’s, the consumer expects great 3D experiences delivered by an army of 3D artists and engineers.
In short, the hardware and software got better, and cheaper. And the dreams remained.
VR’s Second Wave
On August 1, 2012, a man named Palmer Lucky launched a kickstart campaign to bring a new VR Head Mounted Display to market that would be good enough to play 3D games and cheap enough to be bought by gamers. In 2012, Facebook bought Oculus Rift for $2 billion. In 2013, John Carmack joined Oculus Rift as its CTO.
Around the same time, a huge gaming company built on 3D shooters called Valve hooked up with a huge contract manufacturer called HTC and decided to launch a VR Gaming platform of their own. A little electronics and gaming company called Sony has launched yet another VR platform. A newly revitalized Microsoft is launching a related product called HoloLens. And the giant Google launched a $10 VR headset spec called Google Cardboard.
It’s hard to overstate the difference between the 1990s and today. In the 1990s, there was a few tens of millions of dollars pushing VR with slow hardware and custom-made software, with a public that was mostly meh about the whole enterprise. Today, there are billions of dollars pushing VR from several proven, successful, and brilliant companies pushing somewhat-affordable hardware and mature industry-supported software. There is a ready audience in the hungry hard-core gaming set, but there is also a wider audience that has become accustomed to innovative experiences around the $1k price point, especially in 3D and Mobile.