Pictures Under Glass, VR, AR, and Haptics
http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/
Certainly a terrifying vision of the future. Black Mirror and the like would have us believe we're doomed to be holding small flat screens in our hands for eternity. I have a lot to say about this.
Reading (and subsequently being completely swayed by) his posts, I began to think about two things:
1. How user interfaces will be designed for the inevitable AR takeover, and
2. How video game controllers fit into this paradigm of interaction.
I'll go in order.
We'll all be wearing AR glasses at some point in the next twenty-thirty years, likely less. We'll have our own heads-up displays, seeing all sorts of fascinating things, either invisible to everyone else or created as a shared experience, some kind of communal hallucination. This could potentially provide the seismic shift required to shake interaction designers away from Pictures Under Glass, and instead figure out how to make use of these hands we've been dumbing down for so long. To see what AR interfaces may eventually be like, it might be smart to look at what VR interfaces are currently like - which, of course, are a mixed bag this early in the game, only a little more than a year out from the initial release of the Rift and the Vive. There's still plenty of flat surfaces with pointer-based interaction - which is dull, especially in a medium with a literally infinite possibility space and entire extra dimension to work with. Sure, there are some applications where a series of buttons on a flat plane is the most efficient way for the user to do certain things, but there are more creative ways. Google's VR software offerings (Tilt Brush, Blocks) have excellent and inventive ways of re-inventing 2-D interfaces for a 3-D space. In Blocks, there is a plane of tools/options attached to one controller, which you can select by pointing at it with the other - but when you flip over the controller, there's a color picker on the back. Selecting/pointing at any option causes it to physically pop out above the others, which are all 3D models regardless - it retains the ease of use of a 2-d screen, but has affordances that make it a much more pleasant experience. In Tilt Brush, the color picker is one of four flat planes which form a cube around your controller, which you spin around your controller by swiping its touchpad. I can easily see interfaces like this ported to AR devices - given the presence of a controller.
Since Bret seems to value tactile interaction, it's worth discussing controllers. More specifically, the area where VR/AR and controllers meet. The original Oculus Rift CV1 shipped with an Xbox One controller, and while I'm in love with the design of that thing in terms of ergonomics and specific haptic responses, it's not optimal for VR- because there's no way of bringing the controller into the game. It affects the world, but in ways you can only understand because you're the one holding it. Most of the games that use it (Chronos, Lucky's Tale) make you an observer, not a participant. You watch the hero do things from some intangible third-person perspective, you've still got one foot in the real world. What I want to focus on are the Oculus Touch controllers, and Valve's Vive controllers. Games will handle them differently - many try to use them to approximate your hand position down to the finger, and try to fool you into thinking that the 3-d models that seem attached to where you feel your controllers to be are, in fact, your hands. Obviously, they aren't. Some of the best games don't try to model hands at all, instead bringing the controllers into the game by themselves. This works really well in games where the controllers are made full use of as tools - Vanishing Realms makes them a sword and shield, Tilt Brush and Blocks make them into surreal (but physically present and logical) artistic devices.
I suppose that's as good a point as any to segue into haptics - what does it feel like to use these tools? For Pictures Under Glass, of course, the answer is obvious. It feels like you're swiping across a pane of glass. But for Vanishing Realms, holding the controller, even though the weight was off, made me feel a bit more like I was really holding a sword, because I saw a sword that moved where I felt like I was moving this handle I could feel in my hand. This is all a very roundabout way of saying that I truly, sincerely hope AR interfaces don't exclusively turn into just swiping and pinching the air. People are doing some very interesting things with LeapMotion sensors, granted, and I eagerly anticipate the day I can put on some glasses and see a full computer interface complete with multiple resizable screens where there was just a blank desk, but I hope that the human race still finds value for physical buttons in the year 2500. Of course, that doesn't mean buttons are Peak Haptics - quite the opposite, in fact. There's plenty of room to innovate. The main challenge that people seem to be facing right now, and are working through as we speak, is how to make virtual objects feel tangible, especially now that we're removing the glass barrier between our world and the virtual world. I'll take a few examples of my favorite haptic interactions from video games that I think interaction designers should look at - which will, admittedly, mostly be use cases for rumble.
Example A: the Xbox One controller's RT and LT triggers.
Rumble has been ubiquitous since the Nintendo 64 introduced the Rumble Pak, of course, but this specific kind of trigger rumble has more specifically satisfying responses to specific actions. There are rumble motors dedicated to pushing back on the trigger against the user's finger - it's not quite as violent, as, say, racing sim wheels jerking around to simulate the difficulty of controlling a motor vehicle at 120 MPH, but it's enough of a physically stimulating action to feel inherently satisfying. Accelerating sharply or braking with enough force in a Forza game causes the appropriate triggers to rumble. Firing your weapon in Gears 4 or Halo 5 with the right trigger gives you a kick in the same. These make virtual actions more tangible, and reduces the disconnect between player and game. There's a physiological satisfaction to driving in Forza Horizon 3 that few other controller-based driving games can match.
Example B: the Nintendo Switch's HD Rumble tech.
When Nintendo fully detailed the features of the Switch in their livestreamed press conference more than half a year ago, one of the first things they talked about was HD Rumble. They claimed it was a rumble function so precise, you could feel three ice cubes falling into a glass, and rattling around individually as you shook the controller. Currently, implementations are slim, but there are a few notable ones. 1-2 Switch, the requisite gimmick-ridden party game that ought to have been bundled with the system but instead sold for full price, has a minigame which asks players to guess how many marbles are inside their controllers. Players have to tilt their controllers around, feeling the HD Rumble emulate the sensation of ball bearings rolling around inside a box, and guess the number. I haven't personally tried it, but it's allegedly convincing. HD Rumble is also used well for smaller details - in Splatoon 2, smacking onto the ground while in squid form makes the controllers "thump" quickly downwards, gently replicating a jarring impact onto a hard surface. When I played the Japanese demo of Puyo Puyo Tetris, quick-dropping a Tetromino would give a similar "thump". Graceful Explosion Machine, an arcade shooter, creates different physical rumble sensations depending on what weapon is being fired, and what enemy is currently being struck by said weapon. All of this gives an added physical character to the onscreen actions besides the satisfying click of a button press.
Example C: Valve's Haptic Touchpads
Both of the Vive remotes have a haptic touchpad of sorts - a concave circle which knows exactly where it's being touched, can be pressed inward, and can rumble. The touchpad can vibrate in a specific way which, while a finger is moving on it, makes it feel like it's going over a bump of sorts. Plenty of VR games use this circular touchpad for a radial menu of sorts, and the best of those radial menus give a quick pulse when moving from option to option. It's a small detail, but it heavily contributes to immersion, and makes you feel like the menu is that much more real. In The Lab, Valve's assortment of tech demos, one of the most popular is an archery game, which gives you a castle to defend against a series of invaders, your only tool being a bow and arrow. You have to draw arrows from an invisible quiver on your back, then nock the arrow itself into the bow, then pull the string back while holding the trigger, then fire by letting go. Here, the actual act of drawing back the bow is physically tangible - the controllers rumble distinctly each time you pull the bow back a little bit more, and increase in intensity the further back you go. It's not only immersive, but it's physically satisfying to do, and without the feedback, the game would be significantly more difficult - you'd have to pay more attention to the bow itself, to make sure everything is being done correctly and that you're pulling it back far enough. But with the rumble, you know what you're doing without even looking at the bow. It becomes an automatic physical process.
In my perfect world, near-future AR glasses would ship with some kind of haptic remote or device that allows not only for physical interaction with virtual objects, but some kind of tangibility. Even providing some slight physical resistance to a user's actions when grabbing an object could go miles.
I could write even more, but I'd probably devolve into ranting.
Certainly a terrifying vision of the future. Black Mirror and the like would have us believe we're doomed to be holding small flat screens in our hands for eternity. I have a lot to say about this.
Reading (and subsequently being completely swayed by) his posts, I began to think about two things:
1. How user interfaces will be designed for the inevitable AR takeover, and
2. How video game controllers fit into this paradigm of interaction.
I'll go in order.
We'll all be wearing AR glasses at some point in the next twenty-thirty years, likely less. We'll have our own heads-up displays, seeing all sorts of fascinating things, either invisible to everyone else or created as a shared experience, some kind of communal hallucination. This could potentially provide the seismic shift required to shake interaction designers away from Pictures Under Glass, and instead figure out how to make use of these hands we've been dumbing down for so long. To see what AR interfaces may eventually be like, it might be smart to look at what VR interfaces are currently like - which, of course, are a mixed bag this early in the game, only a little more than a year out from the initial release of the Rift and the Vive. There's still plenty of flat surfaces with pointer-based interaction - which is dull, especially in a medium with a literally infinite possibility space and entire extra dimension to work with. Sure, there are some applications where a series of buttons on a flat plane is the most efficient way for the user to do certain things, but there are more creative ways. Google's VR software offerings (Tilt Brush, Blocks) have excellent and inventive ways of re-inventing 2-D interfaces for a 3-D space. In Blocks, there is a plane of tools/options attached to one controller, which you can select by pointing at it with the other - but when you flip over the controller, there's a color picker on the back. Selecting/pointing at any option causes it to physically pop out above the others, which are all 3D models regardless - it retains the ease of use of a 2-d screen, but has affordances that make it a much more pleasant experience. In Tilt Brush, the color picker is one of four flat planes which form a cube around your controller, which you spin around your controller by swiping its touchpad. I can easily see interfaces like this ported to AR devices - given the presence of a controller.
Since Bret seems to value tactile interaction, it's worth discussing controllers. More specifically, the area where VR/AR and controllers meet. The original Oculus Rift CV1 shipped with an Xbox One controller, and while I'm in love with the design of that thing in terms of ergonomics and specific haptic responses, it's not optimal for VR- because there's no way of bringing the controller into the game. It affects the world, but in ways you can only understand because you're the one holding it. Most of the games that use it (Chronos, Lucky's Tale) make you an observer, not a participant. You watch the hero do things from some intangible third-person perspective, you've still got one foot in the real world. What I want to focus on are the Oculus Touch controllers, and Valve's Vive controllers. Games will handle them differently - many try to use them to approximate your hand position down to the finger, and try to fool you into thinking that the 3-d models that seem attached to where you feel your controllers to be are, in fact, your hands. Obviously, they aren't. Some of the best games don't try to model hands at all, instead bringing the controllers into the game by themselves. This works really well in games where the controllers are made full use of as tools - Vanishing Realms makes them a sword and shield, Tilt Brush and Blocks make them into surreal (but physically present and logical) artistic devices.
I suppose that's as good a point as any to segue into haptics - what does it feel like to use these tools? For Pictures Under Glass, of course, the answer is obvious. It feels like you're swiping across a pane of glass. But for Vanishing Realms, holding the controller, even though the weight was off, made me feel a bit more like I was really holding a sword, because I saw a sword that moved where I felt like I was moving this handle I could feel in my hand. This is all a very roundabout way of saying that I truly, sincerely hope AR interfaces don't exclusively turn into just swiping and pinching the air. People are doing some very interesting things with LeapMotion sensors, granted, and I eagerly anticipate the day I can put on some glasses and see a full computer interface complete with multiple resizable screens where there was just a blank desk, but I hope that the human race still finds value for physical buttons in the year 2500. Of course, that doesn't mean buttons are Peak Haptics - quite the opposite, in fact. There's plenty of room to innovate. The main challenge that people seem to be facing right now, and are working through as we speak, is how to make virtual objects feel tangible, especially now that we're removing the glass barrier between our world and the virtual world. I'll take a few examples of my favorite haptic interactions from video games that I think interaction designers should look at - which will, admittedly, mostly be use cases for rumble.
Example A: the Xbox One controller's RT and LT triggers.
Rumble has been ubiquitous since the Nintendo 64 introduced the Rumble Pak, of course, but this specific kind of trigger rumble has more specifically satisfying responses to specific actions. There are rumble motors dedicated to pushing back on the trigger against the user's finger - it's not quite as violent, as, say, racing sim wheels jerking around to simulate the difficulty of controlling a motor vehicle at 120 MPH, but it's enough of a physically stimulating action to feel inherently satisfying. Accelerating sharply or braking with enough force in a Forza game causes the appropriate triggers to rumble. Firing your weapon in Gears 4 or Halo 5 with the right trigger gives you a kick in the same. These make virtual actions more tangible, and reduces the disconnect between player and game. There's a physiological satisfaction to driving in Forza Horizon 3 that few other controller-based driving games can match.
Example B: the Nintendo Switch's HD Rumble tech.
When Nintendo fully detailed the features of the Switch in their livestreamed press conference more than half a year ago, one of the first things they talked about was HD Rumble. They claimed it was a rumble function so precise, you could feel three ice cubes falling into a glass, and rattling around individually as you shook the controller. Currently, implementations are slim, but there are a few notable ones. 1-2 Switch, the requisite gimmick-ridden party game that ought to have been bundled with the system but instead sold for full price, has a minigame which asks players to guess how many marbles are inside their controllers. Players have to tilt their controllers around, feeling the HD Rumble emulate the sensation of ball bearings rolling around inside a box, and guess the number. I haven't personally tried it, but it's allegedly convincing. HD Rumble is also used well for smaller details - in Splatoon 2, smacking onto the ground while in squid form makes the controllers "thump" quickly downwards, gently replicating a jarring impact onto a hard surface. When I played the Japanese demo of Puyo Puyo Tetris, quick-dropping a Tetromino would give a similar "thump". Graceful Explosion Machine, an arcade shooter, creates different physical rumble sensations depending on what weapon is being fired, and what enemy is currently being struck by said weapon. All of this gives an added physical character to the onscreen actions besides the satisfying click of a button press.
Example C: Valve's Haptic Touchpads
Both of the Vive remotes have a haptic touchpad of sorts - a concave circle which knows exactly where it's being touched, can be pressed inward, and can rumble. The touchpad can vibrate in a specific way which, while a finger is moving on it, makes it feel like it's going over a bump of sorts. Plenty of VR games use this circular touchpad for a radial menu of sorts, and the best of those radial menus give a quick pulse when moving from option to option. It's a small detail, but it heavily contributes to immersion, and makes you feel like the menu is that much more real. In The Lab, Valve's assortment of tech demos, one of the most popular is an archery game, which gives you a castle to defend against a series of invaders, your only tool being a bow and arrow. You have to draw arrows from an invisible quiver on your back, then nock the arrow itself into the bow, then pull the string back while holding the trigger, then fire by letting go. Here, the actual act of drawing back the bow is physically tangible - the controllers rumble distinctly each time you pull the bow back a little bit more, and increase in intensity the further back you go. It's not only immersive, but it's physically satisfying to do, and without the feedback, the game would be significantly more difficult - you'd have to pay more attention to the bow itself, to make sure everything is being done correctly and that you're pulling it back far enough. But with the rumble, you know what you're doing without even looking at the bow. It becomes an automatic physical process.
In my perfect world, near-future AR glasses would ship with some kind of haptic remote or device that allows not only for physical interaction with virtual objects, but some kind of tangibility. Even providing some slight physical resistance to a user's actions when grabbing an object could go miles.
I could write even more, but I'd probably devolve into ranting.
Comments
Post a Comment