Learning in Stereo, with Silicon Harlem's Ben Erwin PART 1

Volumetric video and other spatial 3D technologies allows us to add a Z axis to the way we learn. Director of Silicon Harlem Ben Erwin is on the show to discuss teaching in the same three dimensions we all live in.

Julie: Hello, my name is Julie Smithson and I am your XR for Learning podcast host. Ben Erwin is an expert of the XR ecosystem, including hardware and software platforms and events in the rapidly evolving industry. Recognizing the revolution of how we consume information being brought about by spatial computing and volumetric video, Ben is focused on creating new experiences in WebXR and social VR. As director of Silicon Harlem's annual tech enabled community conference since 2018, Ben programs the agenda, develops the marketing, and produces the events. The next one which will be held at the forum at Columbia University on October 16th, 2020. Welcome, Ben. Thanks very much for joining me today.

Ben: Thank you for having me, Julie. It's an honor to be here. I appreciate being on your show.

Julie: Great. Well, I think you and I have had a couple of different conversations that we want to touch base with, and starting with spatial memory. And we're in a time where many of us are developing and working and moving to online and virtual worlds. And I've always said -- and I think people are starting to realize -- that we live in a three dimensional world. Why don't we learn inside of it? And this is where spatial computing comes into play. But even deeper from that is how we respond as a human and how we intake this information and what we do with it. So maybe you can do a little bit of intro to yourself and how you got into knowing more about spatial memory and the applications around it.

Ben: Thank you. I started my business in the late 90s, as the Internet -- as Web 1.0 -- became a thing. And I've been developing all the way through the rise of social and the rise of mobile. And so this new evolution that we're seeing right now -- some call it Web 3, some call it the Fourth Industrial Revolution -- where we're adding 3D to-- as a medium. And it's more than just adding a Z-axis to content. The fact that we're inside of an experience, when you put on a VR headset and you are inside of a room and you have other people there that are also avatars like you, the impression that that makes on your mind is a remarkable difference to any other type of experience. And though 2D communications platforms -- the video conferencing, and all of the ways that we're used to doing it -- as high quality as it is, even the lower fidelity early stage media that we're at inside of social VR -- and just in general -- the medium of 3D has added so much to the way that we perceive it, that I'm very excited about how that is going to be part of our learning process, because when we see things and we are in an area, it imprints our minds differently than it would when we're just in 2D space.

Julie: Just to give our listeners a little bit more premise, "spatial" obviously means the things that are around you. And that's not just objects, that's your environment and also sound as well. So maybe you can talk a little bit more about your memory and how you take these things in, and you remember them more. That's the big change now from learning things in two dimensions and now learning about them and being a part of them in that three dimensional world.

Ben: Part of it -- as you introduced there -- is that it's the room around you, it's the objects in the room. And very importantly, it's sound. And part of spatial audio is the perception of being close to something or what side it's on. And it's more than just a left-right traditional stereo audio effect that we have. It's a positional audio. But when we are sitting down or standing inside of a VR experience, there are several different components to that. There is a horizon. There's a-- it's called a "skybox" or the wall that surrounds the wrapper -- if you will -- of that environment. And then all of the 3D objects in the environment, the way that you relate to things when you see things in stereoscopic 3D, it has a much higher impact. And a great example is the Traveling While Black stereoscopic 360 video experience that is available on Oculus Quest. It was a documentary that came out for the 2019 Sundance Film Festival, right after the Green Book movie had been nominated for an Oscar. And this was the story of the real Green Book. And what really I found incredibly powerful was how the presence of the people that were surrounding you. You're sitting in a booth in a diner and you're listening to Tamir Rice's mother explain what happened the night he was killed. And there are people sitting on the side and the interviewer sitting right next to you. And you're there. And if you were just looking at that through traditional monocular video, then it would be a pretty traditional experience. But when you are in stereo and-- I think that the most important thing about what makes a difference is how relative to your own size it is. So you're looking at a person and they look proportional to your size, they don't look huge, they don't look small. They look like they're right there in front of you. It affects you emotionally and it embeds the experience in you, so that you have a better retention of what is going on. But it's more than just a stereoscopic 3D experience, because that's what they call three-degrees-of-freedom, where you can pitch, yaw and, roll your head. You can turn your head and everything sort of stays in balance, but you can't move within it. Now, in a spatial experience where you have six degrees of freedom, you also add up, down, left, right, and back and forth. And that is really powerful, because you can change your perspective of the room. You are the camera. And so that freedom of choice, to be able to observe something -- and you and I had been part of the Educators In VR a couple of weeks ago -- and this was in an avatar, but you can move around the room, and you're listening and you're watching a presentation on the board, and it's just like you are there. Even if you're a cartoon avatar, it still feels just like you're there.

Julie: So with all of these experiences in this, the immersion of basically touching on all of the senses, you're hearing things, you're seeing things, you're feeling things. This probably is a great segue way into a term that you and I started talking about more at the Educators In VR -- and I presented it as well in my talk -- was more about the humanics, about humans in the way that they interact with the technology, and how they continue to feel and be socially responsible to balance all of these things, as well as the analytics and the data analytics that come with engaging with all of these technologies. Touch a little bit more on that subject in. How does the mind interact with being immersed so much, being in a three dimensional experience?

Ben: One of my favorite personal analogies to how that works is when you've never been to a place, you imagine what that place is like and you can see pictures of that place and more often than not, those pictures are in 2D. And you may have been on a plane before, so you know what it's like to fly and you can travel there. So you can sort of imagine that. But when you actually do it and you get on a plane and you travel and you spend some hours and you get off and you get on a train or a car and you go to a place and you go around, it's like-- a perfect example was a buddy and I, when we were in our early 20s, had a pact to go to Stonehenge. Always wanted to go to Stonehenge. It was a bucket list item from early in my life. And so we went to Stonehenge and this was a long time ago now, but I can still spatially re-imagine that entire trip, including being there at Stonehenge and seeing the rocks and moving around them and seeing the landscape, and several years after that time I went back and visited again, and they had built it out more of a tourist center. And so the spatial memory of my first trip to Stonehenge, where it wasn't all fenced off and it wasn't turned into a tourist trap. Then when I went back and it had a visitor center and a larger parking lot and everything, that is now-- I can-- if I were to go back to Stonehenge in my mind, that whole additional aspect of moving through it is now part of that memory. So it's not just a photograph to me. It's a part of my psyche now, because I spatially moved around Stonehenge.

Julie: The way that the human mind works now is obviously really interesting. And I guess that's what you call a classical humanics. And you and I talked about the Robot-Proof book by Joseph Aoun. And learning a little bit more about humanics and the model of includes two dimensions, the content portion of it, and then the three literacies. But also about the cognitive capacity, of the way that you interpret your presence. And maybe talking a little bit about the cognitive capacities of what we experience in these three dimensional experiences now in virtual worlds. What kind of impact they have on us and how we interpret them.

Ben: With cognitive capacities. I felt that Aoun's four cognitive capacities reminded me of the way that STEM is very left brain heavy with science, technology, engineering and mathematics, and that it's-- at some point, somebody added an A to STEM and made it STEAM. And there is art. And so I find that art, as much as science, is a good representation of the balance of the human mind. Between art and your creative side on one hand -- which is emphasized as part of our humanics as a need -- as well as-- there are several different levels to science that are represented in the T-E-M of STEM. And so music and physical performance and all of these are very much a part of art as technology, engineering, and math is part of science. And so I think that keeping that in balance is very important. And when people earn degrees, they earn a Bachelor of Science or they earn a Master of Arts, or a Master of Fine Arts. But it's-- if you say the Academy of Motion Picture Arts and Sciences, it's-- that is saying, "OK, here is a medium of art that includes arts and sciences" and that's sort of representative of that wholeness of the balance of the mind. And that's one of the things that really attracted me to the Aoun's concept of humanics as well as the humanities, where those are the liberal arts. And I think that the holistic thinking as part of education is very important, to be able to keep those parts of ourself in balance so that we can express ourself creatively. But all art underneath it, if you break it down, you'll find science and math, which allows you to make better art. So it's a full-- it's a continuum. And if we can feed that continuum as part of our education system, then we will do a better job of making whole students.

Julie: Couldn't agree more. with the situation that is unfolding right now, with the pandemic and school closures. I have an 11 and 15 year old at home now, and trying to put program together for them to continue to be educated and provide different methods of learning. One of the things I realized yesterday, in my planning for them was to allow my kids to become creative, which is something that the school systems have lacked in the past and present. Yesterday, I allowed my kids to just be creative and explore their creative minds, and it was everything from baking a carrot loaf, to creating bracelets, to coloring, to painting. And I think the STEM of creative design is in every single one of us. But it's almost hidden. And as you say, with all of these cognitive abilities-- capacities that we do have, being able to balance the creative side with the technical is so important. Is there anything else that I've just said that maybe you want to stem off or engage further with regards to creative design thinking? Because I think that's something that's missing a lot of our systems today.

Ben: Well-- and design thinking is really the operative word there, because Aoun's humanics is-- it specifically mentions systems thinking, but it doesn't specifically mention design thinking. But if you combine the two of them together, then you have your arts and sciences right there. Take any creative software, take Adobe Photoshop or Illustrator, or take Premier After Effects. Very powerful creative tools. If you are using that tool and you want to be very precise, what do you do? You go to the Inspector palette and you change a number. You could draw something and have that be by hand. And that will have its own sort of aspects to it that you created with your hand, with the imperfection of your hand, or even if you have a very talented and accurate hand, it's still a hand-drawn thing. But if you are taking out a polygon tool and you're saying, OK, well, here's a square, here's a hexagon. And then you want to bend it a precise amount, or warp at a precise amount, or scale it a precise amount, then you can just put in that exact number and you can align it to guides, and all of that precision is embedded in art. And to me, that's just a great metaphor for blending arts and sciences in a creative way. There are plenty of artistic things that you can do with data, and data may seem like a very dry-- it's a concept that-- some people don't even think that much about data. But the creativity in how you can structure data to have more meaning and how to analyze it isn't just STEM.

Julie: So we're almost at the near end of our podcast. And I think this deserves a part two of bridging the humanics into the classroom or into learning opportunities. Before we do sign off from this podcast, I do know that you and I are truly passionate on humanics and being able to balance this out. So I hope you'll join me again for another conversation on how everyone must balance their humanics, their creative side, their management of their data, their technology, and remain human through this transition of being in a hyper-connected world. So I hope you'll agree to be on a second podcast and we can can hear your conversation there.

Ben: It would be my pleasure.

Julie: Thanks so much, Ben. Thanks for joining me today.

Ben: Thank you, Julie

PART 2 coming soon!