How Helios Is Using Intel 3D Cameras to Assist People with Vision Impairments

Giving sight to the blind is some kind of miracle. Though that feat continues to elude us, a Bucharest-based Intel® Innovator, Silviu-Tudor Serban, has set out to do the next best thing. Leveraging a background in computer vision and artificial intelligence, he and his team have designed Helios, which uses Intel® RealSense™ 3D cameras to give those with severe visual impairments a clearer view of the world. And for those with total blindness, Helios features like reading assistance, facial recognition, and a haptic interface provide a much enhanced sense of their surroundings.

Motherboard spoke with Serban about his project, and how next-gen technology can give greater independence to the 285 million people across the world who have low vision, or no vision at all.

Tell me about Helios.
What we wanted to do with Helios is use next-gen technologies to radically improve accessibility for visually impaired people. It works similar to a VR set, but instead of displaying computer-generated content, the user receives real-time visual data that is captured by the Intel® RealSense™ 3D camera. Our software processes the data in order to provide the most relevant information for people with low vision. For example, if there are some obstacles nearby, our software can highlight them, and tell the user he or she is going to bump into something. It can also recognize people's faces and assist with reading too.

How did you come up with the idea?
We took part in the Intel® RealSense™ App Challenge, which was a global competition organized by Intel to create the first applications for RealSense. It had a one million dollar prize pool, and we were actually one of the winners. We built a game called Drill Sergeant Simulator, which is when we first got to explore the full range of capabilities for the Intel® RealSense™ camera and SDK. It allowed us to get to know the technology inside out. That eventually led us towards the “Aha moment,” where we knew this camera could be the secret sauce for building something to help the visually impaired.

As someone who's followed emerging technologies within the field of computer vision, what are some things we can do now that we couldn't do five years ago?
We can do more on the go. Five years ago you couldn't do much on mobile. Laptop cameras were of much lower quality. We had Kinect, which is like a big hulk of equipment that had 3D vision. But the Intel® RealSense™ camera is so small, it can fit on your phone or tablet. This is the major difference from five years ago. You have the computing power and the sensors, and it's all so small and power-efficient you can actually place it in the product.

What do you hope to be able to do in a few years that we just can't do yet?
What we're waiting for is the technology to shrink even more. Now we have a wearable prototype, but what we want is to have it smaller and lighter, while keeping all the performance. I think in a couple of years we'll get even better mobility from these kinds of devices.

What kind of equipment is the user carrying?
The current prototype features an Intel® RealSense™ SR300 3D camera, which is mounted on a headset, and an Intel 2-in-1 hybrid laptop, which the user carries in a backpack.

What was most challenging part of making Helios work?
From my point of view, the most challenging part was to see the world from the perspective of the people who we are trying to help: the visually impaired. It's vital for designing the product and actually giving the software features that will make sense for the user, not me or you.

To that end, we've been working with a couple of people who have vision impairments. We have a team member named Mihai -- he's featured in our videos demoing Helios -- who was born with microphthalmia and cataract-microcornea syndrome. Basically, he can see with only one eye, and that’s up to 50 cm. We relied a lot on his insight.

Our meeting was rather serendipitous, actually. I was on the train, returning home after a Helios hardware hacking session with my team. Mihai was carrying a walking stick, and someone was helping him find his seat, which happened to be in the same compartment with me. It was incredibly random. I think the universe is particular in that way, making sure we wind up exactly where we should be.

We started talking. First, about his vision impairment and challenges he faces everyday, then about vision aids and his Master’s thesis on accessibility in museums. Finally, I told him about Helios and he said he would love to try it and help out if possible. He’s been part of our team ever since.

It sounds like he has a very specific type of vision impairment. Do you have to calibrate it so it works with someone who has a different impairment?
There are some afflictions that cover most people: cataracts, glaucoma, Retinitis Pigmentosa, things like that. For the most part, they struggle to see a distant point. For them, Helios works right out of the box. The headgear has an adjustable lense, so that fixes some minor problems. We also have a few presets people can switch between in real-time, to choose whichever is best for them. For example, some people can’t see at night; for them we have a nightvision mode.

Was there a particular moment when you knew this was going to work?
I believe we have had several breakthrough moments while developing the software and hardware, but one of the most meaningful ones was when Mihai tried out the prototype for the first time. He just put the headset on and started using it with zero learning curve. He began describing objects in his line of sight, like tables, chairs, the closet or the door, sometimes touching them for confirmation. Then he gained confidence and went on exploring the area. Meanwhile, I was trying to hold back an ecstatic grin. I just knew we were on the right track.

What if someone can’t see at all? Is there anything Helios can do for them?
We actually have two versions of Helios we’re working on: Helios Light is the current working prototype, which aims to enhance sight for people suffering from low vision; and Helios Touch is our concept for improving spatial awareness for the blind. The secret ingredient for both implementations is Intel® RealSense™ technology, which is used to capture visual data.

The idea behind Helios Touch is to help people with complete blindness by using software that can describe their nearby environment and then communicate with them via haptic impulses. It uses vibration patterns to convey a level of information about what’s around the user, somewhat like painting a picture in broad strokes. If, for instance, an obstacle is encountered, the user can feel it is there before they run into it.

Those with total blindness can also take advantage of other features our software has right now, like reading assistance. That allows them to do things like go to a restaurant on their own and know what’s on the menu. We also have a special feature we call Argos, which is named after the all-seeing guardian giant from Greek mythology. Argos is for remote caregiving. Basically, the user can connect to a friend or relative who they trust, and can ask for help and guidance whenever they need. That friend or relative will have access to a direct visual stream of what the user sees, which they get on their phone.

The Intel® RealSense™ technology also allows for almost instant learning of a person's facial features. So a known person, such as a friend or relative, can thus be easily recognized when they are in the line of sight of the user. More than that, the user can better understand facial cues, such as happiness or surprise, when talking to others.

What's the next step?
Our long-term plan is not only to develop the best possible solution, but also make it affordable. In terms of areas we're working on, we've been focusing a lot on the human-computer interface, improving in terms of computer vision and AI, and refining the algorithms to improve organization.

Though we're developing several other core projects, our greatest focus is on Helios, because we think it has the capacity to make the greatest impact. For people with low vision it’s difficult, or even impossible, to do simple things like reading, writing, shopping, watching television, or recognizing faces. With Helios, all these types of activities become achievable. What we want is to empower people. Imagine people with vision impairment being able to better integrate in the workplace, or more easily access knowledge, or have improved mobility or an easy method of asking a friend for help -- this could change the lives of millions.

The Intel® Developer Zone offers tools and how-to information to enable cross-platform app development through platform and technology information, code samples, and peer expertise in order to help developers innovate and succeed. Join communities for the Internet of Things, Intel® RealSense™ Technology, Modern Code, Machine Learning and Game Dev to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.

from How Helios Is Using Intel 3D Cameras to Assist People with Vision Impairments

How 'Vapor Riders '99' Is Helping Cure Motion Sickness in VR Gaming

Virtual reality is a relatively new frontier for game designers, offering both tremendous potential as well as a bevy of problems to overcome. Matt Rebong and Sam Warner, founders of Well Told Entertainment and members of the Intel® Software Innovators program, are tackling one such obstacle: the motion sickness many users feel when moving through VR games. Some developers have sidestepped this issue by teleporting users between locations; however, to Rebong and Warner, that solution shatters the immersiveness that is one of VR’s chief qualities.

Their new project Vapor Riders ‘99, a multiplayer racing game where users cruise through what looks like the early internet to a very chill vaporwave soundtrack, seeks to mitigate the effects of motion sickness while maintaining, even accentuating, the immersiveness. Motherboard spoke with Rebong and Warner about their game, nausea, nostalgia, and the potential for augmented and virtual reality to bring gamers together again on one couch, where they can lovingly yell at each other, like old times.

Can you tell me how you guys started working together?

Warner: We both studied at Chapman University, graduating class of 2015. Afterward, we got a job developing VR and AR content for a client. We had worked together in the past, but in order to complete this job on time, we put together a team. In the process, we decided we really liked working together, so we made a company out of it, Well Told Entertainment. We’ve been operating for a year now.

Rebong: So the company started with work-for-hire, and we're still doing some of that stuff. But we soon decided that we also want to make our own games in this space. Vapor Riders is a direct result of that. We're looking at the VR market, thinking: ‘What are the issues?’ ‘What is something we can tackle and solve?’

So your starting point with this game was to find a problem to solve? What did you settle on?

Rebong: Locomotion. In VR, especially at the time, people were only teleporting around, since motion sickness was such an issue. The problem is teleporting is fairly immersion-breaking. So we thought, how can we have a game that is fun and doesn't have that movement scheme? We were drawn to track-based movement, because it helps you place yourself, and you know where you're going. That was where we started off.

Warner: We wanted to be able to create games that had more expansion to them: a little more length and depth, so you don't have that immersion-breaking every time you want to change position. A big focus of Vapor Riders was joining the audio, the motion, and the gameplay, into a sense of moving forward uninterrupted. It's kind of racing. It's also kind of chill, flying and gliding and bouncing. There's a lot going on.

The Well Told team.

What was the most difficult part?

Rebong: Like most VR developers, we are struggling with the limited points of data when it comes to input. For a skiing game, we would ideally be tracking more than just the head and hands. So we have to get creative with our assumption about what the user is doing with the rest of their body. Also, every person has a different threshold for VR nausea, so tutorialization and difficulty assessment is also a steep hill to climb. Lastly, networking, and making a scalable, multiplayer experience in a medium where latency and framerate must remain at least at 60hz is quite a juggling act.

When it comes to the motion sickness, how did you work around that?

Rebong: There are things we can do like getting people's height when they play, so it feels more like they're playing the game from their perspective, not disassociated from their body. We’re constantly learning from user experiences, and taking into account what people are saying about the game. It evolves with every play-test.

Warner: We've taken Vapor Riders to two conventions. We went to VR LA, and then we shared a booth with a few other teams in the Intel® Software Innovators Program at the Intel® Developer Forum. At the first one, we got lots of awesome feedback from VR vets.

But when we get to IDF, these are like businessmen, tech people, people who haven't played a videogame in like five years, maybe ten. So we asked, "Have you played with VR before?" And they're like, "I've played a video game before, son. I know what I'm doing." And then they put the headset on, and I kid you not, we had more than six people fall over completely, just laughing and waving their arms and shouting. So we pick them up and take the headset off, and not a single one of them was like, "I'll never do that again. I hated this." They were saying, "That was incredible! Video games are so different, woahh!"

The coolest thing was this Japanese guy that was playing. He just started yelling in Japanese. I had to hold him up, because he kept wanting to fall over backwards. He finished the whole demo, and then he took off the headset, and bowed to me. Then he just left. That was amazing. Those are high points.

As for low points, we for sure do encounter people who just in general don't really like where VR is right now. To answer your question, we're doing a lot of things to try and mitigate nausea, like constant forward velocity, little to no acceleration or deceleration, and a track to guide you.

What was the most surprising thing about developing this game?

Rebong: Maybe the game itself. It was kind of a stab in the dark. It really started with us in an empty landscape just pulling ourselves around in different ways. What we realized was it's actually really fun to just do that. That was kind of the inception of the game.

Warner: That's a core part of what we try to do: Find the fun in an experience first, so it's like a toy to play with, and then build the game on top of that. We put jumps in there. We put in a track. Then it started feeling like a game. Then we put some of our vaporwave soundtracks on, and we just started cruising. This is it!

What's next for the company?

Rebong: At the moment we're balancing Vapor Riders with some of our client work. Because we're still a young design company, we still have to build revenues by doing that kind of development. Last Halloween, for example, we took a little step back from Vapor Riders because we went to a game jam at the Titmouse Studios; Titmouse is an animation company that does a lot of Adult Swim stuff, things like that. The game jam was horror-themed, so we created this little simulator called Escape Bloody Mary. It's a VR experience where you're a little kid, and your friends push you in the bathroom.

Oh, that kind of Bloody Mary. I was imagining brunch in VR. So this is when you conjure a ghoul by saying “Bloody Mary” three times in the mirror...

Rebong: Exactly. It's a virtual escape room experience. It’s available on Steam now. The game marks our first VR title on the market and we couldn't be more excited, especially with all the positive feedback we've been getting.

How big is your team?

Rebong: It's 8 people: half programmers, half artists.

What kind of equipment do your games require?

Warner: Right now we're building for the HTC Vive primarily. We’re looking at Rift and PS VR in the future, hopefully even expanding this racing experience into brick-and-mortar VR arcades.

Vapor Riders is a multi-headset, single-setup experience; what I mean by that is using multiple Vives in the same room-scale setup, so there's only two lighthouses but there could be as many as four or five users standing within them. We’re trying to capture that local multiplayer experience, where everyone’s sitting together in the same room, on the same couch, yelling at each other, but in VR. That's the fun of multiplayer.

I’m sensing some nostalgia here. Does that factor into your creative process?

Warner: Oh yeah, definitely. We're huge Nintendo fans. There's even crazy nostalgia for the early days of Halo, Starcraft…

The epic LAN parties…

Warner: Counterstrike for days, man. There's just something incredibly powerful about knowing you're in the same place as people, playing the same game. We've been talking a lot about VR today, but as a company, we have a huge focus on AR for exactly that reason. Augmented reality and mixed reality allow you to enjoy visual content while still having the presence of another human in the room. That’s important to us.

One of the first projects we worked on was AR for a client in the kid's book space, for instance. A lot of these experiences you can see being expanded into classroom-sized groups of people, who are all enjoying the same content at the same time. So what we’re interested in is how to take this good user experience—which allows them to connect to characters, and connect to each other—and bring it into the mobile market, where people can enjoy it right now.

The Intel® Developer Zone offers tools and how-to information to enable cross-platform app development through platform and technology information, code samples, and peer expertise in order to help developers innovate and succeed. Join communities for the Internet of Things, Intel® RealSense™ Technology, Modern Code, Machine Learning and Game Dev to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.

from How 'Vapor Riders '99' Is Helping Cure Motion Sickness in VR Gaming

Drugs Designed to Aid Alzheimer's Patients Could Enhance Healthy Brains

Written By Joseph Neighbor

Alzheimer’s Disease is perhaps the most famous, least understood medical crisis facing our generation. Though only five million Americans suffer with it, the cost of care exceeds $230 billion a year, making Alzheimer’s the single most expensive disease to treat. And that burden will grow exponentially in the near future. By 2050, the number afflicted will triple. The cost of care will triple, too.

Of the top 10 cause of death in the U.S., Alzheimer’s Disease stands alone as the only one for which we have no cure, nor method of prevention. We don’t even have a way to slow it down. Existing medications treat only symptoms, and there hasn’t been a new drug approved by the FDA in a decade. The degeneration of neurons in the brain remains a mysterious process. We’re running out of time to figure it out.

The Federal government recognizes this. After all, it’s in their interest to find a solution: Medicaid and Medicare picks up roughly 70% of the total cost of care. The long-term solvency of those programs depends on a cure. Accordingly, in 2010, Congress unanimously passed, and the President signed into law, the National Alzheimer’s Project Act, which aims to have a treatment for Alzheimer’s by 2025. The NIH has doubled the amount of funds earmarked for research into the disease over the last two years, bringing the sum to nearly a billion dollars a year—an unprecedented increase.

There are reasons for optimism. Seldom has so much money and effort been directed toward solving a specific medical problem. No doubt this will lead to a better understanding of the disease, which will hopefully lead to more effective therapies. But it will also, presumably, lead to a more general understanding of the basic workings of memory and cognition. This begs the question: If researchers develop a drug to improve how the brain functions, could it be useful for healthy people? Might we unintentionally find a way to hotwire the mind? Could Alzheimer’s research pave the way for the general cognition-enhancing pill so often depicted in science fiction?

Cutting out the beta-amyloid fragment may not be the only benefit to these drugs. Photo: WikiCommons

After all, it’s not unknown in the history of medicine for researchers to set out to solve one problem and accidently solve another. Viagra was originally intended to treat coronary disease. Modafinil—often cited as a magic cognitive-booster, a la Limitless—was designed for narcoleptics. That’s why current laws reserve a good deal of leeway for physicians to prescribe medications for off-label uses. Basically, once a drug is FDA approved for one purpose, doctors can then use it for other things if they think it would be helpful, barring any studies demonstrating that particular off-label use is exceedingly dangerous.

So, legally, prescribing drugs off-label is fair game. This makes it possible, theoretically, for medication designed to treat memory and cognition in Alzheimer’s patients to be prescribed to ordinary people who want a better, more efficient mind.

In fact, some researchers are testing what would happen if we did just that. A study at UC-Berkeley measured the effect of donepezil, the most common drug for treating Alzheimer’s symptoms, on healthy people doing various visual learning tests. Donepezil works by fighting the enzyme in the brain that naturally breaks down acetylcholine, a neurotransmitter. Essentially, the thinking is that more acetylcholine in the cerebral cortex leads to better cognition.

Though the sample size of the study was limited, and the tests focused on very specific forms of cognition, the results were promising. According to Dr. Michael Silver, the primary investigator of the study, there’s evidence that donepezil enhances visual perception, spatial attention, and some forms of perceptual learning.

In 2014, researchers at Boston Children’s Hospital conducted a donepezil study on kids with amblyopia, a defect of the eye that impairs its ability to focus and take in visual information. In order for a child’s brain to develop properly, it needs that visual information. The study found that donepezil chemically re-wired the brain to allow it to process visual stimuli better, as well as increase its neuroplasticity, so patients can learn faster.

That doesn’t mean donepezil is a miracle cognitive booster. “If you could just increase your acetylcholine signaling and get better at perception, attention and learning, how come evolution hasn’t figured that out yet? Why don’t we just have more acetylcholine, all the time?” said Dr. Silver. “The big caveat is maybe you’re worse at other things that aren’t being measured.”

Vitamins are just the beginning of neurological enhancement. Photo: WikiCommons

Tinkering with the delicate chemical balance arrived at by tens of thousands of years of evolution is a dicey enterprise. It’s something that should be pursued with caution. But donepezil is just one drug that happens to exist right now. Now that Alzheimer’s is recognized as an exigent health crisis on the order of cancer and HIV, and research funding is setting new records both here and abroad, it’s possible that a breakthrough era in medicine might be at hand. If we understand the brain well enough to slow or stop the process of neurodegeneration, maybe we can find a way to make the brain operate just plain better for the rest of us.

There will be protests. After all, the point is to develop a drug to treat a disease, not hotwire a healthy brain. From DIY bodyhackers and genetic manipulators to doping students and athletes, attempts at human augmentation are always confronted with thorny ethical questions such as what is cheating, who should have access to these enhancements, and, crucially, whether anyone should have them at all. Often, the discussion comes down to the precise difference between “therapy” and “enhancement.”

Alzheimer’s muddies those distinctions. Amyloid plaques and Tau tangles—the pathological hallmarks of the disease—can be observed 20 years before actual symptoms appear. Surely, a person showing those hallmarks should be entitled to medicine that could halt the disease’s progression, regardless of whether it’s actually impairing his or her life yet. If these therapies, used as a preventive measure, also happen to have ancillary benefits for cognition, memory or learning that allow such a person to excel at school or work, would we deem that an unfair advantage? Besides, if the drug is safe and boosts brainpower, why shouldn’t we all take it?

A firm understanding of the inner workings of the brain continues to elude us. The implications of arriving at that understanding are unimaginable. Perhaps Alzheimer’s Disease could provide the urgency, the direction and the money to propel us not only to its cure, but also to a better brain for humankind.

Made possible by Deus Ex: Mankind Divided. In stores now.

from Drugs Designed to Aid Alzheimer's Patients Could Enhance Healthy Brains