Consider being able to seek professional care or participate in a real classroom experience from the comfort of your own home. This is the type of thought that advancing research at York is tackling, suggesting that augmented reality (AR) and virtual reality (VR) may go beyond how we perceive our world, with possible applications in healthcare and remote education.
Associate professor Gene Cheung at the Lassonde School of Engineering is making use of signal processing tools for point clouds, which is vital in expanding our understanding of AR and VR.
“3D point cloud processing is basically an extension of 2D imaging to 3D imaging,” Cheung explains. “Typical 2D images, such as those captured by a camera, consist of rows and columns of pixels that hold RGB (Red, Green, and Blue) data.”
With the advent of 3D sensors, lasers can be used to shoot onto a physical object and reflect onto the sensor. Based on how many seconds it takes for the bounce to return, it can measure the distance with respect to the camera. Using these distances for each pixel, you can sample the 3D shape of the object. While the traditional array of RGB data remains, each pixel now has the depth and distance relative to the camera. “If you have a 2D image, it looks flat. But now that you have 3D geometry information, you can render that information from different viewpoints, and it looks life-like,” continues Cheung.
When applying 3D point cloud processing to healthcare, Cheung highlights that it can be medically applied to sleep apnea detection, heart/rhythm estimation, and stand pose estimation.
“With geometric data, you can measure an individual’s breathing cycle and respiration, blood movement throughout one’s head (through which you can detect emotions, irregular heartbeats, etc.), and detect the deterioration of a person’s standing position, which is most common among the elderly.”
Moreover, Lauren Sergio, a professor within the Faculty of Health, explores the use of AR and VR in concussion and stroke recovery. Sergio’s goal for AR is to resolve concussions in athletes and working-age adults who have persistent symptoms, and is accomplished by simulating the activities that people engage in on a daily basis.
“With augmented reality, we can actually set up workplace-specific scenarios in a light, controlled manner so we can help the brain adjust little by little,” Sergio explains. VR, on the other hand, could be used for stroke recovery, as strokes can cause long-term motor deficits, which Sergio also aims to correct.
One method Sergio is exploring has patients wearing robotic exoskeletons equipped with VR, allowing them to see an avatar version of the body part. Sergio describes their efforts as “reading the brain waves and using a variety of AI techniques to decode what they are thinking into having the robot help their arm move in the right direction,” aiding their brain’s neuroplasticity to reorganize itself.
This is typically used in conjunction with ongoing physiotherapy treatments as patients “are still going to go to a professional to get started, but now with these systems, medical professionals are able to reach remote communities more easily,” Sergio explains.
The main benefit is that VR systems broaden the range and ease of accessibility for those with reduced mobility, all while being monitored closely by a professional.
“The use of AR and VR in healthcare actually has a long history, and AR in education isn’t all that far behind,” says Dr. Caitlin Fisher, director of the Augmented Reality lab and the Immersive Storytelling Lab at York.
“The thing to remember is that the original money or development in the fields of augmented and virtual reality don’t come from arts or entertainment, they come specifically from industrial, medical, and military spending. A lot of that same research and emerging understanding of user interface design works for training in medicine too,” Fisher continues.
Due to the COVID-19 pandemic, students and workers have been accustomed to remote communication for so long, causing people to feel disconnected from the experience of in-person education and work. “To think that Zoom is like a replica of in-person teaching is completely not true,” Cheung says.
As Cheung’s work in 3D point cloud processing progresses, students may be able to delineate between maintaining the student experience in an online environment, while medical professionals will see expanded accessibility in treating and monitoring patients.
