Seeing the Cardiovascular World in 3D

Imagine a room-sized blood vessel that you can walk through like a hallway. Or a transparent 3D model of a patient’s heart rotating in the air. These are among the holograms created by a team led by Dr. Dayanjan Wijesinghe and cardiothoracic surgeon Dr. Daniel Tang, for cases that require complex surgical planning.

Dr. Daniel Tang, Ali Panahi, computer sciences graduate student, and Dayanjan “Shanaka” Wijesinghe, Ph.D., have worked together to develop AR technology for medical use.

Back in August 2016, Wijesinghe and his students Ali Panahi, a computer science graduate student, and Bansri Rawal, an undergraduate student, started creating the holograms that would allow the visualization of biochemical networks. “Computer screens are just not big enough to fit all of the information,” to see the complex networks, said Wijesinghe, assistant professor in the VCU School of Pharmacy’s Department of Pharmacotherapy and Outcomes Science. “That’s why we decided to completely ditch the computer screen and create a system that allows the biochemical networks to be all around you. You are essentially immersed in the network. You can walk around it.”

As a visitor dons a pair of augmented reality glasses, Wijesinghe’s lab in the VCU’s School of Pharmacy is transformed into various 3D landscapes superimposed on the real world. In one, a comprehensive biochemical network demonstrating the metabolism of omegas 3 and 6 lipids are displayed as an interconnected network of colored molecules in the air that you can walk through and interact with. The name of each molecule appears as you gaze upon it and disappears as you look away. These holographic objects respond to voice commands and can bring up additional information on demand.

Students in his classes love it. “The hologram has the advantage of allowing for this really natural learning because you’re actually interacting with something. It helps the students understand how drugs work from a biochemical network standpoint.”

After creating these augmented reality networks, the team began wondering where next to take their project. They found a new direction in medical imaging and the planning of complex surgeries.

Augmented Reality Comes to Pauley

Wijesinghe’s venture into the cardiac realm began with a conversation over coffee in the faculty lounge with Tang. “The faculty lounge there is probably the best thing that the school has done,” he said. “A lot of ideas bounce back and forth there.”

Soon, his lab was creating cardiothoracic holograms with the aid of Panahi and pharmacy student Vasco Pontinha. The team created its software program, Med-AR, using the Microsoft HoloLens augmented reality platform. Augmented reality, or AR, involves the projection of interactive computer-generated images into a person’s real-world surroundings.

“I think we are the only ones [using the platform who have created applications] for surgical planning of cardiothoracic surgeries,” said Wijesinghe. “There have been other applications—dentists have used it to look at cranial bone structures. Those are really easy because those are hard structures, but the heart is all soft tissue so that’s a different challenge.”

Tang identified several patients who were willing to have their CT scans used to create holographic models that could be used in the planning of their surgeries. One patient had a sarcoma in the center of his heart that had to be removed; another required the repair of leaks around two artificial valves. These first two surgeries took place, successfully, in February 2018.

During each procedure, the surgical team wore AR glasses to review the patient’s anatomical model in 3D. In one, the cancerous spot glows green.

“We used it as a way to discuss the operation with fellows, to help them understand what we were planning to do,” said Tang. “The 3D model provides a more intuitive way to think about the anatomy than you’d get from looking at it in a conventional way [a CT scan on a computer screen]. You can interact with it. You can get your hands on it, turn it around and so forth.”

Multiple viewers can wear the glasses to view the hologram. By pinching your fingers at an object, you can grab it and move it around, even placing it on a patient to understand its relative position in the chest cavity prior to surgery. Additionally, you can give it verbal commands, such as “hide aorta,” to remove structures, allowing an unobstructed view of an area.

With everyone looking at the same image, said Wigesinghe, “the entire surgical team is on the same page.”

What the Future Holds

Some current projects include a kidney cancer hologram and one being developed to show a human cadaver, for use by medical students. More cardiothoracic surgical applications are in the works. Additional projects close to completion include neurosurgical planning tools and a 3D protein structure visualization tool.

Dr. Vigneshwar Kasirajan, an early and enthusiastic supporter of the project, secured a $20,000 grant from the Department of Surgery to support the creation of the Med-AR program. In May, he presented one of the 3D heart models at Pauley’s Consortium Dinner.

An upcoming trial will evaluate the teaching value of the holograms for medical students and surgical residents. “We instinctively think it will offer some opportunity for better education, though we have not yet quantified that,” said Kasirajan.

Wijesinghe, who is applying for additional grants, said his team includes students and faculty from pharmacy, computer science, biomedical engineering, nursing, radiology and even fine arts. “It used to be very difficult to collaborate in a cross-disciplinary manner, but now it’s becoming easier,” he said.

The multidisciplinary aspect of the team appeals to Kasirajan. “It’s a good opportunity to bring groups of people together to share ideas and keep things moving forward.”