Giselle Coelho, M.D., Ph.D. and W. David Freeman, M.D
AI-Surgical Training & Mixed Reality Avatar Laboratory (ASTRAL):
Capturing Mastery and Expanding Surgical Training Without Borders
Award year: 2025 Richard M. Schulze Scholar in Artificial Intelligence
Using volumetric capture and mixed reality, we record a surgeon performing a procedure and recreate that experience so learners can see it in three dimensions, step by step.
They can observe technique, understand anatomy and repeat the procedure in a realistic simulator environment.
What inspired this project?
GC: Many surgeons around the world — especially in underserved regions — will never have access to super-specialized mentorship. So the question became: How can we capture that expertise and make it available to others, without geographic or economic limitations?
In simple terms, what are you building?
GC: We are creating AI-driven surgical avatars that allow a master surgeon’s knowledge to be captured and then used to train others in an immersive way.
Using volumetric capture and mixed reality, we record a surgeon performing a procedure and recreate that experience so learners can see it in three dimensions, step by step. They can observe technique, understand anatomy and repeat the procedure in a realistic simulator environment.
At the same time, we are building an AI component that allows the system to respond to questions and guide the learner, using a curated and validated knowledge base.
The goal is to create a new way of teaching — one that is immersive, repeatable and scalable.
What is volumetric capture and how is it different?
WF: Volumetric cameras capture surgical performance in three dimensions. Rendering engines reconstruct that performance in immersive space. Fine-tuned large language models allow us to build intelligent, procedure-specific AI systems. When combined, this creates a hyperdimensional training environment.
What we are capturing is not just video. It is high-dimensional surgical data — how a master surgeon moves, operates and makes decisions over time. That dataset becomes a replayable training platform.
How do you plan to apply this technology to your project?
WF: Traditional training relies on cadavers, static video or in-person mentorship. Our work adds a new layer — immersive playback and eventually bidirectional holoportation.
There are three pillars to this project. First, the recording platform, which enables four-dimensional capture and even real-time holographic transmission. Second, the playback learning environment, where trainees can rehearse procedures repeatedly without risk to patients. Third, the dataset itself — which has implications for robotics and autonomous systems in the future.
This applies to virtually any procedure at Mayo Clinic. Neurosurgery is the first proof of concept, but the platform is specialty-agnostic. In time, not using this kind of environment for training may feel primitive.
What impact do you hope this will have?
GC: The main goal is to reduce the learning curve in surgery in a safe and effective way. Patients come to Mayo Clinic for expertise. If we can reduce complications and elevate procedural quality through immersive rehearsal and data-driven refinement, that directly benefits patients. This is about scaling excellence.
Instead of learning primarily in the operating room, trainees can practice repeatedly in a realistic environment before treating patients. This increases confidence and improves technical skills.
It also has a global impact. In regions where access to expert mentorship is limited, this technology can bring high-level training directly to those clinicians. That means better preparation and, ultimately, better patient outcomes.
For us, this is about removing geographic, economic and educational boundaries and making high-quality surgical training more accessible.
How has philanthropic support shaped this work?
GC: The support from the Schulze award is essential. It allows us to create the infrastructure at Mayo Clinic — build avatars and develop the AI systems that support them. Without this generous philanthropic funding, it would be very difficult to move beyond small-scale testing.
This is a new way of teaching and learning. With this investment, we can take what we have learned and begin to apply it at a much larger scale.