Using off-the-shelf hardware and a photogrammetry workflow, we photographed kidney stones of different chemical compositions and reconstructed them as manifold meshes with realistic texture in a Universal Scene Description (USD) format. These are assets that can be dropped into a virtual environment, lit and viewed from any angle. Practicing endourologists then rated their fidelity, so that "realistic" carried clinical weight rather than purely technical success. An example of kidney stones in simulation is shown below: the kidney stone USDs are imported into Isaac Sim, laid out on a digital table, and a virtual camera is used to record the kidney stones from a bird’s-eye view to demonstrate the dynamic 3D appearance of the stones in simulation (Video 1).
In practice, we are using these digital assets for two reasons:
- To develop synthetic surgical scenes to improve computer vision and machine learning models related to ureteroscopy and percutaneous nephrolithotomy.
- To develop a virtual reality surgical simulation platform to help train medical students and residents.
The Pipeline
We obtained kidney stones characterized by stone composition, including: calcium oxalate monohydrate (COM), uric acid (UA), cystine, struvite (MAPH/CA), and brushite (CHPD). We photographed each stone inside a custom light box on a motorized rotating stage, using two fixed 4K cameras positioned for stereoscopic capture. Each stone was imaged at 18° increments through a full rotation, and the resulting image sets were reconstructed into textured 3D meshes using a photogrammetry pipeline, then validated for manifold, watertight geometry.
UA and COM stones rendered well and scored highest on ratings by endourologists. Struvite scored lowest on geometry. Cystine failed entirely, suspected to be related to its highly reflective, low-texture surface, which limits feature matching and depth estimation for 3D rendering.
Virtual Reality and Resident Education
In endourology, a trainee's first encounter with a ureteroscope is frequently in an actual patient. Robotic-assisted laparoscopic surgery has moved past this. For example, the da Vinci Skills Simulator is validated and now sits routinely in the training pathway before residents operate on the robot. Ureteroscopy has no widely adopted equivalent. Early virtual-reality trainers were attempted as far back as the early 2000s, but adoption stalled, in part because the fidelity was not there. Realistic stones are one of the missing building blocks for that kind of simulator, but also realistic animation of the lasering of the stone is required!
Synthetic Data for Computer Vision
Computer-vision systems for stone detection, segmentation, sizing, and tracking are advancing quickly, but they are throttled by data. Annotating real intraoperative footage is enormously labor-intensive - requiring hundreds of hours of manual labeling. Furthermore, real datasets are privacy-laden and badly imbalanced, with rare compositions underrepresented.
Synthetic surgical scenes built from validated digital stones change that. When you render a scene yourself, every pixel's ground truth is known for free: segmentation masks, stone type, count, and position come out of the renderer automatically, with no human annotator. That means models can be augmented or pretrained on synthetic data and fine-tuned on a much smaller real set, which is a paradigm already validated for object detection on everyday objects. Just as important, deliberately including diverse compositions, morphologies, and surface behaviors is what lets a model generalize across the messy variety of real cases rather than overfitting to whatever footage happened to be available. Our pipeline and the repository are therefore complementary to, not competitive with, existing computer vision work.
Open Access
All of this only matters if people can use it. The renderings are openly available, the hardware is inexpensive and off-the-shelf, and the workflow is reproducible at academic centers without specialized imaging infrastructure. Future work will need to address stones with highly reflective surfaces to further expand the composition library.
We see a near future in which a resident rehearses a ureteroscopy in a realistic digital simulation before the real procedure, and in which the computer-vision models assisting that procedure were trained, in part, on synthetic scenes built from stones like these.
Written by: Jonathan E. Katz, MD, Assistant Professor of Clinical Urology, Desai Sethi Urology Institute, University of Miami, Miller School of Medicine, Miami, FL
Read the Abstract