A real-time data fusion system updating 3D organ shapes using color information from multi-directional cameras

We have been developing a data fusion system, which allows surgeons to visualize the inner structures of organs during liver surgery. The liver alters its shape dynamically during an operation. In this system, we used stereo cameras to track intraoperative liver deformation. Color markers were attached on to the liver surface and tracked by the stereo cameras. Multi-directional camera views compensated for the lack of surface data during surgical procedures, even if the surface data could not be acquired from one directional camera view. A three-dimensional (3D) organ model that represented inner structures was reconstructed from preoperative image data sets and deformed by the intraoperative data. The model was updated on the video images of the surgical field. We examined the feasibility of this data fusion system in a liver phantom examination and in an in-vivo experiment with a pig liver.