This repository contains library routines and stand-alone programs for various modeling and registration tasks relating to surgery, with a particular focus on 2D/3D registration for intraoperative X-ray navigation. Although xReg was developed from a practical approach and provides an out-of-the-box capability for intraoperative fluoroscopic navigation, it is also very much compatible with research-oriented goals. In addition to accelerating the development of new registration components through the extendable interfaces for ray casting, image similarity computation and iterative optimization, xReg is also useful as an initial baseline for evaluating the neural network approaches du jour.

Much of the functionality provided by xReg is also useful for other computer-assisted surgery tasks. A comprehensive listing of the features of the library are provided below, along with a list of the executable programs that are also included in this repository. Please visit the wiki for descriptions on the use of the library and executable programs. Basic usage of the executables is provided via a walkthrough/tutorial here. Please submit an issue for any problems, feature requests, or suggestions.

This software was originally created while conducting research with Russell Taylor, Mehran Armand, and Mathias Unberath within the Laboratory for Computational Sensing and Robotics at Johns Hopkins University. The current repository represents a rewrite/refactor of the original, internal, version of this software, with a focus on minimizing compile times and being reusable to novice developers and researchers.

• Registration:
  • Efficient ray casters for 2D/3D registration and visualization:
    • Line integral (CPU and OpenCL)
    • Line integral approximation via splatting (CPU)
    • Surface rendering (CPU and OpenCL)
    • Depth maps (CPU and OpenCL)
    • Occluding contours (CPU and OpenCL)
    • Sparse collision detection (CPU)
    • Extendable common interface (CPU, OpenCL, and more)
  • Image similarity metrics for driving 2D/3D registrations:
    • Sum of squared differences (SSD) (CPU and OpenCL)
    • Normalized cross correlation (NCC) (CPU and OpenCL)
    • NCC of Sobel Gradients (Grad-NCC) (CPU and OpenCL)
    • Gradient orientation (GO) (CPU)
    • Gradient difference (Grad-Diff) (CPU)
    • Patch-wise NCC (Patch-NCC) (CPU and OpenCL)
    • Patch-wise Grad-NCC (Patch-Grad-NCC) (CPU and OpenCL)
    • Boundary contour distance (CPU)
    • Extendable common interface (CPU, OpenCL, and more)
  • Various optimization strategies for 2D/3D intensity-based registration:
    • CMA-ES
    • Differential Evolution
    • Exhaustive/Grid Search
    • Particle Swarm Optimization
    • Hill Climbing
    • Wrappers around NLOpt routines:
      • BOBYQA
      • CRS
      • DESCH
      • Variations of DIRECT
      • DISRES
      • NEWUOA
      • Nelder Mead
      • PRAXIS
      • Sbplx
    • Extendable common interface
  • Regularizers for 2D/3D intensity-based registration:
    • Rotation and translation magnitudes
    • Euler decomposition magnitudes
    • Rotation and translation magnitudes of relative pose between multiple objects
    • Relative pose difference from nominal AP pose
    • Landmark re-projection distances
    • Heuristics for automatic global pelvis registration
    • Combination of regularizers
    • Extendable common interface
  • Pipeline for chaining together 2D/3D registrations (intensity-based and feature-based) for solving registration problems with multiple-resolutions and views
  • Perspective-n-Point (PnP) solvers (paired point 2D/3D)
    • Minimization of re-projection distances
    • POSIT
    • P3P using C-arm geometry assumptions
    • RANSAC PnP wrapper
  • Paired Point 3D/3D
  • 3D Point Cloud to 3D Surface ICP
• Mesh Processing:
  • Triangular and tetrahedral mesh representations
• Image/Volume Processing:
  • Interpolation of non-uniform spaced slices
  • Image processing operations leveraging lower-level ITK and OpenCV routines
  • Conversion of Hounsfield units (HU) to linear attenuation
  • Poisson noise
  • Image intensity log transform
  • Piecewise rigid volume warping using label maps
• Numerical Optimization:
  • Configurable line search implementation
  • Implementations of several derivative-free methods, including Differential Evolution, Simulated Annealing, and Particle Swarm Optimization
  • Wrapper around C implementation of CMA-ES optimization
  • Suite of test objective functions
• Geometric Primitives and Spatial Data Structures
  • KD-Tree for points or surfaces of arbitrary dimension
  • Primitives with support for intersection, etc.
  • Fitting primitives to data (circle, plane) with robustness to outliers
• Spatial Transformation Utilities:
  • Rotation and rigid transformation utilities, including lie group/algebra routines
  • Perspective projection (3D to 2D)
  • Calculation of anatomical coordinate frames
  • Point cloud manipulation
• Visualization
  • Interactive 3D scene plotting using VTK
• File I/O:
  • Common DICOM fields
  • DICOM files from Siemens CIOS Fusion C-arm
  • Helpers for HDF5 reading/writing
  • Various mesh formats
  • Various image/volume formats (via ITK)
  • 3D Slicer annotations, FCSV and ACSV
  • Comma separated value (CSV) files
• Basic Math Utilities:
  • Basic statistics
  • Distribution fitting
  • Uniformly distributed N-D unit vector sampling
  • Common interface for probability densities and instances for common distributions:
    • Normal
    • Log-normal
    • Folded normal
• General/Common:
  • String parsing/manipulation utilities
  • Serialization streams
  • Command line argument parsing
  • Timer class for measuring runtimes with a stopwatch-like interface
  • Basic filesystem utilities
  • Runtime assertions
• Hip Surgery:
  • Guessing labels of bones from segmentation volumes
  • Planning and modeling of osteotomies
  • Visualization of osteotomies in 3D
  • Modeling of surgical objects, such as screws and K-wires
  • Support for simulated data creation, including randomized screw and K-wire shapes and poses, and volumetric data incorporating osteotomies, repositioned bones, and inserted screws and K-wires

Some of the capabilities provided by individual programs contained with the apps directory include:

• Image I/O:
  • DICOM conversion and resampling
  • Volume cropping
  • Printing DICOM metadata
• Mesh processing:
  • Mesh creation
  • Mesh display
• Basic point cloud operations:
  • Printing FCSV contents
  • Warping FCSV files
• Registration
  • ICP for 3D point cloud to 3D surface registration
• General utilities for projection data:
  • Advanced visualization of projective geometry coordinate frames with a scene of 3D objects
  • Remap and tile projection data for visualization
  • Tool for creating movie replays of 2D/3D registration processing
  • Extract projection into NIFTY format (.nii/.nii.gz)
  • Insert landmarks (FCSV) into HDF5 projection data
• Hip Surgery: Periacetabular Osteotomy (PAO)
  • Osteotomy planning and modeling
  • Osteotomy 3D visualization
  • Randomized simulation of fragment adjustments
  • Volumetric modeling of fragment adjustments
  • Volumetric modeling of fragment fixation using screws and K-wires
  • Creation of simulated fluoroscopy for 2D/3D registration experiments
  • Examples of 2D/3D, fluoroscopy to CT, registration
    • Single-view pelvis registration
    • Multiple-view, pelvis, femur PAO fragment registration

Although the following capabilities currently only exist in an internal version of the xReg software, they will be incorporated into this repository at a future date:

• Executable for running a multiple-view/multiple-resolution 2D/3D registration pipeline defined using a configuration file
• Intraoperative reconstruction of PAO bone fragments
• Utilities for creation and manipulation of statistical shape models
• Shape completion from partial shapes and statistical models
• More point cloud manipulation utilities
• Python bindings, conda integration
• And more...

• C++ 11 compatible compiler
• External libraries (compatible versions are listed):
  • OpenCL (1.x) (typically provided with your graphics drivers or CUDA SDK)
  • Intel Threading Building Blocks (TBB) (20170919oss)
  • Boost (header only) (1.65)
  • Eigen3 (3.3.4)
  • fmt (5.3.0)
  • NLOpt (2.5.0)
  • ITK (4.13.2)
  • VTK (7.1.1)
  • OpenCV (3.2.0)
  • ViennaCL (1.7.1)
  • Optional: ffmpeg is used for writing videos when it is found in the system path. The OpenCV video writer is used if ffmpeg is not available.

A standard CMake configure/generate process is used. It is recommended to generate Ninja build files for fast and efficient compilation. An example script for building all dependencies (except OpenCL) and the xReg repository is also provided here. The docker directory demonstrates how Docker may be used to build the software.

The software is available for use under the MIT License.

If you have found this software useful in your work, we kindly ask that you cite the most appropriate references below:

Grupp, Robert B., et al. "Pose estimation of periacetabular osteotomy fragments with intraoperative X-ray navigation." IEEE Transactions on Biomedical Engineering 67.2 (2019): 441-452.
----------------------------------------------------------------------
@article{grupp2019pose,
  title={Pose estimation of periacetabular osteotomy fragments with intraoperative {X}-ray navigation},
  author={Grupp, Robert B and Hegeman, Rachel A and Murphy, Ryan J and Alexander, Clayton P and Otake, Yoshito and McArthur, Benjamin A and Armand, Mehran and Taylor, Russell H},
  journal={IEEE Transactions on Biomedical Engineering},
  volume={67},
  number={2},
  pages={441--452},
  year={2019},
  publisher={IEEE}
}

Grupp, Robert B., Mehran Armand, and Russell H. Taylor. "Patch-based image similarity for intraoperative 2D/3D pelvis registration during periacetabular osteotomy." OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis. Springer, Cham, 2018. 153-163.
----------------------------------------------------------------------
@incollection{grupp2018patch,
  title={Patch-based image similarity for intraoperative {2D}/{3D} pelvis registration during periacetabular osteotomy},
  author={Grupp, Robert B and Armand, Mehran and Taylor, Russell H},
  booktitle={OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis},
  pages={153--163},
  year={2018},
  publisher={Springer}
}

Grupp, Robert B., et al. "Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration." International Journal of Computer Assisted Radiology and Surgery (2020): 1-11.
----------------------------------------------------------------------
@article{grupp2020automatic,
  title={Automatic annotation of hip anatomy in fluoroscopy for robust and efficient {2D}/{3D} registration},
  author={Grupp, Robert B and Unberath, Mathias and Gao, Cong and Hegeman, Rachel A and Murphy, Ryan J and Alexander, Clayton P and Otake, Yoshito and McArthur, Benjamin A and Armand, Mehran and Taylor, Russell H},
  journal={International Journal of Computer Assisted Radiology and Surgery},
  pages={1--11},
  publisher={Springer}
}

Grupp, Robert, et al. "Fast and automatic periacetabular osteotomy fragment pose estimation using intraoperatively implanted fiducials and single-view fluoroscopy." Physics in Medicine & Biology (2020).
----------------------------------------------------------------------
@article{grupp2020fast,
  title={Fast and automatic periacetabular osteotomy fragment pose estimation using intraoperatively implanted fiducials and single-view fluoroscopy},
  author={Grupp, Robert and Murphy, Ryan and Hegeman, Rachel and Alexander, Clayton and Unberath, Mathias and Otake, Yoshito and McArthur, Benjamin and Armand, Mehran and Taylor, Russell H},
  journal={Physics in Medicine \& Biology},
  year={2020},
  publisher={IOP Publishing}
}

Grupp, R., et al. "Pelvis surface estimation from partial CT for computer-aided pelvic osteotomies." Orthopaedic Proceedings. Vol. 98. No. SUPP_5. The British Editorial Society of Bone & Joint Surgery, 2016.
----------------------------------------------------------------------
@inproceedings{grupp2016pelvis,
  title={Pelvis surface estimation from partial {CT} for computer-aided pelvic osteotomies},
  author={Grupp, R and Otake, Y and Murphy, R and Parvizi, J and Armand, M and Taylor, R},
  booktitle={Orthopaedic Proceedings},
  volume={98},
  number={SUPP\_5},
  pages={55--55},
  year={2016},
  organization={The British Editorial Society of Bone \& Joint Surgery}
}

Grupp, Robert B., et al. "Smooth extrapolation of unknown anatomy via statistical shape models." Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling. Vol. 9415. International Society for Optics and Photonics, 2015.
----------------------------------------------------------------------
@inproceedings{grupp2015smooth,
  title={Smooth extrapolation of unknown anatomy via statistical shape models},
  author={Grupp, Robert B and Chiang, H and Otake, Yoshito and Murphy, Ryan J and Gordon, Chad R and Armand, Mehran and Taylor, Russell H},
  booktitle={Medical Imaging 2015: Image-Guided Procedures, Robotic Interventions, and Modeling},
  volume={9415},
  pages={941524},
  year={2015},
  organization={International Society for Optics and Photonics}
}