IMNailScrewAR App
The App is low-cost and offers accurate distal locking method in intramedullary nailing, without any radiation exposure.
Augmented Reality Targeting App could be an appealing alternative for distal Locking of tibial intramedullary nails
Intramedullary nailing (IMN) is commonly used to address diaphyseal fractures of long bones. Accurate aiming the distal interlocking holes of the nail for insertion of transfixing screws remains a challenging problem in locked intramedullary nailing requiring intraoperative fluoroscopic guidance. The ‘‘free-hand technique’’ is not sufficiently accurate, because it is conducted by repeated trials using two-dimensional fluoroscopic images. Determining the distal hole using fluoroscopy requires even more radiographic exposure. The greatest level of radiation exposure to the patient and operating room staff was recorded during intramedullary nailing that involves distal interlocking. In an attempt to reduce this potentially harmful radiation burden, several authors proposed fluoroscopy-independent techniques, by proximally mounted target systems. Although these systems improve the operation accuracy, they still rely on radiographic images and the costs of configuring these systems are high.
IMNailScrewAR App
The app allows orthopedic surgeons unimpeded passage of the drill bit through the distal locking hole of a nail by :
watching the planes in AR to navigate in real time. After calibration - independent from nail manufacture - three perpendicular planes appear in AR passing with different hazy transparent colours through the distal holes. A passive sensor (PS) is used for calibration and acts also as a dynamic reference guide because which is constantly recognised and tracked and updating the position of all registered points to new one. In case the IM nail and the proximally mounted target PS is placed in different position as a whole array, all registered points are placed in new position respectively in AR. The surgeon now can navigate and accurate identify the direction of the drill bit toward the distal hole of the nail.
-observing a ‘’radar ‘’ screen which is depicted on screen and act as a navigation map during screw placement. A dot appears inside the radar screen reflecting the position of the tip in relation to half distance between two screw opening (in-out). Manipulating the handle of screw driver the dot size and location in “radar” screen change respectively, in real time. The size of the dot in radar screen is also changed according to the distance to insertion point. The dot in radar screen is starting to enlarge when the tip of the screw is approaching the entrance of hole namely the closer the tip of the screw the bigger the dot in the radar screen.
-seeing in real time current location of the tip of the screw, in relation to hole quadrants which are also depicted (left up, left down, right up, right down quadrant )
-observing current distance measured from tip of the screw during insertion - in mm - to the entrance hole. When the tip of the screw is approaching the proximal the entrance of the hole, distance is reduced. Once the tip of the screw attaches the centre of the entrance hole , distance becomes 0 in mm.
-seeing on screen three colourful columns representing coronal, transverse, sagittal planes, and angles also measured in real time between the previous mentioned planes and the respective planes of tip of the registered screw - values in normal range are taking green colour, otherwise red- which are continuously printed .
The Augmented Reality targeting App can be a robust guidance solution tool for education and training young fellows or residents in orthopaedic surgery by practicing over a benchmark with saw bone before use in real surgery and also might offer an easy to use, precise, low-cost, less time-consuming, complications and radiation-free, technique, allowing novice and expert surgeons alike to perform easily the distal locking screw placement.
Reference
-
Lee M-S, et al . A novel guiding device for distal locking of intramedullary nails. Sensors, 2012 IEEE.
-
Liao H, et al . Precision-guided surgical navigation system using laser guidance and 3D autostereoscopic image overlay. Computerized Medical Imaging andGraphics. 2010; 34(1):46–54.
-
Chu W, et al . Reducing radiation exposure in intra-medullary nailing procedures: Intra-medullary endo-transilluminating (iMET). Injury. 2009; 40(10):1084–7.
-
Nakdhamabhorn S, et al. A novel surgical navigation concept for Closed IntramedullaryNailing of femur using 4-DOF laser-guiding robot. Robotics and Biomimetics (ROBIO), 2011 IEEE International Conference on; 2011: IEEE.
-
Doke T, et al. Fluoroscopy-based laser guidance system for linear surgical tool insertion depth control. International journal of computer assisted radiology and surgery. 2015; 103):275–83.
-
Hoffmann M,et al. Electromagnetic navigation provides high accuracy for transcoracoid-transclavicular drilling. Knee Surgery, Sports Traumatology, Arthroscopy. 2014; 22(9):2237–42.
-
Goodall J. An image intensifier laser guidance system for the distal locking of an intramedullary nail. Injury. 1991; 22(4):339.
-
Krettek C, et al Deformation of femoral nails with intramedullary insertion. Journal of orthopaedic research. 1998; 16(5):572–5.
-
Leloup T, et al . A novel technique for distal locking of intramedullary nail based on two non-constrained fluoroscopic images and navigation. IEEE transactions on medical imaging. 2008; 27(9):1202–12
-
Moor B, et al. Distal locking of femoral nails. Mathematical analysis of the appropriate targeting range. Orthopaedics & Traumatology: Surgery & Research. 2012; 98(1):85–
How can I use this App
For successful accurate distal screw placement during IM nailing with the aid of App, three physical devices: (a) the Case i-Phone - Probe tool (CiPT), (b) the Clamp tool (CT), and (c) Passive sensor (PS) have to be 3D-printed, according to surgeon preference material (Stainless Steel, ULTEM or Peek which can be sterized according to manufacturer own specification and guidelines) - (STL or OBJ files can be downloaded for free from after requesting the relevant files for 3D printing
These devices have concrete dimensions, are calibrated for the app and are necessary for landmark registration and real time positional guidance.
The unsterilized i-Phone should be placed in a sterilized waterproof sealable bag. This is a common practice in surgical fields where non sterile parts are usually placed in sterile bags like arthroscopes, cameras, optical wires, tools etc. Sterile bags are readily available in operation rooms, taking always in to account bag manufacturer specification and guidelines. Plastic coverage should be transparent and not blur the iPhone cameras.
The surgeon should placed the iPhone in the case place of the Case i-Phone - Probe tool (CiPT) which is previously sterilised according to manufacture guidelines for the 3D printed material specification used.
In details:
A. Case i-Phone - Probe tool (CiPT). This device is dedicated for surface registration and the same instrument combined with the next instrument the clamp works as position detector. The tool consist of the case, in which the iPhone sits, and the rod. Rod has predefined length (32 cm) and it is attached perpendicular to the phones case, at the point where factory defined the centrum of the iPhone. This point is by default the origin in 3D dimensions namely 0,0,0 in XYZ axis. The other side of the rod is the side that acts as pointer tip location (z dimension). In the Case i-Phone - Probe tool (CiPT) the case should be according to the dimensions of iPhone model used.
B. Clamp tool (CT). This tool is a simple clamp attached to shaft of screw driver or burr tool that has a slot designed to receive perpendicular the pointer tip side of the case i-Phone-probe tool (CiPT). The clamp can be securely fit to screw drivers shaft trough a hole so rotation of the tool handle is allowed while screw or burring and navigation is possible.
C. Passive sensor (PS). This device is dedicated to act as dynamic reference array during tracking. Once 3d printed it should firmly attached to the proximal part of IM nail instrument or anatomical landmark.
A built-in known QR-code Image should be printed in a common colour printer preferably as adhesive label in order to be easily placed and attached firmly to passive sensor surface plane. The dimension of the printed adhesive QR-code Image Label should be four by four in centimetres in order to perfectly match with passive sensor surface.
build-in QR-code image exported in photo album of the device for color printing and then attached at passive sensor.
How it works. Method-technique
First the Case i-Phone - Probe tool (CiPT) - a device dedicated for surface registration - should be 3D printed, after downloading the appropriate 3D files according to user’s iPhone model from developer’s site. The device consist of the case, in which the iPhone sits and the rod. Rod has a predefined length (32cm) and it is attached perpendicular to the phone’s case. The other side of the rod is the side that acts as mechanical pointer tip location. The pointer tip sphere in augmented reality (AR) by the app should coincide with the mechanical pointer tip location of the rod in order to measure accurate.
Calibration is achieved by simply pressing the direction buttons (+,-) over the screen, the position (XYZ) of pointer tip sphere in augmented reality (red sphere) is adjusted accordingly. The user aim the pointer tip sphere in AR to be aligned in all dimensions and coincide over the mechanical pointer tip location of the rod of the i-phone in Case.
More specifically, by pressing the + or - button in the upper row, the (z) distance is adjusted respectively. It is recommended first to measure manually the distance from tip-pointer to case, by default this is 32 cm, and then calibrate the z distance with the real distance from the case to tip of the pointer. The following x, y - calibration procedure is in two dimensions over the screen’s phone (x,y), aiming to bring the presented red sphere to coincide optically with actual pointer tip. By pressing the + or - button in the intermediate row for the (y) distance (up-down), and likewise the + or - button in the last row for the (x) distance (left-right), the distances are adjusted.
Set up - General principles
For successful unimpeded passage of the drill bit through the distal locking holes of a given IM nail, it is necessary before operative insertion, the distal screws of the Nail to be calibrated. A built-in known QR-code Image should be printed in a common colour printer preferably as adhesive label in order to be easily placed and attached firmly to passive sensor surface plane. The dimension of the printed adhesive QR-code Image Label should be four by four in centimetres in order to perfectly match with passive sensor surface. The printed QR Image is intended to work as unique dedicated marker and it must be printed in certain size (four x four centimetres). The QR-code Image marker is captured and recognised continuously in real time by the App. Over the printed QR-code Image marker in augmented reality, a red sphere and a green plane appear laying over the surface plane once the QR-code Image is recognised. The QR-code Image marker is constantly recognised and continuously tracked by the app as along as the top centred button is on - highlighted. The Passive Sensor (PS) should be firmly mounted on the proximal part of the nail. The QR-code Image attached over PS acts as a dynamic reference base attached to the system. Prerequisite is visual contact through i-Phones screen of QR-code Image. The position of all future register points are updated continuously while the QR-code Image acts as a dynamic reference guide marker. In case the IM nail and the attached PS is placed in different position as a whole array e.g. placement intra medullary in tibia, the registered points in AR of screws are updated in new position respectively.
Calibration of the nail with the two respective distal screw holes should be done before inserting the nail outside, over the surgical table. Once the nail is inserted inside the bone. The PS is recognised by the app and the two registered screw holes are positioned accordingly over real patients tibia in AR. The surgeon precedes to drill passage by seeing the screen readings guided by the app without any fluoroscopic image acquisitions.
Nail Registration phase. A coloured marking sphere appears by touching any point over the screen in augmented reality during registration. By touching the tip pointer of the Case i-Phone - Probe Tool over the intended point and by touching the screen of iPhone simultaneously each time a coloured sphere appears in augmented reality.
In total two distal screws of IM Nail have to be registered sequentially, first the proximal screw and then distal screw. Two points required for each screw registration namely for proximal screw (P1 point - proximal hole of screw and P2 point - distal screw hole, green spheres) and for distal screw (D3 point - proximal hole of screw and D4 point - distal screw hole, light blue spheres).
The surgeon should select first to register the proximal screw (P1-P2) and then the distal (D3-D4) screw. Then, by pressing the corresponding button (P12) or (D34), you can select which screw you prefer first to place, and the selected button remains highlighted respectively. Between the P1P2 and D3D4 distance a purple sphere appears in the middle of the respective distances.
By registering the last point in augmented reality, four perpendicular planes appear passing with different hazy transparent colours one sagittal plane-blue - passing though two purple points, one coronal plane-red - passing through P1, P2, D3, D4, and two transverse planes-green - passing perpendicular to the rest of the planes. Depicting the planes in 3D space over the nail help to identify screw passage in augmented reality (AR).
By pressing the Go button Surgeon can view a grey cylinder projecting from red sphere at the clamp to yellow sphere superimposed in AR over the shaft of the real tool. The length of the presented in (AR) grey cylinder must end at the tip of the screw. The length of grey cylinder which ends at yellow sphere can be adjusted by pressing the + or - of L button over the screen in the same way, and must coincide with the tip of the real screw, in order to be calibrated and work properly.Three perpendicular planes around the tip appear with different hazy transparent colours (sagittal-blue, coronal-red, transverse-green) in augmented reality (AR) over the tip of screw extending around 20 mm for each plane over.
Radar screen. 2D topographical map according to the 3D depth of the corresponding 3D coordinate system (that is defined from the coloured planes), is depicted and act as a radar navigation map during screw placement - A 2D ‘dart board‘ like drawing is composed by concentric circles (latitude) divided by lines (longitude), and reflects the corresponding angle of the screw tip below point in 3D depth, related to the middle distance of P1P2 or D3D4 purple point accordingly. A green sphere now is reflected inside the radar screen as a dot. Manipulating the handle of screw driver the dot size and location in “radar” screen change respectively, in real time. The closer the yellow sphere which represents the real tip position of screw to proximal point P1 or D3 the larger the green dot size inside the radar screen respectively. The optimum for unimpeded screw placement would be when the location of the dot which represents the tip of the screw coincide with the center of the rad screen. A radar screen - A green line in Radar screen reflects the projected location where Longitude value is set to 0°. A Quadrant topographical map (left up, left down, right up, right down) has been developed. The quadrants are extracts from division of the coronal and the sagittal planes passing perpendicular over the corresponding insertion point of the screw at P1or D3 respectively. Each time the tip of the screw is inside the respective quadrant, the name of the relevant quadrant is printed in the screen accordingly.
Real time reading and measurements.
Over the screen also in real time the following dynamic values are presented, respectively:
a. The current location of the tip of the screw, in relation to Quadrants namely (left up, left down, right up, right down) quadrant.
b. Τhe current distance measured from the tip of the screw (yellow sphere) in mm to the point P1 or D3 - (Insertion hole) and current distance to P2 or D4 (output hole) is in mm. When the tip of the screw is approaching proximal to P1 or D3 the current distance is reduced, in real time in mm, depicted on screen. Once the tip of the screw coincide with point P1 or D3 the current distance becomes 0 in mm.
c. The row with red by red tiles, angle subtended by the coronal plane of tip (red) and coronal plane of the registered screw (red), real time angle value - green colour represent normal range 0°±3°, otherwise red.
d. The row with green by green tiles, angle subtended by transverse plane of the tip (green) and the transverse plane (green) of the register screw, real time angle value - green colour represent normal range 90°± 3°, otherwise red.
e. The row with blue by blue tiles, angle subtended by sagittal plane (blue) of the tip and the sagittal plane (blue), real time angle value - green colour represent normal range 90°±3°, otherwise red.
f. Dot reflects in Radar screen in real time also the current position of screw tip location and direction in 3D projection inside smaller semicircular subdivisions areas of the quadrants in relation to the purple point which is the middle of distance of P1P2 or D3D4. The size of the dot in radar screen is also changed according to the distance to insertion point P1 or D3. The dot in radar screen is starting to enlarge when the tip of the screw is in real time approaching the P1 or D3, the enlarge dot is setting off when the tip of the screw is closer to 50 mm in real time, the closer the tip of the screw the bigger the dot in the radar screen.
The powerful undo feature gives the user the freedom to make corrections without resetting the whole procedure. Simply by clicking the undo button the measurement returns to previous chosen point and thus registering once again the same anatomical landmark, without reseting the whole procedure and avoid starting over again.
In quick review:
The following points should be selected orderly and registered at circumference of two distal screw at the Intramedullary Nail before intramedullary insertion:
P1 - first, proximal, screw, proximal hole.
P2 - first, proximal, screw, distal hole.
D3 - second, distal, screw, proximal hole.
D4 - second distal, screw, distal hole.
Screen readings:
Buttons adjustments
Button P12 - Proximal screw hole - highlighted is active for screw placement.
Button D34 - Distal screw hole - highlighted is active for screw placement.
Button switch -on highlighted- Passive sensor (PS) is continuously tracked and recognised.
Undo button - backtrack to last selection.
Save button - screen contents save as an image to the photo album of the device.
Export button - build-in QR-code image exported in photo album of the device for color printing and then attached at passive sensor.
Calibration buttons
z + or - buttons in the upper row —> adjust the (z) distance-depth
y + or - buttons in the intermediate row —> adjust the (y) distance, up-down
x + or - buttons in the last row the + - button —> adjust the (x) distance, left-right
L + or - buttons enlarge or reduce length of grey cylinder ending to yellow sphere in order to be calibrated in AR with the actual screw driver shaft length, namely the tip of the real screw should coincide with the yellow sphere.
All information received from the software output must be clinically reviewed regarding its plausibility before patient treatment! The App indicated for assisting during operation the Operator. Judgment and experience are required to properly use the App. The software is not for primary image interpretation. General knowledge of the location of the pertinent neural and vascular structures is given based on acetabular quadrant system, while not anatomical absolute, should be used with extreme caution by the surgeon during surgery.
Any influence the operators in making decisions during operation remains Surgeons own responsibility and experience. A surgeon must always rely on his or her own professional clinical judgement when deciding whether to use a particular technique when treating a particular patient. App does not dispense medical advice. It is recommended that surgeons must be trained in the use before using it in real surgery.