Original Article

Predictive Validity of the ProMIS Hybrid Simulator in a Urology Residency Training Program

^1,2Amjad Alwaal, MD, MSc, FRCSC ¹Talal M. Al-Qaoud, MD, MSc ¹Richard L. Haddad, MD, ¹Tarek M. Alzahrani, MD, ¹Josee Delisle, RN, BSc, MSc ¹Maurice Anidjar, MD, PhD

• ¹Division of Urology, McGill University Health Centre, Montreal, Quebec, Canada
• ²King Abdul Aziz University, Jeddah, Saudi Arabia

• Submitted: Wednesday, December 17, 2014
• Accepted: Saturday, February 14, 2015
• Published: Sunday, February 22, 2015

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Abstract

Objectives

We assessed the predictive validity of ProMIS hybrid laparoscopic simulator in a urology residency program.

Methods

Between June 2008 and December 2011, we trained 14 urology residents on ProMIS, measuring 5 basic laparoscopic tasks (peg transfer, pattern cutting, EndoLoop placement, extracorporeal suturing, and intracorporeal suturing). Then, we compared their last performance on ProMIS to their first performance on a porcine laparoscopic nephrectomy model. Two independent urologic surgeons with laparoscopic experience rated the resident performance on the porcine models, and kappa test with standardized weight function was used to assess for inter-observer bias. Non-parametric spearman correlation test was used to compare each rater’s cumulative score with the cumulative score obtained on the porcine models in order to test the predictive validity of the ProMIS simulator.

Results

The kappa results showed acceptable agreement between the two observers amongst all domains of the rating scale of performance except for confidence of movement and efficiency. In addition, poor predictive validity of the ProMIS simulator was demonstrated.

Conclusions

We could not demonstrate the predictive validity for the ProMIS hybrid simulator in our urology residency program.

Key words

ProMIS, MISTELS, Laparoscopic simulator, MIS training

Introduction

Modern surgical practice has witnessed a major shift towards minimally invasive surgery (MIS) for obvious reasons related to reduce morbidity and less hospital stay. This has been accompanied by difficulties in surgical training pertaining to how properly train and evaluate surgical residents, as it was clear that open surgical skills do not correlate with laparoscopic surgical skills [1]. For ethical, clinical, and logistical reasons, the operating room (OR) is not the ideal place to start learning those basic laparoscopic skills [2 3]. Therefore, there has been an increasing demand for training simulators that can train the residents, assess their performance on those skills and predict their performance in the OR.

Many laparoscopic simulators have been introduced over the last decade. There are two types of surgical simulators: the virtual reality (VR) simulators and the physical (box) trainers [4]. The VR simulator comes with the disadvantage of lacking haptic feedback [5], and we have already demonstrated in 2012 the lack of construct validity of the LapSim (a VR simulator) [6].

ProMIS is an augmented reality simulator that belongs to the second group of simulators. In 2008, we have proven the construct validity of ProMIS to differentiate between junior and senior urology residents in our urology residency program [7]. In this article, we wanted to prospectively determine the predictive validity of ProMIS by comparing the performance of 14 urology residents in our program to their performance intra-operatively, assessed here through their performance in the porcine laparoscopic nephrectomy model.

Material and Methods

14 urology residents (PGY 1 to 3) residents at McGill University were enrolled in the study between June 2008 and December 2011. They underwent an extensive initial orientation prior to commencement of the study and then they practiced on ProMIS for 1 hour weekly. They had monthly assessment of their performance. They were trained and assessed in 5 different tasks (peg transfer, pattern cutting, EndoLoop placement, extracorporeal suturing, and intracorporeal suturing), based on the widely accredited and validated (Society of American Gastrointestinal and Endoscopic Surgeons, and AmericanCollege of Surgeons)McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) tasks [8]. The parameters of assessment were total time and smoothness of movement. The ProMIS hybrid simulator consists of a Toshiba®computer with a laparoscopic mannequin, which contains 3 camera tracking systems to identify instrument movement inside the simulator from 3 different angles.

The last assessment of the resident performance on ProMIS was compared with their first performance of porcine laparoscopic nephrectomy. Approval from McGill institutional board was obtained prior to the study. The operations were performed at the McGill wet labs and they were recorded on DVDs and their performance was then assessed independently and blindly by two urologic surgeons with an experience in MIS. They gave each resident a rating score from 1 (poor) to 5 (excellent) on 6 pre-defined rating scales of their psychomotor skills (Table 1). These rating scales were based on 2 previously published articles on the global assessment of intraoperative laparoscopic skills [5 9].

Table 1: Summary of 26 patients with a second malignancy (SM)

 

Unnecessary movements

1. Many unnecessary moves

2.

3. Some unnecessary moves

4.

5. Clear economy of movement and maximum efficiency

Confidence of movement

1. Repeated tentative awkward or inappropriate moves with instruments

2.

3.Competent use of instruments but occasionally stiff or awkward

4.

5. Fluent moves with instruments and no awkwardness

Depth perception

1.Constantly overshoots target, wide swings, slow to correct

2.

3.Some overshooting or missing of target, but quick to correct

4.

5.Accurately directs instruments in the correct plane to target

Bimanual dexterity

1.Uses only one hand, ignores non-dominant hand, poor coordination between hands

2.

3. Uses both hands, but does not optimize interaction between hands

4.

5.Expertly uses both hands in a complimentary manner to provide optimal exposure

Efficiency

1.Uncertain, inefficient efforts; many tentative movements; constantly changing focus or persisting without progress

2.

3.Slow, but planned movements are reasonably organized

4.

5.Confident, efficient and safe conduct, maintains focus on task until it is better performed by way of an alternative approach

Tissue handling

1.Rough movements, tears tissue, injures adjacent structures, poor grasper control, grasper frequently slips

2.

3.Handles tissues reasonably well, minor trauma to adjacent tissue (ie, occasional unnecessary bleeding or slipping of the grasper)

4.

5.Handles tissues well, applies appropriate traction, negligible injury to adjacent structures

We prospectively calculated the standardized cumulative score for each rater’s observations, and the performance on the porcine models for each resident. The first part of our statistical analysis involved the assessment of the agreement between the two independent urologic surgeons’ rating of the resident performance on the porcine model using kappa test with standardized weight function to assess for inter-observer bias, agreement, and disagreement. In general, a kappa value less than 0.2 is considered poor agreement, and a value in the range of 0.81 to 1.0 is considered very good agreement [10]. Box whisker plots displaying the inter-quartile range, median, and mode were also constructed. Secondly, we assessed the predictive validity of ProMIS in predicting the resident performance on the porcine model using non-parametric spearman correlation testing in order to compare each rater’s cumulative score with the cumulative score obtained on the porcine models. All statistical analysis was conducted using STATA version 11 (stata), and a p-value <0.5 was deemed significant

Results

As mentioned earlier, the data on 14 residents was analyzed. The kappa results demonstrated acceptable agreement between the two observers amongst all domains of the rating scale of performance except for confidence of movement and efficiency (Table 2). The highest kappa values on agreement were observed on bimanual dexterity and tissue handling and the box whisker plots are shown in (Figure 1)

Table 2. kappa test output for inter-rater agreement.

 

Scale domain	Agreement	Expected Agreement	Kappa value	P
Unnecessary movement	78.12%	66.02%	0.35	0.02
Confidence of movement	75%	65.62%	0.27	0.05
Depth perception	79.17%	72.66%	0.23	0.04
Bimanual dexterity	81.25%	65.62%	0.45	<0.001
Efficiency	71.88%	65.62%	0.18	0.15
Tissue handling	87.50%	79.95%	0.38	0.02

Figure 1. Box plots for unnecessary movement and confidence of movement.

In order to examine the predictive validity of ProMIS in predicting the performance on the porcine models, spearman testing between the each of the ProMIS assessment components and the porcine scores demonstrated poor correlation across all components (Table 3), all correlation p values >0.05), and therefore poor predictive validity.

Table 3: Correlation of ProMIS performance to intra-operative laparoscopic assessment

 

Component	Correlation co-efficient	P-value
Peg transfer	0.27	0.33
Pattern cutting	0.28	0.32
Endo-loop	0.06	0.88
Extracorporeal stitching	0.01	0.98
Intracorporeal stitching	0.06	0.82
Overall score	0.01	0.89

Discussion

Given its integration of a motion tracking system with real laparoscopic tasks, the ProMIS system is considered a hybrid system that was developed as a bridge between traditional box trainers and the VR simulators. This integration allows haptic feedback during performance of laparoscopic tasks, which was demonstrated to be of significant importance [11], while enabling an objective evaluation tool similar to the VR simulators.

Our previously published results were consistent with other studies in the field establishing construct validity of hybrid simulators. Bann et al, showed the ability of the ProMIS system to distinguish expert laparoscopic surgeons from novice surgeons [12]. In general surgery, Van Sickle et al further confirmed the construct validity of the hybrid simulator in distinguishing residents during a simple suturing task [13]. Others have also reached the same conclusions [14 15].

This study however could not demonstrate a predictive validity for this hybrid simulator in the context of a urology residency training program. Of the different types of validity, predictive validity is generally the least studied. In a gynecology residency training program, PGY1 residents who received laparoscopic training on a traditional trainer box performed significantly better on laparoscopic bilateral tubal ligation than their control counterparts [16]. To our knowledge, there have been no other studies in urology or surgical specialties that directly examined the predictive validity of hybrid simulators.

Our study had its own limitations, and it is possible that we could not demonstrate the predictive validity because of our small sample size. The use of one last performance on the simulator and comparing it to one initial performance on the prcine model is also a limitation. Taking an average of 2-3 performances can be a solution. Another limitation is that those residents were actively involved in a surgical training program, and an additional unmeasured exposure to laparoscopic surgery is most likely. We previously showed the lack of predictive validity for ProMIS when we compared a group of medical students’ performance on the hybrid simulator to their later performance on the robotic console, but there was a predictive validity in the subset of students that were trained on both, ProMIS and LapSim [17]. It is hard to extract direct applicable conclusions from that study; given the small sample size and that the study subjects were medical students rather than residents. In addition, their eventual performance was measured on a robotic rather than laparoscopic platform. However, it is always possible to examine the resident performance on both simulators together, ProMIS and LapSim, and compare it to their intra-operative performance. Further studies are required to investigate this hypothesis and examine other simulators in order to identify the best simulator that can achieve the goal of teaching the residents basic laparoscopic skills and properly evaluate their readiness for real operative performance.

Conclusions

We could not demonstrate predictive validity for the ProMIS hybrid simulator in our urology residency program when compared to intraoperative performance in a porcine model.

Authors' Contribution

AA and TA carried out the literature search, performed the analysis, and prepared the manuscript. RH and TMA prepared the scoring system and performed the analysis. JD designed the study and collected data. MA conceived the idea, designed the study, and edited the final manuscript. All authors read and approve the final manuscript for submission.

Conflict of Interests

The authors declare that there are no conflicts of interests.

Ethical Considerations

The study was approved by the institute ethics committee

Acknowledgement

None

References

[1]Figert PL, Park AE, Witzke DB, Schwartz RW. Transfer of training in acquiring laparoscopic skills. Journal of the American College of Surgeons. 2001;193(5):533-7 [Pubmed]

[2]Scott DJ, Bergen PC, Rege RV, Laycock R, Tesfay ST, Valentine RJ, et al. Laparoscopic training on bench models: better and more cost effective than operating room experience? Journal of the American College of Surgeons. 2000;191(3):272-83 [Pubmed]

[3]Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. The American Journal of Surgery. 1999;177(1):28-32 [Pubmed]

[4]Haluck RS, Krummel TM. COmputers and virtual reality for surgical education in the 21st century. Archives of Surgery. 2000;135(7):786-92 [Pubmed]

[5]Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. British Journal of Surgery. 2004;91(2):146-50.[Pubmed]

[6].Kovac E AR, Quirouet A, Delisle J, Anidjar M. Construct validity of the LapSim virtual reality laparoscopic simulator within a urology residency program. Can Urol Assoc J. 2012;6(4):253-9.

[7]Feifer A, Delisle J, Anidjar M. Hybrid Augmented Reality Simulator: Preliminary Construct Validation of Laparoscopic Smoothness in a Urology Residency Program. The Journal of Urology. 2008;180(4):1455-9.

[8]Fried GM, Feldman LS, Vassiliou MC, Fraser SA, Stanbridge D, Ghitulescu G, et al. Proving the Value of Simulation in Laparoscopic Surgery. Annals of Surgery. 2004;240(3):518-28 [Pubmed]

[9]Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. The American Journal of Surgery. 2005;190(1):107-13 [Pubmed]

[10]Acock A. A gentle introduction to stata.

[11]Seymour NE, Gallagher AG, Roman SA, O'Brien MK, Bansal VK, Andersen DK, et al. Virtual Reality Training Improves Operating Room Performance: Results of a Randomized, Double-Blinded Study. Annals of Surgery. 2002;236(4):458-64 [Pubmed]

[12]..Bann SD, Khan MS, Darzi AW. Measurement of Surgical Dexterity Using Motion Analysis of Simple Bench Tasks. World J Surg. 2003;27(4):390-4 {pubmed]

[13].Sickle KR, Iii DAM, Gallagher AG, Smith CD. Construct validation of the ProMIS simulator using a novel laparoscopic suturing task. Surg Endosc. 2005;19(9):1227-31 {pubmed]

[14].Aggarwal R, Hance J, Undre S, Ratnasothy J, Moorthy K, Chang A, et al. Training junior operative residents in laparoscopic suturing skills is feasible and efficacious. Surgery. 2006;139(6):729-34 [Pubmed]

[15].Broe D, Ridgway PF, Johnson S, Tierney S, Conlon KC. Construct validation of a novel hybrid surgical simulator. Surg Endosc. 2006;20(6):900-4 [Pubmed]

[16]..Banks EH, Chudnoff S, Karmin I, Wang C, Pardanani S. Does a surgical simulator improve resident operative performance of laparoscopic tubal ligation? American Journal of Obstetrics and Gynecology. 2007;197(5):541.e1-.e5.

[17].Feifer A, Al-Ammari A, Kovac E, Delisle J, Carrier S, Anidjar M. Randomized controlled trial of virtual reality and hybrid simulation for robotic surgical training. BJU International. 2011;108(10):1652-6.[Pubmed]