Validation of a Novel Simulation-Based Test in Robot-Assisted Radical Prostatectomy
Abstract
Purpose: To investigate validity evidence for a simulator-based test in robot-assisted radical prostatectomy (RARP).
Materials and Methods: The test consisted of three modules on the RobotiX Mentor VR-simulator: Bladder Neck Dissection, Neurovascular Bundle Dissection, and Ureterovesical Anastomosis. Validity evidence was investigated by using Messick's framework by including doctors with different RARP experience: novices (who had assisted for RARP), intermediates (robotic surgeons, but not RARP surgeons), or experienced (RARP surgeons). The simulator metrics were analyzed, and Cronbach's alpha and generalizability theory were used to explore reliability. Intergroup comparisons were done with mixed-model, repeated measurement analysis of variance and the correlation between the number of robotic procedures and the mean test score were examined. A pass/fail score was established by using the contrasting groups' method.
Results: Ten novices, 11 intermediates, and 6 experienced RARP surgeons were included. Six metrics could discriminate between groups and showed acceptable internal consistency reliability, Cronbach's alpha = 0.49, p < 0.001. Test–retest reliability was 0.75, 0.85, and 0.90 for one, two, and three repetitions of tests, respectively. Six metrics were combined into a simulator score that could discriminate between all three groups, p = 0.002, p < 0.001, and p = 0.029 for novices vs intermediates, novices vs experienced, and intermediates vs experienced, respectively. Total number of robotic operations and the mean score of the three repetitions were significantly correlated, Pearson's r = 0.74, p < 0.001.
Conclusion: This study provides validity evidence for a simulator-based test in RARP. We determined a pass/fail level that can be used to ensure competency before proceeding to supervised clinical training.
Get full access to this article
View all available purchase options and get full access to this article.
References
1. Rawla P. Epidemiology of prostate cancer. Rev World J Oncol 2019;10:63–89.
2. Goldenberg MG, Lee JY, Kwong JCC, et al. Implementing assessments of robotic-assisted technical skill in urologic education: A systematic review and synthesis of the validity evidence. BJU Int 2018;122:501–519.
3. Morris C, Hoogenes J, Shayegan B, et al. Towards development and validation of an intraoperative assessment tool for robot-assisted radical prostatectomy training: Results of a Delphi study. Int Braz J Urol 2017;43:661–670.
4. Danish Prostate Cancer Group DaProCa: Clinical guideline for radical prostatectomy in prostate cancer [In Danish]. 2019.
5. Ficarra V, Wiklund PN, Rochat CH, et al. The European Association of Urology Robotic Urology Section (ERUS) survey of robot-assisted radical prostatectomy (RARP). BJU Int 2013;111:596–603.
6. MacCraith E, Forde JC, Davis NF. Robotic simulation training for urological trainees: A comprehensive review on cost, merits and challenges. J Robot Surg 2019;13:371–377.
7. Brewin J, Ahmed K, Challacombe B. An update and review of simulation in urological training. Int J Surg 2014;12:103–108.
8. Abboudi H, Khan MS, Guru KA, et al. Learning curves for urological procedures: A systematic review. BJU Int 2014;114:617–629.
9. Mcgaghie WC, Issenberg SB, Barsuk JH, et al. A critical review of simulation-based mastery learning with translational outcomes. Med Educ 2014;48:375–385.
10. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Mastery learning for health professionals using technology-enhanced simulation: A systematic review and meta-analysis. Acad Med 2013;88:1178–1186.
11. Bjerrum F, Thomsen ASS, Nayahangan LJ, et al. Surgical simulation: Current practices and future perspectives for technical skills training. Med Teach 2018;40:668–675.
12. Ghaderi I, Manji F, Soo Park Y, et al. Technical skills assessment toolbox a review using the unitary framework of validity. Ann Surg 2015;261:251–262.
13. Jørgensen M, Konge L, Subhi Y. Contrasting groups' standard setting for consequences analysis in validity studies: Reporting considerations. Adv Simul 2018;3:1–7.
14. Downing SM. Reliability: On the reproducibility of assessment data. Med Educ 2004;38:1006–1012.
15. Mazzon G, Sridhar A, Busuttil G, et al. Learning curves for robotic surgery: A review of the recent literature. Curr Urol Rep 2017;18:89.
16. Alemozaffar M, Narayanan R, Percy AA, et al. Validation of a novel, tissue-based simulator for robot-assisted radical prostatectomy. J Endourol 2014;28:995–1000.
17. Whittaker G, Aydin A, Raison N, et al. Validation of the RobotiX mentor robotic surgery simulator. J Endourol 2016;30:338–346.
18. Harrison P, Raison N, Abe T, et al. The validation of a novel robot-assisted radical prostatectomy virtual reality module. J Surg Educ 2018;75:758–766.
19. Sethi AS, Peine WJ, Mohammadi Y, et al. Validation of a novel virtual reality robotic simulator. J Endourol 2009;23:503–508.
20. Bjerrum F, Sorensen JL, Konge L, et al. Randomized trial to examine procedure-to-procedure transfer in laparoscopic simulator training. Br J Surg 2016;103:44–50.
21. Maruthappu M, Gilbert BJ, El-Harasis MA, et al. The influence of volume and experience on individual surgical performance: A systematic review. Ann Surg 2015;261:642–647.
22. Downing SM, Yudkowsk R. Validity and its threats. In: Yudkowsky R, Downing SM, eds. Assessment in Health Professions Education. London: Routledge Taylor & Francis Group, 2009, pp. 21–55.
23. Borgersen NJ, Naur TMH, Sørensen SMD, et al. Gathering validity evidence for surgical simulation: A systematic review. Ann Surg 2018;267:1063–1068.
24. Havemann MC, Dalsgaard T, Sørensen JL, et al. Examining validity evidence for a simulation-based assessment tool for basic robotic surgical skills. J Robot Surg 2019;13:99–106.
25. Goldenberg M, Lee JY. Surgical education, simulation, and simulators—Updating the concept of validity. Curr Urol Rep 2018;19:52.
26. Downing SM. Validity: On the meaningful interpretation of assessment data. Med Educ 2003;37:830–837.
27. Prebay ZJ, Peabody JO, Miller DC, et al. Video review for measuring and improving skill in urological surgery. Nat Rev Urol 2019;16:261–267.
28. White LW, Truong M, Lendvay TS, et al. Crowd-sourced assessment of technical skills: differentiating animate surgical skill through the wisdom of crowds. J Endourol 2015;29:1183–1188.
29. Kwong JC, Lee JY, Goldenberg MG. Understanding and assessing nontechnical skills in robotic urological surgery: A systematic review and synthesis of the validity evidence. J Surg Educ 2019;76:193–200.
Information & Authors
Information
Published In
Copyright
Copyright 2021, Mary Ann Liebert, Inc., publishers.
History
Published online: 12 August 2021
Published in print: August 2021
Published ahead of print: 10 March 2021
Published ahead of production: 2 February 2021
Topics
Authors
Author Disclosure Statement
No competing financial interests exist.
Funding Information
No funding was received for this article.
Metrics & Citations
Metrics
Citations
Export Citation
Export citation
Select the format you want to export the citations of this publication.
View Options
Get Access
Access content
To read the fulltext, please use one of the options below to sign in or purchase access.⚠ Society Access
If you are a member of a society that has access to this content please log in via your society website and then return to this publication.