Tag Archive for: validation studies

You are here: / validation studies

Posts

Article of the Week: Performance of robotic simulated skills tasks is positively associated with clinical robotic surgical performance

/by

Every Week the Editor-in-Chief selects an Article of the Week from the current issue of BJUI. The abstract is reproduced below and you can click on the button to read the full article, which is freely available to all readers for at least 30 days from the time of this post.

In addition to the article itself, there is an accompanying editorial written by a prominent member of the urological community. This blog is intended to provoke comment and discussion and we invite you to use the comment tools at the bottom of each post to join the conversation.

Finally, the third post under the Article of the Week heading on the homepage will consist of additional material or media. This week we feature a video from Alvin Goh, discussing his paper.

If you only have time to read one article this week, it should be this one.

Performance of robotic simulated skills tasks is positively associated with clinical robotic surgical performance

 

Monty A. Aghazadeh*,†, Miguel A. Mercado*,†, Michael M. Pan†, Brian J. Miles
‡ and Alvin C. Goh*,‡

 

*Methodist Institute for Technology, Innovation, and Education (MITIE), Houston Methodist Hospital, †Scott Department of Urology, Baylor College of Medicine, and ‡Department of Urology, Houston Methodist Hospital, Houston, TX, USA

 

Read the full article

Objective

To compare user performance of four fundamental inanimate robotic skills tasks (FIRST) as well as eight da Vinci Skills Simulator (dVSS) virtual reality tasks with intra-operative performance (concurrent validity) during robot-assisted radical prostatectomy (RARP) and to show that a positive correlation exists between simulation and intra-operative performance.

Materials and Methods

A total of 21 urological surgeons with varying robotic experience were enrolled. Demographics were captured using a standardized questionnaire. User performance was assessed concurrently in simulated (FIRST exercises and dVSS tasks) and clinical environments (endopelvic dissection during RARP). Intra-operative robotic clinical performance was scored using the previously validated six-metric Global Evaluative Assessment of Robotic Skills (GEARS) tool. The relationship between simulator and clinical performance was evaluated using Spearman’s rank correlation.

dffdbd

Results

Performance was assessed in 17 trainees and four expert robotic surgeons with a median (range) number of previous robotic cases (as primary surgeon) of 0 (0–55) and 117 (58–600), respectively (P = 0.001). Collectively, the overall FIRST (ρ = 0.833, P < 0.001) and dVSS (ρ = 0.805, P < 0.001) simulation scores correlated highly with GEARS performance score. Each individual FIRST and dVSS task score also demonstrated a significant correlation with intra-operative performance, with the exception of Energy Switcher 1 exercise (P = 0.063).

Conclusions

This is the first study to show a significant relationship between simulated robotic performance and robotic clinical performance. Findings support implementation of these robotic training tools in a standardized robotic training curriculum.

Read more articles of the week

Editorial: Robotic simulation: are we ready to go?

/by

In a study in this issue of BJUI, Aghazadeh et al. [1] were able to show that there was a significant relationship between simulated robotic performance and robotic clinical performance. The authors conclude that this supports the implementation of such robotic training tools in a standardized robotic training curriculum. Evidently, particularly in minimally invasive surgery, we must assess and learn from our surgical errors in order to prevent them in future [2]. For this, the dream scenario would be the ability to simulate a surgical procedure before being allowed to perform the real operation on the patient, similarly to pilots being trained on a flight simulator [3]. A robotic (virtual) simulator would be the ideal way to provide training on the respective robot-assisted procedure [1].

The authors are to be congratulated in their effort to prove the transfer validity of their training model using 12 previously validated robotic skill tasks consisting of four inanimate models and eight selected virtual reality tasks on the da Vinci Skills Simulator [1]. Endopelvic fascial dissection during robot-assisted radical prostatectomy was chosen to assess clinical robotic performance. Trainees were evaluated using the Global Evaluative Assessment of Robotic Skills (GEARS) scoring system, which distinguishes among depth perception, bimanual dexterity, efficiency, force sensitivity, autonomy, robotic control in relationship to the case difficulty using scores ranging from 1 to 5.

Nevertheless, there is still a long way to go! The authors have already conceded that there are some limitations to their study. It involved a small cohort of urological surgeons of different levels of expertise. It was not a randomized study comparing the impact of virtual simulator training on the surgical outcome, therefore, the authors could only show that they might be able to predict a surgeon’s performance during a real case based on his/her performance at the simulator. Based on this, the simulator training could theoretically pre-assess surgeon’s quality in robot-assisted surgery.

The authors probably chose endopelvic dissection because it represents a relatively easy part of robot-assisted radical prostatectomy. Accordingly, it remains unclear whether such basic tasks are really the best models to prepare a surgeon for this operation. Using low-fidelity training models, including vesico-urethral anastomosis, we were able to demonstrate the transfer validity of a six-step training programme [4]. It would be interesting to see the impact of the simulator training on clinical performance of robot-assisted vesico-urethral anastomosis because a valuable exercise for this already exists on the da Vinci Skills Simulator.

Despite this progress, we must recognize that we are far away from equalling the simulator training of pilots [3]. It is much easier to simulate turbulence during a flight than significant bleeding during robot-assisted surgery. Trials in simulating a complete laparoscopic procedure are still in their infancy [5]. This may differ with regard to other minimally invasive procedures such as ureteroscopy, where sophisticated trainer exercises exist; however, even in this field there is actually no proven evidence of the significant impact of simulators on shortening the learning curve. For all ‘hands-on’ simulation there will be always the problem of soft-tissue engineering and navigation [6]. Robotic simulators have the advantage compared with laparoscopy of three-dimensional video technology and the fact that there is no tactile feedback for the surgeon.

Evidently, information technology is continuously advancing and we may one day have a robotic simulator that can compare with the flight simulators used in aviation; however, to date, only training on basic skills can be provided and evaluated using the existing simulation systems in urology.

Read the full article
Jens Rassweiler
Department of Urology, SLK Kliniken Heilbronn, University of Heidelberg, Heidelberg , Germany

 

1 Aghazadeh MA, Mercado MA, Pan MM, Miles BJ, Goh AC.
Performance of robotic simulated skills tasks is positively associated with clinical robotic surgical performance. BJU Int 2016; 118: 475–81

 

2 Rassweiler MC, Mamoulakis C, Kenngott HG et al. Classification and
detection of errors in minimally invasive surgery. J Endourol 2011; 25:
1713–21

 

3 Sommer KJ. Pilot training: what can surgeons learn from it? Arab J Urol 2014; 12: 32–5
4 Klein J, Teber D, Frede T et al. Development, validation and operating room of a six-step laparoscopic training program for the vesicourethral anastomosis. J Endourol 2013; 27: 349–54

 

5 Shamim Khan M, Ahmed K, Gavazzi A et al. Development and
implementation of centralized simulation training: evaluation of
feasibility, acceptability and construct validity. BJU Int 2013; 111:
518–23

 

6 Baumhauer M, Feuerstein M, Meinzer HP, Rassweiler J. Navigation in
soft tissue surgery: perspectives and limitation. J Endourol 2008; 22: 751–66

 

Video: Performance of robotic simulated skills tasks is positively associated with clinical robotic surgical performance

/by

Performance of robotic simulated skills tasks is positively associated with clinical robotic surgical performance

Monty A. Aghazadeh*,†, Miguel A. Mercado*,†, Michael M. Pan†, Brian J. Miles
‡ and Alvin C. Goh*,‡

 

*Methodist Institute for Technology, Innovation, and Education (MITIE), Houston Methodist Hospital, †Scott Department of Urology, Baylor College of Medicine, and ‡Department of Urology, Houston Methodist Hospital, Houston, TX, USA

 

Read the full article

Objective

To compare user performance of four fundamental inanimate robotic skills tasks (FIRST) as well as eight da Vinci Skills Simulator (dVSS) virtual reality tasks with intra-operative performance (concurrent validity) during robot-assisted radical prostatectomy (RARP) and to show that a positive correlation exists between simulation and intra-operative performance.

Materials and Methods

A total of 21 urological surgeons with varying robotic experience were enrolled. Demographics were captured using a standardized questionnaire. User performance was assessed concurrently in simulated (FIRST exercises and dVSS tasks) and clinical environments (endopelvic dissection during RARP). Intra-operative robotic clinical performance was scored using the previously validated six-metric Global Evaluative Assessment of Robotic Skills (GEARS) tool. The relationship between simulator and clinical performance was evaluated using Spearman’s rank correlation.

dffdbd

Results

Performance was assessed in 17 trainees and four expert robotic surgeons with a median (range) number of previous robotic cases (as primary surgeon) of 0 (0–55) and 117 (58–600), respectively (P = 0.001). Collectively, the overall FIRST (ρ = 0.833, P < 0.001) and dVSS (ρ = 0.805, P < 0.001) simulation scores correlated highly with GEARS performance score. Each individual FIRST and dVSS task score also demonstrated a significant correlation with intra-operative performance, with the exception of Energy Switcher 1 exercise (P = 0.063).

Conclusions

This is the first study to show a significant relationship between simulated robotic performance and robotic clinical performance. Findings support implementation of these robotic training tools in a standardized robotic training curriculum.

Read the full article

Article of the week: Moving up a GEAR: evaluating robotic dry lab exercises

/by

Every week the Editor-in-Chief selects the Article of the Week from the current issue of BJUI. The abstract is reproduced below and you can click on the button to read the full article, which is freely available to all readers for at least 30 days from the time of this post.

In addition to the article itself, there is an accompanying editorial written by prominent members of the urological community. This blog is intended to provoke comment and discussion and we invite you to use the comment tools at the bottom of each post to join the conversation.

If you only have time to read one article this week, it should be this one

Face, content, construct and concurrent validity of dry laboratory exercises for robotic training using a global assessment tool

Patrick Ramos, Jeremy Montez, Adrian Tripp, Casey K. Ng, Inderbir S. Gill and Andrew J. Hung

USC Institute of Urology, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA

Read the full article
OBJECTIVES

• To evaluate robotic dry laboratory (dry lab) exercises in terms of their face, content, construct and concurrent validities.

• To evaluate the applicability of the Global Evaluative Assessment of Robotic Skills (GEARS) tool to assess dry lab performance.

MATERIALS AND METHODS

• Participants were prospectively categorized into two groups: robotic novice (no cases as primary surgeon) and robotic expert (≥30 cases).

• Participants completed three virtual reality (VR) exercises using the da Vinci Skills Simulator (Intuitive Surgical, Sunnyvale, CA, USA), as well as corresponding dry lab versions of each exercise (Mimic Technologies, Seattle, WA, USA) on the da Vinci Surgical System.

• Simulator performance was assessed by metrics measured on the simulator. Dry lab performance was blindly video-evaluated by expert review using the six-metric GEARS tool.

• Participants completed a post-study questionnaire (to evaluate face and content validity).

• A Wilcoxon non-parametric test was used to compare performance between groups (construct validity) and Spearman’s correlation coefficient was used to assess simulation to dry lab performance (concurrent validity).

RESULTS

• The mean number of robotic cases experienced for novices was 0 and for experts the mean (range) was 200 (30–2000) cases.

• Expert surgeons found the dry lab exercises both ‘realistic’ (median [range] score 8 [4–10] out of 10) and ‘very useful’ for training of residents (median [range] score 9 [5–10] out of 10).

• Overall, expert surgeons completed all dry lab tasks more efficiently (P < 0.001) and effectively (GEARS total score P < 0.001) than novices. In addition, experts outperformed novices in each individual GEARS metric (P < 0.001).

• Finally, in comparing dry lab with simulator performance, there was a moderate correlation overall (r = 0.54, P < 0.001). Most simulator metrics correlated moderately to strongly with corresponding GEARS metrics (r = 0.54, P < 0.001).

CONCLUSIONS

• The robotic dry lab exercises in the present study have face, content, construct and concurrent validity with the corresponding VR tasks.

• Until now, the assessment of dry lab exercises has been limited to basic metrics (i.e. time to completion and error avoidance). For the first time, we have shown it is feasibile to apply a global assessment tool (GEARS) to dry lab training.

Read more articles of the week

 

Editorial: Validating dry lab exercises for robotic surgical skills training

/by

Standardising and structuring of robotic surgery curricula: validation and integration of non-technical skills is required

Kamran Ahmed and Oliver Brunckhorst

Surgical simulation has advanced tremendously over the last two decades with the development of laparoscopic and robotic surgery. Because these procedures have a steep learning curve and because of the reduced training times experienced by trainees, safe adjuncts to operating room training are required [1]. Simulation training is a novel approach to surgical training and has been validated as a training and assessment tool and has been shown to improve a surgeon’s performance in the operating room.

In the present paper, Ramos et al. [2] evaluate the face, content, construct and concurrent validity of robotic dry laboratory (dry lab) exercises. They developed similar tasks to those included in the validated virtual reality da Vinci Skills Simulator using the da Vinci Surgical System in a dry lab environment. They also explored the applicability of Global Evaluative Assessment of Robotic Skills (GEARS) to assess dry lab performance. Good responses from the expert cohort with regard to realism and usefulness as a training tool confirmed the face and content validity of the dry lab exercises, whilst concurrent validity was also established, with experts outperforming novices in all but one of the individual metrics. Finally the simulator composite score achieved from the virtual reality simulator and GEARS scores from the dry lab exercises moderately correlated, thereby also establishing concurrent validity.

The present study raises an interesting question about the definitions of what constitutes an expert in robotic surgery. There are no consistent definitions of a ‘novice’ or an ‘expert’. A recent review has shown that the learning curve in the literature varies from 80 to 250 cases in robot-assisted laparoscopic prostatectomies, depending on previous open or laparoscopic experience and the outcome measures used [1]. With this in mind, Ramos et al. classified ‘experts’ as those performing >30 cases as the primary surgeon. This should be taken into consideration as the experts were so vital in establishing the face, content and construct validity of the dry lab curriculum. Additionally, although it has been shown that simulation models are valid and reliable for the initial phase of training and assessment in urological procedures, this is not the case for advanced and specialist level skill learning.

The present study provides validated dry lab exercises which could be incorporated into a robotic surgery training curriculum. With several simulators now having been validated for robotic surgery, integration of these within a structured robotic surgery curriculum is required. Steps have been taken to set out an effective curriculum similar to that which exists for laparoscopic surgery. The Fundamental Skills of Robotic Surgery is a simulation-based curriculum which has been validated and has a proven educational impact [3]. The Fundamentals of Robotic Surgery curriculum has also recently been laid out, but this is still in the process of validation [4]. One of the biggest initiatives in urology, however, was recently announced by the European Association of Robotic Urology Section Congress 2013 [5]. A multinational push to implement a standardized curriculum set out a multi-step curriculum which uses various techniques, including online theoretical training, simulation and observation and finally fellowship prior to certification, then allowing the surgeon to perform independent surgery. The next step for these curricula is for them to be fully validated and implemented, with the patient outcomes then being analysed.

Another aspect worth considering is the integration of non-technical skills within these curricula which has occurred in few of them. Full-immersion simulation (Fig. 1) offers the opportunity for training surgeons to develop critical skills such as communication, coordination and leadership. For the more experienced surgeon, at the later stages of a curriculum, full crisis simulation can help develop decision-making and team-working skills in difficult situations. It has been shown that integration of full-immersion simulation in simulation training is feasible and effective [6]. Surgeons performing robotic surgery need to develop the non-technical aspects alongside their technical skills, and full-immersion simulation certainly has a role to play in this and needs to be integrated within the curricula. The development of the Distributed Simulator offers a validated and low-cost method [7] of introducing full-immersion simulation into robotic surgery curricula and is an option that requires further investigation for its effectiveness in urology.

Figure 1. Aspects of non-technical skills developed through full-immersion simulation [8, 9].

In conclusion, the present study offers further examples of tasks that could be incorporated into robotic surgery training curricula. Whilst curricula exist, implementation and analysis of their effect on patient outcomes are the next steps. The integration of non-technical skills within these is important, and full-immersion simulation has an important role to play within robotic training curricula.

Kamran Ahmed*† and Oliver Brunckhorst*†

*MRC Centre for Transplantation, King’s College London, King’s Health Partners, and †Department of Urology, Guy’s Hospital, London, UK

Read the full article

References

  1. Abboudi H, Khan MS, Guru KA et al. Learning curves for urological procedures: a systematic review. BJU Int 2013; doi: 10.1111/bju.12315. [Epub ahead of print]
  2. Ramos P, Montez J, Tripp A, Ng CK, Gill IS, Hung AJ. Face, content, construct and concurrent validity of dry laboratory exercises for robotic training using a global assessment tool. BJU Int 2014; 113: 836–842
  3. Stegemann AP, Ahmed K, Syed JR et al. Fundamental skills of robotic surgery: a multi-institutional randomized controlled trial for validation of a simulation-based curriculum. Urology 2013; 81: 767–774
  4. Smith R, Patel V, Satava R. Fundamentals of robotic surgery: a course of basic robotic surgery skills based upon a 14-society consensus template of outcomes measures and curriculum development. Int J Med Robot 2013; doi: 10.1002/rcs.1559 [Epub ahead of print]
  5. Khan R, Ahmed K, Mottrie A et al. Towards a standardised training curriculum for robotic surgery: a consensus of an international multidisciplinary group of experts. Poster presented at the EAU Robotic Urology Section Congress Stockholm, Sep 3–5 2013
  6. Shamim Khan M, Ahmed K, Gavazzi A et al. Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity. BJU Int 2013; 111: 518–523
  7. Kassab E, Tun JK, Arora S et al. ‘Blowing up the barriers’ in surgical training: exploring and validating the concept of distributed simulation. Ann Surg 2011; 254: 1059–1065
  8. Flin R, Yule S, Paterson-Brown S, Maran N, Rowley D, Youngson G. Teaching surgeons about non-technical skills. Surgeon 2007; 5: 86–89
  9. Undre S, Sevdalis N, Healey AN, Darzi A, Vincent CA. Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg 2007; 31: 1373–1381
Read more articles of the week
© 2024 BJU International. All Rights Reserved.