A three-pronged approach to evaluating robotic surgery

Introduction

Numerous factors influence a technology’s diffusion (1) and, as with most innovations, the adoption of technology in healthcare can be divided into three phases: (I) assessment (e.g., assessing efficacy, effectiveness under real-world conditions, costs), (II) deployment (e.g., training, incorporation into organizational culture and existing processes, impact on workflows), and (III) post-implementation monitoring (2). Robotic surgery has received a great deal of attention in the literature, yet barriers to its adoption remain (3,4).

Historically, the first elective laparotomy was accomplished by Ephraim McDowell in 1807 for an ovarian cyst (5). Revolutionary endoscopic techniques during the 19th and 20th centuries allowed surgeons to access the abdominal cavity via smaller “keyhole” incisions but, for many years, there was heavy resistance to changes in practice and only few early adopters welcomed laparoscopic surgery (6,7). Following the introduction of video-laparoscopy in the 1980s, the procedure was gradually accepted (6,7). However, while minimal invasive surgery (MIS) grew (8,9), its use was limited in complex cases such as the treatment of many gynecologic cancers, where the laparotomy remained the primary surgical approach (4,10-13).

The next major evolution came with the development of a computer-assisted “robotic” interface (4,13), approved by the United States Food and Drug Administration (FDA) in 2000 and cleared for use in gynecology in 2005 (13). At the time that robotic surgery was being introduced at our center [2007], the state of MIS for the primary treatment of gynecologic cancers had remained very limited (15% of confirmed endometrial, 0% cervical, and 0% ovarian cancer cases, similar to published data (10,14). The implementation of a robotics program was expected to drive the use of MIS upwards by rendering the technique more feasible for surgeons, thereby allowing more patients to benefit from the procedure. While there was early traction for robotics in gynecological procedures, field data was lacking. The start of the robotics program at our center was therefore implemented as part of a clinical research mandate to evaluate its use.

The current article outlines the results of this twelve-year research. A three-pronged approach was undertaken to evaluate how robotic surgery affected: (I) surgical and clinical outcomes, (II) patient-reported outcome measures, and (III) implications for the hospital including resource use and costs.

Surgical and clinical outcomes

Endometrial cancer

A critical first step when evaluating a new intervention in oncology is to measure its impact on survival and cancer recurrence. All patients who underwent robotic staging were compared to a historical cohort of patients operated via laparotomy, and results demonstrated lower recurrence rates in the robotic cohort within two years of surgery (15). In a subsequent study, propensity score matching on the basis of several risk factors was employed. After 36 months of follow-up, no significant differences were found in recurrence rates or progression-free survival (odds ratio of 0.6 for the robot cohort relative to the historical laparotomy cohort, P=0.2). Patients in the robotic cohort trended towards higher overall survival though this did not reach statistical significance (odds ratio of 0.4, P=0.06).

Previously, we’ve corroborated some of the benefits associated with robotics including fewer postoperative complications (15), lower blood loss (15,16), and a shorter hospital stay (15,16). Some of the improvements in perioperative outcomes that we noted with the introduction of robotic surgery are highlighted in Figure 1. Furthermore, clinical benefits and outcomes were maintained among older patients (75 years of age or older) (19,20).

Figure 1 Perioperative outcomes following laparotomy and robotic surgery in gynecologic oncology. *, retrieved from Lau et al. [2012] (15) comparing pre-robotic era (n=133 open, n=27 laparoscopy) to robotic cohort (n=143) for endometrial cancer; ^†, retrieved from Matanes et al. [2018] (17) comparing open to robotic surgery for cervical cancer; ^‡, unpublished results comparing open surgery cases in the pre-robotic era (prior to the first use of robotics for ovarian cancer) to robotic surgery for stage III–IV ovarian cancer, based on data reported by Abitbol et al. [2019] (18).

Since 2010, we studied the value of sentinel lymph node dissection compared to full lymphadenectomy (21-25). We reported on detection rates of sentinel nodes using a mixture of tracers including indocyanine green (ICG) (22), which can be visualized with near-infrared light using the robotic system’s integrated fluorescence imaging tool (Firefly^®) (26). The cohort of patients who underwent sentinel node dissection had improved recurrence-free survival at the pelvic sidewalls in comparison to a historical cohort where full lymph node dissection was performed (propensity-score-adjusted hazard ratio of 0.32, P=0.007) (25). We detected sentinel lymph nodes in atypical areas where lymph nodes are not routinely removed, such as the presacral space and along the hypogastric vein (23). The prognostic effect of this finding has yet to be studied.

In comparison to a matched historical cohort of patients who underwent laparotomy for endometrial cancer, patients who underwent robotic surgery were associated with a significantly diminished use of over-the-counter analgesics (acetaminophen, ibuprofen, naproxen), opioids (from 71 mg morphine IV equivalents in the historical laparotomy cohort to 12 mg in the robotic cohort), patient-controlled and/or continuous epidural analgesia (from 98% to 2%), and an associated reduction in direct costs related to pain treatments over the course of the postoperative hospital stay (16). Cohn et al. [2016] similarly demonstrated lower opioid consumption in the first 24 hours after robotic surgery compared to laparotomy (27). What’s more, some have demonstrated reduced opioid use following robotic compared to laparoscopic surgery for endometrial cancer (28,29) though others have found no significant difference (30). A potential reason to explain the findings of less pain after robotic surgery, as alluded to by Leitao et al. [2013], may be the reduced pressure of the robotic trocars against the abdominal wall (28).

Cervical cancer

In light of recent findings from a retrospective observational study (31) and a randomized controlled trial (32) demonstrating worse survival outcomes among patients who underwent minimally invasive radical hysterectomy compared to open surgery, we re-evaluated survival and oncologic outcomes in our patient population (17). Patients who underwent robotic-assisted radical hysterectomy for early stage cervical cancer were compared to a historical cohort of patients who underwent open surgery (17). No significant differences were found in overall survival or progression-free survival and patients in the robotic cohort tended to have fewer postoperative complications (P<0.001) including lower rates of wound complications, less intraoperative blood loss (median 82 vs. 528 mL, P<0.001), and a shorter hospital stay (median 1 day vs. 1 week, P<0.001) (see Figure 1) (17). This supported findings from an earlier study in our division comparing outcomes following robotic and open radical hysterectomy (33). Despite these results, data from the LACC trial are disconcerting (32) and although some have brought up certain shortcomings of the trial and the generalizability of the findings across settings (34-36), there is a need to make better sense of the data. As we continue to move towards an era of precision medicine, results from randomized controlled trials need to be interpreted carefully at the patient level (36).

Epithelial ovarian cancer

Unlike other indications, the role of robotics in the management of epithelial ovarian cancers is still under debate (37). Over three quarters of patients with ovarian cancer are diagnosed at an advanced stage (38,39). The paradigm for staging and debulking ovarian cancer remains maximal cytoreductive surgery (40-42), which has traditionally entailed full exploration of the abdominal cavity via laparotomy (43). Early reports have indicated that robotic surgery may be feasible in selected cases where complete cytoreduction is achievable (37,44-46), though the key factor is “selected cases” rendering direct comparisons between open and robotically-assisted debulking surgeries less credible.

In our patient cohort, overall and progression-free survival were superior among patients who were operated by robotic surgery for advanced ovarian cancer (18). However, unlike uterine and cervical cancer that saw a jump in rates of MIS at our center following the introduction of robotics [from 15% laparoscopy to over 95% MIS in endometrial cancer; 0% to 100% robotically assisted MIS in cervical cancer], the uptake of robotic surgery in ovarian cancer has been more moderate (from 0% to an annual peak of 71% across all stages) (18). To construct a more equitable comparison, we compared all the patients in the era before robotics, during which all patients had a laparotomy, to all the patients in the era where some had robotics and some had open surgery (18). For the entire cohort of patients with stage III or IV ovarian cancer before and after the introduction of robotics, there were no significant differences in overall survival (median 40 vs. 47 months before and after the introduction of robotic surgery, P=0.6) or progression-free survival (median 13 vs. 16 months, P=0.4) (18). While there are limitations to a pre-post analysis, the data indicates that oncologic outcomes did not seem to have been comprised after carefully selecting the patients who were offered robotic debulking surgery. Consistent with data in endometrial and cervical cancer, among patients who were considered suitable for robotic debulking surgery, perioperative outcomes were improved in patients with ovarian cancer in comparison to laparotomy cases performed before the application of robotics in ovarian cancer (Figure 1), including lower blood loss (145 vs. 507 mL among robotic cases and laparotomy cases in the pre-robotic era, respectively), lower transfusion rates (4% vs. 31% intraoperatively; 20% vs. 55% postoperatively), and a shorter hospital stay (median of 1 day vs. 1 week) (18).

Similar results were noted for early stage (I–II) ovarian cancers: after a medium follow-up of 50 and 69 months among robotic (n=29) and laparotomy (n=30) cases, respectively, there were no significant differences in overall or progression-free survival between the robotic and laparotomy cohorts as well as between the pre-robotic and robotic eras (unpublished data).

Patient-reported outcome measures

To address the shortcomings of traditional clinical endpoints, patient-reported outcomes have emerged to increase patient-centeredness (47-50) and enable patient empowerment (47,50). Patient-reported outcomes may include anything reported by patients including symptoms, quality of life (QOL), pain, satisfaction, and anything that patients find meaningful with regard to their health (47-49).

QOL, in itself, is also a broad construct that generally refers to an individual’s well-being (51,52). Conceptually, the idea behind QOL has appeared throughout history (51,52), and Western (53) as well as Eastern (54) philosophers have reflected on the meaning of happiness and/or pleasure. Health-related QOL refers to QOL as it pertains to health (49,51), and can be broken down into four dimensions: physical, functional, emotional/psychological, and social well-being (52).

During the first two years of our robotics program, our division undertook a pilot study on patients’ health-related QOL following robotic surgery for endometrial cancer in order to monitor the effects of the new intervention on their recovery (55). At their first postoperative visit three to four weeks after surgery, patients reported high satisfaction (93% were very satisfied with the surgery), reported resuming regular activities within 2 weeks (mean 11 days), and 67% reported feeling no pain (55). Subsequently, we introduced a new patient survey consisting of standardized and validated health-related QOL instruments as well as additional questionnaires to explore the impact of robotic surgery on other aspects of QOL (56). Using the FACT-G, a validated cancer-specific health-related QOL questionnaire where patients are asked a series of questions regarding how they have been feeling “during the past 7 days” (Cella et al., 1993) (57), it was noted that while overall health-related QOL scores dropped within the first week after surgery for any gynecologic cancer, they returned to preoperative levels within three weeks (Figure 2) (56). Similar results were noted for patient-rated pain: both pain severity and its interference with daily life dropped at the first follow-up one week after surgery and returned to baseline by the second follow-up three weeks after surgery (Figure 2) (58). In addition, patients reported high satisfaction with the surgery even at the first follow-up one week after surgery (58).

Figure 2 Impact of robotic surgery on quality of life and pain in gynecologic oncology. *, adapted from Abitbol et al., 2019 (58). Data represents mean pain severity and pain interference scores (on scale of 0 to 10 from least to most pain) using Brief Pain Inventory [BPI, see (59)]. ^†, adapted from Abitbol et al., 2014 (56). Data represents mean total cancer-related quality of life (QOL) score (on scale of 0 to 108 from lowest to highest QOL) using FACT-G questionnaire [see (52)].

The protocol of the previous study eventually served as a template for what would become a pan-Canadian multicenter trial to prospectively compare outcomes between laparotomy, laparoscopy, and robotic surgery for apparent early stage (clinical stage I–II) endometrial cancer (GOC2; ClinicalTrials.gov Identifier: NCT01480999). Though subjects were not randomized, in comparison to laparotomy, the MIS arm (laparoscopy and robotics) had significantly better HRQOL outcomes across a variety of measures in the short term (1 and 3 weeks) after surgery (60,61). The FACT-G, as described above, as well as the EQ-5D, a commonly used preference-based instrument to assess generic health status (61,62), demonstrated sustained differences (P<0.05) in favor of MIS up to three months postoperatively (61). The Brief Pain Inventory [BPI, adapted from (59)] was used as in our previous study (58), with the MIS cohort demonstrating lower pain scores at one week after surgery (P<0.05 for all aspects of pain severity) and less pain interference up to three weeks postoperatively (P<0.0001 and P=0.0008 one and three weeks after surgery, respectively) (61).

Hospital outcomes

Hospital costs were estimated for endometrial (15), cervical (33), and ovarian cancer, and across all three, robotic surgeries were, on average, less expensive than laparotomies in the immediately preceding historical cohorts. Excluding acquisition and maintenance costs of the robots, the average hospital costs decreased from $10.4 thousand to $7.6 thousand (2010 Canadian dollars, CAD) for the surgical treatment of endometrial cancer (15), $11.8 to $8.2 thousand (2010 CAD) for cervical cancer (33), and from $24 to $14.9 thousand (2017 CAD) for ovarian cancer. This data is summarized and adjusted for inflation in Table 1.

Across the three studies mentioned above, the driving force for lower costs in the robotic cohort was the reduced length of stay vis-à-vis laparotomy. The difference in the perioperative process times between open and robotic cases for the treatment of ovarian cancer is shown in Figure 3, illustrating that while robotic cases tended to take longer in the operating room, patients flowed through the recovery room and the inpatient unit at a faster pace. This prompted our team to take a bird’s eye view and evaluate how the introduction of the robotics program affected the entire gyn-oncology inpatient ward, including any and all admissions under the division (63). Despite a greater number of surgical admissions in the period encompassed by the robotics program, surgical admissions occupied less of the ward in terms of number of bed-days, reflecting the faster turnover of patients from the increased use of MIS (63). Implications of the findings suggest that the program helped accommodate the growth of the division and was associated with improved operational efficiency by the freeing up of surgical beds and decreased resource utilization (63). This was especially pertinent in our setting given capacity constraints (64).

Figure 3 Impact of robotic surgery on perioperative processes in ovarian cancer. Data adapted from Abitbol et al., 2019. Data illustrates the perioperative process flow times for patients who underwent robotic surgery and open surgery for ovarian cancer. Time patient spent in the operating room was approximated by total anesthesia time. Data labels are median number of hours. *, to limit the patient selection bias during the period where robotics was offered, the open surgery cohort represents patients who underwent laparotomy in the pre-robotic era prior to the first use of robotic surgery for ovarian cancer. PACU, Post-Anesthesia Care Unit.

To evaluate whether the robotics program, as a whole, was a worthwhile investment, capital budgeting methods were applied and the net present value (NPV) of the program was crudely estimated and compared to a hypothetical scenario wherein the robotics program was not implemented, i.e., what it would have cost to accomplish the same number of cases using the same case-mix of laparotomy and laparoscopy maintained throughout the years. From the perspective of the division of gyn-oncology alone, even after including the department’s share of the robots’ upfront costs and their annual maintenance costs, the NPV of the costs associated with the robotics program were less than that of the supposed status-quo where most surgeries would have presumably been accomplished by laparotomy, entailing a return on investment (ROI) of 15%.

Unlike a conventional NPV analysis (65), the above analysis was both retrospective and forward looking, sunk costs were included, and average hospital costs based on internal accounting data were used in place of proper incremental cash flows. Additional limitations include the possibility that the use of laparoscopy would have increased without robotics and that the difference in length of stay between MIS and laparotomy—the major source of cost savings—could have diminished owing to developments in surgical recovery processes like Enhanced Recovery After Surgery (ERAS) protocols (66), which would have in turn reduced the value proposition of robotic surgery for the hospital. Nevertheless, the analysis provides a framework for healthcare organizations to model out an investment in new medical interventions or equipment.

Discussion

The current article describes a synopsis of our division’s experience with robotic surgery. We took a three-pronged approach in our evaluation of the robotics program: clinical, patient-reported, and organizational and economic outcomes. The robotic system rapidly permeated our surgical practice and enabled the performance of MIS in cases where it was previously not offered or where it was clinically judged to be too high a risk in our patient population at the time (e.g., cytoreductive surgeries, radical hysterectomies, periaortic lymphadenectomies, patients with comorbidities like morbid obesity, etc.). Researchers in other specialties have similarly noted the expanding applicability of surgical robotics to new frontiers (67-70).

Donald Berwick has suggested how “in health care, invention is hard, but dissemination is even harder” (Donald M. Berwick, 2003, p. 1970) (71). The growth of our robotics program (over 1,500 surgeries to date) was driven by a continuous effort to track the relevant data on our use of robotics at both the patient level (including traditional clinical endpoints, patient-reported outcomes, care trajectories) and the hospital level (including workflow, resource utilization, costs). Across all surgically managed gynecologic cancers, the greater use of MIS was associated with satisfactory oncological results, improved operative and clinical outcomes, and notably improved patient-reported outcomes. The assessment of patient-reported outcome measures is increasingly being reported as a tool to complement traditional endpoints and to personalize measures of quality of care (47-50). While some have highlighted concerns with popular QOL instruments (72-74), in the absence of standardized instruments, it is nevertheless important to “ask the patient” (Fayers and Machin, 2002:42) (51) to ensure a patient-centered assessment of care.

This meticulous data-driven approach to evaluate the impact of our interventions follows the philosophy of Ernest Codman, a pioneer of outcomes tracking who, at the time (early 1900s), faced enormous backlash by his peers for unremittingly recording and publishing treatment outcomes including surgical errors (75). As many healthcare organizations move towards value-based payment models (76-78), diligent outcomes management is as important as ever to ensure quality care (78,79) and a positive patient experience (79).

With hospitals under constant financial pressure, the effects of introducing a new technology on costs are important to measure internally. At the hospital level, the average costs of robotic surgeries were found to be less expensive than those of open surgeries in our retrospective cohorts. Generally, a cost-effective intervention is defined as one where the incremental outcomes outweigh the incremental costs of that intervention over another and it is judged to be worth the added costs (80). This was not an objective of the current analyses and no normative assertions were meant regarding the adoption of robotic surgery. In our setting, with decent multi-specialty usage of the robotic platform and a large delta between the length of hospital stay between robotic cases and open surgeries in the historical cohorts, the robotics program was found to be cost saving. While the assumptions underlying our findings are important to take into account (e.g., retrospective analyses and use of historical controls, changes in clinical practice that would have diminished postoperative length of stay and resource use, depreciation and amortization assumptions of the robots and their routine maintenance, etc.), the robotics program was found to at least not be a white elephant in our setting. Nonetheless, contextual factors are necessary to accurately appraise the adoption of any new technology. Fittingly, the Donabedian model provides a framework to measure quality of care at three levels: outcomes, process, and structure (81).

In addition to the outcomes described in this review, “not everything that can be counted counts, and not everything that counts can be counted” (William Bruce Cameron), and organizations often consider other factors when deciding to adopt a technology like robotic surgery, such as the need to remain competitive (82) and as a way to market themselves (83).

The Institute for Healthcare Improvement developed the Triple Aim framework to address policy objectives across three dimensions: population health (i.e., health outcomes), patient experience, and costs (84). We are now seeing an evolution from the Triple Aim to the Quadruple Aim to also incorporate the well-being of healthcare providers (85). Indeed, the introduction of a new medical technology greatly affects the user experience. Gaster described a general surgeon’s second live laparoscopic cholecystectomy: “the surgeon was sweating . . . learning a new procedure had never been so difficult” as “the new technology prevented him from using some of his most basic surgical skills” (Gaster, 1993:1280) (86). While a new technology may always entail a learning curve, the rapid success of surgical robotics has largely been attributed to a superior interface (4,13,87,88). The advantages of robotics over laparoscopy, however, go beyond a more ergonomic interface. Rather, the integration of a computer system has implications for the development of surgical innovation (4,89-92).

Although robotic surgery has been swiftly adopted as a natural evolution of conventional MIS, it was originally spearheaded by aerospace and military institutions with the ambitious goal of high-tech long-distance surgery (93). Unlike conventional straight-stick laparoscopy, computer-supported surgical robotics can allow other innovative uses to be tacked on more easily. Some of the potential applications that have been described include enhanced training modalities using virtual reality (VR) (91,94) as well as machine learning (95) and deep learning (96) algorithms, image guidance (97), long-distance surgery (89,98), and even the automation of surgical tasks (92). As such, as surgery continues to digitize, we could anticipate leveraging advancements in the technology sector.

Conclusions

As with any surgical technology, adoption should be evidence-based and carried out cautiously (99). We outlined our experience with robotic surgery, initially implemented as part of a clinical research mandate, and rapidly proliferated to a majority of—yet carefully selected—surgical cases in our division. Continued research is encouraged to validate clinical, patient-reported, and economic outcomes of robotic surgery, to delineate for whom this option is most appropriate, and to establish ways to optimize processes and workflows around these surgeries. More upstream, ongoing research and development in the areas of surgical robotics and computer-assisted surgery are anticipated to further innovate the field.

Acknowledgments

Funding: This work was made possible in part by donations from the Israel Cancer Research Fund, The Anne-Marie and Mitch Garber Fund, and the Susan and Jonathan Wener Fund.

Footnote

Conflict of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://gpm.amegroups.org/article/view/10.21037/gpm.2019.07.04/coif). WHG serves as an unpaid editorial board member of Gynecology and Pelvic Medicine from Jun 2018 to May 2020. SL and WHG report other (support for proctoring) from Minogue Medical, personal fees from Intuitive Surgical, outside the submitted work. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All studies from our division that are referenced in the current review received research ethics approval from the institutional review board of Jewish General Hospital.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Rogers E. Diffusion of innovations. 3rd ed. New York: The Free Press, 1983.
2. Institute of Medicine (US) and National Research Council (US) Committee on New Approaches to Early Detection and Diagnosis of Breast Cancer. 7: Translating New Technologies into Improved Patient Outcomes. In: Joy JE, Penhoet EE, Petitti DB, editors. Saving Women's Lives: Strategies for Improving Breast Cancer Detection and Diagnosis. Washington, DC: National Academies Press, 2005.
3. Benmessaoud C, Kharrazi H, MacDorman KF. Facilitators and barriers to adopting robotic-assisted surgery: contextualizing the unified theory of acceptance and use of technology. PLoS One 2011;6:e16395 [Crossref] [PubMed]
4. Tinelli A, Malvasi A, Gustapane S, et al. Robotic assisted surgery in gynecology: current insights and future perspectives. Recent Pat Biotechnol 2011;5:12-24. [Crossref] [PubMed]
5. Ellis H. The first successful elective laparotomy. J Perioper Pract 2015;25:207-8. [Crossref] [PubMed]
6. Spaner SJ, Warnock GL. A brief history of endoscopy, laparoscopy, and laparoscopic surgery. J Laparoendosc Adv Surg Tech A 1997;7:369-73. [Crossref] [PubMed]
7. Kelley WE Jr. The evolution of laparoscopy and the revolution in surgery in the decade of the 1990s. JSLS 2008;12:351-7. [PubMed]
8. Begos DG, Modlin IM. Laparoscopic cholecystectomy: from gimmick to gold standard. J Clin Gastroenterol 1994;19:325-30. [Crossref] [PubMed]
9. Masoomi H, Nguyen NT, Dolich MO, et al. Laparoscopic appendectomy trends and outcomes in the United States: data from the Nationwide Inpatient Sample (NIS), 2004-2011. Am Surg 2014;80:1074-7. [PubMed]
10. Wright JD, Neugut AI, Wilde ET, et al. Use and benefits of laparoscopic hysterectomy for stage I endometrial cancer among medicare beneficiaries. J Oncol Pract 2012;8:e89-99. [Crossref] [PubMed]
11. Naumann RW, Coleman RL. The use of adjuvant radiation therapy in early endometrial cancer by members of the Society of Gynecologic Oncologists in 2005. Gynecol Oncol 2007;105:7-12. [Crossref] [PubMed]
12. Marcus HJ, Hughes-Hallett A, Payne CJ, et al. Trends in the diffusion of robotic surgery: a retrospective observational study. Int J Med Robot 2017;13: [Crossref] [PubMed]
13. Advincula AP, Wang K. Evolving role and current state of robotics in minimally invasive gynecologic surgery. J Minim Invasive Gynecol 2009;16:291-301. [Crossref] [PubMed]
14. Hoekstra AV, Jairam-Thodla A, Rademaker A, et al. The impact of robotics on practice management of endometrial cancer: transitioning from traditional surgery. Int J Med Robot 2009;5:392-7. [Crossref] [PubMed]
15. Lau S, Vaknin Z, Ramana-Kumar AV, et al. Outcomes and cost comparisons after introducing a robotics program for endometrial cancer surgery. Obstet Gynecol 2012;119:717-24. [Crossref] [PubMed]
16. Abitbol J, Cohn R, Hunter S, et al. Minimizing pain medication use and its associated costs following robotic surgery. Gynecol Oncol 2017;144:187-192. [Crossref] [PubMed]
17. Matanes E, Abitbol J, Kessous R, et al. Oncologic and surgical outcomes of robotic versus open radical hysterectomy for cervical cancer. J Obstet Gynaecol Can 2019;41:450-8. [Crossref] [PubMed]
18. Abitbol J, Gotlieb W, Zeng X, et al. Robotic surgery at the time of interval surgery for advanced ovarian cancer after neoadjuvant chemotherapy. Int J Gynecol Cancer 2019; In press.
19. Lavoue V, Zeng X, Lau S, et al. Impact of robotics on the outcome of elderly patients with endometrial cancer. Gynecol Oncol 2014;133:556-62. [Crossref] [PubMed]
20. Zeng XZ, Lavoue V, Lau S, et al. Outcome of robotic surgery for endometrial cancer as a function of patient age. Int J Gynecol Cancer 2015;25:637-44. [Crossref] [PubMed]
21. Press JZ, Gotlieb WH. Controversies in the treatment of early stage endometrial carcinoma. Obstet Gynecol Int 2012;2012:578490 [Crossref] [PubMed]
22. How J, Gotlieb WH, Press JZ, et al. Comparing indocyanine green, technetium, and blue dye for sentinel lymph node mapping in endometrial cancer. Gynecol Oncol 2015;137:436-42. [Crossref] [PubMed]
23. How J, Boldeanu I, Lau S, et al. Unexpected locations of sentinel lymph nodes in endometrial cancer. Gynecol Oncol 2017;147:18-23. [Crossref] [PubMed]
24. How J, Lau S, Press J, et al. Accuracy of sentinel lymph node detection fol-lowing intra-operative cervical injection for endometrial cancer: a prospective study. Gynecol Oncol 2012;127:332-7. [Crossref] [PubMed]
25. How J, Gauthier C, Abitbol J, et al. Impact of sentinel lymph node mapping on recurrence patterns in endometrial cancer. Gynecol Oncol 2017;144:503-9. [Crossref] [PubMed]
26. Rossi EC, Kowalski LD, Scalici J, et al. A comparison of sentinel lymph node biopsy to lymphadenectomy for endometrial cancer staging (FIRES trial): a multi-centre, prospective, cohort study. Lancet Oncol 2017;18:384-92. [Crossref] [PubMed]
27. Cohn DE, Castellon-Larios K, Huffman L, et al. A prospective, comparative study for the evaluation of postoperative pain and quality of recovery in patients undergoing robotic versus open hysterectomy for staging of endometrial cancer. J Minim Invasive Gynecol 2016;23:429-34. [Crossref] [PubMed]
28. Leitao MM Jr, Malhotra V, Briscoe G, et al. Postoperative pain medication requirements in patients undergoing computer-assisted (“Robotic”) and standard laparoscopic procedures for newly diagnosed endometrial cancer. Ann Surg Oncol 2013;20:3561-7. [Crossref] [PubMed]
29. Shashoua AR, Gill D, Locher SR. Robotic-assisted total laparoscopic hysterectomy versus conventional total laparoscopic hysterectomy. JSLS 2009;13:364-9. [PubMed]
30. Turner TB, Habib AS, Broadwater G, et al. Postoperative pain scores and narcotic use in robotic-assisted versus laparoscopic hysterectomy for endometrial cancer staging. J Minim Invasive Gynecol 2015;22:1004-10. [Crossref] [PubMed]
31. Melamed A, Margul DJ, Chen L, et al. Survival after minimally invasive radical hysterectomy for early-stage cervical cancer. N Engl J Med 2018;379:1905-14. [Crossref] [PubMed]
32. Ramirez PT, Frumovitz M, Pareja R, et al. Minimally invasive versus abdominal radical hysterectomy for cervical cancer. N Engl J Med 2018;379:1895-904. [Crossref] [PubMed]
33. Halliday D, Lau S, Vaknin Z, et al. Robotic radical hysterectomy: comparison of outcomes and cost. J Robot Surg 2010;4:211-6. [Crossref] [PubMed]
34. Bentley JR. Minimally-invasive radical hysterectomy for cancer of the cervix: the perspective of the society of gynecologic oncologists of Canada (GOC). J Obstet Gynaecol Can 2019;41:143-5. [Crossref] [PubMed]
35. Kanao H, Aoki Y, Takeshima N. Unexpected result of minimally invasive surgery for cervical cancer. J Gynecol Oncol 2018;29:e73 [Crossref] [PubMed]
36. Cantrell LA, Pfisterer J, Boggess J, et al. Interpreting randomized clinical trials in gynecologic oncology surgery: does one size fit all? Am Soc Clin Oncol Educ Book 2019;39:342-50. [Crossref] [PubMed]
37. Lucidi A, Chiantera V, Gallotta V, et al. Role of robotic surgery in ovarian malignancy. Best Pract Res Clin Obstet Gynaecol 2017;45:74-82. [Crossref] [PubMed]
38. Howlader N, Noone A, Krapcho M, et al. SEER Cancer Statistics Review, 1975-2014. National Cancer Institute Bethesda, MD. Available online: https://seer.cancer.gov/archive/csr/1975_2015/results_merged/sect_21_ovary.pdf, based on November 2016 SEER data submission, posted to the SEER web site, April 2017.
39. Doubeni CA, Doubeni AR, Myers AE. Diagnosis and management of ovarian cancer. Am Fam Physician 2016;93:937-44. [PubMed]
40. Griffiths CT. Surgical resection of tumor bulk in the primary treatment of ovarian carcinoma. Natl Cancer Inst Monogr 1975;42:101-4. [PubMed]
41. Hoskins WJ, McGuire WP, Brady MF, et al. The effect of diameter of largest residual disease on survival after primary cytoreductive surgery in patients with suboptimal residual epithelial ovarian carcinoma. Am J Obstet Gynecol 1994;170:974-9; discussion 979-80. [Crossref] [PubMed]
42. Bristow RE, Tomacruz RS, Armstrong DK, et al. Survival effect of maximal cytoreductive surgery for advanced ovarian carcinoma during the platinum era: a meta-analysis. J Clin Oncol 2002;20:1248-59. [Crossref] [PubMed]
43. Schorge JO, Eisenhauer EE, Chi DS. Current surgical management of ovarian cancer. Hematol Oncol Clin North Am 2012;26:93-109. [Crossref] [PubMed]
44. Magrina JF, Zanagnolo V, Noble BN, et al. Robotic approach for ovarian cancer: perioperative and survival results and comparison with laparoscopy and laparotomy. Gynecol Oncol 2011;121:100-5. [Crossref] [PubMed]
45. Feuer GA, Lakhi N, Barker J, et al. Perioperative and clinical outcomes in the management of epithelial ovarian cancer using a robotic or abdominal approach. Gynecol Oncol 2013;131:520-4. [Crossref] [PubMed]
46. Chen CH, Chiu LH, Chen HH, et al. Comparison of robotic approach, laparo-scopic approach and laparotomy in treating epithelial ovarian cancer. Int J Med Robot 2016;12:268-75. [Crossref] [PubMed]
47. Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the patient-reported outcomes (PRO) harmonization group meeting at the Food and Drug Administration, February 16, 2001. Value Health 2003;6:522-31. [Crossref] [PubMed]
48. Weldring T, Smith SM. Patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs). Health Serv Insights 2013;6:61-8. [Crossref] [PubMed]
49. Osoba D. Health-related quality of life and cancer clinical trials. Ther Adv Med Oncol 2011;3:57-71. [Crossref] [PubMed]
50. Bergman S, Feldman LS, Barkun JS. Evaluating surgical outcomes. Surg Clin North Am 2006;86:129-49. x. [Crossref] [PubMed]
51. Fayers PM, Machin D. Quality of Life: The assessment, analysis and interpretation of patient-reported outcomes. 2nd ed. Chichester: John Wiley & Sons, 2007.
52. Cella DF, Tulsky DS. Quality of life in cancer: definition, purpose, and method of measurement. Cancer Invest 1993;11:327-36. [Crossref] [PubMed]
53. Russell B. A history of western philosophy. New York: Simon and Schuster, 1972.
54. Zhang G, Veenhoven R. Ancient Chinese philosophical advice: can it help us find happiness today? J Happiness Stud 2008;9:425-43. [Crossref]
55. Lau S, Aubin S, Rosberger Z, et al. Health-related quality of life following robotic surgery: a pilot study. J Obstet Gynaecol Can 2014;36:1071-8. [Crossref] [PubMed]
56. Abitbol J, Lau S, Ramanakumar AV, et al. Prospective quality of life outcomes following robotic surgery in gynecologic oncology. Gynecol Oncol 2014;134:144-9. [Crossref] [PubMed]
57. Cella DF, Tulsky DS, Gray G, et al. The functional assessment of cancer therapy scale: development and validation of the general measure. J Clin Oncol 1993;11:570-9. [Crossref] [PubMed]
58. Abitbol J, Lau S, Ramanakumar AV, et al. Evaluating postoperative pain and satisfaction among women treated by robotic surgery for gynecologic cancer. Gy-necol Pelvic Med 2019;2:6. [Crossref]
59. Cleeland CS, Ryan KM. Pain assessment: global use of the Brief Pain Inventory. Ann Acad Med Singapore 1994;23:129-38. [PubMed]
60. Ferguson SE, Gotlieb WH, Gien LT, et al. GOC2: A multicenter prospective trial comparing open, laparoscopic and robotic surgical outcomes in women with endometrial cancer. Part B: Patient-reported outcomes. Gynecol Oncol 2017;145:28-9. [Crossref]
61. Ferguson SE, Panzarella T, Lau S, et al. Prospective cohort study comparing quality of life and sexual health outcomes between women undergoing robotic, laparoscopic and open surgery for endometrial cancer. Gynecol Oncol 2018;149:476-83. [Crossref] [PubMed]
62. Brooks R. EuroQol: the current state of play. Health Policy 1996;37:53-72. [Crossref] [PubMed]
63. Leung A, Abitbol J, Ramana-Kumar AV, et al. Outside the operating room: how a robotics program changed resource utilization on the inpatient Ward. Gynecol Oncol 2017;145:102-7. [Crossref] [PubMed]
64. Sutherland JM, Crump RT. Alternative level of care: Canada’s hospital beds, the evidence and options. Healthc Policy 2013;9:26-34. [PubMed]
65. Damodaran A. Measuring Investment Returns. Available online: http://people.stern.nyu.edu/adamodar/pdfiles/cf2E/invret.pdf. (Accessed 29 April 2019)
66. Ljungqvist O, Scott M, Fearon KC. Enhanced recovery after surgery: a review. JAMA Surg 2017;152:292-8. [Crossref] [PubMed]
67. Gundlapalli VS, Ogunleye AA, Scott K, et al. Robotic-assisted deep inferior epigastric artery perforator flap abdominal harvest for breast reconstruction: a case report. Microsurgery 2018;38:702-5. [Crossref] [PubMed]
68. Boggi U, Signori S, De Lio N, et al. Feasibility of robotic pancreaticoduodenectomy. Br J Surg 2013;100:917-25. [Crossref] [PubMed]
69. Byrd JK, Duvvuri U. Current trends in robotic surgery for otolaryngology. Curr Otorhinolaryngol Rep 2013;1:153-7. [Crossref] [PubMed]
70. Escobar Dominguez JE, Ramos MG, Seetharamaiah R, et al. Feasibility of robotic inguinal hernia repair, a single-institution experience. Surg Endosc 2016;30:4042-8. [Crossref] [PubMed]
71. Berwick DM. Disseminating innovations in health care. JAMA 2003;289:1969-75. [Crossref] [PubMed]
72. Tax C, Steenbergen ME, Zusterzeel PL, et al. Measuring health-related quality of life in cervical cancer patients: a systematic review of the most used questionnaires and their validity. BMC Med Res Methodol 2017;17:15. [Crossref] [PubMed]
73. Carr AJ, Higginson IJ. Are quality of life measures patient centred? BMJ 2001;322:1357-60. [Crossref] [PubMed]
74. Campbell R, Quilty B, Dieppe P. Discrepancies between patients’ assessments of outcome: qualitative study nested within a randomised controlled trial. BMJ 2003;326:252-3. [Crossref] [PubMed]
75. Neuhauser D. Ernest Amory Codman MD. Qual Saf Health Care 2002;11:104-5. [Crossref] [PubMed]
76. Porter ME, Teisberg EO. Redefining health care: creating value-based com-petition on results. 1st ed. Boston: Harvard Business School Press, 2006.
77. Mahendraratnam N, Sorenson C, Richardson E, et al. Value-based arrangements may be more prevalent than assumed. Am J Manag Care 2019;25:70-6. [PubMed]
78. . AMCP partnership forum: advancing value-based contracting. J Manag Care Spec Pharm 2017;23:1096-102. [Crossref] [PubMed]
79. Manary MP, Boulding W, Staelin R, et al. The patient experience and health outcomes. N Engl J Med 2013;368:201-3. [Crossref] [PubMed]
80. Cohen JT, Neumann PJ. The cost savings and cost-effectiveness of clinical preventive care. Robert Wood Johnson Foundation. Available online: https://pdfs.semanticscholar.org/cfd7/3393b42d19f5d6418060189c0b506df1c215.pdf
81. Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005;83:691-729. [Crossref] [PubMed]
82. Aggarwal A, Lewis D, Mason M, et al. Effect of patient choice and hospital competition on service configuration and technology adoption within cancer surgery: a national, population-based study. Lancet Oncol 2017;18:1445-53. [Crossref] [PubMed]
83. Schiavone MB, Kuo EC, Naumann RW, et al. The commercialization of robotic surgery: unsubstantiated marketing of gynecologic surgery by hospitals. Am J Obstet Gynecol 2012;207:174.e1-7. [Crossref] [PubMed]
84. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood) 2008;27:759-69. [Crossref] [PubMed]
85. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med 2014;12:573-6. [Crossref] [PubMed]
86. Gaster B. A piece of my mind. The learning curve. JAMA 1993;270:1280. [Crossref] [PubMed]
87. Peiretti M, Zanagnolo V, Bocciolone L, et al. Robotic surgery: changing the surgical approach for endometrial cancer in a referral cancer center. J Minim Inva-sive Gynecol 2009;16:427-31. [Crossref] [PubMed]
88. Conrad LB, Ramirez PT, Burke W, et al. Role of Minimally invasive surgery in gynecologic oncology: an updated survey of members of the society of gynecologic oncology. Int J Gynecol Cancer 2015;25:1121-7. [Crossref] [PubMed]
89. Avgousti S, Christoforou EG, Panayides AS, et al. Medical telerobotic systems: current status and future trends. Biomed Eng Online 2016;15:96. [Crossref] [PubMed]
90. Rassweiler JJ, Autorino R, Klein J, et al. Future of robotic surgery in urology. BJU Int 2017;120:822-41. [Crossref] [PubMed]
91. Seixas-Mikelus SA, Kesavadas T, Srimathveeravalli G, et al. Face validation of a novel robotic surgical simulator. Urology 2010;76:357-60. [Crossref] [PubMed]
92. Shademan A, Decker RS, Opfermann JD, et al. Supervised autonomous robotic soft tissue surgery. Sci Transl Med 2016;8:337ra64 [Crossref] [PubMed]
93. Satava RM. Robotic surgery: from past to future—a personal journey. Surg Clin North Am 2003;83:1491-500. xii. [Crossref] [PubMed]
94. Schreuder HW, Wolswijk R, Zweemer RP, et al. Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 2012;119:137-49. [Crossref] [PubMed]
95. Hung AJ, Chen J, Gill IS. Automated Performance Metrics and Machine Learning Algorithms to Measure Surgeon Performance and Anticipate Clinical Outcomes in Robotic Surgery. JAMA Surg 2018;153:770-1. [Crossref] [PubMed]
96. Wang Z, Majewicz Fey A. Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery. Int J Comput Assist Radiol Surg 2018;13:1959-70. [Crossref] [PubMed]
97. Hughes-Hallett A, Mayer EK, Marcus HJ, et al. Augmented reality partial nephrectomy: examining the current status and future perspectives. Urology 2014;83:266-73. [Crossref] [PubMed]
98. Gong Q. China completes world’s first 5G remote surgery in test on animal. South China Morning Post. Available online: https://www.scmp.com/video/china/2176104/bureaucracy-still-hampers-academic-research-china-40-years-after-opening
99. Wilson CB. Adoption of new surgical technology. BMJ 2006;332:112-4. [Crossref] [PubMed]