Artificial intelligence (AI) is a rapidly evolving field which will have implications on both individual patient care and the health care system. There are many benefits to the integration of AI into health care, such as predicting acute conditions and enhancing diagnostic capabilities. Despite these benefits potential harms include algorithmic bias, inadequate consent processes, and implications on the patient-provider relationship. One tool to address patients’ needs and prevent the negative implications of AI is through patient engagement. As it currently stands, patients have infrequently been involved in AI application development for patient care delivery. Furthermore, we are unaware of any frameworks or recommendations specifically addressing patient engagement within the field of AI in health care.
We conducted four virtual focus groups with thirty patient participants to understand of how patients can and should be meaningfully engaged within the field of AI development in health care. Participants completed an educational module on the fundamentals of AI prior to participating in this study. Focus groups were analyzed using qualitative content analysis.
We found that participants in our study wanted to be engaged at the problem-identification stages using multiple methods such as surveys and interviews. Participants preferred that recruitment methodologies for patient engagement included both in-person and social media-based approaches with an emphasis on varying language modalities of recruitment to reflect diverse demographics. Patients prioritized the inclusion of underrepresented participant populations, longitudinal relationship building, accessibility, and interdisciplinary involvement of other stakeholders in AI development. We found that AI education is a critical step to enable meaningful patient engagement within this field. We have curated recommendations into a framework for the field to learn from and implement in future development.
Given the novelty and speed at which AI innovation is progressing in health care, patient engagement should be the gold standard for application development. Our proposed recommendations seek to enable patient-centered AI application development in health care. Future research must be conducted to evaluate the effectiveness of patient engagement in AI application development to ensure that both AI application development and patient engagement are done rigorously, efficiently, and meaningfully.
Artificial intelligence (AI) is a broad term referring to advanced computational methods that allow machines to mimic the functions of human cognition, such as learning or problem-solving [1]. AI is rapidly emerging as a technology that will impact numerous sectors, including health care. There is immense promise for AI to improve health care by surpassing the performance of health care providers, including assisting in the diagnosis of conditions such as melanoma and diabetic retinopathy [2, 3], predicting onset of acute conditions such as inpatient delirium or cardiac arrest [4, 5], and communicating with patients to address common questions as a chatbot [6, 7].
While AI has the potential to improve patient outcomes and health equity, potential harms exist. These include concerns about where patient data is being shared, the impact on the patient-provider therapeutic relationship, algorithmic bias, and the proper consultation of key stakeholders in AI development, among others [8]. Our ability to use AI in health has outpaced critical normative discussions among key stakeholders regarding how AI technologies should be responsibly developed and used in health care [9].
In Canada, the Canadian Institute for Health Research Strategy for Patient Outcomes Research defines patient engagement as “occurring when patients meaningfully and actively collaborate in the governance, priority setting, and conduct of research as well as in summarizing, distributing, sharing and applying its resulting knowledge” [10]. There is a growing consensus that patient engagement has a crucial role in health care delivery [11]. Effective patient engagement can improve patient outcomes, quality of life, and safety, as well as decrease hospital admissions and health care costs [12,13,14]. Engagement can also potentially lead to improvement in the acceptability of AI technology and support its transition into clinical practice [15]. Three in four patients feel that scientific developments should act in line with what is most important to patients and their families [16]. Patients are motivated to be involved in the development of new technologies as seen by our current understanding of the user-centered design space as being critical to developing digital health technology that users can and want to use [17]. Despite the goal of inclusive co-design, patients tend to still occupy passive roles in research through interviews or observations [18].
Specifically, within AI development, patient engagement has been largely overlooked. From a systematic review our research group is currently conducting, we found that very few studies on AI-related health care applications affecting patient care reported patient engagement in any form within its development. Additionally, of the patients who have been engaged, most identified as White, medically stable and have had a high enough educational attainment to have a fundamental understanding of AI [19]. We interpret these findings to be the product of a lack of a patient engagement framework or recommendations of how, when, and in what ways patients can be meaningfully engaged within this complex field. In doing so, we may work towards reaping the benefits of patient engagement within the field of AI and health care as a whole. Therefore, this articles aims to detail our findings and interpretations of how patient engagement can meaningfully be conducted within the field of AI, from the patient perspective, in addition to a preliminary framework for future patient engagement within this field.
This study utilizes an exploratory qualitative design using focus groups to engage patients on their perspectives of how patients can best be engaged in the development of AI in health care. We used the Sittig and Singh 2010 conceptual framework to guide our focus group question guide development [20]. As it currently stands, there are no conceptual frameworks addressing patient engagement research in AI, as such this paper will build towards the foundation of a conceptual framework to be utilized in future research.
This study was conducted virtually within Canada, and more specifically in the Greater Toronto area. Canada is a high-income country with a publicly financed single-payer universal health care system and diverse ethnicities. More than 85% of Canadians over age 12 have a primary care provider [21]. As of 2019, only 1.1% of Canadian physicians in any discipline reported using AI tools in patient care [22].
Participants were required to be 18 years of age or older, speak English, and have seen their health care provider in the last year. This definition included patients who have visited any type of provider, including nurse practitioners, social workers, physiotherapists, physicians and more. Given the setting of the study within the COVID-19 pandemic, participants required reliable devices and internet to participate in our virtual focus groups. Participants were on-boarded to the study via phone, consent was obtained verbally, and a demographic survey was administered. We employed the concept of maximum variation sampling – a technique otherwise used to identify dimensions of variation and selecting cases which fulfil this variation – to have diverse perspectives based on age, ethnicity, socioeconomic status, sex, gender, chronic illness, and geographical location across Canada.
Participants were recruited from posters in Unity Health Toronto family medicine clinics, social media (Twitter and Kijiji), and through emailing various community organizations.
Given the novelty and complexity of AI, patient education prior to study participation was critical. Prior to participating in focus groups, participants were asked to complete a 30-min educational module on the fundamentals of AI, including the different types of AI, examples of its uses in clinical practice, and important ethical considerations. This module was created in joint efforts with an educator at the University of Toronto to review for readability. After the completion of this educational module, participants were asked to complete a feedback survey. The revised module has since been published on the Rise 360 platform and is open access for public use (see Additional File 1).
Semi-structured 90-min focus groups were conducted virtually on Zoom from July–September 2021. Our focus group question guide included questions about what stage of development patient engagement should occur, what barriers exist to engagement, and what tools or training are necessary for patients, among others (see Additional File 2). Fieldnotes were recorded and focus groups were audio-recorded and transcribed verbatim. Data was managed in Microsoft Excel. Data collection ended when thematic saturation was attained.
Data collection and data analysis were performed simultaneously. This was a deductive qualitative analysis. We conducted a content analysis on all transcripts, where a team of coders (SA and JM) independently coded the same transcript manually using pre-conceived codes from the literature, and a codebook was developed. SA and JM both independently used the codebook in order to analyze the remaining transcripts line by line and each transcript analysis was combined across coders into main themes and subthemes. Member-checking with participants was performed as needed.
We have divided our results section into prominent areas of the patient engagement continuum, namely participant demographics, the need for patient engagement, patient recruitment, timing of engagement, engagement methods, patient education/training, the overall engagement process, and evaluation of the engagement. Please see Table 1 for representative participant quotes for each section.
Some of the most prominent themes from our discussions with patients were: the need for patient engagement, education on AI, interdisciplinary collaboration in AI as it pertains to patient engagement, equity diversity and inclusion (EDI), and quality improvement in patient engagement.
It has been well-established that patient engagement is needed in health care, and the results of this study reiterate that it is similarly important in the field of AI in health care. Patient engagement, when done meaningfully, has the opportunity to ensure development is in alignment with patient needs, provides insight into reassuring patients concerns with respect to AI development, and create longstanding collaborative relationships between health care providers and the patients they serve.
AI development can be described as a life cycle comprising various stages from conception to production and implementation. One such model of this cycle is that developed by DeSilva and Alahakoon (2022) which discusses the CDAC AI life cycle comprising 3 phases: design, develop and deploy. The first stage within the design phase is problem identification [23]. Based on our data, we found that patients would prefer to be engaged at problem identification stages as a means of prioritizing and strategizing their own health needs. Participants within our study highlighted that there are many benefits to encouraging patient collaboration at problem identification stages including improvements to resource allocation. By partnering with patients in early stages, there may be benefits to avoiding future iterations of improvements due to latent patient feedback.
In addition to patient engagement being beneficial from an operational perspective, there are also benefits from an AI acceptability standpoint by both informing patients of technological advancements and being able to adequately address patients concerns through engagement [22, 23]. In a study investigating AI-led chatbot services in health care, researchers found that the employment of user-centered approaches to address patient concerns assisted in improving both user experience and utilization [24]. Within our study, patients discussed concerns of AI use, specifically as it pertains to data consent processes, representation of data and algorithmic justice, data privacy and storage ethics. These concerns have similarly been discussed in the literature [25]. To address these reservations, collaborative study design principles such as user-centered design and patient engagement can be used. It has been found that health and technology literacy contributes to people's perceptions of AI, and assists in building trust, further re-iterating the critical role AI education has in the ways in which patients interact with and adopt AI [26].
Patient engagement work should serve as a condition that needs to be met by AI researchers rather than an after-thought as it relates to the AI cycle of development.
During our study, we drew importance on educating our participants about AI prior to entering focus group discussions. We created an introductory AI educational module which discusses the fundamentals of AI.
A main take-away from this work was that AI is a new, rapidly-evolving, and complex field that often makes patients feel unprepared, uncomfortable, and uneducated, oftentimes not even trying to engage in such a complex topic. It has been proposed in the literature that in order to successfully engage patients, there must be patient orientation and education about the topic, and on-going support [27]. This statement may be that much more important with the steep AI learning curve, which may necessitate more rigorous and longitudinal training throughout the engagement process, with the creation of accessible and easily understood educational modalities, than engagement studies may speak on their own lived experiences of a disease, for example.
As it currently stands within the literature, there are many papers discussing the importance of educating health care providers about AI in medicine from the perspective of digital literacy, specifically as it concerns medical students, physicians and nurses; however, none of which discuss educational pathways for patients themselves [28,29,30,31,32,33,34]. While educating health care providers is critical from an AI stewardship perspective, the lack of accessible learning modalities for patients re-iterates paternalistic structures in medicine whereby health care providers are the holders of knowledge. This directly contrasts the general principles of patient engagement as listed by the Ontario Patient Engagement Framework, which serves to empower patients and permit self-advocacy [35]. We hope that with the creation of our module we may begin to create more accessible educational pathways for patients and the general public to learn more about AI.
The nature of AI development in health care is complex. There are a multitude of stakeholders within the field of AI including developers, data managers, clinicians, and ethicists, among others. In order to successfully research, develop and implement beneficial AI interventions, there must be interprofessional collaboration across these groups. As it currently stands however, there are no established strategies for interprofessional collaboration within the field of AI [36].
Within our discussions, participants acknowledged the importance of interdisciplinary collaboration and its potential integration in patient engagement methods by having experts answer patients' questions and provide contextual insight throughout the engagement process. While this may provide added benefit from an educational standpoint, and may assist in clarifying key concepts for patients, doing so may result in the creation of power dynamics. As such, if this method is to be adopted, these power dynamics must be mitigated. Data is power, and from which lends itself to being analyzed through decolonizing lenses of mitigating power-dynamics. One such indigenous research framework which both addresses and overcomes power dynamics of western research methodologies are Talking Circles. The purpose of Talking Circles is to build relationships across members of the Circle, share power, elicit stakeholder voice, sharing of ideas to solve problems and assist in shared design. The Circle method itself entails the researcher creating a safe space for participants to express viewpoints. This method places emphasis on the physical and spatial orientation of participants as equals, in addition to dedicating time to acknowledge individual participants' power and privilege in relation to the topic being discussed [36, 37]. As mentioned by Brown and Di Lallo, the Circle has potential to be used to mitigate power imbalances between participants and researchers, and among participants themselves [37]. General principles of talking circles can be applied within the context of overcoming power-dynamics within interdisciplinary collaboration work in AI, particularly those involving patients.
With respect to our recent systematic review demonstrating the lack of patient engagement specifically within marginalized communities, building longitudinal patient engagement relationships with members of marginalized communities is imperative. The inclusion of the voices of patients experiencing marginalization may serve as a method to combat the well-known implications that AI may have in worsening health inequities in health care, specifically with respect to a lack of represented in the data, and a lack of prioritization of anti-oppressive practices by AI researchers and developers [38]. A paper by Leslie et al. details the cascading effects of health inequities as they present in AI system development, namely the usage and perpetuation of discriminatory data and sampling bias, biased design and deployment, unethical applications of these biased models and the real world implications on health outcomes [39]. Often within our study we found participants referencing the earlier aspects of the cascade, with discriminatory data sampling and representation within the samples themselves and the broader picture of how this may impact their health. Additionally, participants discussed the importance of centering the patient’s experiences in patient engagement work as a skill to be valued, and in prioritizing the contributions of patients experiencing marginalization, AI engagement can be done meaningfully [40]. The notion of employing recruitment practices that purposely sample patients from marginalized communities is important; however, it is critical to avoid tokenistic practices when doing so [15]. In order to achieve this, research groups may instead collaborate with the communities they wish to engage in longitudinal relationships, with transparency and accountability to garner trust and improve patient engagement uptake. Building long-standing reciprocal relationships between researchers and patients being engaged can assist in fostering mutual respect, creating expectations and further informing future research priorities [41]. These research partnerships can also assist in mitigating language, socioeconomic, and cultural barriers which otherwise may impede patient participation in engagement. However, relationship building must be preceded by training from researchers themselves on anti-oppression and cultural humility [42].
The topic of quality improvement in patient engagement practices is an area of limited research both in terms of what to evaluate and how to do so [27, 40, 43, 44]. Currently, there are notable gaps in research assessing patient engagement, which may be attributable to the delayed impact patient engagement has on an individual and systems level, in addition to a lack of an agreed upon evaluation framework [40]. Furthermore, there is little research measuring the validity of indicators currently used for patient engagement. A study conducted by Vat et al. suggested the use of a coherent set of measures for effective patient engagement rather than a single measure such as recruitment rate [40].
In our study, patient participants described the importance of concordance of researcher and patient values as it concerns patient engagement research, specifically with respect to transparency, and trust as being important features of meaningful patient engagement. This finding has been supported by previous research suggesting 9 principles of quality improvement in patient engagement work, specifically discussing transparency, integrity, respect, and continuous re-evaluation [16]. From our study, subjective and objective measures of determining patient engagement success were outlined. Subjectively, participants outlined the use of surveys, either at the end of the engagement process, or as a continuum throughout the engagement process. Using these surveys, key areas of inquiry include the patient's subjective level of participation, preparedness for participation, and if they have in any way benefited from their participation. From this, we can understand that it is important that patients feel competent and heard when participating. Also emphasizing the importance of reciprocity in patient engagement, where the patients themselves can benefit from the process. Additionally, patients highlighted that a marker of quality patient engagement may rest in the researcher’s perceptions of the quality of the data itself. Objectively speaking, patients described that quality patient engagement may be measured by comparing the representation of social determinants of health within the sample of surveyed participants relative to the population as a whole. These principles can further reiterate and inform current models of patient engagement evaluation more broadly such as the Public and Patient Engagement Evaluation tool which includes a socio-demographic survey, an evaluation of communication and supports for participation, ability to express views while engaging, and perceived level of input/influence in the patient engagement initiative [43].
While patient engagement is critical for health care innovation, it is also important to acknowledge the time and financial resources required for its success which may cause tensions with research or clinical teams. Developing a better understanding of the markers for good patient engagement can assist in making the case to researchers and other stakeholders of its importance [40, 45, 46]. Given the significant investment and corresponding speed at which the field of AI is developing, it is critical to ensure that the implemented patient engagement practices are continuing to be evaluated. We argue that in this setting, evaluation is just if not as important as the methods of engagement themselves to ensure that patients are being appropriately consulted and that researchers are held accountable.
This study was a Greater Toronto Area-based study that emphasized the re-centering of the patient voice in artificial intelligence innovation in health care. Our study comes with several strengths. Our educational module for patient participants prior to the focus groups took an extra step in engaging patients to ensure they could contribute to thoughtful discussion. This module can now be freely used by other research teams and the public. We conducted an open-ended focus group approach where we primarily empowered patients to guide the discussion to topics that were important to them. We engaged a diverse group of patient participants based on age, socioeconomic status, and race, as well as various personal experiences with disability and chronic illness which may provide differing opinions on AI applications in their care. We believe our study results are generalizable to an international audience, but future research must be conducted in other countries with different health care and technology development systems.
This study does not come without limitations. First, we cannot completely eliminate the risk of introduction of bias to participants in this a priori study, where patient engagement is known to be beneficial in health research. To mitigate this bias, we used an open-ended focus group guide. Participants disagreed with one another, and questioned the need for patient engagement in AI in health research all together, making us comfortable that there was not strong social desirability bias or agreement bias in our cohort. To mitigate confirmation bias from our a priori literature review, we used multiple coders and acknowledged our role/goal of the study up front. Second, our patient participant sample was not representative of the general public’s educational background, which could have an impact on the understanding and perspectives on AI applications. Furthermore, our study sample did not contain any 1) self-identifying Indigenous people who may have unique views on AI and data in Canada, based on historical discrimination and colonialism, and they may further refine our proposed recommendations, or 2) people specifically identifying as caregivers, who are also under the definition of patient partner and have important perspectives for their loved one’s experiences. Given that the vast majority of participants were located within the Greater Toronto Area, future research should also seek out to engaging patient voices across the country and internationally.
In order to develop a holistic understanding of patient engagement practices in AI, we acknowledge the importance of incorporating the voices of an interdisciplinary group of participants, including health care workers and policy makers. Due to the ongoing pandemic and demands on health care workers at that time, their recruitment was not feasible and is an important area of future research. Despite this however, we believe the incorporation of interdisciplinary voices can assist in the further adaptations of our current guidelines on patient engagement in AI.
AI in health care is a field that will continue to see rapid developments and have long standing implications on health and the health care system. In order to ensure that innovation continues to meet the needs and address issues critical to patients, quality patient engagement is required. We hope that our research assists in starting a dialogue on effective, representative and inclusive patient engagement practices within the field of AI in health care so that it becomes the standard of innovation.
