perspectives 2017: sandra monteiro
TRANSCRIPT
Sandra MonteiroMcMaster University, Touchstone Institute
Plenary IV: Research on Impacts and Outcomes
Perspectives on Successful Transitions for Internationally Educated Health Professionals
Sandra Monteiro, PhDMarch 2, 2017
Impact and Outcomes
Measuring Impact – typically means measuring change
Measuring Outcomes – typically means measuring learning, skill, competence
2/22
What makes a transition program
successful?
3/22
What are the outcomes of a transition program?Customizing program evaluationSandra MonteiroDavid Rojas
4/22
Agenda
• Describe the pre-residency education program
• Who should define success?– Examples of studies on predicting and
measuring outcomes• What are we doing to measure
“successful” outcomes? But also “failed” outcomes?– Innovative way to think about program
evaluation
5/22
Pre-Residency Program at Touchstone Institute
Provide international medical graduates with an exceptional learning experience on medical training and practice in Canada with a focus on:
– Patient-Centred Communication
– Collaboration– Professionalism– Health Advocacy– (Medical Expert)
6/22
Our Curriculum
7/22
Defining Success Touchstone Institute?
– Integrate best and defensible practices in education and assessment
– Able to address stakeholders’ needs– Provide international medical graduates with an exceptional
learning experience
Medical Residents?– Preparation for a successful residency– Prefer not to attend
Public? Residency Programs? Regulators? – Residents are ready to provide high quality care
8/22
Markers of success in education
Focused feedback
Goal directed teaching
Opportunities for deliberate practice9/22
Markers of success in education Focused feedback
• Information on their strengths • Opportunities to improve • Not specific to specialty
Goal directed teaching• Whose goals should we evaluate?
Opportunities for deliberate practice• Small Group Simulations • Not everyone is equally engaged
10/22
Markers of success in residency
From 2005 to 2009, … the CMG pass rate was 95%, while the IMG pass rate was 76%
Can the PRP do anything to address this? What factors influence these outcomes?
*Thomson, G., & Cohl, K. (2011). IMG SELECTION. Independent review of access to postgraduate programs by international medical graduates in Ontario. Volume 1: Findings and Recommendations; Volume 2: Analysis and background.
11/22
Factors that influence outcomes
Selection and admissions
12/22
13/22
• Medical Council Evaluating Exam Scores– Simulated office orals– Short answer management problems
• Demographics of Medical Residents– Country of birth (Human Development Index)– First language– Professional experience; internship
13/22
• First language other than English related to positive outcome on exam
• Prior professional experience and country of birth, slight benefit for outside Canada
Studies of outcomes have an impact• The development of a multi-institution data
sharing, processing, and governance agreement between Ontario PGME Programs and the CFPC and RCSPC
14/22
Factors that influence outcomes
Challenges in social integration
15/22
Program Evaluation in Health Professions Education
18/22
• Identify what works• Often focused on satisfaction ratings
Program Evaluation in Health Professions Education
Outcome based
• Rigid model• Focus only
on one outcome
Theory based
• Rigid model• Acknowledges
complexity but is unable to capture it
Holistic program evaluation
•Flexible model•Understand emergence•Acknowledges and aims to capture complexity
18/22
Holistic evaluation
Parker, K. (2013). A better hammer in a better toolbox: Considerations for the future of program evaluation
Acknowledges and aims to capture complexity through emergent properties
How do we define emergence?
Emergent properties are those unexpected behaviors exhibited by the system that cannot be explained by elements, or the interaction of the components in the system.
Emergence from Design
•Director•Coordinator
Emergence from Expectations
•Students•Instructors
Stakeholders
20/22
Example of unintended outcomeNot engaging with simulated sessions
Trainee dissatisfaction with PRP
• Not enough specialty content; doesn’t apply
• Frustration with transition program, seen as a hurdle and a burden to overcome
• Lack of communication to trainees
21/22
Future Directions
• Communicate with all stakeholders to better define goals of the PRP and areas for improvement
• Address intended but failed outcomes and unintended negative outcomes
22/22
Thank you
2017 Perspectives SymposiumMarch 2, 2017
#Perspectives2017