KATHERINE JONES
Trust and Policy Implications In Predictive Modeling
This project is the result of a partnership between the Michigan Medicine Department of Learning Health Sciences
(DLHS) and the University of Michigan School of Public Health (SPH) Innovation Studio. As Policy Designer within
a multidisciplinary team of four including business, medicine, and design, I co-led research and strategy over the
course of a three month internship during the summer of 2018.
Problem
Approach
Interviews
Insights
Outcomes
Conclusion
Our objective was to identify key trust and policy implications for the use of a platform known as
the Knowledge Grid (KGrid). Intended for use in healthcare settings, KGrid is an online repository
of predictive models aimed at accelerating the time frame it takes medical research knowledge
to become common practice.
We utilized a human-centered design approach to explore our problem space including interviewing stakeholders, conducting secondary research, and leveraging design methods. For our design process we utilized the Double Diamond method focusing primarily on future strategy recommendations.
1
2
3
4
5
6
7
Expert
Interviews
Secondary
Research
Primary
Interviews
Public
Survey
Market
Analysis
Data
Synthesis
Final
Report
DISCOVER
DEFINE
DEVELOP
DELIVER
Model Persona Card
Porter's Five Forces
Stakeholder Map
Business Model Canvas
Model Journey Map
Speculative Design
Business Model
Final Report
Governance Model
Implementation Model
Stakeholder Persona
We conducted 52 interviews targeting stakeholders from diverse professional backgrounds including clinical providers, predictive model developers, healthcare administrators, IT personnel, and ethicists. Our interviews focused on gaining a deeper understanding of stakeholder workflow, experience with predictive modeling, and concerns regarding the use of clinical decision making tools to determine the desire for KGrid.
Early in the interview process we came to realize our participants were unable to clearly identify their own trust factors for evaluating predictive models. To supplement our interviews, we developed a generative research tool we call a Model Persona.
Model Persona. Concept developed collaboratively. Designed by Sam Bertin.
Used as a prompt for participants, the card assisted in gathering deeper insights into how clinical providers evaluate and trust predictive modeling tools in current practice. We entered each interview with four of these cards, all with varying information (e.g. varied patient population, levels of validation, different reviews, different institutions, and varied update date). We then displayed one card at at time and asked our participants to identify anything negative or untrustworthy.
It became evident clinical providers have a clear and consistent process to vetting and trusting predictive models. The following describes the stages with which a clinician evaluates a model:
1. Population Data
Is the data used to create and validate the model similar to the
patient population of health system seeking a new model?
2. Evidence Based
What is the methodology behind the model?
Where do the case studies derive from? Is there proof of concept?
3. Peer Review
How is the evidence documented and is it peer reviewed?
If peer reviewed, what is the reputation and credibility of the journal etc.?
4. Current Use
What is the reputation of health system which developed the model?
How other health systems using the model?
5. Model Update
Are the health outcomes improving over time?
Has the science behind the model changed since inception?
6. Model Developer
What is the reputation and credibility of model developer?
Has the developer created additional models?
To articulate both the trust and policy barriers to implementing predictive models with the use of KGrid, we created a model journey map. This map shows the journey a health institution must take to create, verify, and implement a predictive model. Each stage describes the action needed to be taken and the roles involved. The content for this map was collected during our primary interviews.
Model Journey Map. Concept developed by Katherine Jones and co-designed with Sam Bertin
Process photos of Model Journey Map.
Throughout the model journey map we placed personas to articulate the roles, responsibilities,
and preferences of key stakeholders. Personas were also utilized as a research tool to aid in our understanding and development of the proposed governance model.
Personas developed collaboratively. Co-designed by Katherine Jones and Sam Bertin
For the implementation of KGrid, interviewees emphasized the need for mechanisms to ensure platform security. Expectations remain for the KGrid team to provide leadership in the establishment of early forms of governance until a formal multi-disciplinary governing body can be established. We developed a proposed governance model to be rolled out in three phases over the course of KGrid growth and future development.
Proposed Governance Model for KGrid. Concept and design by Katherine Jones.
We presented our final recommendations to our partners, the KGrid team, and stakeholders from DHLS and Michigan Medicine. Both our partners and the KGrid team are working towards incorporating our recommendations into the future development of KGrid.