Trust and Policy Implications In Predictive Modeling

This project is the result of a partnership between the Michigan Medicine Department of Learning Health Sciences
(DLHS) and the University of Michigan School of Public Health (SPH) Innovation Studio. As Policy Designer within

a multidisciplinary team of four including business, medicine, and design, I co-lead research and strategy over the 
course of a three month internship during the summer of 2018.

Problem

Approach

Interviews

Insights

Outcomes

Conclusion

Our objective was to identify key trust and policy implications for the use of a platform known as
the Knowledge Grid (KGrid).
Intended for use in healthcare settings, KGrid is an online repository
of predictive models aimed at accelerating the time frame it takes medical research knowledge
to become common practice.

We utilized a human-centered design approach to explore our problem space including interviewing stakeholders, conducting secondary research, and leveraging design methods. For our design process we utilized the Double Diamond method focusing primarily on future strategy recommendations.

1

2

3

4

5

6

7

Expert
Interviews

Secondary
Research

Primary
Interviews

Public
Survey

Market
Analysis

Data
Synthesis

Final
Report

DISCOVER

DEFINE

DEVELOP

DELIVER

Model Persona Card

Porter's Five Forces

Stakeholder Map

Business Model Canvas

Model Journey Map

Speculative Design

Business Model

Final Report

Governance Model

Implementation Model

Stakeholder Persona

We conducted 52 interviews targeting stakeholders from diverse professional backgrounds including clinical providers, predictive model developers, healthcare administrators, IT personnel, and ethicists. Our interviews focused on gaining a deeper understanding of stakeholder workflow, experience with predictive modeling, and concerns regarding the use of clinical decision making tools to determine the desire for KGrid. 

Early in the interview process we came to realize our participants were unable to clearly identify their own trust factors for evaluating predictive models. To supplement our interviews, we developed a generative research tool we call a Model Persona

model personas

Model Persona. Concept developed collaboratively. Designed by Sam Bertin.

Used as a prompt for participants, the card assisted in gathering deeper insights into how clinical providers evaluate and trust predictive modeling tools in current practice. We entered each interview with four of these cards, all with varying information (e.g. varied patient population, levels of validation, different reviews, different institutions, and varied update date). We then displayed one card at at time and asked our participants to identify anything negative or untrustworthy.

It became evident clinical providers have a clear and consistent process to vetting and trusting predictive models. The following describes the stages with which a clinician evaluates a model:

1. Population Data

Is the data used to create and validate the model similar to the
patient population of health system seeking a new model?

2. Evidence Based

What is the methodology behind the model?
Where do the case studies derive from?  Is there proof of concept? 

3. Peer Review

How is the evidence documented and is it peer reviewed? 

If peer reviewed, what is the reputation and credibility of the journal etc.?

4. Current Use

What is the reputation of health system which developed the model?
How other health systems using the model?

5. Model Update

Are the health outcomes improving over time?
Has the science behind the model changed since inception?

6. Model Developer

What is the reputation and credibility of model developer?
Has the developer created additional models?

To articulate both the trust and policy barriers to implementing predictive models with the use of KGrid, we created a model journey map. This map shows the journey a health institution must take to create, verify, and implement a predictive model. Each stage describes the action needed to be taken and the roles involved. The content for this map was collected during our primary interviews. 

Map.png

Model Journey Map. Concept developed by Katherine Jones and co-designed with Sam Bertin

IMG-0013.JPG
IMG-0014.JPG
IMG-0016.JPG

Process photos of Model Journey Map. 

Throughout the model journey map we placed personas to articulate the roles, responsibilities,
and preferences of key stakeholders. Personas were also utilized as a research tool to aid in our understanding and development of the proposed governance model.

press to zoom

press to zoom

press to zoom

press to zoom
1/7

Personas developed collaboratively. Co-designed by Katherine Jones and Sam Bertin 

For the implementation of KGrid, interviewees emphasized the need for mechanisms to ensure platform security. Expectations remain for the KGrid team to provide leadership in the establishment of early forms of governance until a formal multi-disciplinary governing body can be established. We developed a proposed governance model to be rolled out in three phases over the course of KGrid growth and future development.

Artboard 1.png

Proposed Governance Model for KGrid. Concept and design by Katherine Jones.

We presented our final recommendations to our partners, the KGrid team, and stakeholders from DHLS and Michigan Medicine. Both our partners and the KGrid team are working towards incorporating our recommendations into the future development of KGrid.

"I've heard excellent feedback from the people who
got to see the final presentation. Really well done. 
Next steps are already in motion."

- Jodyn Platt, DLHS