AI, Dashboards, and Human Decisions: A Conversation with Melanie Tory

In this week’episode of the PolicyViz Podcast, I talk with Melanie Tory, Professor of the Practice at Northeastern University, about how people actually use dashboards in the real world — and why that use often looks very different from what designers intend. Her research reveals that dashboards frequently serve as a starting point for accessing data rather than tools for answering questions directly, with many users simply exporting data to Excel to do their real analytical work. We also explore her work on AI-enabled healthcare systems designed to help clinicians monitor patient risk in intensive care units, including how to visualize uncertainty in ways that busy medical teams can process quickly. And we close with a look at her emerging research on how people are beginning to use generative AI tools for data visualization tasks. It’s a thought-provoking conversation about the gap between the tools we build and the ways people actually work with data.

Resources

Guest Bio

Melanie Tory is Professor of the Practice at Northeastern University. Her team focuses on
empowering people to do more with data, through the design and evaluation of novel
visualization techniques, human-data interactions, and technology interfaces. She is especially
focused on visualizations for health and engineering applications, and the interplay between
visualization and AI. In her previous role at Tableau, Melanie managed an applied user research
team and conducted research in natural language interaction with visualizations, ultimately
commercialized as Tableau’s Ask Data feature. She also worked as a faculty member in
visualization at the University of Victoria, where she explored topics such as collaborative and
personal visual analytics. Melanie earned her PhD in Computer Science from Simon Fraser
University and her BSc from the University of British Columbia. Tory serves as Overall Papers
Co-Chair for IEEE Visualization 2025-26 and has previously served as associate editor for IEEE
Computer Graphics & Applications and IEEE Transactions on Visualization & Computer
Graphics. She was inducted into the IEEE VGTC Visualization Academy in 2025.

Different Ways to Listen to the Show!

New Ways to Support the Show!

With more than 250 guests and 10 seasons of episodes, the PolicyViz Podcast is one of the longest-running data visualization podcasts around. You can support the show by downloading and listening, following the work of my guests, and sharing the show with your networks. If you’re interested in financially supporting the show, you can sign up for my Patreon platform, make a one-time payment via PayPal, or shop one of the show’s sponsors.

Transcript

00:01.82
Jon
Well, hello, Professor.

00:03.64
Melanie Tory
Hello. thanks for having me on the show.

00:04.76
Jon
oh but oh It’s great to have you. Thanks for coming on. It’s been a while since I saw you last. I don’t even know, did I even see you at at Viz last year?

00:14.52
Melanie Tory
I was at Viz.

00:15.13
Jon
you

00:15.52
Melanie Tory
I think I might have seen you there, but I saw a lot of people there.

00:18.78
Jon
Yeah. A lot of people.

00:20.22
Melanie Tory
Yeah.

00:21.56
Jon
A lot of people. And at one point, a lot of people in a very small bar, which was which was super entertaining.

00:26.14
Melanie Tory
yeah

00:27.83
Jon
But ah yeah, I spent a lot of time. I think that was the West Coast Viz Party. ah I think the organizers were trying desperately to get rid of their drink tickets so they could go home or back to the hotel.

00:40.10
Jon
so um But

00:40.57
Melanie Tory
that That’s pretty funny. I stayed for a while, but not the whole thing.

00:42.87
Jon
um

00:45.94
Jon
but that was it.

00:46.53
Melanie Tory
I was pretty tired.

00:47.90
Jon
ah um Okay, so you’ve got a lot of work going on There’s one particular paper of yours that’s now ah you know two three years old that i I definitely want to talk about about dashboards. um But why don’t we start with introductions for folks who are not familiar with you or not familiar with your work? So you know where are you and and you know what does your work tend to focus on?

01:11.10
Melanie Tory
Yeah, sure. So I’m a professor of the practice at Northeastern University. and i’m based at the Rue Institute, which is Northeastern’s campus in Portland, Maine.

01:22.07
Melanie Tory
Northeastern actually has a bunch of different campuses all over the place. So this is one of them. And we’re a little bit unusual, I suppose you might say, in that we focus on three things.

01:35.13
Melanie Tory
One is research. So I lead a research team here. But we also have graduate education and we have an embedded entrepreneurship accelerator. So that makes us a little bit different as a university.

01:47.03
Melanie Tory
And part of our mission is to work really closely with industry and foster a bit of economic activity in the state of Maine to help grow the tech economy here.

01:59.16
Melanie Tory
So that means that my group does sort of pure research in data visualization and human data interaction. But we also do a lot of applied work with company partners, nonprofits, and so on to try to do a little bit more apply to practice applied and practical things than you might get in a normal university.

02:20.79
Jon
And as most of that work with the business communities that tend to be in Maine and New England, or is it just everywhere?

02:30.52
Melanie Tory
Could be anywhere, but we’ve mostly focused, yeah, Maine and New England.

02:34.30
Jon
Yeah. Yeah. Okay. That makes sense. That makes sense. um Okay. So I reached out initially. because I’ve had this question in my head for a while about whether dashboards are worth our time to create. um And I found your study with Lynn Bartram and others, finding their data voice practices and challenges of dashboard users.

02:56.54
Jon
um And then i recently posted a blog post on my thoughts about you know why, especially now in our current media environment, why I don’t think they’re they’re very useful. so um I want to dive into this, but I want to sort of maybe ask you first to give folks a summary of the work so they know kind of where we’re coming from.

03:18.68
Melanie Tory
Sure. So this particular study was an interview study we did with people who are users of dashboards. So these are, and and this was in particular dashboards that are internal to organizations. So this wasn’t looking at really at public facing dashboards, which might be a little different, but we were curious to know How are people actually using dashboards in practice?

03:45.37
Melanie Tory
And is that actual use aligned with the way that the visualization community and the visualization tool developer community is thinking that people are using dashboards?

03:58.36
Melanie Tory
I was working at Tableau Software at the time, so we were a dashboard building company.

03:58.46
Jon
Yeah. Mm-hmm. Mm-hmm.

04:03.64
Melanie Tory
And we had a lot of contact with analysts who build dashboards because they were our direct users. But we actually didn’t have much insight into the people who were ultimately using the dashboards because those were in some ways a hop away, right?

04:20.34
Melanie Tory
There are our customers’ customers, if you will.

04:23.19
Jon
he

04:24.31
Melanie Tory
And so we wanted to know, you know, Are the dashboards that they get meeting their needs? is it

04:31.54
Jon
he

04:32.25
Melanie Tory
Or is there more we could be doing to support these folks in working with data?

04:36.62
Jon
Right. Okay. And so this was primarily a qualitative study and you talked to 20, 24 people who are in a variety of sectors.

04:40.94
Melanie Tory
Right.

04:46.49
Melanie Tory
Yeah. Right. It was 20-ish people, and ralph the majority were dashboard end users, I think, Maybe seven of the 20 were analysts who were building dashboards because we also wanted the perspective of that what did analysts think that they could be doing to help their users be more effective.

05:07.89
Jon
Right.

05:11.83
Melanie Tory
Right. And so, yeah, it was across variety of sectors, across a variety of tools. We didn’t limit it to Tableau. We were interested in dashboards of any kind. dashboards, whether they’re Power BI or Tableau or custom built, largely all kind of function the same from the end user’s point of view anyway, so it didn’t matter.

05:30.49
Melanie Tory
And yeah, we were curious to know what works, what doesn’t, and are they being used the way we think they are?

05:30.55
Jon
Yeah.

05:38.74
Jon
So I want to give you a chance to talk about what works and what doesn’t, what the bottom line was, but I do find it striking that Tableau, but I’m guessing this is similar for lots of dashboarding companies and you know beyond Tableau and Power BI, that like the even though the client the the the user is two steps away, that that’s still not a focus issue.

06:03.58
Jon
of the I mean, i’m I’m guessing like the marketing team sort of focus on it, but like, I just find that fascinating that like the like, people still have to click, like, does the checkbox work? Does the dropdown work?

06:16.63
Jon
Like, I’m not sure I have a real question here for you, but like, I just find it fascinating that that was not a focus of the of the company.

06:23.45
Melanie Tory
Yeah, I’m with you on that. It was surprising to me too.

06:25.18
Jon
Yeah.

06:26.49
Melanie Tory
But in some ways it makes sense because the people who are deeply using the product are the analysts using the desktop product to build the dashboards.

06:26.52
Jon
Yeah.

06:34.04
Jon
Yeah.

06:36.57
Melanie Tory
And we can kind of assume that they know what their end users needs are.

06:40.60
Jon
Right.

06:42.36
Melanie Tory
And so that they’re also a harder population to reach, right?

06:42.78
Jon
Yeah.

06:46.24
Melanie Tory
Because we have all of the people who were in the database are all the analysts and the data admins and so on, and not the end users.

06:46.26
Jon
For sure.

06:50.23
Jon
Yeah.

06:53.85
Jon
Right. Right, right, right. Yeah, that that totally makes sense. and they’re And they’re the ones that are paying for the product. So I get it. It’s just, a yeah, anyway. Okay, so what’s the what’s the what’s the bottom line here? What did what did you all find?

07:10.62
Melanie Tory
Well, what was super interesting was we kind of found that dashboards weren’t being used the way that everybody in the biz community thought they were right.

07:21.17
Melanie Tory
We all kind of thought you, what the user will go to the dashboard and answer their questions and the dashboard, you know, the dashboard designer, that’s their role is to design the dashboard so that all the users questions can be answered.

07:21.34
Jon
ah

07:29.21
Jon
Mm-hmm. Mm-hmm. Mm-hmm.

07:37.18
Melanie Tory
And so we the model prevailing mental model was user goes to the dashboard, answers their question, all is done. But what we really found out is that it was almost like the dashboard was the portal to the data.

07:52.31
Melanie Tory
Like that was the place that the user started. They go there, get their data, and then they do whatever they need to do with it. And there’s some questions they can just answer there. That’s great. But there were a whole bunch of other things that people wanted to do the The dashboard was just the starting place.

08:11.22
Jon
I mean, so do you think that there’s a fundamental difference between these folks who are using these for their internal work purposes and a more public dashboard? And I’m just gonna pick one.

08:29.40
Jon
just because it gives us a grounding. So the OECD had this better life index. It was a custom build.

08:34.09
Melanie Tory
Sure.

08:34.76
Jon
It was a beautiful thing. You know, it’s really cool, but just to give us like something to think about. So that was on a website where you could, you know, move sliders and you can pick and you can filter and, you know, sort of your your basic thing. But that was developed for the public to use. Do you think the folks in your study that their behavior is fundamentally different than someone going to the OECD website?

09:00.89
Melanie Tory
I think only in that the folks in our study were kind of maybe more purpose-driven. Like they had to use this data to get part of their job done.

09:11.16
Jon
Right.

09:12.63
Melanie Tory
Whereas that might be true for some people going to that OECD dashboard, but a lot of people might be going there just out of interest.

09:13.18
Jon
Mm hmm.

09:16.92
Jon
Sure.

09:20.76
Jon
Yeah.

09:20.79
Melanie Tory
And it’s it’s not of critical importance to something they need to deliver on.

09:26.57
Jon
Right.

09:26.70
Melanie Tory
That’s probably the major difference.

09:28.69
Jon
Mm hmm. That makes sense. Yeah. So so these folks who have a task, they need to answer a question or a set of questions. they’re primarily using these dashboards as as an entryway to actually just get to the data.

09:43.35
Jon
Do you, did you find that it, like, what was the reason that that’s what people were doing was because the dashboards weren’t designed well?

09:44.06
Melanie Tory
Yeah.

09:54.14
Jon
was because, you know I remember there was one person that was like doing gym membership stuff. So I’m guessing that person isn’t like you know like a computer scientist or like a PhD in stat.

10:06.45
Jon
No shade on people working in gyms.

10:07.51
Melanie Tory
yeah

10:08.25
Jon
But you know you know not a know deep data person. um Yeah.

10:15.19
Melanie Tory
Yeah, I think that was true of most of the people in our study, right?

10:18.11
Jon
Mm-hmm.

10:18.67
Melanie Tory
These are folks who, you know, data’s got to be part of their job, but it’s not their day-to-day life.

10:19.10
Jon
Yeah.

10:25.19
Melanie Tory
It’s not everything they do in their job.

10:26.95
Jon
Right.

10:27.51
Melanie Tory
And for the most part, they’re not data experts. then We had a couple of folks who had been data experts and then moved their way into leadership roles. So they did have that expertise, but no time.

10:35.80
Jon
you

10:39.30
Melanie Tory
But most of the people were average folks who’d you know weren’t data experts, but had to use the data in some way to answer questions, to build reports for their managers, et cetera, things like this.

10:55.58
Jon
And yet they’re still going to the dashboards for the most part and downloading the data and exploring it or making the graph or the slide that they need to make.

11:05.50
Melanie Tory
Yeah, that was a super common theme. We called it the data dump, actually. And even some of our participants called it, you know, taking a data dump.

11:15.94
Melanie Tory
You go to the dashboard because that’s where you get your data and you dump the data out into Excel or some other spreadsheet and then you can answer your questions with it.

11:18.78
Jon
yeah

11:25.40
Melanie Tory
We even had some analysts tell us that they had built dashboards that were essentially data dump interfaces.

11:33.87
Jon
Mm hmm.

11:34.14
Melanie Tory
And they were almost embarrassed to tell us that this is what they had used their tool like Power BI or Tableau to produce because that’s not the intent of those tools, but it it served a need for their users, right?

11:36.70
Jon
Oh.

11:49.24
Melanie Tory
Their users had a really simple way to go and do what was essentially a database query, but without having to know any SQL, get their data, and then it’s flexible and they can do whatever they want with it.

11:49.78
Jon
Yeah.

11:54.47
Jon
Right.

11:57.11
Jon
Right.

12:01.73
Jon
Sure, right. so they’re So they’re using it as a way to make the data more familiar feeling and looking for the end user, not necessarily for the purpose of exploration.

12:15.35
Melanie Tory
Yeah, it was a real mix of things, right? Like there were a lot of questions that people had that the dashboard was designed well to answer using all the visuals that you would expect in a dashboard.

12:17.70
Jon
Mm-hmm.

12:22.84
Jon
Mm-hmm.

12:26.82
Melanie Tory
And and those scenarios are great. But then there were also questions analytical questions people had that the dashboard wasn’t designed well.

12:37.69
Melanie Tory
to be able to meet. So that would be one reason that they would have to go through awkward workarounds or dump out the data to try to answer those questions.

12:44.09
Jon
Yeah. Mm-hmm.

12:47.93
Melanie Tory
And there were a whole lot of use cases that didn’t involve analysis, but rather sort of communication where, for example, they would have, someone would have to get the the numbers for the current month for their sales, let’s say,

13:05.75
Melanie Tory
and repackage that in some way for leadership or for distribution throughout the organization. and that might mean you know rolling it up to a higher level of detail because leadership doesn’t want to see all those detailed numbers.

13:21.09
Melanie Tory
They just want the big picture. It might mean sort of spinning a story around those numbers.

13:23.20
Jon
Mm-hmm.

13:26.60
Melanie Tory
right like I don’t just want to tell you what the number is this month. I want to tell you my take on why it is that way, what was going on, like what’s the context.

13:32.15
Jon
Yeah.

13:37.24
Melanie Tory
I want to repackage it maybe, make it look prettier or make it look different for my audience. so So they were doing a lot of this kind of repackaging and communication of the data in addition to just answering questions with it.

13:42.78
Jon
Mm-hmm.

13:50.52
Jon
Mm. So so let’s I want to dig in on just sort of these ah two of these groups um on the dashboards that were well designed and then on the communication side.

14:01.78
Jon
Because I think a lot of people would say,

14:02.13
Melanie Tory
Mm-hmm.

14:04.76
Jon
a poorly designed dashboard is therefore a bad dashboard. um But it sounds like in in your study, at least, even the well-designed dashboards didn’t quite meet the user’s needs. So do you think that is just not understanding your end user? Is that just a dashboard, dashboards don’t quite do what we think they are supposed to do? like what what is What do you think is going on with that group?

14:35.67
Melanie Tory
I think it’s kind of both of those things, depending on the use case.

14:38.01
Jon
Yeah.

14:38.87
Melanie Tory
So I’ll give you one example of a dashboard that I think was not well designed for the purpose that the end user was using it for.

14:39.06
Jon
Yeah.

14:48.31
Melanie Tory
So this one ah this one user worked at a telecom company and had to get monthly numbers. kind of the use case I was talking about before. And so the dashboard that she was using was designed to give all of the numbers for a given month. So she could go in, set a bunch of filters, set it to like, i want to see the numbers for January. And she’s got all the numbers there she needs.

15:12.66
Melanie Tory
But what she really needed to do for her report to leadership was to compare the current month to last month and show the difference. And this particular dashboard would only show you the snapshot of one month at a time.

15:27.51
Melanie Tory
So she had to go through this really super awkward workflow of loading up January, set that filter, write down all the numbers, either in a notebook or in a separate spreadsheet manually.

15:38.26
Jon
enough Wow.

15:41.50
Melanie Tory
change the filter to February, write down all of those numbers, and then compute the difference.

15:43.26
Jon
Yeah.

15:46.39
Melanie Tory
And that’s a ah really simple thing that the dashboard could have been designed to make that direct comparison, saving her tons and tons of work had the dashboard designer known that it would be used for that comparative use case.

15:49.85
Jon
Yeah.

16:02.58
Melanie Tory
So that I think is a case of failure on the dashboard designer’s part, at least to understand who are all of the users of this book dashboard and what are all of the ways of using

16:02.79
Jon
For

16:12.92
Jon
sure. Right. and And do you think that also applies to the users who needed to create some sort of other product, more of the narrative storytelling briefing book slide deck. Like, I guess if if if if person a is the dashboard designer and they know that, you know, person person’s B through Q have to make PowerPoint slides out of the dashboard.

16:39.50
Jon
I feel like you would build that dashboard in a way to facilitate that and use, but also maybe not.

16:46.82
Melanie Tory
Yeah, I think that that’s true to some extent, that that maybe they could have been able to do that a little bit more.

16:48.01
Jon
Mm hmm.

16:54.43
Melanie Tory
But I also think that this is a place where our current dashboard tools just fail to support the kinds of flexible use that the end users would really like to have.

16:54.78
Jon
Yeah.

17:07.00
Melanie Tory
right? Like, wouldn’t it be nice if you could go into your dashboard and as an end user without having to go into some kind of deep edit mode and understand all of the inner workings of the tool that built the dashboard, could you just like change the encoding of a bar chart into a pie chart because that’s how you want to see it?

17:29.75
Jon
Yeah.

17:30.04
Melanie Tory
Or could you change the color scheme to make it what you think is beautiful for your end users?

17:35.44
Jon
Yeah.

17:36.49
Melanie Tory
Or could you filter out parts or do roll-ups? Like there’s a lot of, or add annotations, add story around it.

17:43.65
Jon
Mm-hmm.

17:45.79
Melanie Tory
Like if they had more of this flexible sort of mash it up,

17:46.23
Jon
yeah

17:51.26
Melanie Tory
functionality to them, then these folks would have been able to do a lot of more of that work in the dashboarding tool without having to sort of fail out and go back to a more manual process in Excel or PowerPoint or whatever else.

17:53.14
Jon
he

18:04.09
Jon
Yeah.

18:07.25
Jon
Right. So the last question on this paper I wanted to ask you is a little beyond the study because your study was focusing on internal use cases. But did the work lead you to have more thoughts about dashboards for external purposes? So we could go back to the OECD or just like Tableau Public, right? Or or whatever, like Did it give you any thoughts or insight on how on how you think people are using dashboards sort of generally speaking?

18:39.56
Melanie Tory
I think it probably speaks to dashboard use in general, even though we studied the more narrow population.

18:46.68
Jon
Mm-hmm.

18:49.53
Melanie Tory
My suspicion is that dashboards are useful for a fixed set of ah functions that they they support, right?

18:59.88
Jon
Mm-hmm.

19:00.02
Melanie Tory
They’re good at distributing information to a wide audience.

19:05.38
Jon
Mm-hmm. Mm-hmm. Mm-hmm.

19:05.74
Melanie Tory
They’re good at being a sort of source of truth, right? They document the state of the data at the current time or perhaps through past history, right?

19:16.53
Melanie Tory
They’re good at those sort of circulation of information functions. They’re not necessarily great tools for data exploration, certainly not for answering novel questions that the designer didn’t think of. A designer is probably not going to think of all of the possible questions that someone might want to answer with their dashboard.

19:40.16
Jon
Right.

19:40.44
Melanie Tory
And so it’s just it’s sort of look it’s a property of the way dashboards work.

19:46.74
Jon
Yeah.

19:47.22
Melanie Tory
They’re a little bit inflexible and fixed to a small set of things that they do well and a bunch of other things that they just don’t do or don’t do well.

19:58.30
Jon
Yeah. i mean, one of the points that I made it in my post was that You know, more than half of people now, especially globally, but even in the United States are are primarily using their mobile phone for their internet access. And I’m just not sure that dashboards…

20:17.23
Jon
I might even say I’m not sure. Dashboards just don’t work on mobile phones, right? Like that’s just not, they’re just too small to do all the filtering and searching and dropdown stuff like that. And um i wonder if you think, and we we’ll talk about AI in a second because you know we’ve got to talk about AI these days, but I wonder if you think that the way in which consumption of data and media is changing changes how we should be thinking about creating dashboards in this public space.

20:26.26
Melanie Tory
Yeah.

20:47.45
Jon
For this public use.

20:50.58
Melanie Tory
Yeah, I like I said, I think dashboards serve their purpose. But we shouldn’t assume that they’re going to meet all needs of people with data. like People have way more flexible and interesting questions that we can’t anticipate, and a dashboard can’t handle that.

21:09.24
Jon
Right. Yeah. um OK. um Really interesting. i’m sure there’s a lot of dashboard creators who are yelling at us right now and shaking their fists. So we’ll we’ll hear from them later on. um I want to turn to some of the the newer work that you’re doing. um You’ve got kind of like two very different papers I wanted to ask you about. One is the heart interface, which I think has An abbreviation, yes.

21:40.95
Jon
Healthcare enabled by AI in real time. So this is going to get us to a number of different things at at one point. So why don’t we start there? um What is that?

21:48.68
Melanie Tory
Sure.

21:49.48
Jon
Who is that project with? and And what is that project focusing on?

21:53.27
Melanie Tory
Yeah, that’s a really big collaborative project I’ve been working on since I started here at Northeastern. It’s a collaboration between us here at the Rue Institute and ah a local hospital, Maine Health, as well as a ah healthcare care technology company, Nihon Code and Digital Health Solutions.

22:14.39
Melanie Tory
And this project started with the idea that we might be able to enhance patient care in intensive care settings, particularly the cardiothoracic ICU, where all patients go to recover after they’ve had open heart surgery.

22:31.23
Jon
Mm-hmm. Mm-hmm.

22:31.54
Melanie Tory
And the idea was, could is it possible that an ai would be able to help the care team better care for these patients by predicting ahead of time if they were going downhill and heading for some adverse outcome.

22:48.95
Melanie Tory
Could we predict that ahead of time before the humans could have let the humans know And then maybe they might be able to intervene earlier and and save some of these patients from pretty nasty outcomes.

23:04.73
Jon
Right.

23:04.86
Melanie Tory
And so this project was all about, yeah I’m not an AI developer at all. I couldn’t build this kind of predictive AI.

23:12.21
Jon
Mm

23:12.86
Melanie Tory
But fortunately, I have great colleagues who can. And so the idea was to build an AI predictor, build an interface around that. That was a my team’s job.

23:23.38
Melanie Tory
that could then be ultimately deployed in the ICU and tested through a clinical trial, which we haven’t got to yet, but that’s ultimately the goal, to see if humans plus an ai could have more effective outcomes than humans alone.

23:30.54
Jon
hmm.

23:39.83
Jon
Interesting. And so this is doing predictive analytics on the real time, ah i guess, healthcare or health information from the patient.

23:50.04
Melanie Tory
Exactly. So it’s it’s being fed the patient’s health record, any vital signs data that’s collected, all that stuff you see hooked up to you know heart rate monitors and so on in hospitals.

23:51.14
Jon
Right.

24:01.83
Jon
Mm-hmm.

24:03.12
Melanie Tory
All of that information is being fed in in real time and used to make predictions about a range of possible negative outcomes like heart failure, kidney injury, and so on.

24:16.63
Jon
And is there a ah viz component to this? Because because the way you’ve described it sounds like there’s a big predictive analytics ah challenge. There’s a big AI sort of data ingestion problem.

24:29.37
Jon
There’s obviously like communicate it to the healthcare workers, but like, is there a viz part to it too? That’s like trying to pull it all together.

24:37.69
Melanie Tory
Yeah, because that bit about communicating it to the healthcare workers is exactly where the biz comes in, right?

24:41.69
Jon
Yeah. Yeah.

24:43.77
Melanie Tory
So what we ultimately wind up with is a data set that is changing over time of all of the patients in the ICU, their current predicted risk scores for a range of like nine or 10 different possible negative outcomes.

24:44.54
Jon
Yeah.

25:02.16
Melanie Tory
um and uncertainty information about the best likely outcomes, because what can happen is that, you know, the algorithm might be predicting that it’s highly likely that they’re going to go into a septic shock, but maybe there’s missing values in the patient record, like lab tests that haven’t come back yet.

25:16.63
Jon
Right.

25:22.87
Melanie Tory
And we know that the the result of those lab tests could affect the score up or down. And so that leaves us with a range of risk scores. that have to be communicated as along with the risk score themselves.

25:36.50
Melanie Tory
So it actually becomes a pretty big data problem. It’s just that there happens to be an AI in the loop generating all this data.

25:39.70
Jon
yeah

25:44.10
Jon
Right. um ah So the uncertainty piece has got to be pretty important here.

25:54.84
Melanie Tory
Yeah.

25:55.22
Jon
And communicating uncertainty to even healthcare workers has got to be pretty a pretty hard nut to crack. So how are you, at least it sounds like kind of early-ish stages, but how are you thinking about communicating uncertainty metrics of uncertainty to, I mean, I’m not going to say non-data people because i think healthcare workers are are pretty steeped in data all the time, but they’re also really busy.

26:18.73
Jon
um So to like understand, at a glance, understand this like, you know, median number, but also that it’s plus or minus, you know, blah, blah, blah percent.

26:19.67
Melanie Tory
yep

26:28.89
Melanie Tory
Yeah, that was exactly the challenge that we tackled in this particular paper is how to convey those the risk scores and their uncertainty values around them.

26:38.71
Jon
Hmm.

26:39.19
Melanie Tory
And one of the first things that we learned in talking to our collaborators at the was that they didn’t really want to see a raw number score.

26:49.75
Melanie Tory
They like to see the number, but what was more important to them was to classify it into sort of threshold categories, right? They think in terms of patients are high risk, medium risk, or low risk.

27:03.34
Jon
Oh.

27:03.90
Melanie Tory
they And they think about this across a bunch of metrics already that they monitor in the hospital.

27:08.22
Jon
Right.

27:10.22
Melanie Tory
And they even give them colors, right? So that’s like red is always high risk, right?

27:14.78
Jon
Yeah, right. Yeah. Yeah.

27:17.99
Melanie Tory
is low risk. We got away from the red green color scale in our interface. But they even think about these categories as colors, right? They talk about red alarms.

27:25.27
Jon
Yeah. Right.

27:27.32
Melanie Tory
When an alarm rings and it’s a bad one about something dire, like that’s a red alarm.

27:32.41
Jon
Yeah, yeah. Code red versus code white versus code brown. Yeah, I remember those. Yeah.

27:37.18
Melanie Tory
Yeah, I forget where we were going with this question.

27:38.06
Jon
Yeah.

27:39.34
Melanie Tory
I think you were asking about how we conveyed uncertainty, right?

27:42.20
Jon
Uncertainty. Yeah. Yeah. so So it’s… it’s it’s for your For their purposes, it’s not so much plus or minus one standard v deviation. It’s is this high, medium, low. Like places them in a bucket.

27:59.32
Melanie Tory
Yeah, exactly.

28:00.02
Jon
Yeah.

28:00.16
Melanie Tory
And so what that God is thinking about is it’s not when we convey uncertainty, it’s not so much, you know, plus or minus five points. What matters is does the the window of uncertainty potentially cross the threshold into the higher or lower category, right? So what would be quite worrisome is if a low risk patient could actually be a medium risk patient because we’re missing data about that person.

28:30.20
Jon
Right.

28:32.67
Melanie Tory
So that was kind of the critical thing that we wanted to convey.

28:35.18
Jon
Gotcha.

28:36.91
Melanie Tory
And so we kind of simplified down the way that we were conveying uncertainty to focus on those boundary crossings.

28:40.04
Jon
Mm hmm.

28:45.50
Melanie Tory
And what we ended up doing in our interface was creating these sort of pills fur eat for each patient. We would show a pill for each possible negative outcome that they might be at risk for, filtering out all the ones that aren’t a problem for that patient.

28:58.93
Jon
Ah, okay.

29:02.23
Melanie Tory
So like, let’s say they’re at risk of renal failure.

29:02.31
Jon
Right.

29:04.79
Melanie Tory
We have a little pill with the name renal failure on it, and it’s colored based on their risk category.

29:08.89
Jon
e

29:10.00
Melanie Tory
And then we would append to the end of that pill a little blip of color of the neighboring category if the uncertainty could possibly cross the boundary.

29:22.62
Melanie Tory
So that’s how we made a sort of like super duper simplified representation of uncertainty for this group.

29:23.35
Jon
and

29:27.51
Jon
right

29:29.94
Jon
Interesting. Interesting. um You might not be able to answer this question, but it but it is a question that everybody’s talking about, which is, are all of our jobs at risk because of AI? And the way you describe this project is is interesting because…

29:49.72
Jon
it actually sounds more like a tool for health care workers to use and sort of summarizing a whole bunch of health data. It’s not replacing anybody. But like these sorts of tools, do you think that and maybe you’re not maybe not far enough along in the project to actually talk about this yet.

30:08.68
Jon
But I’m curious whether you think this sort of project, this sort of tool will facilitate job loss or job gain. Because I could see it going kind of or nothing or or neither. But I can see it going in all sorts of different directions.

30:24.54
Melanie Tory
Yeah, we were pretty deliberate about this when we were conceptualizing this project, that this was never intended to be a tool that would replace healthcare care workers or their decisions.

30:35.06
Jon
Yeah.

30:37.09
Melanie Tory
we We were even really careful that, you know, it’s not even making direct care recommendations. It’s not even suggesting actions that they could take.

30:42.90
Jon
he

30:46.51
Melanie Tory
It’s meant just as another information source that can help them

30:49.78
Jon
Right.

30:52.02
Melanie Tory
think about what’s going on with their patients so that they can hopefully make better decisions. And that was a deliberate choice on our part to keep the healthcare care workers in charge, right?

30:57.42
Jon
Right.

31:02.62
Melanie Tory
Like to us, they are the decision makers here.

31:06.68
Jon
Yeah.

31:07.22
Melanie Tory
They’re gonna be in charge.

31:07.58
Jon
Yeah.

31:08.18
Melanie Tory
Our job is to inform them with more information.

31:10.46
Jon
Right. Right.

31:12.38
Melanie Tory
And I don’t know about you, but if I’m in the hospital after cardiac surgery, I want the experienced physician caring for me.

31:16.66
Jon
Yeah, that’s right. Yeah, that’s right.

31:19.38
Melanie Tory
I don’t all want an AI making decisions.

31:21.40
Jon
Yeah, no, right.

31:22.06
Melanie Tory
Yeah. Yeah.

31:22.48
Jon
That’s right. That’s right. Yeah, the person who has to cut my chest open, that’s the person that I want making decisions. Yes, 100%. We have one more project that is a work in progress, but just wanted to to let you mention it has a very cool ah title, Vibe Modeling.

31:38.32
Jon
um and And that one is also on AI, right?

31:39.57
Melanie Tory
yeahp

31:41.92
Jon
And and practitioners.

31:43.90
Melanie Tory
It is. So I guess a theme of my work for a while has been getting out of my academic bubble and trying to understand how people are using data in the world and how people are using tools in the world and what we as tool developers or researchers could be doing to make their lives easier.

31:50.58
Jon
yeah.

31:54.90
Jon
Yeah.

32:03.19
Melanie Tory
And since everybody is now starting to use AI for everything,

32:08.12
Jon
yep

32:08.12
Melanie Tory
The natural question for me was, well, how are people using things like large language models, generative ai for data visualization work?

32:18.94
Melanie Tory
and Is that working? Is it effective?

32:21.18
Jon
Yeah.

32:22.02
Melanie Tory
Or what needs to change to actually make it work? Because we know how error prone these kind of models are. played with them myself. They do kind of bad jobs some of the time.

32:33.05
Melanie Tory
so But we know people are using them.

32:33.34
Jon
Yeah.

32:35.21
Melanie Tory
so We just simply wanted to know, you how are people using and AI for data viz work? and what are all the problems that are going on? So this was another interview study, kind of similar to the dashboard user study, just trying to understand current practice and challenges.

32:49.20
Jon
Mm-hmm.

32:52.57
Jon
And do you, I know it’s early, but like, what is your level of concern with doing an academic study, especially a qualitative study, which tends to take time and then how fast the AI models are changing?

33:02.98
Melanie Tory
Yep.

33:06.28
Melanie Tory
It’s true. it is a big problem.

33:08.15
Jon
Yeah.

33:09.02
Melanie Tory
we We do want to get this work out fairly quickly because of that reason.

33:14.04
Jon
Right.

33:14.78
Melanie Tory
But I still think even if the AI models change and get better, it’s still interesting to see where in the Viz process people are using these models and what kind of recommendations we could make or what kind of training programs we we could create to help people use them in safer, more effective ways.

33:23.29
Jon
Yeah, for sure.

33:30.10
Jon
ehi

33:34.71
Jon
For sure. Yeah. Awesome. Melanie, thanks so much for for coming on the show. Before i let you go, if people have more questions about any of these projects or you know if they’re working on you know risk scores and hospitals in their area, what’s the best way to get in touch?

33:51.77
Melanie Tory
Oh, they can absolutely reach out to me. I’ll get you my email. It’s on my web page too. can get you the the link to our our human data and interaction group web page.

33:57.00
Jon
Great.

34:02.21
Melanie Tory
And there’s also a contact form on there if people want to reach out and get in touch. So lots of ways.

34:08.50
Jon
Awesome. Awesome. Terrific. Thanks for coming on the show. It was great to see you. And good luck with all these. And I very much appreciate that dashboard paper. It was great to see you. Thanks so much.

34:18.33
Melanie Tory
Thanks so much for having me. It’s been fun.