The last Data Stories podcast featured a fantastic discussion with Enrico Bertini, Alberto Cairo, Robert Kosara, and Moritz Stefaner about storytelling in data visualization. I’ll save my post about storytelling for another time, but right now I want to write about Alberto Cairo’s “So What?” test.
Alberto defines his “So What?” test as follows:
“A test that we can apply to any visualization, I believe, is the ‘So What?’ test…..I see some visualizations about that make me ask myself, ‘And?’”…. (around minute 37)
In this part of the discussion, Moritz argues that there are parts of newspapers, for example, that don’t meet the “So What?” test; stories that don’t touch you or aren’t meaningful or don’t impact you. Alberto responds:
“They still try to make clear why you should care…..The very fact that the articles are there is a suggestion that the journalist thinks that you should care. But sometimes that is explicit in the article.”
Alberto has made this point before. In his terrific talk at Chicas Poderosas (if you haven’t watched it, it’s a great introduction to visualization) he discusses a student project (Visualizing Buffy) that tracked the screen time of each character in the show Buffy the Vampire Slayer (in some ways, the project is similar to Santiago Ortiz’s Lostalgic project). Here’s what Alberto said about that project:
“Think about the topic [of the Visualizing Buffy project]….It satisfies the needs of Buffy’s fans, but I believe the student could have done something different and much more worthwhile using the exactly the same technology [d3]. And again, I’m not saying that we should not be focusing on these kinds of things. These kinds of graphics are fun, they will bring attention to the work, etc., but perhaps they are not that important. Let’s reduce the amount of time we spend on these types of things; let’s increase the time we spend on things like that [healthcare.gov].” (around minute 52)
Let me begin with what I see as a contradiction in these statements: that the existence of a newspaper article suggests it has passed the “So What?” test, but that criterion does not extend to visualizations. Why? Do journalists have some sort of monopoly on saying what is important? Newspapers have entertainment sections, right? They probably covered Buffy in its heyday—was that coverage wasted ink?
Here’s a screen shot of the Most Read articles from the Washington Post on Thursday, May 21st. Do all of those articles meet Alberto’s “So What?” threshold? The meteor shower may not actually help address today’s social or political challenges, but it’s certainly interesting to many people. I, for one, don’t find a story about meteor showers as interesting as the story about inequality, but that doesn’t mean it doesn’t have value.
I’m being a little tongue-in-cheek here. I don’t think Alberto actually believes these products have no value. But I think we should be careful in writing off visualizations—or newspaper articles, academic journal articles, books, games, toys, sports, or really anything—that doesn’t meet some sort of arbitrary “So What?” threshold. Because the threshold you and I put on something may be completely different than someone else’s “So What?” threshold.
Take, as an example, Aaron Koblin’s Flight Patterns project (also discussed in the Data Stories podcast). The Flight Patterns project was popular for many reasons: it was relatable, it was beautiful, and it was stunning to see air traffic as it flowed from day to night and back again. But does it meet the “So What?” test? The graphic itself is not annotated—it doesn’t point out particular patterns, outliers, or stories. Those things could be added, of course, but were not. So in that form, perhaps Flight Patterns does not meet the (or should I say “a”) “So What?” test.
Now, let me put the Flight Patterns project in a different perspective. About a year ago, during a presentation about data visualization, I showed part of Flight Patterns to an audience largely made up of analysts from different government agencies. Two Federal Aviation Administration analysts pointed out that it could be used to examine U.S. airspace and flight patterns in ways the agency has not yet explored. In that perspective, Flight Patterns clearly passes the “So What?” threshold, right?
My point here is twofold. First, none of us are in a position to truly judge the “So What?” question because it will meet one threshold for some and a different threshold for others. I could really care less about characters in the Buffy series, but perhaps there are people, say, in the entertainment industry who could use the visualization to better understand character development in a successful television show. Similarly, I have little interest in marketing data, for example, but there are hundreds upon hundreds of infographics dealing with marketing. For some people—perhaps, even, people working in social policy or public policy fields—that information in those visualizations is valuable. Second, the serious and the fun can live together*. I can read The New Yorker and US Weekly (I don’t actually read either) and have a completely happy existence.
I’ll make just one final point that I’ve been struggling with for a while. There are a lot of visualizations out there that use big data or social media data. And some of them look great: mapping languages on Twitter certainly looks cool; mapping friends on Facebook is pretty. But I haven’t seen these projects or data applied to social or policy challenges (if you’ve seen one, please leave it in the comments below). On the one hand then, I think this is a real shame because I think (or maybe hope) there are ways to use these data to reveal something about public policy, economics, or governments. But on the other hand, perhaps these visualizations serve a different purpose: Maybe they simply illustrate our connections with one another, what it is to live in the 21st century—the human condition, if you will. So that itself may be a worthy goal.
I’ve spent most of my professional life conducting economic research on what I believe are important public policy challenges: among them, inequality, immigration, disability, and food stamps. But I hesitate to discount many journal articles or data visualization projects simply because they don’t deal with serious issues or don’t interest me. That is to say, I have my own “So What?” test; I just hesitate to give it to anyone else.
*Note: While I was writing this post, Chris Cillizza at The Washington Post wrote a nice semi-rant about why journalism isn’t broken. In it, he touches on some of the points I’ve made here.
Thanks for writing about this, Jon, and for your kind words. You’re right on this: “Let me begin with what I see as a contradiction in these statements: that the existence of a newspaper article suggests it has passed the “So What?” test.” That’s indeed a contradiction. I should have been more careful.
Regarding the other points, I am not saying we should get rid of visualizations like the Buffy one. I love it, actually. Those kinds of projects are fun and do have a purpose. I’m a fan of AintItCool.com’s, for instance. I’m just saying that perhaps, when we have the choice and the time to invest in a visualization or news story or any other kind of communication, we could think more about themes that have the potential of improving the lives of many people out there.
I believe that the “value” of the topic a visualization covers can be estimated in a pretty objective manner, by the way. I like to use this analogy: Given the choice of investing a certain amount of money and time in developing a technology like the iPod OR a technology like nets to repel or kill the Anopheles, the mosquito that transmits malaria, which one matters more to more people? Which one increases the well-being of more people to a greater extent?
Relevance and value are relative and context-dependent, of course, but only to a certain point. I have recently written a couple of chapters and articles on the ethics of visualization for two different books and magazines in which I discuss these things at length. I was working on them when Moritz, Enrico, Robert and I did the podcast. I will try to share them once they’re out, but here you have an excerpt (not edited yet!!!!) of one of them, just for you to understand where I’m coming from:
“Rule utilitarianism, which is the ethical theory that I’m adopting in this chapter, is similar to deontological ethics in the sense that the morality of an action is judged against a set of rules. But these rules are not a priori or absolute. Rule utilitarians devise rules that are more guidelines than inflexible decrees. For a rule utilitarian, for instance, telling the truth is morally good, but not because there’s an a priori rule against lying or because being dishonest is non-virtuous. It’s because we have solid evidence to think that lying decreases the well-being of the people who see, read, and believe our lie, so avoiding lies is good behavior. We can make up a rule out of that.
But lying is not always bad, as we’ve already seen. When telling the truth conflicts with protecting the lives of other people (another ethical rule that we can derive from evidence,) lying is appropriate.
My reasoning in this chapter is therefore based on the following assumptions:
1. Morally good actions are those that increase the well-being of as many people as possible, either directly or indirectly.
2. Accurate, useful information, presented visually in a compelling way, is likely to increase awareness of relevant matters among the audience the visualization is created for, and improve their understanding and knowledge. We have abundant evidence to support this idea.
3. Good understanding of relevant matters can inform future decisions, so it is likely to increase the chances of people conducting fruitful, happy lives. This is, understanding can have a positive influence on their well-being.
4. Therefore, it is the obligation of the designer of visualizations to create graphics that (a) are aimed at sparking interest in relevant matters, (b) are based on a thorough and accurate depiction of those issues, (c) are built in a way that enables comprehension. To do this, designers ought to base their decisions on available scientific evidence or, in case that this is not available, on judgments derived from their experience and personal observations.
This is the moral rule —in the utilitarian sense— this whole chapter is based on.”
Thanks for commenting, Alberto. The ethics and morality surrounding data visualization is, of course, an important topic worth considering and I look forward to seeing the longer pieces.