Michael Kubovy is Professor Emeritus of Psychology at the University of Virginia. His publications in cognitive science span the fields of decision-making, visual and auditory perception, the psychology of art and the psychology of pleasure. In his work on perception he has used mathematical models to answer challenging questions raised a century ago by the Gestalt psychologists. In his book The Psychology of Perspective and Renaissance Art he uses art history and perceptual research to illuminate each other. He is currently working on a book called The Structure of Lives, which explores implications of the idea that the first-person experience of our lives can be decomposed into continuous, concurrent, and asynchronous strands.
Professor Kubovy I was born in France on the day the Nazis marched into Paris (June 1940). First, eight years in NYC, then Israel (but because his father became an ambassador for Israel, he spent time in Czechoslovakia and did his high school studies in Buenos Aires at a French school). He went back to Israel after high school, studied aeronautical engineering (never finished), studied psychology (up to PhD) and philosophy (up to MA) at the Hebrew University of Jerusalem, where his studies were interrupted by the Six Days War (he fought in Jerusalem). he then took a post-doc at the University of Illinois (Urbana-Champaign), and was planning to teach in Israel when he got an assistant professorship at Yale. He spent 10 years there, seven years at Rutgers (New Brunswick), and then at UVa from 1987 to 2006, when he retired. Kubovy is married to Judith Shatin, an outstanding composer of contemporary classical music (acoustic and electronic). He has a son, Itamar, who is Executive and Artistic Director of Pilobolus Dance Theatre, and two grandchildren, Betty (15) and Theo (14).
I came across Professor Kubovy’s work as I was reading some papers about Gestalt Principles. We talk about his work in Gestalt psychology, what it is and how it can be applied to data visualization, and some of the early founders of the Gestalt school of thought. It’s one of the more fascinating conversations I’ve had about Gestalt Principles and I’m sure you’ll enjoy it!
A century of Gestalt psychology in visual perception, Kubovy, Wagemans and others.
Human Perception, Kubovy & Bertamini
Support the Show
I recently revised the steps in my Patreon page, so please check it out and see if you’d like to become a financial supporter of the show. If even for a few months, your support helps me cover audio editing services, transcription services, and more. You can also support the show by sharing it with others and reviewing it on iTunes or your favorite podcast provider.
Jon Schwabish: Hi everyone. Welcome back to the PolicyViz podcast. I am your host, Jon Schwabish. And on this week’s episode of the show, I’m very excited to talk about your stall principles with Professor Michael [Kabovi 00:00:21] from University of Virginia. I found Dr. Kabovi through some research I was doing on Gestalt, uh, to learn a little bit more about how we perceive information, how we perceive visuals. And, uh, he has been doing this sort of work for a very long time, and so I was really excited to sit down and chat with him and, um, I learned a lot, and it was a really a fun conversation to have. So I hope you will enjoy it as well. So here’s my interview with Dr Kabovi. Hi, Michael. Thanks for coming on the show. I appreciate you taking the time out.
MK: Well, thank you for inviting me. It’s a pleasure to be here.
JS: I’m really excited to talk about the work that you’ve done. What sounds like work you’ve done for a long time on gestalt and visual information and how we perceive things, um, I wanted to give you a chance to talk about, uh, your background and your work and maybe how you got into this particular topic?
MK: So it’s really interesting how one develops an intellectual interest by meandering through various fields of science and how one’s intellectual, uh, life develops. So I didn’t know that I wanted to do psychology. I was convinced that I wanted to do philosophy. And uh, it turns out that, uh, at the Hebrew University in Jerusalem where I was studying, there was a researcher by the name of Daniel Kahneman who eventually got the Nobel Prize for economics and his, his book Thinking, Fast and Slow is a big hit. Well, so he was teaching at the Hebrew University, and I was in his class, and one day he saw me at the little store where we were, that he happened to be living in the same building as I was, and I was shopping at this little store and he said, “Uh, you’re in my class, aren’t you?” And I said, “Yes.” And he said, “Well, I need an assistant.” And it turns out that what he needed is a person who was pretty big who could lift things. And I’m not going to go into details, but uh, but it ended up turning into an extraordinary collaboration. And to make a long story short, at one point there was a hiatus in the work that we were doing and he asked me to, uh, organize his reprint collection. And so I did it very slowly, because I was reading all the papers and I came across a paper by a guy at Bell Labs called Béla Julesz who invented the random dot stereograms. I’m sure you’ve seen those where there’s a picture that you show to the left eye and a picture that you show to the right eye and they’re sort of random dots. And when you look at them together, uh, some form emerges. And so I was looking at these pictures through these red and green glasses and suddenly I saw this triangle float up out of this random mass of dots. And I said to myself, “Oh my God, I am watching my brain compute.”
MK: If that’s possible, that’s what I want to do for the rest of my life. So the, the idea of doing science on one’s experience of the world is a very different thing than what, uh, many psychologists do. Psychologists are most interested in describing behavior from a third person point of view. They observing you, they might put you into an experimental situation and look at how you respond to this or that. But the study of experience is what is most interesting to me, and I think a fundamental problem for, uh, for psychology and it shows the traces of my interest in philosophy, because the problem of other minds is a big problem in philosophy. How can you tell that someone is seeing red the way you see red is the sophomore problem that people discuss late at night after smoking a joint. So, um, there is actually an answer to that question that it’s very interesting. If you want, I can take a minute to tell you about it.
JS: Yeah. Yeah.
MK: So suppose I gave you the following, uh, task. I give you pairs of colors, and I show you four colors at a time in two pairs, and I ask you is the pair on the right more similar to each other than the pair on the left, and you have to decide which one is more similar. And so you get a, a collection of inequalities of pairs and from that through a process called metric or non-metric multidimensional scaling, you can actually reconstruct the geometry of color space as it’s represented by the mind. And it turns out that this color space is circular as we typically represent it, but it gets reconstructed from these judgments that people make in this task. Now in this color circle, everyone has red more or less across the circle from green and blue more or less across the circle from yellow. So if the color circle that is yours and the color circle that is mine, both have red and green on a diameter and blue and yellow on an orthogonal diameter, then the structure of your color space is the same as the structure of my color space. So I don’t know if you see red the way I see red, but I do know that your red is to your green as my red is to my green.
JS: I see. So it’s not about the specific value of the, of the individual color. It’s relative to other colors.
MK: It’s relative to other colors. But that’s really all one can say about experience. Experience is never absolute.
MK: The loudness of something is loud if it’s in a very quiet environment than not at all if it’s in a noisy street.
MK: So everything is relative in experience. And so this is actually the ultimate answer to the question: do you see red the way I see red, because if we slightly reinterpret the question, we have an answer. So I’m very interested in gestalt because gestalt psychologists were the first one to propose a way of getting at experience. They were wonderful thinkers who started out with an idea that’s very interesting. This idea of gestalt comes from a philosopher, psychologist by the name of von Ehrenfels and he was an Austrian from Graz, and he talked about the following. He talked about a melody and he said, if you hear a melody, you can pay attention to every single note. You can say it’s exactly on pitch or it’s slightly off, it’s loud, it’s soft, it’s tambour, it’s such and such, whatever. You can pay attention to the individual notes. But there is an emergent property, which we call the melody that is of a different quality than the notes themselves. It is in that sense that the whole is different from the sum of its parts. And so that idea that comes from von Ehrenfels’s, uh, notion of a melody being different than simply the collection of its notes led Wertheimer to start exploring these kinds of questions in vision.
JS: I see. So the actual roots are in audio.
MK: It, the roots, yes. The best early example of a gestalt is in audio, but the research in gestalt in audio perception waited for a long time because it’s much, much harder with primitive tools. So here’s what happened. Uh, Wertheimer started doing experiments on apparent motion. So what he would do would have a flashing light on the left of the screen and a flashing light on the right of the screen and they were alternating. And if you flash them at the proper rate, you’ll see apparent motion that is you’ll see the dot moving from right to left and left to right back and forth. But at a certain rate, something miraculous happens. You stop seeing the dot, the dot itself. You see pure motion. In other words, you are seeing coming back to the example of the melody, you are hearing the melody without hearing the notes.
MK: And that was such a revelation. Uh, Wertheimer wrote a very important paper on this. And uh, around 1912 and Gestalt psychology was off and running, and it was, it was a very important movement with hundreds of papers written, if not thousands, fascinating work until about 1933, 1934 when two of the three, uh, major gestalt psychologists, Wertheimer, Koffka and Köhler, uh, two of those Wertheimer and Koffka were Jewish, and so they had to flee and Köhler fled as well, and they came to the states. And what happened was in the states, that was the heyday of behaviorism. And so anyone studying experience was absolutely anathema to what these psychologists were doing. So gestalt psychologists went subterranean and uh, they couldn’t publish in the major journals. It was just, uh, just terrible. And so it took, well after the three major gestalt psychologists have died, it took a few of us. I was one of the people who sort of rediscovered and revived Gestalt psychology under the name of perceptual organization. And so I edited a book called Perceptual Organization. And in the two years after that, two other books appeared on the same topic. So it was clear that the time had come to uh, relax the sort of, uh, dominance of the behaviorism and we could start talking about experience.
JS: Was there an event that led to this resurgence in this or was it just the natural evolution of, of a scientific field?
MK: It’s very strange the way I described it at that conference where I talked to a former student of mine at Yale, I got together with him and I said, “Jim, this is amazing. People are talking about Gestalt psychology. And then they’re apologizing to the audience for, having mentioned the word as if it were dirty word.”
MK: Uh, it was just not, not kosher to use the word gestalt because it had to do with experience and what do we know about experience?
MK: So it was I think psychology waking up to the insufficiency of a third person perspective that one needed several things, one needed to understand experience, and one needed to understand nonlinearity, which produces emergent properties. So that’s what, what I think triggered this need among psychologists to use the word gestalt. And a bunch of us made the use legitimate. And, uh, and since then there’s been fascinating, extraordinarily interesting work on various aspects of emergent properties, gestalt, perceptual organization, the relationship among perceptual dimensions, et cetera, et cetera that can fill a book. And in fact, there is a, uh, Johan Wagemans who had been a postdoc with me in the, uh, 90s, has, uh, over the past few years edited an Oxford Handbook of Perceptual Organization. So it’s full of extraordinary articles on the topic. It’s an amazing reference.
JS: Can you give me a sense of the type of experiments that you and your colleagues start and have continued to do over the last several years? You know, you talked about Wertheimer and his early work where it’s, you know, a left eye and a right eye, but, but what are the methods that you use now that the more modern methods?
MK: Well, so I’ll give you two examples, one from auditory perception and one from visual perception. So in auditory perception, the question is, can you get an emergent property, you remember I mentioned Bela Julesz where you present something to the right eye and the left eye. Well, what I did was something analogous by presenting noise to the right ear and a slightly different noise to the left ear. And when you listen to them together, out of this noise would emerge the tune of daisy, daisy, daisy. And so I was able to figure out how to create auditory emergent properties, uh, that were similar to things that were happening in vision. So that’s one example. Uh, in the 90s, I started worrying about the fact that gestalt psychology was not quantified. In other words, one couldn’t really talk about the strength of a gestalt. One couldn’t measure, uh, gestalts. So with some colleagues, I developed an approach using collections of regular dots. Now imagine dots in rows and columns, just an array of rows and columns. Now suppose that the columns are closer to each other than the rows. And because of that, the rows are denser than the columns. And so what you’ll see is the pattern organized by rows.
MK: Now suppose you color every other column a different color. So you have red, green, red, green, red, green in columns, and you have organization by density or by proximity in rows. Now you have a competition between grouping by similarity and grouping by proximity.
MK: And by using that technique and, and manipulating very carefully with very short exposures, uh, stimuli that lasted a 100, 200 milliseconds, um, I could get people to report how things were organized. And it turned out to be an extraordinarily productive approach because it allowed me and my students to start quantifying Gestalt psychology. And so over the 15 years from 1995, I published with various students a number of papers that I think moved the field forward because it turned it into a quantifiable discipline that dealt both with motion perception and with grouping a phenomena, et cetera.
JS: Right. So let’s, uh, take the gestalt principles and apply them then to graphics and, and visualizing data. When you think about your work and how people now think about creating their visuals and using data, be it publicly or within organizations, how do you think about the way that maybe they use or don’t use these principles? And maybe the way they talk about gestalt and sort of, uh, you know, I think, I think a lot of people in the, in the data visualization feel, for example, talk about gestalt in a very different way than, than the way you’re talking about it, right? It’s a very, it’s a very applied, um, manner of speaking.
MK: Yeah, absolutely. So let me give you an example. Uh, there’s this very famous book designer, graphic artist by the name of Tufte. Um, and so he’s been very interested in graphics. And one of the principles that he’s proposed is that you want to minimize the ink to information ratio. You want to have as little ink as possible to convey the information.
MK: And, uh, I don’t know if your audience knows what a box plot is. I hope so, but, but, uh, a box plot is a way of showing how data are distributed in a very schematic way. So the central 50% of the data are enclosed in a box and then a certain proportion of the data that are outside those 50% are represented by an upper tail of, uh, above that box and the proportion below represented by a tail. And it’s called in fact by the guy who invented it, a box and whiskers plot. So you have the box and you have the line on top and the line on bottom. And it’s, it’s a complicated story that, uh, the exact details of which I don’t want to go into. But what Tufte did with the box plot, he said, “You don’t need both sides of the box. You don’t need a closed box. So all you need is the whiskers. And one side of the box and may be align in the middle that represents the median and, and then the other whisker.” So what he did was to create this very sparing in ink box plot that violates the laws of gestalt. In other words, because it’s not a closed form because it’s very, very hard to extract the information from it. So that’s an example I think of where you’ve got to respect the visual system and you can’t let, uh, principles of design get in the way of how the visual system and the mind work.
JS: Right. Right. And going back to your earlier comments about color, I wonder when you think about people, how they use color in their design, because we may perceive the color red differently, but what’s important, as you mentioned, is it’s relative.
JS: It’s relative to other colors. So when you look at graphs and you, and you think about the way people use color, how do you think about that applying it to the early study of the way we think about a relative, uh, appearance of color?
MK: Well, first of all, people often just use off the shelf colors.
JS: Yes. Yeah.
MK: And they use very bright colors. And the first rule I think is use much softer colors than the ones that the primary colors that you’d naturally would. And that, um, uh, Excel for instance offers you by default. So what you want is to decide how many colors you’re going to need. And then there are various tools that will give you equally spaced colors around the color circle so that they will be maximally distinguishable. And in the R language, uh, there are many tools that give you perceptually equally spaced colors and…
JS: Right. So it’s about, it’s about the spacing, not necessarily about the exact cue that you [indiscernible 00:25:01].
MK: Right. For instance, there’s a package in R called ggplot, and this package uses colors by default very nicely. So it seems to me that the issue of color, people need to pay attention more attention to color and not use brutal colors like the red, green and yellow. If you, I, um, I happen to be looking at the icon of the Chrome browser.
MK: And it has red, yellow, and green. Very bright and pretty ugly actually. Um, I think that subdued colors work much, much better because they don’t draw attention away from other elements in the graph that are equally important. Like the axes, like notations, you don’t want the color to be blindingly obvious and to be the first thing that you look at. You want to be able to look at details. Another, uh, notion that comes from Gestalt psychology is the following. Imagine that you have a graph with two lines and they’re both sloping upwards. Now it is actually quite difficult to gauge the difference between two sloping lines. I mean, you can tell that they’re getting further apart.
MK: So there’s a technique to maximize the perceptibility of the difference. So imagine that you have sitting on the bottom line, sloping line, you have a deck of cards. So, of course, the deck of cards isn’t even because the line is sloping. It’s like a sloping surface and the cards are all vertical. But now they’ve been cut so that they correspond to the top line. So now you have cards of different sizes that represent the difference between the two lines.
MK: Now take that deck and drop it onto a table so that the bottom line is now horizontal. Now you’re comparing a zero slope to some other slope that represents the difference in slopes. And now by representing the difference, see this as a theme. Representing differences is enormously important. And so by leveling one line, the reference line, you get a very clear idea, much clearer idea of how different the, the slope is.
MK: Um, I could give more examples. I don’t want to, yeah.
JS: I also wonder, I mean I think that the top two examples is a great one. And I wonder if there are other basic things even abstracting from the specific chart type. Are there other basic things that we as data and graphic designers are not thinking carefully enough about? For example, the one that pops to mind immediately is, is just the axes of a chart. Like, you know, some people have, you have a line for the y-axis, a line for the x-axis.
JS: Some people have the secondary axes so that they, they enclose the box. Some people leave that y-axis, the first y-axis off altogether. I mean is there a perceptual organization way of thinking about those sorts of basic parts of a chart that maybe we’re not thinking carefully enough about?
MK: Well, this is a matter of taste. I think that there comes those issues. I don’t think that there are hard and fast rules about, I think that you need to make sure that your axes are clear that you labeled them right, that you have a proper distance between ticks, et cetera, et cetera. But it seems to me that exactly how you designed the, the axes is less important than how you represent the contents of the, the graph.
MK: For example. I have one more example. If you have data that curve, it might be a good idea to try to transform the y-axis so as to make the data straight because then if you’re trying to represent the noise around that line, you’ll get a much better sense of what that noise is like. So I think you’re asking a really interesting question about the axes. And there’s some people that think that having, for instance, two y axes with different scales, when sometimes one scale is the transformation of the other, some people think that it’s very distracting and unhelpful, but I think it’s a matter of context. It’s a matter of how experienced your reader is in looking at a certain kind of graph. So we trained ourselves to look at graphs very quickly and very effectively within a certain category of graphs. So if you’re looking at stock market data, you’re very accustomed to all the symbols and to extracting sort of the general trend from the wiggly pattern. If you’re a scientist, you look at things in a different way.
MK: So here’s one other issue. There’s the issue of do you want to emphasize the trend of data like you might with the variations in a stock or do you want to emphasize the difference between things like in a scientific experiment where you have Group A and Group B and you want to see is there a difference between the two groups?
MK: And very often people in the latter case, they use, um, bar graphs. And I’d like to argue against bar graphs for a minute. Can I do that?
JS: Abs… abs… absolutely. Oh, yeah, sure. Well, let me take… we take arguments against any chart type on this show.
MK: Okay, good. Good. So the problem with bar graphs is that the conventional wisdom is that a bar has to start at zero. Otherwise, you are, uh, misleading the reader how to lie with statistics is, uh, is famous for giving examples of how people distort things by not having a zero at the bottom of the, but think of, and here I come back to Tufte’s ink idea. Think of how much space is wasted and suppose that you’re dealing with the measurement of the speed of light. So you’re dealing with 186 million, uh, miles per second. And how do you graft that from zero? And you want to represent, uh, differences between measurements of the speed of light in different experiments. The bar graph isn’t going to show you the difference. In other words, it depends on what you’re trying to do. If you’re trying to convince someone that a stock is rising and you don’t start from zero, then you can distort the trend that may be insignificant, may be tiny, but in many cases you actually want to put to apply a microscope to the differences and then you don’t need to see the zero point because what you’re interested in are the differences. And so…
JS: Right. The differences are what’s important.
MK: That’s right. And then a bar graph is not the thing to do. You see.
JS: Well, you could plot a bar chart as the difference between your observations, right?
MK: You could do that.
MK: You could do that or you have or you don’t have bars at all. You have points.
JS: Right. Right. You have points. Right. Then you’re not in the world of having to have a zero, start the graph at 0.
MK: Another bugaboo I think is this idea of connecting dots that belong to different categories. One of the reasons people use bar graphs is they believe it stresses the categorical nature of the data. So one bar is August and one bar is September, et cetera, et cetera. Um, or one bar is men. But another bar is women. And so if you had dots and do you want to connect them with a line to emphasize the difference between them, people will scream and say, oh, but you can’t connect them. If one bar is men and one bar is women, what’s halfway between men and women in this picture? It makes no sense. But you’ve got to understand the semantics of what you’re doing here. What you’re doing is drawing dots and connecting them not In order to show that there’s a continuous path from man to woman, but that there is a slope between one point and the other. And I want to emphasize it. So I think a little bit of judgment goes a long way in doing great graphs.
JS: Yeah. Yeah. That’s fascinating. Um, before I, I let you go, um, I know you’re, you’re writing a new text, a statistics textbook.
JS: And I wonder if you could talk a little bit about that and how you bring in, in particularly the data visualization that that, you know, embodies a lot of the things you’ve been talking about into that textbook.
MK: Yeah. So let me make, first, why do we need another textbook? Well, um, we need another textbook because there’s a revolution going on in statistics and the revolution is very recent. In the journal American statistician, there’s a editorial that says we recommend the journals forbid the use of significant, the word significant in reporting of data and people are hooked on the notion of p values and significance, and it’s been exceedingly harmful. So one thing that this textbook offers is an alternative to what is called Null Hypothesis. Statistical testing. Uh, it will not use the word significance. It will use p values minimally. Um, but as to graphics, one of the most important features of the book is any time we discuss a model, we will use graphical tools to do something that we call model critique. So for instance, we have tools to see if the residuals from the model are normally distributed. And so what we do is we teach the students to use certain tools in a routine way to see if data deviate in any important way from the model that they’re proposing. So we use graphics a lot and we show by example. And because this is a textbook based on R, and there will be R code all over the place, they will have examples of good practice. We’re not going to preach, but we’re going to give examples of what to do. And uh, we hope that by Osmosis and by practicing with our examples that we give in the text and on our website, we hope that people will, uh, sort of acquire best practices in their data analysis work.
JS: That’s great. When will the book come?
MK: Well, we’re aiming to give it to the publisher in about a year.
MK: So I think that publication date might be 2021.
JS: Okay. I’ll be looking forward to it. So I’m going to have to tag it on Amazon at some point, so we’ll, we’ll wait for it. So, um, that sounds great. I know it’s a, I know it’s a big undertaking to do a textbook, so I’m sure. Yeah, I’m sure you and your students are working hard on it. So, um, but it sounds great and I’m excited to see how you blend all the psychology research along with the statistics. It should be a great addition to the library.
MK: Well, thank you.
JS: Uh, professor, thanks so much for coming on the show. This has been really, uh, fascinating and uh, I appreciate you taking the time and chatting with me.
MK: Oh, I appreciate it. You’ve, I’ve been looking at the stuff that you’ve put out. I’ve looked at your site and wonderful interviews, so I’m very pleased to be one of them.
JS: And thanks everyone for tuning into this week’s episode of the show. I hope you learned a lot. I hope you enjoyed that. [Indiscernible 00:40:07] links up to the various papers and books that we talked about in the show. If you’d like to support the podcast, please consider checking out my Patreon page. Uh, I recently changed the tiers on the Patreon page, so, uh, you can, uh, support the show with as little as a dollar a month. Um, I’ve got new PolicyViz podcasts, Mo, uh, mugs. If you want to, uh, if you want to support the show and get some swag or if you, um, want to just support the show in other ways, please consider sharing it. Please consider reviewing it on iTunes or your favorite podcast provider. So until next time, this has been the PolicyViz podcast. Thanks so much for listening.