Interaction

Winter 2007

Interacting with our computers

L.A Cicero

Scott Klemmer, co-director of the Human-Computer Interaction group, works with a class on writing a profile of a make-believe student who is interacting with the software designed by the group.

Since the rise of industrialization, or perhaps even earlier, machines have been regarded as potential threats to our humanity, something oppressive, worthy of suspicion, certainly external. Machines were not our friends.

If they are not exactly our friends today, our relationship with them, what practitioners call the user interface, has become a matter of understanding, communication, even empathy. We and our ubiquitous machines must interact in ways that make sense, both in terms of our sensibility and in terms of results.

Courtesy Pamela Hinds

Zoe, the "robotic astrobiologist," works in the Atacama desert in Chile for a group of researchers in Pittsburgh.

The field of Human Computer Interaction (HCI), which at Stanford is a group within the Computer Science Department, is dedicated to minimizing the barrier between human cognition and experience, on the one hand, and software and hardware, on the other. Researchers work at both ends of this relationship—the social and the mechanical—and in between. They come from the social and behavioral sciences, engineering, computer science and education.

Like “environment,” which implicitly suggests a sphere in which nature and humans interact—that is, a terrain both external and internal to us—HCI embodies a relationship and, implicitly, responsibility.

HCI means different things to different people, depending where they are on the scientific spectrum and on whether they are interested in the relationship between a particular user and his technology or among many users who communicate via technology. Though we have a one-on-one relationship with our machines, our relationships with each other and, in fact, with ourselves, are also mediated by them.

Scott Klemmer, an assistant professor of computer science and co-director of the HCI group, said HCI “studies people acting through technology. The difference between HCI and other areas of computer science is that for much of computer science, the metric of success is the system [speed, capacity, etc.], while for us what matters is the user experience.”

With a background in design and computer graphics, Klemmer felt that an element was missing from many of his computer science classes when he was an undergraduate. “It was all about how we implement technology, and I wanted to know why we implement it. I wanted to create tools to enable designers and users to be more creative and to think about how computing can be better integrated into the practical logic of everyday life.”

L.A. Cicero

Scott Klemmer, left, with Matt James and Ryan Park, in Klemmer's project-based class, Human-Computer Interaction Design Studio.

Conceptually, his work lies at the intersection of computer users, computer scientists and designers. It is a big intersection, with lots of interaction, for there is virtually no part of our lives that is not linked to computer technology and that couldn’t be linked better. With the goal of enabling a prototyping culture, an expression heard often at the Design Institute (see related story), Klemmer has worked on such projects as a pen-and-notebook system that combines the best of paper and computer record-keeping; field research tools that make sense for researchers in the wild; a study of technological mash-ups (composites of online or hardware sources) for “opportunistic design”; and a system that balances physical and virtual design with the simultaneous development of hardware and software. At the heart of his work always is a preoccupation with human needs in the real, everyday world.

Getting there from here

Two components of everyday life studied by a psychologist who works on HCI projects are route maps and assembly instructions, that bane of every casual furniture shopper. Clearly, this is an area where the technology (in this case, visualization) does not respond to ordinary human cognition.

Barbara Tversky

Psychology Professor Emerita Barbara Tversky and her collaborators in computer science started off by working on maps. They wanted to use computers to represent cognitive design principles in algorithms that could then automate the generation of effective visualization. In other words, computers could be made to understand and illuminate how people actually visualize directions.

With undergraduates as her guide—they were asked to draw a map to a nearby Taco Bell—Tversky figured out how people think in sequences and hierarchies and how much information they actually need in order to get where they need to go. Then, graduate students Maneesh Agrawala and Chris Stolte produced the algorithms that could generate maps. (Both have since earned their doctoral degrees.)

From there the team bravely moved on to assembly instructions, specifically for a television cart. The process was much the same; once again, undergraduates were observed, and “computer-generated instructions won hands-down” compared to manufacturer’s instructions that came in the box or to the best hand-drawn ones, Tversky said. (The resulting software, developed by the graduate students, is called LineDrive.)

Tversky’s fellow traveler in much of her work has been Pat Hanrahan (adviser to Agrawala and Stolte), described by one of his colleagues as the world’s best when it comes to visualization. Hanrahan, the Canon USA Professor in the School of Engineering, has twice been honored by the Academy of Motion Picture Arts and Sciences (the Oscar people) for scientific and technical achievements in digital imaging. He says he “builds tools,” which include rendering software and graphics hardware that transform vast amounts of data into visualizations. Both Simbios and the Institute for Computational and Mathematical Engineering consult with Hanrahan, who was trained as a biophysicist.

L.A. Cicero

Pat Hanrahan, a professor in the Computer Science Department, works with scientists, engineers and physicians to help them use visualizations to improve their work, convey information and support reasoning.

“I was always interested in scientific visualization, how people envision math and how mathematicians can share their information,” he said. “If you think historically, you see techniques that we think are obvious, but they weren’t; they took forever. Bar charts, for example, started being used only 150 years ago.” Tversky too points out that “metaphorically visible” visualizations such as pie charts were late arrivals.

Hanrahan says he uses graphical techniques to convey information or support reasoning. His collaborators include psychologists, engineers, physicians, mathematicians and physicists. All those people need to visualize their data and their concepts; computers can make that happen, but someone has to ensure that the results make sense to humans and respond to the questions they’re asking.

Of course, different people ask different questions. As Tversky might say, they have different mental representations of space.

“The way you picture things depends on the questions you ask,” Hanrahan said. “So that’s an HCI point of view, trying to help people solve specific problems, not just make cool pictures. We build tools.”

The flexible definition of HCI means that different universities house it in different ways. HCI is a degree-granting institute within Carnegie Mellon’s computer science school; a subgroup within Berkeley’s department of electrical engineering and computer sciences; a nucleus of courses within MIT’s Media Lab; and a degree-granting program sitting between the School of Psychology, the School of Literature, Communication and Culture and the College of Computing at Georgia Tech.

Artificial intelligence

One of Klemmer’s closest faculty associates in the HCI group is Terry Winograd, artificial intelligence pioneer, founder of Computer Professionals for Social Responsibility and adviser to untold numbers of students who have gone off and changed the world. His shift from artificial intelligence to HCI and design was as much a philosophical one as a mechanical one, he said.

Terry Winograd

“Design and research are two ways of thinking,” he said. “With research you ask, which is the faster mouse? And you can test it. But with design, experience is what’s important. You can’t be precise about that, you can’t measure it. You have to meet needs that are not well specified. An old person will just say, ‘I want this computer to help me.’ So you learn by talking, observing, watching.”

Computers may have been getting smarter a few decades ago, Winograd said, but they weren’t getting any easier to use. They were not meeting those ill-specified but nonetheless crucial human needs.

He was not alone in his observation. The Department of Mechanical Engineering began making institutional moves decades ago in recognition of design deficits; down the road, those changes would lead to the establishment of the Design Institute. At the same time, the Computer Science Department, which moved out of the School of Humanities and Sciences and into the School of Engineering in 1986, began looking at the “why” instead of just the “how” of computing.

Two new concepts began attaining prominence: ubiquity and empathy. By 1990, when the HCI group was formed, computers were no longer bulky things sitting on desks. They were small and mobile, and their technology was not even confined to artifacts called “computers.” Chips and handhelds and GPS devices began showing up in the most unexpected locations. It was the start of the era of ubiquitous computing, or ubicomp, defined by Klemmer as “technology that supports embodied cognition and that is integrated into everyday life.”

And with ubiquity comes the recognition that all people in all places at all times might require or benefit from technology and therefore have to be able to use it in a productive and enjoyable way. Enter the social scientists.

As Pamela Hinds, associate professor of management science and engineering (MSE) says, “It’s very easy for people to design things for people similar to themselves.” Most people, however, are not like computer engineers.

Hinds, who has a PhD in organizational science and management, is co-director of the Center for Work, Technology and Organization in MSE. Just as many HCI experts work on the interface between technology and individual users, people in Hinds’ field work on the effects of technology on groups and teams.

Here the question is not just how a technology affects a user, but how it affects workers’ ability to communicate with and understand each other, often at a distance. The disciplinary underpinnings of such work can be found in the behavioral sciences: anthropology, sociology, communication and psychology.

As an example, Hinds is working with colleagues at Carnegie Mellon on a human-robot project that links Chile to Pittsburgh. In the Atacama Desert, in northern Chile, a four-wheeled, solar-powered robotic astrobiologist named Zoë goes about picking up biological and geological data for a group of scientists in Pittsburgh. For several years, Hinds’ team has been observing both ends of the conversation to track the inevitable missed signals and misunderstandings, which are as much behavioral or cognitive issues as technical ones.

Essentially, robots have to be trained to be more perceptive about humans and to provide enough contextual information to the scientists so that the latter are able to form sound conclusions. In Hinds’ words, the robots have to be “creative communicators.” They have to know what to say and when to say it.

In this study, Hinds relies on common ground theory, which social scientists use to assess the chances of successful collaboration. In this case, Hinds’ team reported, “the interactive process … was problematic.” Just as a human being needs to know what another person’s knowledge, attitudes and expectations are in order to have a fruitful conversation, so too with robots.

Sharing knowledge

In a similar fashion, certain technologies may enable humans to share knowledge, not just in the technical sense of moving a file from one place to another but in the sense of generosity or inspiration. These matters are addressed by the subfield of HCI called computer-supported cooperative work, which is Hinds’ specialization. Groupware, social bookmarking, blogs and wikis are all examples.

Jan Chong, one of the students in the Center for Work, Technology and Organization, for example, is examining two software development teams; the members of one are all in the same space, while the members of the other are dispersed. She is interested in the degree to which technology helps them share knowledge. Beyond the technical aspects, she is interested in the ways in which people conceive of knowledge-seeking.

“What is the best environment for people to seek knowledge in?” she asked. “We don’t really know.”

In a related vein, Kathy Lee, a second-year student in MSE, is looking at social bookmarking. (Lee has a master’s degree in HCI from the University of Michigan’s School of Information, formerly known as library science, where many HCI programs are housed.)

“Why do people share? Why not? How do people react to social pressure, to the knowledge that someone else is close by?” she asked, referring to the online community created by social bookmarking. If you are part of this community, you can track the back-and-forth suggestions and annotations. If you see that someone benefited from one of your bookmarks, you may be more likely to give back, and Lee wants to measure that.

Chong and Lee describe themselves as inhabiting a space somewhere between computer science and the social sciences—and being very happy to be there, though aware of the difficulties.

Being on the job market, for example, which Chong is, means figuring out what other universities call this hybrid field. It might be computer science or information sciences or management sciences or business or HCI itself. The Chronicle of Higher Education lists jobs by schools or departments, not by field, so there is no efficient way to determine where the openings are. Chong finally did it backward, going to websites of departments that interested her, regardless of what they were called, and checking for openings.

“The biggest challenge is learning to speak everyone else’s language,” she said. “My work crosses so many disciplines, but translating that is really difficult. Some people say it’s too technical, other people say too organizational.

“So you just have to highlight certain things, and suddenly everyone is interested. You have to orient people to see the interesting parts. It’s funny, because I see the whole picture. But when I talk about my work, I have to learn to say it’s A or it’s B or it’s C.”

Hinds considers herself fortunate to be in an engineering school. Most people who do similar work are in business schools and, for the purposes of evaluation, they publish mostly in business journals. Likewise, if she were in a sociology department (which she could be), she probably would publish in sociology journals.

“The real challenge,” observed Lee, “is when people say, ‘What is your work on?’ So you say, ‘It’s on this and on that.’”

“There’s tension inherent in any interdisciplinary field,” Chong agreed. “There are many perspectives about the end result. You wonder, where am I putting my eggs? There are tensions about the direction of the field, but it’s a good conversation.”

Bridging the worlds of the physical and the digital seems only natural to someone like Klemmer, who has worked in multiple disciplines since the beginning of his schooling.

“Perhaps civilization’s biggest screw-up came when René Descartes said, ‘I think, therefore I am,’ separating mind and body,” he said. What Descartes didn’t know is that it’s all happening in the interface.