People watch other people shake boxes for science: Here’s why

When researchers asked hundreds of people to watch other people shake boxes, it took just seconds for almost everyone to understand what the shaking was about.

The deceptively simple work by perception researchers at Johns Hopkins University is the first to demonstrate that people can understand what others are trying to teach just by observing their actions. Published in magazine Proceedings of the National Academy of Sciencesthe study reveals a key but overlooked aspect of human cognition, and one with implications for artificial intelligence.

“Just by looking at how someone’s body moves, you can tell what they’re trying to learn about their environment,” said author Chaz Firestone, an assistant professor of psychology and brain sciences who studies how vision and thought interact. “We do this all the time, but there’s very little research on it.”

Recognizing another person’s actions is something we do every day, whether it’s guessing where someone is heading or detecting what object they’re reaching for. These are known as ‘pragmatic actions’. Numerous studies have shown that people can quickly and accurately identify these actions just by observing them. Johns Hopkins’ new work examines a different kind of behavior: “epistemic acts” that take place when someone tries to “learn” something.

For example, someone may put their foot in a pool because they are going for a swim, or they may put their foot in a pool to test the water. Although the actions are similar, there are differences, and the Johns Hopkins team hypothesized that observers would be able to detect another person’s “epistemic goals” just by observing them.

In several experiments, researchers asked a total of 500 participants to watch two videos in which someone picked up a box full of objects and shook it. One shows someone shaking a box to find out the number of objects in it. The other shows someone shaking a box to find out the shape of the objects inside. Almost every participant knew who was shaking for number and who was shaking for form.

“What’s surprising to me is how intuitive this is,” said lead author Sholei Croom, a graduate student at Johns Hopkins. “People can really understand what others are trying to understand, which shows how we can make these judgments, even though what we’re looking at is very noisy and changes from person to person.”

“When you think about all the mental calculations that someone has to do to understand what someone else is trying to learn, it’s an extremely complex process. But our findings show that this is something that people do easily,” Firestone added.

The findings could also inform the development of artificial intelligence systems designed to interact with humans. A shopping robot assistant, for example, that can look at a customer and guess what they’re looking for.

“It’s one thing to know where someone is headed or what product they’re reaching for,” Firestone said. “But it’s another thing to deduce if someone is lost or what information they’re looking for.”

In the future, the team would like to pursue whether people can monitor someone’s epistemic intention versus their pragmatic intention – what they do when they dip their foot in the pool. They are also interested in when these observational skills emerge in human development and whether it is possible to build computational models that describe exactly how observed physical actions reveal epistemic intention.

The Johns Hopkins team also included Hanbei Zhou, a sophomore studying neuroscience.