I once was a research assistant on a project that called on participants in an experiment to make a decision that depended on the expected value of a randomly drawn object. The scenario and instructions were printed on a bit of paper and we asked players to put a checkmark next to their choice. So far, so good.

Back in the grad student dungeon, I was counting up the responses and quickly noticed something interesting. The majority of respondents had done a little calculation in the margins of the paper to help them decide what to choose—and an awful lot had done their calculation wrong. I don’t mean they’d tried to calculate the wrong thing or misunderstood. They had a good idea of what they’d like to know in order to make their choice, but they made math mistakes, typically in calculating probabilities.

I don’t know exactly how many people in the experiment did this, but it was more than just a few. Unfortunately, the data on what people wrote in the margins wasn’t admissible anyway, since we were quite clear in the instructions about what we were going to use from the responses. And anyway, in the end the project relied on data from the computer lab version of the experiment, for unrelated reasons.

It stuck with me, though. I still think about it from time to time, especially when I’m teaching game theory, as I am again this semester (hello, ECN350!).

Uniquely inferring what strategy a person was using in a game is a non-starter when all you observe is outcomes. But when you add the marginalia there’s something deeper—even if you could get at a person’s strategy, you can’t get at what they were thinking when they formulated it.

When I’m teaching, I use data from Kahneman, Knetsch, and Thaler to introduce these idea in the simple setting of the ultimatum game. A neat working paper by Pedro Dal Bó and Guillaume Fréchette is one example of a project that seeks to solicit strategies from players rather than moves in a repeated game setting. In that paper, the authors use the solicited strategies to see if the “standard” tools for inferring strategies from observed payoffs hold up. I think this one is especially interesting because repeated games are potentially subject to quite complicated “heat of the moment” effects.

In a pedagogical sense, I think soliciting strategies is also the right way to run in-class experiments. The concept of a strategy in game theory is not an easy one to teach or to grasp. The “complete contingent plan” concept is not really at all like how it feels to actually play a game in real time. And yet all of the mathematics of game theory is built on the complete contingent plan, not the real-time version. For example: to what extent is playing a mixed strategy in rock-paper-scissors like or unlike implementing a “randomization” in real play? To what extent does the assumption of consequentialism hold up to the tension between real time play and the complete contingent plan?

There are, I think, several benefits to having students write down strategies to be implemented on their behalf rather than having them play in real time.

1. All students get to be active participants. With real time play, the majority of students are asked to think along or act as observers. Sometimes that in itself might be a valuable exercise, but it puts students in different roles and risks losing peoples’ attention.
2. The game only gets played once all students understand its structure. If a student’s written instructions are incomplete or conditions on things that aren’t known, it doesn’t constitute a strategy and will be found out.
3. it emphasizes the game theoretic meaning of the word “strategy”. The difference between strategies and real time play is in itself interesting and belongs in the pedagogy of games, but only if the technical idea of a strategy is well understood first.

I for one certainly used to rely on real time play in classes. It’s easier to administer and more dramatic. These days I prefer to have all students write down instructions for me to play on their behalf, and draw at random whose strategies will be used in the demonstration. Maybe this isn’t the most stunningly original realization I’ve ever had, but I’m glad I had it eventually.