What economists do: theory versus empirics

Down to dirty work today, as I make the bold claim to start talking about the guts of the economics profession. What are we up to? The first distinction in economics research methodology is ‘theory’ versus ’empirics’. Specialization has gotten to us in a big way here, in that theorists and empiricists don’t really associate at all.

So what’s what? Both methods are trying to attack similar questions – what happens if this changes, how do I achieve this, what is the relationship between these things – but use very different standards of proof. A theoretical ‘proof’ is to create a simplified model of reality to speculate on how the things might be related, while empiricists dig into big datasets to try to find the real-world relationship, the common problem being that things are pretty complicated. When economists talk about “applied economics”, they are using a label for the practice of statistical analysis of data in empirical economics research, so in some sense “applied” is not really an informative word here.

When we actually want to answer questions, say for policy analysis or just because we care, it is obviously smart to draw on diversity and explore the theoretical reasoning behind the relationship you’re interested in as well as whatever suggestive real-world evidence exists. Being that this isn’t what economic research papers do, this isn’t what economists do, though: we all do either one or the other whenever we write a research paper. Every economist is, first and foremost, a theorist or an empiricist (or both, but you see what I mean – they are distinct concepts at all moments).

The problem for empiricists is, in a way, harder than for theorists, because finding meaningful relationships in real data is surprisingly difficult, and assuming something away is a much more technical proposition when you have to kill it in your actual data rather than just in your abstraction. For example, if I see that the airport built a new terminal and that house prices went down, I can certainly argue that one caused the other, but actually proving it is a very different proposition. Econometrics is the branch of economics that tries to develop methods to analyze data where it’s difficult to infer causality. Of course, this problem is common to all statistical analysis, not just economics, and it is surely true that really strong evidence is revealed without fancy techniques.

A lot of economists do “applied economics”. Now this is going to be mostly just an anecdotal claim, but it’s certainly plausible to argue that the things that made economists decide to become economists seldom include a burning desire to trawl through huge datasets and run a bunch of regressions; the questions that can be answered in this way are interesting, sure, but the work itself is not a lot of fun. On top of that, despite the positivist teaching of economics, the proportion of time spent on the empirical methods is very, very small compared to the proportion of economics research that is empirical. Not that this is a bad thing: there isn’t a huge amount you can say about empirical methods before you’re actually in a position to use them (and again: not that much fun), but it might be presenting a drastically skewed picture of what it means to be an economist.

There’s actually a bit of a rift within empirical economics about the role of theory, which is a different matter entirely – I’ll try to paraphrase to the best of my ability. That rift concerns the seed of the empirical test being done – should it be explicitly associated with a theoretical model of the relationship you’re looking for in the data (that’s ‘structuralist’), or should the data be allowed to speak for itself and leave models out of it (‘reduced form’)? Now, the funny thing is that, as we know, it’s possible to write down a self-contained and consistent theoretical model that proves any relationship you want; the value of the model depends entirely on how you judge the value of its own little world. Thus, employing theory as some kind of dual proof while doing empirical work is actually redundant; it can offer some clarification of what you think might be driving the relationship you’ve found in the data, but it’s not especially helpful to say “hey, I found this empirical evidence – and look, the model says the same thing!”.

Which, again, is different from the idea of puzzling out a theoretical idea then trying to find evidence to see if it’s true or not. This kind of thing is actually not incredibly popular, perhaps because of the vastly different worlds theorists and empiricists orbit in – different methods, different seminars, different journals. The paradox is thus that very little empirical economics research actually tests theoretical economic hypotheses. Does each approach lend itself to different questions, never the two to meet, or is it in fact just that we don’t like following on each others’ coattails?

Back to the big point. Let’s say I’m a research economist and I’m thinking of a question like this: “would a national health service be good for the United States?” What I will not end up doing is writing an answer to that question, drawing on the arguments and evidence from a variety of sources. The economist’s role in answering such questions depends on which flavor of economist he is. The theorist might end up asking “how would it change the problem for an individual if they were faced with a national health service rather than the current system?” She might create a little model of a person facing choices between spending their money on health care or on other things, who goes on to interact with an insurance company in one instance or the new health service in the other, and figure out how that person’s choices might plausibly change.

The empiricist might end up asking something like “how does the size of a deductible affect people’s health care spending?”, since this might tell us something about the zero-deductible world of national health care, or “how do wait times affect health outcomes?”. Note that to answer the original question – should the US switch systems – using any kind of data, or indeed any kind of theoretical model, is staggeringly complicated and difficult.

Neither type of economist actually writes about the answer to the big question in their academic research. Instead, they go to the questions that their method might be able to answer, making just one brushstroke on the painting of the argument, and for theorists and empiricists, those questions are very seldom the same.

Measurement

Just after talking about the euphemistic use of “economy” yesterday, I found an even better one here:

“Nearly all Italians drink bottled water rather than the piped stuff. The industry is worth an estimated 3.2bn euros (£2.38bn) a year to the Italian economy.”

It would be very refreshing if they’d just say “GDP”, since that’s what they mean. That wouldn’t make it any less understandable either, because “economy” is equally vacuous. Let’s play the show and tell game again: what does the quotation mean?

It can’t mean that “if no bottled water was sold, people would spend 3.2bn euros less” – I’m sure they’d find another way to spend it. It can’t mean “worth 3.2bn euros a year to the Italian resource allocation”, because that’s not a sentence. It can’t mean that “Italian workers/producers would get 3.2bn less in wages/payments a year”, because I’m sure that they could do something else besides produce bottled water.

My best guess is “the Italian bottled water industry makes sales worth 3.2bn every year”. Why, oh why, can’t the reporter simply say that? It’s not remotely the same thing as any of the other suggestions I just made, yet I guess they’re all technically possibilities if we read “economy” as “system of production and consumption” or something like that. If I want to be really obnoxious I could ask whether the reporter has measured every consequence of the hypothetical disappearance of the Italian bottled water industry to come up with his figure.

More to the point, let’s forget about the absurdity of the quotation in itself and ask why the “worth” of any effect on the “economy” measured in money? This screams a confusion of metric and quality, a cardinal sin of positive science; even if I could get an accurate figure for the effect of something on “Gross Domestic Product”, I still think “worth” is too loaded a term.

A big chunk of the gulf between theoretical economics and empirical testing of real-world relationships is the metrics we use. Our abstractions work (or can, or should work) in a world where we measure outcomes agnostically: if you care about this. Theoretical economists can play in imaginary worlds all day, exploring the “relationships” between fundamentally unmeasurable things under their assumptions. On the other hand, some imaginary concept like “utility” is singularly useless if we want to actually talk about the real world. Empirical economists must deal with this problem somehow: if you want to talk about the effect of this measurable thing on that measurable thing you must explicitly ignore the intangible (like, perhaps, satisfaction).

Then what conclusions can we draw? This affects that, but not how “good” it is. This is, again, the reason why economics can never be a technocratic prescription of what “should” be done; we simply have no real-world metric to answer the question, and our theoretical metrics are unobservable. It’s the power and beauty of the science – we don’t have the answers. Is someone pretending to? Just for fun, I Googled “what’s wrong with GDP”. When our metrics are the sole determinant of policy, of course the metric – and, by extension, economics – comes under intense attack.

Now that’s all well and good until we get to economics teaching, practice and discussion which ignores this important conclusion. I don’t deny the challenge of constant vigilance to make sure student, reader, researcher know that we’re dealing with only what we can measure, but nothing short of a commitment to acknowledge the limitations of measurement at every turn will be enough to dispel the notion that the science of economics can tell us what to do.