Heckman Curve Update Update

tl;dr: “The policy conclusion we draw from our analysis is that age is not a short cut for identifying where governments should, or should not, invest. There are many well‐studied interventions for children that are worthy candidates for public funding based on efficiency considerations. However, the same is also true of many interventions targeting youth and older people. . . .”

Here’s the background. David Rea writes:

Just letting you know what eventually happened with our exchange with James Heckman over our ‘Heckman Curve’ paper in the Journal of Economic Surveys.

When we last emailed, James Heckman had written a response to our paper, and this had been accepted for publication in the JES.

The journal then offered us the opportunity to write a reply, which we did, and this was published as an early view.

After this James Heckman decided to withdraw his paper. We don’t know why he changed his mind about publication, but it did seem a little unusual.

If you are interested, here is a bit more background on the debate that didn’t eventuate. This is the Heckman Curve as published in Science in 2006:

Our paper attempted to see if more recent data showed a similar pattern. We used a dataset of benefit cost ratio estimates for 314 well-studied interventions from the Washington State Institute for Public Policy database.

We couldn’t find any evidence of a Heckman Curve in the data.

To be clear, the dataset has many early childhood interventions that generate large estimated benefits relative to the program cost. There are many early intervention programs for children that are worthwhile investments from an efficiency point of view. However, the key point is that there are many well-studied youth and adult interventions that are also very cost effective.

Heckman’s unpublished response to our paper contained several criticisms. The central argument was that we were misinterpreting his work. He stated that the Heckman Curve does not describe how the return on investment of human capital interventions differs by age. It is instead a theoretical proposition or thought experiment about an optimal portfolio of best-practice investments in human capital.

We actually thought that this was a very useful clarification. Many people, particularly in the policy community, understand the Heckman Curve as describing how average rates of return of human capital interventions differ by age. Clarifying this point has important real-world implications for social policy. It’s a funny thing to say given that he is critical of our work, but we think it would be useful for James Heckman to publish his paper.

We discussed David Rea and Tony Burton’s paper, “New evidence on the Heckman Curve,” last year.

Apparently the response by Heckman will never be published.

But, in any case, here’s the reply by Rea and Burton to the response that Heckman withdrew:

Clarifying the Nature of the Heckman Curve

David Rea and Tony Burton

Abstract: In response to our paper, James Heckman states that the Heckman Curve does not describe how the average return on investment of programs differs by the age of recipients. This clarification is useful as many people in the policy community have understood the Heckman Curve in this manner.

In our paper, we interpreted the Heckman Curve as a proposition that (a) social policy interventions targeted at early childhood would generate benefit cost ratios that were higher than other age groups, and (b) interventions targeted at older age groups would often have benefits smaller than their costs (Rea and Burton, 2020).

Our characterisation of the Heckman Curve was based on the paper by James Heckman entitled ‘Skill formation and the economics of investing in disadvantaged children’. This was published in Science in 2006 and stated that ‘early interventions targeted toward disadvantaged children have much higher returns than later interventions such as reduced pupil teacher ratios, public job training, convict rehabilitation programs, tuition subsidies, or expenditure on police. At current levels of resources, society overinvests in remedial skill investments at later ages and underinvests in the early years’ (Heckman, 2006, p.1902).

The natural interpretation of the quoted statement is that Professor Heckman was discussing interventions, investments in human capital and was advising decision‐makers inside and outside government to target investments in a particular way.

The empirical component of our paper used a dataset of benefit cost ratios estimated by the Washington State Institute for Public Policy. Analysis of the dataset showed no readily apparent relationship between the estimated benefit cost ratios of interventions and the age of recipients.

The policy conclusion we draw from our analysis is that age is not a short cut for identifying where governments should, or should not, invest. There are many well‐studied interventions for children that are worthy candidates for public funding based on efficiency considerations. However, the same is also true of many interventions targeting youth and older people. A number of early intervention programs have been shown to be cost effective, as have a range of ‘remedial’ or ‘second chance’ programs targeting older individuals. Good public policy requires a case‐by‐case assessment of the evidence and benefit cost analysis for each intervention being considered.

In his comment on our paper, Heckman states that we have misinterpreted the Heckman Curve (Heckman, 2020). He states that it is not a proposition about how the average return on investment of programs differs by the age of recipients. We welcome this clarification and believe it is a useful outcome of the exchange between us. Many people including ourselves have previously understood the Heckman Curve to be advice on public investment in human capital programmes. It is helpful to understand that his work in this area is not meant to be relevant to public investment decisions in the manner we describe.

References

Heckman, J.J. (2006) Skill formation and the economics of investing in disadvantaged children. Science 312 (5782): 1900–1902.

Heckman, J.J. (2020) Comment on “new evidence on the Heckman Curve” by David Rea and Tony Burton. Journal of Economic Surveys.

Rea, D. and Burton, T. (2020) New evidence on the Heckman Curve. Journal of Economic Surveys 31 (2).

So, it all seems clear. Heckman published a pretty graph in 2006 that was misinterpreted by many people as implying that “early interventions targeted toward disadvantaged children have much higher returns than later interventions such as reduced pupil teacher ratios, public job training, convict rehabilitation programs, tuition subsidies, or expenditure on police. At current levels of resources, society overinvests in remedial skill investments at later ages and underinvests in the early years.” But that was a mistake. In fact, the data don’t seem to support the claim that human capital investments are most effective when targeted at younger ages, and Heckman appears to agree with this, to the extent of wanting to emphasize that his curve is not a proposition about how the average return on investment of programs differs by the age of recipients.

On the other hand, Heckman still has this up at his webpage:

The Heckman Curve

This graphic shows that the highest rate of economic returns comes from the earliest investments in children, providing an eye-opening understanding that society invests too much money on later development when it is often too late to provide great value. It shows the economic benefits of investing early and building skill upon skill to provide greater success to more children and greater productivity and reduce social spending for society.

This graphic is formatted for use on social media and insertion into presentations, handouts and press releases.

Summary

I’m not sure what to make of Heckman’s advice. On one hand, on his website he is clear that his eponymous curve shows that “The earlier the investment, the higher the return.”

But in his (now unpublished) response to Rea and Burton, he wrote:

None of this says that the benefit-cost ratios or internal rates of return are necessarily higher for all younger-age interventions. . . . The Curve is a technological frontier across programs (best practice) and not an average across all programs, however poorly executed . . . Policy makers need advice on best practice, not on average practice.

So we should modify the advice on his website to “In best practice, the earlier the investment, the higher the return.”

I’m concerned, though, because I think that the statistical methods used by Heckman and his collaborators cause them to overestimate effect sizes. Early-childhood intervention supposedly increasing adult earnings by 42%. Noisy studies with huge standard errors, thus the statistical significance filter yields high effect sizes. The upshot is that I have no idea what are the effects of these supposed best-practice interventions—and I think Heckman has no idea either. The main difference between Heckman and me here is that he’s expressing a lot more confidence than I am in those noisy estimates.

I have two problems with the statement, “Policy makers need advice on best practice, not on average practice.” First, we really don’t know what best practice is. These interventions that Heckman is so sure are best practice, maybe aren’t. Second, even if these interventions were best practice, to be scaled they’d have to be applied in the real world, i.e., average practice. It would seem naive to think that whatever particular interventions had been tried in some experiment many years ago could just be applied directly in new settings.

In any case, I agree with Rea that these are all good things to be discussing in the open. He and Heckman and Charles Murray and Aaron Edlin and Anna Dreber and Rachael Meager and anyone else are welcome to reply in comments.