Decision-making under uncertainty: heuristics vs models

This post is by Phil Price, not Andrew.

Sometimes it’s worth creating a complicated statistical model that can help you make a decision; other times it isn’t. As computer power has improved and modeling capabilities have increased, more and more decisions shift into the category in which it’s worth making a complicated model, but often it still isn’t. If you’re trying to make a decision about something that is affected by many different factors which interact in unknown ways and are controlled by parameters whose values you don’t know very well, it’s probably not worth your trouble to try to make a detailed model.

To take a current example, if you want me to predict how many Americans will have died of COVID-19 by the beginning of June, 2021, I’m not going to try write a model that simulates all of the political, social, medical, and environmental factors that go into that number, I’m just going to make up something that seems reasonable to me based on my general sense of how all these things interact. Presumably I could learn just a little bit more by making that complicated model — at least it might help me understand what the most important parameters are — but in practice the uncertainty in the numbers coming out of such a model is going to be so large that I don’t see how it could be worth the trouble.

Although I’m pretty sure it would not be worthwhile to try to build a model to answer the question posed above, there are plenty of other cases in which making a model is well worth the trouble.

Which brings me to my current predicament: I have a client who wants some advice (on how to decide how much electricity to buy in advance at a fixed price, rather than on the day they use it at a variable price) and I keep going back and forth about whether it’s worth trying to build a detailed model for this. I just don’t know how much there is to gain from such a model, compared to just using some rules of thumb to make the decisions, and I think that even figuring this out will take a lot of work. How should I proceed?

The problem involves electricity bills for a large company that uses a lot of electricity (their annual bill is something like $100 million). Unlike homeowners, they are exposed to real-time fluctuations in electricity prices. If there’s an unusually warm spring in the Sierra Nevada mountains, all the snow melts into water and has to be released through the dams, so there’s a lot of hydropower and electricity gets cheap for a while in Northern California…but then that hydro isn’t available later in the year, so if there’s a warm summer and the natural gas “peaker” plants are in heavy use, the electricity price will be highly dependent on the cost of natural gas, which could be high because…well, you get the idea. 

The company had normally been content to pay whatever the market price happened to be, with fairly predictable seasonal variability and occasional mildly pleasant or mildly unpleasant surprises …until 2018, when some usual events led to extremely high prices for a short time at a few facilities. When I say ‘extremely high’ I mean it: the price per MWh was about 200 times the typical price for a few hours, and about 30 times normal for several days. That was bad enough — spending an extra month’s electricity budget for those facilities in a few days — but of course there was no guarantee that that period was the worst it could ever be. What if it happened again, with even higher prices and for an even longer duration? It had taken some pretty unusual events to lead to those high prices, so it’s not like this is likely, but it’s possible. 

The company responded by starting to buy some electricity several months in advance. There’s a standard market for this; you can buy such-and-such an amount of MWh for next June at a specific delivery location for $y per MWh. The basic idea is that you are worried that the price of electricity next June might be exceptionally high, but some electricity producer somewhere is worried that the price might be exceptionally low, so you’re both willing to make a deal. Actually there are third-party energy traders who are heavily involved in the market too.

I am part of a three-person consulting team that is advising the company. We started by considering a single facility, i.e. a single warehouse. Historically, how predictable are electricity prices, and what does the distribution p(price | predicted price) look like? Here we take the market price as the predicted price. Similarly, how predictable is a facility’s electric load, and what does the distribution p(load | predicted load) look like? We have a simple model based on historical data, and we can quantify how well it works. 

Actually, we need p(price, load | predicted price, predicted load), since there is correlation between the error in the predicted price and the error in the predicted load. For instance, unusually hot weather can lead to higher energy prices (because higher demand for air conditioning) and higher electric load in the company’s facilities (ditto).  

Even for a single facility it’s hard to know exactly how to model all of this: we only have a few years of useful data — because both the facilities and the electric industry itself have changed a lot in the past five years, and are continuing to change — so that’s only a couple of dozen summer months for example. That’s not really enough to know how to parameterize the problem, e.g. to characterize the tail of the distribution that is relevant for 1-in-20 or even 1-in-10 events. But the problem here is at least something we can wrap our heads around, and it’s not too hard to model: Once we learn the price at which we can buy electricity next June, we generate a lot of possible futures, meaning what the distribution of real-time prices next June might look like, and how much energy the facility might need, and we have an objective function that we optimize conditional on those distributions. We are comfortable with this, and so is our client. 

But now we want to move on to optimizing over many facilities and many months. It’s a big company with facilities all over the country, so a lot of things will at least partially average out on a company-wide scale. Electricity prices are correlated across the country, but they are not perfectly correlated (coal, hydro, wind, solar, nuclear, and natural gas prices don’t vary in lockstep, and there are transmission losses and transmission bottlenecks that stop electricity from flowing freely all across the country.)  If the company as a whole wants to avoid spending far more than expected for energy, they are already partially covered simply by being spatially diverse. They don’t need to make sure the energy prices are under control at every single facility, as long as they aren’t crazy-high at too many of them.  Similarly, they can absorb high prices for a month or two, as long as it doesn’t bust their budget for the year (or maybe for half a year). Suppose the company wants to (try to) make sure their electricity budget next year doesn’t exceed by more than 15% what they have budgeted. It’s pretty clear that the optimal decision for the company, as far as the amount of electricity to buy in advance, is going to be less than the amount that would be obtained by trying to make sure they don’t go over 15% at any facility, in any month.

So we’ve started working on how to do the multi-facility, multi-month generalization of the problem we’ve already solved…and it’s a bit of a mess. The problem is mathematically straightforward, it’s just that when we get to the modeling decisions we don’t really trust them. We need the variance-covariance matrix for the errors in the predicted prices between the facilities, and the variance-covariance matrix for the errors in the predicted electric load between the facilities, and we don’t have nearly enough data to estimate those with any confidence. We’re more or less OK about the diagonals of those matrices, which is what goes into the single-facility single-month stuff we’ve already done, but the off-diagonal elements are going to be very poorly estimated. And of course the results will be conditional on the functional forms we’ve chosen for the price and load variability, for which we care a lot about the upper tails but have few data up there. 

One option is: quit complaining and write the model. We’ll have a fairly big, fairly ugly, fairly complicated model and will have really uncertain estimates of a lot of the parameters, but that’s just life sometimes. And we can use the model to do some sensitivity analysis to figure out what additional information might help us. 

Another option is to just sort of wing it. The company could do the single-month, single-facility optimization for each facility and then just cut the recommended purchase for each facility in half, for example.  Or they could purchase the full recommended amount at the facilities that face the highest price uncertainty and not purchase any at the other facilities (there is regional variation in price volatility). We could look at historical data and come up with some heuristics that seem to work OK. 

Trying to write the whole model would be kinda fun, but honestly I’m not sure I’d trust the results more than I would trust some heuristics based on expert judgment that we can elicit from someone who is familiar with electricity futures trading. 

So I’m putting this out there for advice, if any of you have it. 

This post is by Phil