Modelers Behaving Badly

On Joseph C. Sternberg's recommendation in the WSJ, I read John Kay and Mervyn King's Radical Uncertainty: Decision-Making Beyond the Numbers. Its March 2020 release was serendipitous timing for a book detailing why experts often fail so badly to predict outcomes of grave consequence, like the Covid-19 global emergency whose consequences were first widely felt in the U.S. and Europe that month -- largely because of policies taken in response to the grim predictions of modelers. In the book, Kay and King explain why modelers are so often wrong -- largely because we use their models for a purpose for which they are not designed.

Here is an example Kay and King offer of modelers behaving badly:

"By the 1960s, greater effectiveness of modern fishing technology had led to overfishing in many locations. In 1968 an unsustainable catch of 480,000 tons of cod was harvested from the shores of Newfoundland, and cod stocks began to decline. Many countries reacted to overfishing by extending their territorial waters to protect their domestic fisherman. In 1977 Canada took control of almost all the waters of the Grand Banks.  The country set out in organized Canadian fashion to promote a fishing industry that would revive the flagging economies of Nova Scotia and Newfoundland. The government provided subsidies for trawler construction and charged the Dominion Fisheries Office with determining the total allowable catch, that catch to be set to ensure not only the survival but the growth of stocks and the gradual expansion of the industry. The Fisheries Office developed complex models on which its recommendations were based, but cod stocks continued to decline. For the year 1992, the total allowable catch was set at 145,000 tons. That proved to be the last year of commercial cod fishing on the Grand Banks. The fish are gone, and in the course of that year, the industry finally closed. Today only around 5000 tons of cod are caught annually on rod and line by recreational and artisinal fishermen.

"It would be wrong to hold the modelers solely responsible for the collapse of the Grand Banks fisheries. Greedy fishermen and mendacious politicians should take most of the blame. But the modeling exercises were used to justify the actions and inactions of politicians and big industry.  The original impulse to determine catch limits based on insights from modeling and environmental science was effectively inverted: the evidence generated by the models ended up justifying the policy rather than actually protecting fish stocks. The modelers were complicit in an environmental disaster."

The economist Thomas Sowell provides in his popular and accessible (and thus, most sophisticated readers would likely conclude, discreditable) book Basic Economics many other examples why expert-guided leadership fares no better, and often much worse, than plain common sense. Specifically on the topic of modelers behaving badly, Sowell offers this example:

"Planners can easily make false projections, either from ignorance or from various political motives, such as seeking more power, re-election, or other goals. For example, during the 1970s, government scientists were asked to estimate the size of the American reserves of natural gas and how long it would last at the current rate of usage. Their estimate was that the United States had enough natural gas to last for more than a thousand years! While some might consider this good news, politically it was bad news at a time when the President of the United States was trying to arouse public support for more government programs to deal with the energy “crisis.” This estimate was repudiated by the Carter administration and a new study begun, which reached more politically acceptable results."

These two anecdotes are not enough to draw the conclusion that models are useless (or worse), or that modelers are inept (or worse). They should suggest, however, that leadership must be informed by more than mere models, and that modelers must be measured more than by their credentials.