Trust in forecasting

File:City of London skyline at dusk.jpgStephen King (global economist at HSBC) made some profound comments about forecasting in The Times (London) (paywall) yesterday.

He points out that it is only a year since the International Monetary Fund (IMF) criticised UK economic strategy and forecast 0.7% GDP growth in 2013 and 1.5% in 2014. The latest estimate for 2013 is growth is 1.9%. The IMF now forecasts growth for 2014 at 2.4% and notes the strength of the UK economy. I should note that the UK Treasury’s forecasts were little different from the IMF’s.

Why, asks King, should we take any notice of the IMF forecast, or their opinions, now when they are so unapologetic about last year’s under estimate and their supporting comments?

The fact is that any forecast should come attached to an historic record of previous forecasts and actual outcomes, preferably on a deviation from aim chart. In fact, wherever somebody offers a forecast and there is no accompanying historic deviation from aim chart, I think it a reasonable inference that they have something to hide. The critical matter is that the chart must show a stable and predictable process of forecasting. If it does then we can start to make tentative efforts at estimating accuracy and precision. If not then there is simply no rational forecast. It would be generous to characterise such attempts at foresight as guesses.

Despite the experience base, forecasting is all about understanding fundamentals. King goes on to have doubts about the depth of the UK’s recovery and, in particular, concerns about productivity. The ONS data is here. He observes that businesses are choosing to expand by hiring cheap labour and suggests macroeconomic remedies to foster productivity growth such as encouraging small and medium sized enterprises, and enhancing educational effectiveness.

It comes back to a paradox that I have discussed before. There is a well signposted path to improved productivity that seems to remain The Road Not Taken. Everyone says they do it but it is clear from King’s observations on productivity that, in the UK at least, they do not. That would be consistent with the chronically poor service endemic in several industries. Productivity and quality go hand in hand.

I wonder if there is a preference in the UK for hiring state subsidised cheap labour over the rigorous and sustained thinking required to make real productivity improvements. I have speculated elsewhere that producers may feel themselves trading in a market for lemons. The macroeconomic causes of low productivity growth are difficult for non-economists such as myself to divine.

However, every individual company has the opportunity to take its own path and “Put its sticker on a lemon”. Governments may look to societal remedies but as an indefatigable female politician once trenchantly put it:

The individual is the true reality in life. A cosmos in himself, he does not exist for the State, nor for that abstraction called “society,” or the “nation,” which is only a collection of individuals. Man, the individual, has always been and, necessarily is the sole source and motive power of evolution and progress.

Emma Goldman
The Individual, Society and the State, 1940

Advertisement

Adoption statistics for England – signals of improvement?

I am adopted so I follow the politics of adoption fairly carefully. I was therefore interested to see this report on the BBC, claiming a “record” increase in adoptions. The quotation marks are the BBC’s. The usual meaning of such quotes is that the word “record” is not being used with its usual meaning. I note that the story was repeated in several newspapers this morning.

The UK government were claiming a 15% increase in children adopted from local authority care over the last year and the highest total since data had been collected on this basis starting in 1992.

Most people will, I think, recognise what Don Wheeler calls an executive time series. A comparison of two numbers ignoring any broader historical trends or context. Of course, any two consecutive numbers will be different. One will be greater than the other. Without the context that gives rise to the data, a comparison of two numbers is uninformative.

I decided to look at the data myself by following the BBC link to the GOV.UK website. I found a spreadsheet there but only with data from 2009 to 2013. I dug around a little more and managed to find 2006 to 2008. However, the website told me that to find any earlier data I would have to consult the National Archives. At the same time it told me that the search function at the National Archives did not work. I ended up browsing 30 web pages of Department of Education documents and managed to get figures back to 2004. However, when I tried to browse back beyond documents dated January 2008, I got “Sorry, the page you were looking for can’t be found” and an invitation to use the search facility. Needless to say, I failed to find the missing data back to 1992, there or on the Office for National Statistics website. It could just be my internet search skills that are wanting but I spent an hour or so on this.

Gladly, Justin Ushie and Julie Glenndenning from the Department for Education were able to help me and provided much of the missing data. Many thanks to them both. Unfortunately, even they could not find the data for 1992 and 1993.

Here is the run chart.

Adoption1

Some caution is needed in interpreting this chart because there is clearly some substantial serial correlation in the annual data. That said, I am not able to quite persuade myself that the 2013 figure represents a signal. Things look much better than the mid-1990s but 2013 still looks consistent with a system that has been stable since the early years of the century.

The mid 1990s is a long time ago so I also wanted to look at adoptions as a percentage of children in care. I don’t think that that is automatically a better measure but I wanted to check that it didn’t yield a different picture.

Adoption2

That confirms the improvement since the mid-1990s but the 2013 figures now look even less remarkable against the experience base of the rest of the 21st century.

I would like to see these charts with all the interventions and policy changes of respective governments marked. That would then properly set the data in context and assist interpretation. There would be an opportunity to build a narrative, add natural process limits and come to a firmer view about whether there was a signal. Sadly, I have not found an easy way of building a chronology of intervention from government publications.

Anyone holding themselves out as having made an improvement must bring forward the whole of the relevant context for the data. That means plotting data over time and flagging background events. It is only then that the decision maker, or citizen, can make a proper assessment of whether there has been an improvement. The simple chart of data against time, even without natural process limits, is immensely richer than a comparison of two selected numbers.

Properly capturing context is the essence of data visualization and the beginnings of graphical excellence.

One my favourite slogans:

In God we trust. All else bring data.

W Edwards Deming

I plan to come back to this data in 2014.