13 November 2013
Nothing draws more criticism from both the public and the media than a government project that has gone over budget. This alone is enough to categorise it as a failure, whether or not it has achieved its aims. The assumption tends to be that this failure is the result of incompetent implementation rather than whether the cost estimate was wrong in the first place.
Too often, major government projects are seen as a 'done deal', with investment approvals just a hoop to jump through. The areas that deserve additional scrutiny – particularly financial modelling – mostly just receive a cursory glance. Instead the approvers focus on affordability, and whether the estimated cost falls within budgets, and do not see their role as questioning the underlying analysis. This lack of scrutiny is made worse because the project team has probably relied on 'someone from finance' or an external consultant to put together the financial model.
Financial modelling is a specialist skill and most people would rather have their fingernails removed than try to understand the inner workings of a large and complex Excel model. But figures do need to be checked. Indeed, the public accounts committee said in the wake of the West coast mainline re-franchise fiasco that the Department for Transport's senior management did not sufficiently probe the information provided by the project team. As a result, no one spotted that inflation had been left out of the calculations.
It was also recently reported that the much-cited analysis by two Harvard professors which investigated the relationship between public debt and economic growth, and on which George Osborne based his austerity programme, included a basic spreadsheet error. This shows small oversights can have huge ramifications, and these high-profile examples are likely to be just a small fraction of the total.
Clearly the answer is not to train every project manager to be a spreadsheet expert who can conduct their own line-by-line audit of every model. However every reviewer needs some awareness to allow them to check what they are being told makes sense. There are five key questions they should ask to help them do this.
1. Do the numbers feel right when compared with other similar projects? Following the Holyrood inquiry, Lord Fraser remarked of the Scottish Parliament that "this unique one-off building could never ever have been built for £50m and I am amazed that for so long the myth has been perpetuated it could". The final cost was £414m. If your financial model is telling you that your project is going to be significantly cheaper than a comparable scheme, then there is probably something wrong.
2. Are there any patterns or trends in the costs and are they showing what is expected? In any project there should be a general idea of how the cost profile is likely to change over time. There might be a large spend at the start and then a fairly steady operational phase where costs rise gently year on year with inflation. Any anomalies, such as a sudden unexplained increase or decrease, should be probed as they may indicate an error.
3. Where does the data come from? Determined efforts must be made to ensure all source data is reliable. Real cost information from actual suppliers is obviously always best but not always available, particularly in the early stages of a project. Where estimates are used, they should have some sort of evidential basis and be current. The public accounts committee challenged the Department for Transport's calculation of HS2 benefits for business travellers, which was based on survey information more than ten years old and assumed that business travellers cannot and do not work on trains using modern technology.
4. Are assumptions appropriate and have they been applied correctly? There will always be elements of a cost estimate that have been based on assumptions, whether purely financial (such as the rate of inflation in five years' time) or more operational (such as changes in customer behaviour). There may be little evidence available on which to base these assumptions, but there should always be a clear rationale for the figure chosen.
5. What sensitivity analysis has been performed on key drivers? Testing the sensitivity of an output to a small change in an input helps to provide information about its variability. Testing how the outputs change by varying input values can help weed out errors in the model. If you expect a change in a key parameter to have a significant impact on the output and it does not, you need to ask why.
Best practice dictates that all financial models should be quality assured and independently tested Although no amount of auditing will ever rid the world of incorrect or incorrectly applied logic entirely, it is important is that errors with material impacts are minimised to an acceptable level of tolerance.
Reviewers cannot be afraid to question numbers. They do not have to understand every bit of code; they just need to be able to ask the right questions to reduce the risk of an uncomfortable appearance before the public accounts committee.
Katie Smith is a sourcing and procurement expert at PA Consulting Group