All Models are Wrong but Some are Useful

Reading Time: 2 minutes

Body-Mass Index (BMI) can be helpful in giving an indication of whether someone is overweight, at risk of developing negative health effects and can give a clear call-to-action to make changes from a dietary or exercise perspective.

However, take BMI to extremes and you end up concluding that the Rock, and at least half of the England rugby team (world class professional athletes at the top of their game) are overweight and in need of serious interventions – clearly the wrong conclusion (false positive).

It’s the same with over-reliance on any single-parameter test and this is very much the case in investing with use of VaR (Value-at-Risk). It’s helpful in that it can be used to generate a clear signal on risks in the portfolio, but over-relying on it can get you into trouble in any number of ways: it can give false confidence, it can make you too short-termist with investments and risk eroding your real value in the long run and it can end up with you saying patently ridiculous things like “we suffered a 25 standard deviation events, several days in a row” as David Vinar of Goldman Sachs infamously said in 2007.

Just in the way that a qualified health professional will look at a broader spectrum of indicators other than just BMI, a good investment adviser or consultant will look at the broader picture too when forming advice, but that doesn’t mean to say that VaR is not useful to have as a quick metric to provide a decent view on portfolio risk, compare two portfolio and act as a guideline when contemplating actions to change risk.

The key to successful use of any model for advanced users is not aiming for the perfect model or a better model, but better understanding a model’s flaws and shortcomings and being mindful of the limitations. If a tool works 90% or 60% of the time then this is probably better than nothing (or relying on superstition or gut feel, which can frequently lead us astray)

Understanding these limitations usually boils down to what are the key assumptions and how can they be wrong. For VaR, that looks like:

  1. Correlation between assets. This is a key determinant of VaR and is not knowable with any real degree of accuracy but VaR results will be quite sensitive to it. Estimates from historical data can be useful, but the confidence level around these needs to be bourn in mind
  2. Calibration time period. Regimes and paradigms exist in financial markets for long periods then change rapidly. If your model is calibrated in one period it might be way off in another due to changes in volatility, correlations or distributions.
  3. Distribution of asset returns. Normal distributions make for easy-to-calculate numbers, but black swans are more common than we might think
  4. False sense of security. Any quantitative estimate of risk can lead to a seductive narrative because we badly want to believe that the future is under our control. It isn’t, we must accept that models are just guides in a world of uncertainty.

In defence of VaR, at least the shortcomings are well known and can be fairly easily interrogated, compared to more complex and longer term (multi-period) models where the moving parts and key assumptions can be harder to get a handle on.

“All models are wrong, but some are useful”

George Box
Posted in GK

Leave a Reply