Tools, Techniques, Essays and News that will help you pick the best CTAs, Hedge Funds and Alternative Investment managers. No nonsense. No excuses. No easy answers.
Another Kind of Model Risk
Six Questions and Answers about Model Risk
Model Risk is currently being presented as a new investment management concept, with all the usual hype that entails, but at its base is nothing new at all.Model risk is the risk we don’t know what we are doing, in whole or in part, when we build an investment model.This suggests six questions.
Does the basic concept make sense? As investment concepts go, model risk is not a bad one.Consider, as an example, a bet I used to offer arbitragers.I would borrow a pack of paper matches, rip out a match and mark one side of it with a line.The side with the line was heads.The side without the line was tails.I would then offer them the following bet.I would flip the match into the air and the side that faced up when the match landed would be the winner.I would also let them bet on heads or tails or both at three to one odds; one hundred dollars each side.Before you go on, think about the bet.What would you do and why?
The person on the other side of the bet usually expects to lose $200 on every flip of the match and win $300, for a net gain of $100 per throw.This analysis is sound, so long as there is no model risk, so long as the match can only land heads or tails.But if the person throwing the match bends the match before he throws it, and nothing in the rules prevents this, the probability of the match coming down on its side is just under 100%, which means that the person on the other side of the bet can expect a net loss of just under $200 a throw.
What kind of models does Model Risk apply to? The literature I have read assumes that the only models of interest are pricing models.However, I find nothing in the concepts and little in the techniques that can only be applied to such models.To the contrary, I would apply these techniques to almost everything a trader does. How serious is the problem? The example above is nothing more than a bar bet, but the problem itself is common and cancerous.I have personally listened to more investment presentations than I can remember and, in my experience, most fatal investment errors are errors of omission; the trader is not wrong about his facts, his set of facts are incomplete. Is Model Risk an Operational Risk? Model Risk is sometimes defined as an operational risk, which, in turn, is often defined as ‘risk without reward’.As long as operational risk is defined that way, then model risk is an operational risk.But I see nothing to be gained from that classification and much to be lost.The skills necessary to manage most operational risks are the skills of the business manager and accountant.The skills necessary to manage model risk are the skills of scientists, statisticians, philosophers, programmers, magicians and others I am not clever enough to think of.I would classify model risk as all other investment risks. Does Model Risk offer anything other than a name? It does, indeed.One article on model risk lists potential problems as inapplicability of modeling, incorrect modeling, correct model but incorrect solution, correct model inappropriate use, badly approximated solution, software and hardware bugs, and, finally, unstable data.Let’s consider a few of these issues.
Most investment programming is fairly squishy.For example, one shrink-wrapped technical analysis software package used to have, and may still have, for all I know, a moving average module that will allow you to keep your short positions long, long after the price has risen above the relevant moving average.In almost all cases, the software correctly switches you from short to long, but if the price rises to exactly the price of the moving average and then rises above it, the software never makes the switch.Moving averages are one of the simplest possible models.More complex models, such as those of convertible bonds or exotic options are much more likely to be wrong and the errors will be much harder to find.Worse yet, a reasonable person can only assume that for every error he finds, there are another ten or a thousand he hasn’t found yet. Proper use of programming quality assurance and quality control techniques, not to mention superior programming techniques, such as defensive programming, feedback analysis and object-oriented programming, can reduce errors by one or two orders of magnitude (ten or a hundred fold).Serious error reduction, however, requires double programming.Double programming is just what the name suggests.Two programmers, working from the same specifications, write each piece of code.They do not talk to each other or compare notes in any way.Someone else compares the results (not the code) and when the results differ, he or she has to dig through the code and figure out why.
If programming problems now seem difficult and expensive to manage, the reader has made real progress.The reader needs to keep in mind, however, that programming problems are the easiest of the modeling risk issues to manage.Consider a more difficult problem: the model is presumed to be conceptually correct, but the mathematician may have derived a wrong solution.Unless the mathematician revisits his work or allows someone else to check it, the error may never be found.If the mathematician publishes his work, some other mathematician may find and announce the error—or he may simply trade against the error.This is sometimes called ‘model arbitrage’. If the error is conceptual, rather than mathematical, it may never be found.One billionaire options trader once claimed the Black-Scholes model had two missing variables.He made that claim decades ago and I have yet to hear anyone suggest what those variables are.If he is right and the source of his wealth suggests he is, many very bright, educated and persistent people have missed something basic.
Does naming the problem ‘Model Risk’ add anything? Shakespeare to the contrary, naming the problem ‘Model Risk’ may actually be an innovation.Names are important.I have been writing about what is now known as model risk since 1983 without noticeable effect.Perhaps the reason I have had so little effect here, is because I never gave the problem a name, I never gave my reader something to hang on to.
On the other hand, calling the problem ‘Model Risk’ might actually destroy the concept.Some of the people who are writing about model risk seem to think that model risk is a short list of problems that can be managed with a short list of techniques.This is to be expected, of course, but it doesn’t mean that these people aren’t being silly.The real problem is everything of which the model builder is unaware.The model builder who can compile a complete list of everything of which he is unaware can eliminate the problem.The rest of us are still stuck in the mud.
Comments and Critiques
"If your mother says she loves you, check it out." --Old reporters' motto; also our motto. Copyright (C) 2005, 2006, 2007, 2008, 2009 Fred Gehm. All rights reserved.
