Evidence-Based Investment Analysis: An Introduction.
In his 1996 inaugural speech as President of the Royal Statistical Society, Adrian Smith argued that Evidence-Based Practice should be the standard for all public policy. Mr. Smith used evidence-based medicine as his example, which is fitting because medicine is where these issues were first raised and where most of the development work has been done. Considering the stories we all read concerning the results of this medical study or that medical study, the reader may be surprised that this is an issue. But in fact doctors often prescribe treatments that deviate from the best medical practices.
Evidence-based medicine attempts to base medical decisions on the best available medical evidence. In contrast to the medical industry, the investment industry has no such standard. I suggest that the investor who has such a standard or who, at least, understands what kinds of evidence must be taken seriously and what kinds of evidence can be laughed at has a considerable advantage over investors who cannot.
While there are important differences between medical and investment knowledge, doctors and investors face exactly the same problem. Each must judge his industry's knowledge; that is, its laws, its theories, its rules of thumb, its purported facts and somehow grade this material with nonsense and crap on one side of the scale, certainty on the other side and everything else in various places in between.
Medicine has several such standards. For example, the U.S. Preventive Services Task Force has rates evidence on a three level scale where level one consists of evidence obtained from at least one properly designed random controlled trial (see sidebar). Level three consists of the opinions of respected experts. As in launching a rock, one counts down. The UK National Health Service has a similar scale.
I don’t mean to suggest that it is not possible to quibble about details; that, after all, is my stock in trade, but both of these scales are essentially correct. Unlike religion or politics, say, the quality of scientific study is indisputable within broad bounds. There is no doubt that a triple blind study (see below) is better than a double blind study, which is better than a single blind study. There is no doubt that a large sample is better than a small sample. There is no doubt that a study based on a broadly representative sample (all races, ages, and sexes) is better than one based on a sample that is more narrowly defined (19 year old, white males).
For several reasons, the medical industry does better research than we do. Medical researchers can perform experiments; they can vary the dosage of a drug and see how things change, if at all. Investment researchers can only perform studies. We can look and see what has happened when the Fed has lowered interest rates, for example, but we can manipulate nothing. None of this is our fault; by the way, astronomers and geologists face similar issues. But our work is necessarily of lower quality than that of other scientists. We need to keep that in mind when we evaluate our studies. And when we talk to our clients.
More, we can learn from the other sciences. Researchers use blinding to reduce bias. In a single blind study, a doctor gives a patient one of two or more drugs. The patient knows he is taking, say, one of two blood pressure medicines, but he does not know which. In a double blind study, neither the patient nor the doctor knows which drug any individual patient is taking. In a triple blind study, neither the patient, the doctor nor the statistician analyzing the study knows which drug any individual patient is taking. In a triple blind study, the statistician compares the results of drug one against drug two. Unless the statistician has a particular affection for certain integers, his analysis should be unbiased. Compare this with our own studies. If we were smart, and we aren’t, we would write our software so that it blinds us to what the various parameters mean. Our biases, our hopes, our fears make traitors of us all.
I intend to write further on this topic. In a future essay, I will suggest a set of standards. For several reasons, medical industry standards will not work for us.
| U.S. Preventive Services Task Force system for ranking evidence:
- Level I: Evidence obtained from at least one properly designed randomized controlled trial.
- Level II-1: Evidence obtained from well-designed controlled trials without randomization.
- Level II-2: Evidence obtained from well-designed cohortt or case-controll analytic studies, preferably from more than one center or research group.
- Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
- Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.
|