Tough Minded Investment Manager Analysis

Home

Managing Your Investment Manager

That's Mr. Ponzi to you.

F3 Funds

Lock Down Your Investment Manager

When Should You Pull Your Money Out?

Doing Due Dillegence over the Phone

Is Your Investment Manager Lying to You?

Six Questions to Help You Evaluate Investment Gossip

What kind of Performance Data do Investors Need?

Avoiding the Gambling Addict

Investment Styles

Oxford Metrica and Cluster Analysis: Fund Style Analysis du Jour

Holdings Analysis

Investment Technique

Artificial Intelligence

Fat-Tailed and Skewed Asset Price Distributions

Model Risk

Veryan Allen

Evidence-Based Investment Analysis

An Introduction to EBIA

Quantitative Analysis

Global Quant Meltdown

Trust Me, I'm a Quant!

An Explanation for the Quant Meltdown

Five Things the Quant Meltdown Should Teach Investors

Risk Management

Do Fund Managers Understand Operational Risk?

Operational Risk: How Bad is the Problem

Chris Mushell and Frontiers in Operational Risk Management

Was Societe Generale’s Meltdown Preventable?

Kroll Report on Corporate Fraud

Activist Investing, Private Equity and Venture Capital.

Activist Investing

Michael Jenson and Private Equity

New Taxes on Private Equity?

The Future

2008 Emerging Technology

Manufacturing

Batteries

Melting Artic Ice

Investment Book Reviews

Carbon Finance

Evidence Based Technical Analysis

Fat-Tailed and Skewed Asset Return Distributions

Fortune's Formula

How to Select Investment Managers

The Misbehavior of Markets

Sound Practices for Hedge Funds

Managing Yourself and Others

An Article, Containing a Good Idea, But One You Had Better Sleep On

Eating Smart

Listening to Professor Cory

Extraordinary Comebacks

Humor, Time Wasters and Stuff That's Just Plain Weird

Greg Newton

Alfred Lawson--Nutcase

The Stand Up Economist

Futures from Nature

A Few Kind Words About Ayn Rand

Marc Dreier

A Version of Artificial Intelligence that Works!

Quant Gifts

Random Investment Humor Links

Books By Fred Gehm

About

About FredGehm.com

Endorsements and Slurs

Tools, Techniques, Essays and News that will help you pick the best CTAs, Hedge Funds and Alternative Investment managers. No nonsense. No excuses. No easy answers.

Evidence-Based Investment Analysis: An Introduction.
In his 1996 inaugural speech as President of the Royal Statistical Society, Adrian Smith argued that Evidence-Based Practice should be the standard for all public policy.  Mr. Smith used evidence-based medicine as his example, which is fitting because medicine is where these issues were first raised and where most of the development work has been done.  Considering the stories we all read concerning the results of this medical study or that medical study, the reader may be surprised that this is an issue.  But in fact doctors often prescribe treatments that deviate from the best medical practices.

 

Evidence-based medicine attempts to base medical decisions on the best available medical evidence.  In contrast to the medical industry, the investment industry has no such standard.  I suggest that the investor who has such a standard or who, at least, understands what kinds of evidence must be taken seriously and what kinds of evidence can be laughed at has a considerable advantage over investors who cannot.

 

While there are important differences between medical and investment knowledge, doctors and investors face exactly the same problem.  Each must judge his industry's knowledge; that is, its laws, its theories, its rules of thumb, its purported facts and somehow grade this material with nonsense and crap on one side of the scale, certainty on the other side and everything else in various places in between.

 

Medicine has several such standards.  For example, the U.S. Preventive Services Task Force has rates evidence on a three level scale where level one consists of evidence obtained from at least one properly designed random controlled trial (see sidebar).  Level three consists of the opinions of respected experts.  As in launching a rock, one counts down.  The UK National Health Service has a similar scale.

 

I don’t mean to suggest that it is not possible to quibble about details; that, after all, is my stock in trade, but both of these scales are essentially correct.  Unlike religion or politics, say, the quality of scientific study is indisputable within broad bounds.  There is no doubt that a triple blind study (see below) is better than a double blind study, which is better than a single blind study.  There is no doubt that a large sample is better than a small sample.  There is no doubt that a study based on a broadly representative sample (all races, ages, and sexes) is better than one based on a sample that is more narrowly defined (19 year old, white males). 

 

For several reasons, the medical industry does better research than we do.  Medical researchers can perform experiments; they can vary the dosage of a drug and see how things change, if at all.  Investment researchers can only perform studies.  We can look and see what has happened when the Fed has lowered interest rates, for example, but we can manipulate nothing.  None of this is our fault; by the way, astronomers and geologists face similar issues.  But our work is necessarily of lower quality than that of other scientists.  We need to keep that in mind when we evaluate our studies.  And when we talk to our clients. 

 

More, we can learn from the other sciences.  Researchers use blinding to reduce bias.  In a single blind study, a doctor gives a patient one of two or more drugs.  The patient knows he is taking, say, one of two blood pressure medicines, but he does not know which.  In a double blind study, neither the patient nor the doctor knows which drug any individual patient is taking.  In a triple blind study, neither the patient, the doctor nor the statistician analyzing the study knows which drug any individual patient is taking.  In a triple blind study, the statistician compares the results of drug one against drug two.  Unless the statistician has a particular affection for certain integers, his analysis should be unbiased.  Compare this with our own studies.  If we were smart, and we aren’t, we would write our software so that it blinds us to what the various parameters mean.  Our biases, our hopes, our fears make traitors of us all.

 

I intend to write further on this topic.  In a future essay, I will suggest a set of standards.  For several reasons, medical industry standards will not work for us.


U.S. Preventive Services Task Force system for ranking evidence:

  • Level I: Evidence obtained from at least one properly designed randomized controlled trial.
  • Level II-1: Evidence obtained from well-designed controlled trials without randomization.
  • Level II-2: Evidence obtained from well-designed cohortt or case-controll analytic studies, preferably from more than one center or research group.
  • Level II-3: Evidence obtained from multiple time series with or without the intervention. Dramatic results in uncontrolled trials might also be regarded as this type of evidence.
  • Level III: Opinions of respected authorities, based on clinical experience, descriptive studies, or reports of expert committees.

The UK National Health Service system for ranking evidence:

  • Level A: consistent Randomised Controlled Clinical Trial, Cohort Study, All or None, Clinical Decision Rule validated in different populations.
  • Level B: consistent Retrospective Cohort, Exploratory Cohort, Ecological Study, Outcomes Research, Case-Control Study; or extrapolations from level A studies.
  • Level C: Case-series Study or extrapolations from level B studies
  • Level D: Expert opinion without explicit critical appraisal, or based on physiology, bench research or first principles

If you liked this essay, please let others know; Digg this site!

Digg.com is a social networking site that allows members to vote for their favorite web pages.  If you are a Digg member, you can help me by voting.  If you are not a member, clicking the button on the left will let you join and vote.  Joining Digg and voting takes only a minute.  Please vote.  Thank you for your support.



Comments and Critiques


"If your mother says she loves you, check it out." --Old reporters' motto; also our motto.
Copyright (C) 2005, 2006, 2007, 2008, 2009 Fred Gehm.  All rights reserved.