Too often, statistical evidence is based on questionable methodology
“There are very few things which we know which are not capable of being reduced to a Mathematical Reasoning, and when they cannot, it’s a sign our Knowledge of them is very small and confused….”
—John Arbuthnot, “Of the Laws of Chance,” 1692
Much testimony in personal-injury matters is, by necessity, based on statistical reasoning. Statistical analysis can be, in many instances, the only reasonable way to predict what would have occurred absent the tort. The probability of a future outcome such as life expectancy or lifetime earnings is extrapolated by examining data reflecting past experiences. Too often, however, conclusions are drawn based on inappropriate data or inappropriate methodology. It is apparent that many lawyers and courts have a very small and confused knowledge of statistical analysis and what constitute appropriate data. Here is a list of some areas where problems with data and methodology arise in economic-damages testimony in injury cases:
Average Earnings by Profession. Many industries are extremely well-documented in terms of data showing earnings. Baseball, that vast repository of statistical data, is a prime example — player salaries, as well as their on-field performance, are a matter of public record and a matter of great public debate. Detailed salary data for other professions is readily available. However, much testimony on probable future earnings of a plaintiff based on salary surveys for a specific occupation overlooks the probabilities that the plaintiff will enter that occupation, or achieve a certain level within that occupation. Testimony based on the assumption that a gifted high school athlete would have become a professional baseball player earning millions of dollars annually would be certainly challenged in court, yet testimony to the effect that a CPA can expect to earn the average of what accountants in Big Five accounting firms earn may be accepted without consideration of the fact that accountants working in the largest national accounting firms, like professional baseball players, represent a minuscule fraction of all CPAs.
Any testimony on average earnings in a profession should be considered in light of the barriers to entry into the profession and the washout rate among members of that profession. Stockbrokers, for example, may enjoy spectacular annual salaries but also shorter careers, like baseball players.
Female Worklife. “Worklife” is the amount of time a person can, on average, expect to be employed over his or her lifetime. The U.S. Department of Labor’s Bureau of Labor Statistics publishes “Worklife Estimates: Effects of Race and Education.” The latest worklife publication by the BLS contains worklife estimates based on a sample taken in 1979-1980. The estimates are presented in tables broken down by sex, age, race and schooling completed. The tables are often relied on by economists and others testifying on lost future income. However, the tables reflecting worklife estimates for women are now an inappropriate basis for lost-income testimony since the data captures women in the work force who were 50 or older in 1980. Female work-force participation has changed so dramatically since 1980 that the tables for men are now a much more reliable basis for predicting current probable work-force participation by women.
Multiple Regression Analysis. Statisticians often construct models to predict an event or make inferences about the real world using a small sample of data. The effectiveness of statistical models relies on the skill with which variables on which they are predicated are chosen. The models are usually built around regression analyses, which compare the relationship between two or more independent variables and a dependent variable. Properly applied, regression analysis can measure the effect of each variable, how important the variable is in relationship to other variables and what effect the variable would have but for the intervention of other variables.
Models based on regression analyses do not show that A caused B. Rather, they show the relationship between A and B— whether A is positively or negatively correlated to B, or if there is no correlation. For example, in a model used to predict whether exercise has a positive or negative relationship to longevity, the relationship between exercise and other independent variables such as age, eating habits, race, and geographical location could be examined. Exercise might prove to be either a more or less significant factor in longevity. Such models can be eloquent demonstrations of hidden relationships, highly useful, for example, for explaining the effects of exposure to toxic chemicals. However, elaborate multiple regression analyses can also act as a type of shell game, hiding weak or disingenuous testimony under layers of arcane methodology.
In one case in which this author was involved, an elaborate multiple regression analysis conducted by plaintiffs in a class-action matter supposedly demonstrated a positive correlation between IQ points allegedly lost as a result of exposure to toxic chemicals and diminished lifetime earnings — a seemingly logical assumption. When the statistical model was dissected and carefully examined, though, defense economists discovered that the model was the statistical equivalent of a house of cards. The model was based on the assumption that each lost IQ point correlated with the loss of about one month of education. This model served as the basis for loss claims of $60,000 to $500,00 per plaintiff. However, since no plaintiff in the class had lost more than 6 IQ points, all the model showed in reality was that no more than six months of schooling had been lost by any plaintiff. The model failed to show that any plaintiff missed graduation from high school as a result of this predicted six-months-or-less loss. Cross examined on the model, the statistician who designed it admitted that the model made no distinction between an extra few months of school in the 10th grade versus an extra few months of school in the 12th grade. Thus, the multistage model, while statistically correct in each of its constituent stages, was shown to be, overall, an extremely unreliable as a predictor of lost income.
“Hedonic” Damages Testimony. Many jurisdictions have accepted the value of life’s enjoyment as an element of damages separate and distinct from pain and suffering, and some jurisdictions have accepted testimony from experts who claim to use government studies on risk, wages for dangerous jobs and other data to calculate statistically the value of life’s enjoyment. Various experts have placed life’s value, above and beyond lost income, at between $1 million and more than $8 million.
One basic assumption behind this “hedonic” testimony is that life’s enjoyment can be extrapolated from statistical studies examining correlations between wages and risk. However, a relatively obscure academic study intended to provide supporting evidence to the hedonic-damages theory showed that there is no statistically significant relationship between wage and risk. Therefore, hedonic testimony falls far short of the standards for scientific evidence called for in Daubert v. Merrell Dow Pharmaceuticals, 113 S.Ct. 2786 (1993). The study, “No Evidence of Compensating Wages for Occupational Fatalities,” was conducted by J. Paul Leigh, a professor at San Jose State University, and was published in Industrial Relations, Vol. 30, No. 3 (Fall 1991).
Any defendant in an action where hedonic claims are made should consider seeking a motion in limine excluding hedonic testimony. Such testimony was emphatically excluded in Ayers v. Robinson, 887 F.Supp. 1049 (N.D. Ill.,1995). The court found that the proffered testimony was woefully short of the Daubert guidelines. In its strongly worded opinion, the court found that the hedonic experts’ calculation of a “benchmark” value of human life was unscientific, since it derived from adjusted data and is based on flawed reasoning.
These are just a few areas where testimony based on statistical analysis may not be as reliable as advertised by the testifier. This is not to say that all statistical analysis is bogus, as many of the mathematically disinclined have been conditioned to think by books such as “How to Lie With Statistics,” or the famous Disraeli quote about “lies, damn lies and statistics.” Statistical analysis is, when properly performed, a powerful, useful tool. However, any litigant confronted with testimony based on statistical reasoning should maintain a healthily skeptical attitude.
___________
This article is reprinted, with permission, from the September 2001 edition of Medical Malpractice Law & Strategy, Copyright © 2001 NLP IP Company. All rights reserved. Further duplication without permission is prohibited.