Debunking junk science: techniques for effective use of biostatistics.

• Author: Parker, Bruce R.

Numbers and statistical jargon may make jurors' eyes glaze over, but defense counsel must be alert to show the errors of plaintiffs' experts

DEFENSE counsel can attack junk science through the effective use of biostatistical evidence. It can be used against plaintiffs' experts both in cross-examination and in using defense experts to explain why plaintiffs' theories are incorrect. This article will focus primarily on how to use statistical evidence to cross-examine plaintiffs' experts effectively.

Biostatistical analysis is, like other disciplines, shrouded in jargon that is hard to cut through. Effectively using biostatistical data(1) requires cutting through the jargon and understanding the statistical concepts.

The first sections of this article discuss statistical concepts.(2) There is concentration on experimental design, since statistical data is no better than the study that produced it, and there is focus on factors that can negatively affect the results of an experiment and how scientists attempt to "control" for these factors.(3) Next is a primer on statistical analysis. It explains many of the statistical concepts discussed in medical literature and used by experts to support their opinions and the process by which researchers statistically analyze data to determine whether the experiment produced a "significant" result.(4) Last, there are examples of how experts and attorneys mislead juries and courts with statistical testimony. Strategies are offered for effectively cross-examining an expert who relies upon erroneous statistical data.

STUDY DESIGN FACTORS

1. Research Design

   One of the goals of researchers is to determine whether relationships exist between or among variables. They achieve their goal by designing experiments and accurately recording the data from the experiment. Counsel must review scientific literature and expert testimony based on experimental (either laboratory or clinical) data to consider whether the article or testimony is flawed by poor study design. Pointing out errors in study design is an excellent way to challenge expert testimony under Daubert(5) and at trial.

   1. Reliability

      Reliability is similar to the concept of reproducibility. It refers to how well the research design produces results that are the same, or very similar, each time the data are collected. An easy way to think of reliability is to consider a scale. A "reliable" scale will report "the same weight for the same object time and again."(6) This does not mean that the scale is accurate--it may always report a weight that is too high or too low--but it always makes the same error each time.

   2. Validity

      Validity is synonymous with accuracy, and it has internal and external components. Whether the data properly measure the group sampled is a reflection of its degree of internal validity. To the extent the data can be generalized, they have external validity. A study that has high internal validity, but is nevertheless not generalizable, can be misleading.(7)

      The concepts of validity and reliability are interrelated. A researcher can have an experimental design that produces reliable, but invalid results--that is, the scale always reports that you weigh 175 pounds, when you in fact weigh 180--but you cannot have valid results that are not reliable.(8)

   3. Sensitivity

      The sensitivity of a test refers to the percentage of times that the test correctly gives a positive result when the individual tested actually has the characteristic or trait in question. For example, the sensitivity of a test that is designed to determine high red cell counts is the percentage of people who have high red cell levels and who test positive.

      When the test correctly reports that a person has high red cell counts, the result is a true positive. Conversely, when the test reports that a person does not have high red counts when, in fact, that person does, the result is a false negative. The numerical value of a test's sensitivity is obtained by dividing the number of true positives by the total of true positives and false negatives in the sample.(9)

   4. Specificity

      The specificity of a test refers to the percentage of times a test correctly reports that a person does not have the characteristic under investigation. When a test shows that a person who has a normal red cell count is negative, the result is a true negative. A false positive result occurs when the test incorrectly reports a high red cell count, when in fact that person is normal. Specificity is determined by dividing the number of true negatives by the total of true negative plus false positive responders.(10)

   5. Predictive Value

      Although the sensitivity and specificity of a test give a crude measure of its accuracy, they do not tell a physician the probability that an individual who tests positive actually has the condition being measured. This is provided by the positive predictive value of the test. The positive predictive value expresses the probability that an individual with a positive test result does, in fact, have the trait, while the negative predictive value expresses the likelihood that an individual with a negative test result does not have the characteristic in question.(11)

      The predictive value of a test is depends on the prevalence of the condition in the group tested and the test specificity.(12)

   6. Sampling

      If researchers could ask all people in the world who drink one or more glasses of milk per day whether they suffer or have suffered from cancer, there would be no need for a statistical analysis to determine if milk is associated with cancer. The researcher could simply look at the data and determine, with complete confidence, whether a relationship exists. However, obtaining information from everyone who drinks milk would be impossible. As a result, researchers select a sample of individuals to study, and then they statistically analyze the data obtained from these individuals to extrapolate findings to the an entire population.

      There are several different ways in which researchers "sample" a population, but "the result of a sampling study is no better than the sample it is based on."(13) The major trap that must be avoided when a researcher samples a population is bias, and the researcher must eliminate or control for it. An excellent opportunity exists to discredit an expert whose opinion is predicated on studies that fail to avoid this problem.

      1. Selection Bias

         Selection bias is the failure when recruiting participants to obtain a fair and true cross-section of the population under investigation.(14) Selection bias will affect the validity of a study if it results in an overrepresentation of one type or class of individual.(15)

         A classic example of selection bias that jurors readily understand is the 1936 Literary Digest presidential poll, which predicted that Alf Landon, the Republican candidate, would defeat Franklin Roosevelt, the Democratic candidate, 57 to 43 percent. In fact, Roosevelt won the election by 62 to 38 percent. The sampling model was flawed by a bias that was inherent in the manner in which participants were recruited for the poll. Names were chosen from "telephone books, rosters of clubs and associations, city directories, lists of registered voters and mail order listings."(16) However, in 1936, only the wealthy had telephones, and the people whose names were on the other lists also tended to be more affluent and Republican. Thus, despite the fact that the responses were statistically significant, the data were useless because of design flaws in the sampling model.

         Another example jurors understand is that of a researcher asking pedestrians for their opinion on whether people in large cities are less polite than they were 15 years ago. As two men approach, the researcher must choose whom to question. One is nicely dressed, with a clean shave and a smile, while the other is in blue jeans, a stained undershirt, three days growth and a scowl on his face. Many interviewers would probably choose to approach the well-dressed man. Selecting subjects in this manner, known as "interviewer bias," would not generate a true cross section of the population since less well-dressed, surly looking men are being systemically excluded.(17)

         In some instances, bias is generated simply by human desire to give pleasing answers to an interviewer. Male interviewers probably get different responses from female subjects than female interviewers would on sensitive personal issues. An interviewer aware of the study hypothesis may project more empathy with the exposed subjects than controls, thereby evoking greater trust. A greater feeling of trust among the exposed group will generate more revealing and complete answers than from the controls.

      2. Random Sampling

         A good study is one that uses a sampling technique that obtains a representative sample of the population being studied. A truly representative sample is one in which every source of bias has been removed. Therefore, researchers try to control for as many of the different sources of bias as is practicable under the circumstances.

         The most effective way to control for sampling biases is to use a purely random sample, which is obtained by selecting participants in such a way that each member of the population being studied has an equal chance of being selected. By using this method a researcher eliminates all selection bias.(18)

         Obtaining a purely random sample, however, is usually impossible because people cannot be forced to participate in a study. To the extent it is possible, it is often prohibitively expensive. For these reasons, researchers have devised ways to obtain samples that approximate purely random samples. None of these methods, however, provides a researcher with the level of confidence that the sample is free of bias as does a purely random sample.

   7. Controlled Experiments

      "Controlled experiments are, far and away, the best vehicle for establishing a causal relationship."(19)...