The Use of Pilot Tests and Pretests in Consumer Surveys

• Author: Ivan Ross
• Pages: 11-26

11

2

The Use of Pilot

Tests and Pretests in

Consumer Surveys

By Ivan Ross

INTRODUCTION

In the social and behavioral sciences,the termpretest often refers to

apreliminary small-scale investigation of the proposed questionnaire,

while the termpilot test includesmore: asmall-scale investigation that

triesout surveyparameters in addition tothe questionnairedesign,

including the screening procedure, the determination of likelyincidence

and responserate, the choice of surveymodality,and soon.1However,

the termsareoften used interchangeablytodescribe preliminary

investigationsof anykind aimed attrying out anyaspectof asurvey

thatisbeing considered asextrinsic evidence in trademarkordeceptive

advertising litigation.2In thischapter,the termpilot (test/work) will

be used in thisbroadersenseasincluding preliminary empirical

investigationsof anykind.

1. Apretest isdefined asa“trial runofaquestionnaire”on page G5 of the Glossary

in CMD, J., &RG,M R (John Wileyand

Sons,7thed. 2007), whereason page 65the same authors define pilotstudiesas

“...surveys using alimited numberof respondents and often employing less rigorous

sampling techniquesthan areemployed in large, quantitativestudies.”

2.Some research experts define pretests asincluding bothpretests and pilottests,

e.g., “Pretests aresmall scale rehearsalsof the datacollection conducted beforethe main

survey.The purposeofapretest istoevaluatethe surveyaswell asthe datacollection

and respondentselection procedure.”R G,, SM

247(John Wiley&Sons2004).

Section II

12

Apilottest hasanumberof potential benefits.Forexample, itmayreveal that

the proposed surveymethodologywill operateeffectivelybecausethe questionsare

clearand easilyunderstood byrespondents and the method of soliciting participation

producesahigh yield. Oron the otherhand, apilottest mayuncoverpotential

problemssuch asambiguity in the questionsordifficultiesin the sampling frame’s

ability tolocatequalified respondents—in eithercasecalling forarevision in the

surveymethodology.Undercertain conditions,apilottest mayindicatethataclaim

ordefenseisnotlikelytobesupported byafull survey,informing the clientand the

attorneythatdropping orsettling the disputemaybe warranted.

The purposeofpilottests istoincreasethe reliability and validity of the

measurementof the state-of-mind constructs the client/attorney,inconsultation with

the expert,determine tobegermane tothe matterathand. Reliability refers towhether

the results of the surveycould be reproduced byconducting the surveyasecond

time, while validity refers towhetherthe surveyactuallymeasureswhatitpurports

tomeasure. Validity in the context of Lanham Actsurveys isparticularlyconcerned

withwhetherthe results arebothunbiased and unambiguous astoconclusionsreached

about the state-of-mind intended tobemeasured.

The factthatproperlyconducted pilotworkwasconducted should increasethe

court’s perception of the carethe expert took in ensuring thatthe surveywasdesigned

tobeasreliable and valid aspossible. Pilotworkisconsistentwiththe scientific

perspectivethatthe Supreme Court expressed in Daubert3becauseits purposeisto

increasethe quality of surveyevidence.

Naturallythe question of whethersuch efforts wereconducted objectivelyorfor

a“darkerpurpose” 4isaproperfocus forcross-examination. Nevertheless,courts on

some occasionsshould be suspicious of asurveywhereno pilottest orpreliminary

workbyanyothername wasconducted. Pilotwork, properlyconducted, should be

viewed asevidence of scientific objectivity ratherthan creativeconniving.

Asof November20,2011, areviewof Lanham Actcaselawidentified 25cases

in which the term“pilot” wasused in describing such preliminary work, sixin which

the term“pretest” wasused, and one in which both“pretest” and “pilot” wereused. In

most cases,the court eitherappeared tofind the results of the pilotworktohavesome

valueasempirical evidence ordid notmake specific commenton its valueone way

orthe other.5In afewcases,the pilotorpretest results wereevaluated bythe court as

3.See Gary Ford, The Impactof the Daubert Decision on SurveyResearch Used in Litigation, 24J.

P. P. &M. 234–52(Fall 2005), forathoughtfuldiscussion of the impactof Daubert standards

on surveyresearch.

4. LouisVuitton Malletierv.Dooney&Bourke, Inc., 340F. Supp. 2d415 (S.D.N.Y. 2004).

5. E.g., Storck USA, L.P. v.FarleyCandyCo., 797F. Supp. 1399 (N.D. Ill. 1992);HersheyFoods

Corp. and Homestead, Inc. v.Mars,Inc., 998 F. Supp. 500 (M.D. Pa. 1998);The Dreyfus Fund Inc., etal.

v.The Royal Bank of Canada, 525F.Supp. 1108(S.D.N.Y. 1981), 213U.S.P.Q. (BNA) 872; American

Home Products v.The Procter&Gamble Company,SyntexUSA Inc., and Procter-SyntexHealth

Products Co., 871F.Supp 739(D.N.J. 1994);Sands,Taylor&Woodv.TheQuakerOats Co., 8U.S.P.Q.

2d(BNA) 1457,1990WL 251914 (N.D. Ill.1990);Inc. Pub. Corp. v.Manhattan Magazine, Inc., 616F.

Supp. 370 (S.D.N.Y. 1985);P&G v.Haugen, 506 F. Supp. 2d883(2007);ReddyCommunications,Inc. v.