Methodological Expectations for Studies Using Computer Simulation

• DOI: http://doi.org/10.1111/jbl.12128
• Date: 01 June 2016
• Published date: 01 June 2016

Methodological Expectations for Studies Using Computer

Simulation

W. David Kelton

University of Cincinnati and Naval Postgraduate School

Simulation is often used in papers and studies across diverse fields like logistics, supply chains, health care, manufacturing, and defense.

But simulations must be properly done, including input and model building, designing/analyzing the simulations, and model verification/

validation. Unfortunately, simulation studies are not always done well, even though great effort could have gone into the model building and

coding. This paper specifies, in brief outline, what authors and researchers need to do, when using simulation as a main tool, to build a convinc-

ing case for their findings and conclusions. Considerations on the input side of the model are enumerated, including specification of input distri-

butions and processes; nonstationarity; random-number generation; and generating realizations of random variables and random processes. On

the output side are issues of statistical analysis of simulation output; comparison, selection, and ranking of simulated scenarios; variance reduc-

tion; and optimum seeking. Involving matters on both the input and output sides are the essential activities of verification and validation. The

intent is to establish expectations on what acceptable papers need to do if using simulation, and to serve as a guideline to applied simulation

studies. Such papers and studies will then be more valid, more precise, more useful, and ultimately more convincing.

Keywords: simulation; simulation methodology; statistical design and analysis; input; output; verification; validation

Computer simulation is a powerful an d popular tool for both

research and applications in di verse areas including logistic s,

supply chains, health care, manuf acturing, and defense, to name

just a few. Simulation’s appeal li es in its ability to tolerate rea l-

istic and thus often-complicated modeling assumptions, in clud-

ing uncertainty and nonstationar ity, even if those assumptions

result in a model of great complexity that would b e completely

intractable to exact mathematical -analytical approaches like

queueing theory. As a result, simu lation models can usually be

more valid than oversimplified stylized m odels of the same real

system, and thus more accurate and ulti mately more useful. For

instance, many queueing-type mod els need to assume exponen-

tial probability distributio ns in various places to enable pushi ng

through exact mathematical de rivations (exploiting the conve-

nient memoryless property of this dist ribution), but the expo-

nential distribution, with its mod e of zero, is unrealistic (maybe

downright ridiculous) for modeling of ma ny queueing aspects

like service-time durations; th is unrealistic input assumption can

propagate through such models to render their outputs, res ults,

and conclusions simply invalid. On th e other hand, simulation

can easily use any probability distribution needed to match real-

ity, and any logical structure at any point in the model. So ft-

ware for both modeling and statistica l analysis of simulation has

advanced markedly (e.g., Kelto n et al. 2015a,b); of course,

these advances have been matched by dramat ic increases in the

performance/cost ratio of computi ng hardware, often making a

proper simulation study the meth od of choice. For more on

comparison of simulation versus an alytical models, see Lucas

et al. (2015).

But, quoting from that paper, “As Wagner wrote in 1969,

results from stochastic simulations are indeed uncertain statistical

estimates. It is fair to criticize some simulation projects for

failing to recognize this, and thus failing to deal with it appropri-

ately. This has, perhaps deservedly, sometimes given simulation

a bad name”(Lucas et al. 2015, 296). Going a step further, this

has also likely contributed to bias (sometimes even prohibition)

in some journals against simulation, which may be historically

understandable but is now unfair, unnecessary, and harms the

real-world validity and relevance of published papers, not to

mention inhibits the careers of junior faculty. The purpose here

is to establish expectations for practices and methods for sound

simulation studies that should effectively neutralize such criti-

cism and bias against simulation, and build a convincing case

that such studies can be accurate, precise, and believable. Indeed,

aproperly done simulation study is often preferable to those

oversimplified unrealistic stylized models built mostly for the

sake of an exact mathematical analysis, rather than for the sake

of high-fidelity modeling of reality.

Most of the (warranted and fair) criticism of simulation stems

from the lack of adequate statistical design and analysis in some

simulation studies. There are statistical issues on both the input

and output sides of a stochastic simulation model, some of which

are enumerated below; there are also issues concerning proper

verification and validation of simulation models, discussed as

well below. These comments were compiled mostly with

dynamic stochastic discrete-event simulations in mind, but would

also mostly apply as well to other simulation-modeling para-

digms, notably to agent-based simulation (Kasaie and Kelton

2015), and to stochastic static simulations as well. Also, while

there is of course a large literature on simulation methods, there

is no attempt here to compile any kind of complete review or

bibliography; rather, the reader is encouraged to consult general

comprehensive simulation textbooks (e.g., Banks et al. 2010;

Kelton et al. 2015a,b; Law 2015), which in turn include long

lists of references, as well as the open-access full-text refereed

papers in the annual Proceedings of the Winter Simulation Con-

ference (http://informs-sim.org/, accessed January 31, 2016),

especially the tutorial tracks and sessions.

Corresponding author:

W. David Kelton, Department of Operations, Business Analytics,

and Information Systems, University of Cincinnati, Cincinnati, OH

45221-0130, USA; E-mail: david.kelton@uc.edu

Journal of Business Logistics, 2016, 37(2): 82–86 doi: 10.1111/jbl.12128

Published 2016. This article is a U.S. Government work and is in the public domain in the USA