Assessment of Complex Performances in Digital Environments

• Published date: 01 May 2019
• Date: 01 May 2019
• DOI: http://doi.org/10.1177/0002716219846850
• Subject Matter: Toward the Future: Theories of Knowing and Implications for Assessment

846850ANN
The Annals of The American AcademyComplex Performances in Digital Environments
research-article2019
Digital technologies hold the potential to transform
educational assessment. recent advances reveal that
digital environments will support the development of
learning and assessment activities in ways that will both
increase the inferential fidelity of assessments and
change the form of assessments altogether. Digital
technologies can also automate data collection and the
production of assessment inferences on a massive scale.
here, we discuss the wide variation in digital learning
Assessment of experiences and explain how they are transforming
traditional language for discussing assessment. We
Complex
argue that the predigital constraints on assessment have
skewed our thinking about assessment and give exam-
ples of new and novel approaches. Second, we discuss
Performances how digital environments can allow us to capture and
make inferences from simple or complex learning
in Digital
activities in new ways. Third, we point to advances in
machine learning and AI that have the potential to
change current and future assessment practices. Finally,
Environments we argue for balancing enthusiasm for digital environ-
ments against the challenges of making appropriate
assessment inferences.
Keywords: assessment; machine-learning; games;
digital
By
JOhN T. BEhrENS,
KrISTEN E. DICErBO,
A primary goal of educational assessment is
to infer attributes of the learner from
and
observation of their performances and activities
PETEr W. FOLTz
in natural or contrived contexts (Behrens and
John T. Behrens is vice president of AI Product
Development at Pearson and adjunct assistant research
professor at the University of Notre Dame. He leads
product development and research teams that create,
interpret, and improve digital learning and assessment
experiences by integrating advances in computational,
learning, and data sciences.
Kristen E. DiCerbo is vice president of Learning
Research and Design at Pearson, leading and bringing
research insights to the design of digital learning envi-
ronments. Her personal research program centers on
the use of games and simulations to understand what
learners know and can do.
Correspondence: John.Behrens@Pearson.com
DOI: 10.1177/0002716219846850
ANNALS, AAPSS, 683, May 2019 217

---

218
ThE ANNALS OF ThE AMErICAN ACADEMY
DiCerbo 2014b). historically, this has been accomplished by presenting students
with small tasks or questions (often called “items”) to elicit these performances,
responses, or answers, as seen, for example, in paper-based testing in which a
series of questions serves as the context for performances. All assessment activi-
ties are bound by their operational and technological limits. For example, the
experience of answering questions presented by paper-and pencil forms is a stark
contrast to the complex digital environments many students engage in such as
online three-dimensional multiplayer games or instantaneous social networking
across the globe, or video or audio interactions occurring on mobile devices.
These rapid changes in the standards of personal experience and expression
move expectations and possibilities for educational assessment from predigital
discrete sets of questions and answers to complex environments and perfor-
mances that are digitally embedded. In this article we explain the needs and
opportunities for this changing landscape and recommend shifts in understand-
ing for the future of assessment.
This article addresses this challenge in four sections. First, we discuss the
importance of understanding assessment broadly, beyond the traditional ideas of
testing, with special attention to unpacking implicit assumptions concerning
assessment that limit our vision for the use of digital contexts. Second, we discuss
variation in digital environments and how their characteristics support the work
of inference in assessment goals, especially in the area of complex performances.
Third, we give examples of areas in which there is special opportunity and pos-
sibility because of advances in computing and inference. In the final section, we
temper the enthusiasm with discussions of limitations and concerns
Assessment, Testing, and Inference
What is assessment?
While educational assessment has often been formulated in terms of charac-
terizing learner knowledge, skills, and abilities (KSAs), we prefer to broaden the
understanding to reflect that assessment can include affective, motivational, and
behavioral attributes (Behrens and DiCerbo 2014b). Knowledge, skills, and abili-
ties are only a subset of the broad range of attributes relevant for the learning
process. Across the range of educational assessment uses, we see needs for psy-
chosocial information; cognitive and neurological information; as well as new
emphasis on so-called twenty-first-century skills, including collaboration and
technological acumen.
In substantially large digital environments, such as gaming platforms or learn-
ing management systems, there may be opportunities to collect data and make
Peter W. Foltz is vice president of AI Research & Development, in the AI Product and Solutions
group at Pearson and a research professor at the University of Colorado’s Institute of Cognitive
Science. He leads R&D on AI-based approaches applied to assessment and feedback of complex
cognitive processes, such as writing, collaboration, and problem-solving.

---

COMPLEx PErFOrMANCES IN DIgITAL ENvIrONMENTS
219
inferences regarding a broad range of learning-relevant human attributes, such
as engagement and persistence in addition to the traditional emphasis on knowl-
edge, skills, and abilities.
Testing is not equivalent to assessment
While assessment activity comes in many forms depending on the goals,
requirements, and resources available, a dominant pattern of assessment data
collection is that of “the test.” For those educated in traditional American public
schools, the test is such a dominant pattern of assessment that it is considered the
canonical form and often considered to map one-to-one against the concept of
assessment. Because of the rapid growth of assessment in a broader range of digi-
tal environments, we differentiate between the idea of the test and assessment in
general and define a test as a particular context established to elicit activity and
collect data for assessment purposes.
A homework assignment, created with the goal of supporting practice, and an
end of the chapter test created to assign a grade illustrate the distinction between
a test and another form of assessment. Each of these activity modes has preset
expectations that might consist of almost identical behaviors. The teacher would
be interested in assigning a set of tasks and likely providing performance feed-
back, and the learner would complete the tasks assigned. Typically, the test has
more specific constraints (e.g., time and location) added to the performances so
that limitations and scope of the inference are more clear. The identity of the
homework completer may not have to be rigorously verified if the only implica-
tion is that the student receives less valuable feedback. On the other hand, the
identity of the test taker is likely to require rigorous verification so we have faith
that the claims about the test taker are attributed to the right learner. In each
case, the activity is a form of assessment and at the level of specific activities (e.g.,
calculating math problems or writing short answers about a reading assignment),
the test and the homework may look nearly identical.
While the quiz and test highlight the assessment nature of assignments in
educational contexts, we (DiCerbo and Behrens 2014) have argued that as daily
life becomes increasingly digital, the opportunity for unobtrusive assessment
from our digital footprints offers the potential for new opportunities. Consider,
for example, an adult’s credit score, which is the result of applying a measure-
ment model unobtrusively to one’s consumer behavior. This is a form of “stealth
assessment” (e.g., Shute and ventura 2013) that occurs in the context of the com-
merce of our daily life and provides a characterization of an important attribute
in the life of a modern economic actor. The activity is a form of assessment that
results in inference from observation to create an inferred attribute, but the
inference is entirely based on observed behavior from financial transactions.
While the outcome is a gross level characterization and the system is well fed by
adult credit-related behavior, this assessment system is indicative of the type of
value that can be created for assessment purposes without creating a test. This
example is also valuable insofar as it can point to the potential ubiquity of assess-
ment systems, as well as to the dangers that lurk in such systems, including the

---

220
ThE ANNALS OF ThE AMErICAN ACADEMY
potential for abuse based on identity crime, severely high-stakes impact due to
data corruption or error, and potential for systematic bias in the statistical
machinery.
The role of inferential distance
A key attribute of any assessment is the inferential distance (the length of the
logical chain) required to get from observation to inferential claim. When the
evidentiary warrants and amount of background knowledge required to make an
...