Should patients use online reviews to pick their doctors and hospitals?

• Published date: 01 December 2022
• Author: David A. Hyman,Jing Liu,Bernard S. Black
• Date: 01 December 2022
• DOI: http://doi.org/10.1111/jels.12338

ORIGINAL ARTICLE

Should patients use online reviews to pick their doctors

and hospitals?

David A. Hyman

1

| Jing Liu

2

| Bernard S. Black

3

1

Georgetown University Law Center, Washington, DC, USA

2

East China University of Political Science and Law, Shanghai, China

3

Pritzker School of Law and the Kellogg School of Management, Northwestern University, Evanston,

Illinois, USA

Correspondence

Jing Liu, East China University of Political Science and Law, Shanghai, China

Email: jingliu0226@outlook.com

Abstract

We compare the online reviews of 221 “Questionable”Illinois and Indiana physicians

with multiple paid medical malpractice claims and disciplinary sanctions with matched

control physicians with clean records. Across five prominent online rating services, we find

small, mostly insignificant differences in star ratings and written reviews for Questionable ver-

sus control physicians. Only one rating service (Healthgrades) reports on paid medical mal-

practice claims and disciplinary actions and it misses more than 90% of these actions. We also

evaluate the online ratings of 171 Illinois hospitals and find that their ratings are largely

uncorrelated with the share of hospital-affiliated physicians with paid medical malpractice

claims and disciplinary sanctions. Online ratings have limited utility in helping patients avoid

physicians with troubled medical malpractice and disciplinary records, and steering patients

away from hospitals at which more physicians have paid medical malpractice claims and dis-

ciplinary sanctions.

INTRODUCTION

Online reviews are pervasive—but it is one thing to use them to pick a TV

(where the stakes are modest), and quite another to use them to pick a physician

or hospital (where the stakes can be life or death). Can consumers rely on online

reviews to identify and avoid the worst physicians and hospitals in a state?

This paper can be downloaded without charge from the Social Science Research Network electronic library at:

https://ssrn.com/abstract=3720180

DOI: 10.1111/jels.12338

©2022 Cornell Law School and Wiley Periodicals LLC.

J Empir Leg Stud. 2022;19:897–935. wileyonlinelibrary.com/journal/jels 897

Prior studies have found that online reviews focus on attributes of service

quality that patients can readily observe (e.g., waiting time, demeanor of physi-

cian and nursing staff, billing practices) and reflect overall patient satisfaction,

rather than more objective measures of quality of care. But online reviews might

still be useful to identify true “outlier”physicians—that is, physicians with

extremely problematic medical malpractice and disciplinary records. When it

comes to the medical malpractice and disciplinary systems, the past often pre-

dicts the future—meaning that a problematic record of paid medical malpractice

claims and disciplinary sanctions likely reflect durable physician characteristics

that patients might reasonably care about. Similarly, online reviews might be

useful to identify “outlier”hospitals—where the likelihood of being treated by a

physician with a paid medical malpractice claim or disciplinary sanction is much

higher than at other hospitals.

One obvious strategy is for online rating services to provide all publicly

available information regarding physicians’medical malpractice and disci-

plinary records, along with the hospitals with which they are affiliated. Alter-

natively, if overall patient satisfaction (as measured by online reviews) is

substantially different forphysicianswithextremely problematic medical

malpractice and disciplinary records, then online ratings might still be useful

in steering patients toward physicians with clean records. We extend the

extensive prior literature by comparing the star ratings and written reviews

of a cohort of outlier physicians versus matched control physicians with

clean records. We also evaluate whether online rating services accurately

report the medical malpractice and disciplinary records of our sample of out-

lier and control physicians.

Using data from Illinois and Indiana, we identified 221 “Questionable”phy-

sicians with multiple paid medical malpractice claims and disciplinary sanc-

tions.

1

We compare the online reviews of these Questionable physicians with

matched control physicians with clean records. Quantitative and qualitative

reviews were collected from five popular online rating services: Healthgrades.

com,RateMDs.com,Vitals.com, WebMD, and Yelp.com. For all five rating

services, users submit star ratings and written reviews.

We find only modest differences in the star ratings of Questionable versus

control physicians; most differences are statistically insignificant. We also use

text analysis methods to study the written reviews and find no material differ-

ences between reviews for Questionable versus control physicians. However,

consistent with prior studies, we find clear distinctions between written reviews

for physicians who received one-star ratings versus those who received five-star

1

Public Citizen used data on disciplinary sanctions to identify “Questionable”physicians in all 50 states. See, for

example, Public Citizen Health Research Group, 20,125 Questionable Doctors, Health Letter, September, 2000,

https://www.citizen.org/wp-content/uploads/hl_200009.pdf. As described below, we adopt their terminology but

use more restrictive criteria for identifying Questionable physicians.

898 DOCTOR ONLINE REVIEWS

ratings. We extend this research by showing that one cannot use star ratings or

written reviews to meaningfully distinguish physicians with multiple paid medi-

cal malpractice claims and disciplinary actions from physicians with pristine

records. Stated differently, patient evaluations of the physicians in our dataset

(as reflected in online reviews) are effectively disconnected from the outputs of

the two major systems that Illinois and Indiana (and every other state) use to

enforce quality of care standards.

Strikingly, only one of the five online rating services (Healthgrades) reports

the number of paid medicalclaims and disciplinary sanctions for rated physi-

cians. However, we find that they miss over 90% of such incidents.

What about hospitals? We compared the online ratings of 171 Illinois hospi-

tals by a branch of the federal government (Hospital Compare) and by 2 private

rating services (Healthgrades and Leapfrog) with a hospital ranking system

based on the share of all hospital-affiliated physicians with paid medical

malpractice claims and disciplinary actions from Hyman et al. (2021). Hospital

Compare rates hospitals using a one- to five-star rating system, and Leapfrog

gives hospitals letter grades (A–F)—both based on various clinical performance

measures. Healthgrades gives out several different awards, including recognition

of certain hospitals as “America’s Best”;“clear leaders”in “patient safety excel-

lence”; and delivering an “outstanding patient experience.”

In earlier work, we ranked Illinois hospitals from 1 to 171 based on the share

of hospital-affiliated physicians with paid medical malpractice claims and disci-

plinary sanctions, with lower figures (i.e., 1) being better (i.e., lowest share of

hospital-affiliated physicians with paid medical malpractice claims and disciplin-

ary sanctions) and higher figures (i.e., 171) being worse (i.e., highest share of

hospital-affiliated physicians with paid medical malpractice claims and disciplin-

ary sanctions). We emphasize that our hospital ranking is based on the medical

malpractice and disciplinary records of all hospital-affiliated physicians, and not

just the share of hospital-affiliated physicians who are Questionable.

We view this ranking as a robust measure of the quality of care provided by

hospital-affiliated physicians, which we treat as a measure of hospital quality.

As with our findings for physicians, we show that hospital online ratings

(whether from the federal government or from private rating services) are

largely disconnected from the outputs of the two primary systems that Illinois

uses to enforce quality of care standards.

Our findings for online ratings of physicians and hospitals highlight the mul-

tidimensional nature of quality and quality assessment in health care—while

also raising serious questions about the validity and utility of existing online rat-

ings. If our goal for online ratings is to steer patients away from problematic

physicians and hospitals, we will need a bigger (and better) boat.

2

2

Cf. Jaws (1975) (“You’re going to need a bigger boat.”).

DOCTOR ONLINE REVIEWS 899