Education research is
under the microscope
Quality of data crucial as
schools base more decisions on studies
by Joshua Benton, The Dallas Morning
For more articles
like this, visit
Education research has been a punching
bag for decades – largely for good
Streams of studies serve up
contradictory facts. Anecdotes and hearsay are often as
prominent as hard data. And some scholars, motivated by
ideology, seem to reach their conclusions before the
research even starts.
"When you look back, there have been
entire movements in education launched on the basis of a
few anecdotes, a lot of rhetoric, and not much evidence,"
said Gerald Sroufe, director of government relations for
the American Educational Research Association.
But now the field is facing
unprecedented pressure that could change the nature of
research into what works in schools.
A new federal law requires schools to
base dozens of policy decisions on research or risk losing
federal money, and a bill pending in Congress would, for
the first time, attempt to set quality standards for
government-funded education research.
"There's some very good work out there,
but there's a lot of very bad work, too," said Stanford
professor Richard Shavelson. "You just have to figure out
how to separate the two."
The federal education bill passed in
January uses some variant of the phrase "scientifically
based research" 110 times.
It requires states and districts to use
research to determine their approaches to everything from
teacher training to the hiring of security guards. The
Bush administration has spoken often about its desire to
make education an "evidence-based field."
That, in turn, has fostered the desire
to establish quality-control measures for research.
"I've seen some colossal missteps in
education – things like New Math and 'schools without
walls' – that were seemingly never tested before being
tried," said Rep. Michael Castle, R-Del., chairman of the
House education reform subcommittee and author of the
Teachers looking for ways to improve
their skills find that different researchers give
contradictory evidence for what works and what doesn't.
Academics line up on both sides of every major issue –
vouchers, testing, social promotion – and are often
accused of cooking the data to get the results they want.
Reformers such as Mr. Castle would like
the field of education research to look more like the
world of medical research.
Until the 1950s, doctors operated
largely on anecdotal evidence about what worked and what
didn't. As independent professionals, many felt that they
– not some distant academic researcher – were best able to
decide the effectiveness of treatment.
"The prevailing attitude was that each
doctor was his own experimenter," said Robert Baruch, an
education researcher at the University of Pennsylvania.
That's not far from how some teachers feel today, he said.
Over time, physicians were convinced
that controlled clinical trials could lead to better
medicine. Now, medical research is strictly regulated by
government – the Food and Drug Administration – and by an
academic community that conducts rigorous peer reviews of
scholarly work in prestigious publications such as The
Journal of the American Medical Association.
Mr. Castle's bill, which passed the
House on April 30, would require peer reviews of all
federally funded studies and would model research
regulations on those of other federal agencies, such as
the National Institutes of Health and the National Science
The bill is now sitting in the Senate
committee chaired by Sen. Edward Kennedy, D-Mass. A
Kennedy spokesman said that Senate work on the bill will
begin in September, but that the committee will likely
create its own bill rather than work from Mr. Castle's.
Some academics agreed that the overall
quality of education research needs to be raised but said
they are wary of government standards.
"The intent is appropriate, but how it
gets translated into law is questionable," Dr. Shavelson
said. "I don't think it's the role of the federal
government to define science."
For example, the most successful style
of medical experiment has been the randomized trial, in
which a group is randomly divided into two parts: One gets
a certain treatment and the other doesn't. But local
control of schools makes random assignment difficult for
education studies, he said.
"When the researchers come in and say,
'We want you to abandon the way you teach and try this new
way for three years so we can see if it works,' are
parents going to go along with that? Or will local control
come into play and say 'We're in charge, we're not going
to [do] that'?"
Maintaining strict research conditions
in a school is notoriously difficult: students move,
principals get fired, policies and demographics change. In
one recent long-term experiment led by Boston University
researcher Christine Rossell, half of her subjects had
withdrawn from the study within three years.
"The so-called hard sciences –
chemistry, physics – are misnamed," she said. "Those
should be called the easy sciences. You can keep ions and
molecules in a beaker for decades. They don't have any
human rights. They don't move from one school district to
another in the middle of a research study and mess up your
data. Researching real human beings who don't always
follow directions – that's a hard science."
"People think you can just look at test
scores and figure out what works," she said. "Doing a
strict scientific study is extremely difficult and gets
very, very expensive."
According to a National Research Council
study released this year, federal funding for educational
research has dropped from more than $400 million in 1973
to about $130 million today. Only about one-tenth of 1
percent of education funding in America goes to research.
"If we were a drug company or GM or
Ford, spending that little money on research, we wouldn't
stay in business," Dr. Baruch said.
Even without government intervention,
some private sources are trying to improve research
quality. Dr. Baruch leads the Campbell Collaboration, a
2-year-old organization whose goal is to evaluate existing
education research and "screen the good stuff from the
poor stuff." He said that sometimes fewer than 10 percent
of the studies they evaluate meet the highest standards
the collaborative looks for.
"One of our biggest problems is that
it's all called 'research,' no matter how good it is," Dr.
Sroufe said. "Once you see the headline 'Researchers show
such-and-such,' it's difficult for the public to get much
deeper than that."