Education research is under the microscope

Quality of data crucial as schools base more decisions on studies

by Joshua Benton, The Dallas Morning News, 07/29/2002

For more articles like this, visit http://www.EducationNews.org.

Education research has been a punching bag for decades – largely for good reason.

Streams of studies serve up contradictory facts. Anecdotes and hearsay are often as prominent as hard data. And some scholars, motivated by ideology, seem to reach their conclusions before the research even starts.

"When you look back, there have been entire movements in education launched on the basis of a few anecdotes, a lot of rhetoric, and not much evidence," said Gerald Sroufe, director of government relations for the American Educational Research Association.

But now the field is facing unprecedented pressure that could change the nature of research into what works in schools.

A new federal law requires schools to base dozens of policy decisions on research or risk losing federal money, and a bill pending in Congress would, for the first time, attempt to set quality standards for government-funded education research.

"There's some very good work out there, but there's a lot of very bad work, too," said Stanford professor Richard Shavelson. "You just have to figure out how to separate the two."

The federal education bill passed in January uses some variant of the phrase "scientifically based research" 110 times.

It requires states and districts to use research to determine their approaches to everything from teacher training to the hiring of security guards. The Bush administration has spoken often about its desire to make education an "evidence-based field."

That, in turn, has fostered the desire to establish quality-control measures for research.

"I've seen some colossal missteps in education – things like New Math and 'schools without walls' – that were seemingly never tested before being tried," said Rep. Michael Castle, R-Del., chairman of the House education reform subcommittee and author of the pending bill.

Teachers looking for ways to improve their skills find that different researchers give contradictory evidence for what works and what doesn't. Academics line up on both sides of every major issue – vouchers, testing, social promotion – and are often accused of cooking the data to get the results they want.

Peer reviews

Reformers such as Mr. Castle would like the field of education research to look more like the world of medical research.

Until the 1950s, doctors operated largely on anecdotal evidence about what worked and what didn't. As independent professionals, many felt that they – not some distant academic researcher – were best able to decide the effectiveness of treatment.

"The prevailing attitude was that each doctor was his own experimenter," said Robert Baruch, an education researcher at the University of Pennsylvania. That's not far from how some teachers feel today, he said.

Over time, physicians were convinced that controlled clinical trials could lead to better medicine. Now, medical research is strictly regulated by government – the Food and Drug Administration – and by an academic community that conducts rigorous peer reviews of scholarly work in prestigious publications such as The Journal of the American Medical Association.

Mr. Castle's bill, which passed the House on April 30, would require peer reviews of all federally funded studies and would model research regulations on those of other federal agencies, such as the National Institutes of Health and the National Science Foundation.

The bill is now sitting in the Senate committee chaired by Sen. Edward Kennedy, D-Mass. A Kennedy spokesman said that Senate work on the bill will begin in September, but that the committee will likely create its own bill rather than work from Mr. Castle's.

Some academics agreed that the overall quality of education research needs to be raised but said they are wary of government standards.

"The intent is appropriate, but how it gets translated into law is questionable," Dr. Shavelson said. "I don't think it's the role of the federal government to define science."

Raising bar

For example, the most successful style of medical experiment has been the randomized trial, in which a group is randomly divided into two parts: One gets a certain treatment and the other doesn't. But local control of schools makes random assignment difficult for education studies, he said.

"When the researchers come in and say, 'We want you to abandon the way you teach and try this new way for three years so we can see if it works,' are parents going to go along with that? Or will local control come into play and say 'We're in charge, we're not going to [do] that'?"

Maintaining strict research conditions in a school is notoriously difficult: students move, principals get fired, policies and demographics change. In one recent long-term experiment led by Boston University researcher Christine Rossell, half of her subjects had withdrawn from the study within three years.

"The so-called hard sciences – chemistry, physics – are misnamed," she said. "Those should be called the easy sciences. You can keep ions and molecules in a beaker for decades. They don't have any human rights. They don't move from one school district to another in the middle of a research study and mess up your data. Researching real human beings who don't always follow directions – that's a hard science."

"People think you can just look at test scores and figure out what works," she said. "Doing a strict scientific study is extremely difficult and gets very, very expensive."

According to a National Research Council study released this year, federal funding for educational research has dropped from more than $400 million in 1973 to about $130 million today. Only about one-tenth of 1 percent of education funding in America goes to research.

"If we were a drug company or GM or Ford, spending that little money on research, we wouldn't stay in business," Dr. Baruch said.

Even without government intervention, some private sources are trying to improve research quality. Dr. Baruch leads the Campbell Collaboration, a 2-year-old organization whose goal is to evaluate existing education research and "screen the good stuff from the poor stuff." He said that sometimes fewer than 10 percent of the studies they evaluate meet the highest standards the collaborative looks for.

"One of our biggest problems is that it's all called 'research,' no matter how good it is," Dr. Sroufe said. "Once you see the headline 'Researchers show such-and-such,' it's difficult for the public to get much deeper than that."

E-mail jbenton@dallasnews.com