There is a “dirty little secret” among consultants who assess health care spending. They can make cost savings for Medicaid programs such as North Carolina’s showcase Community Care of North Carolina a moving target to reach whatever conclusions the hiring agency wants, experts in the field say.

“I would not let any state agency, whether Medicaid or any other, do [its] own cost justifications,” said Al Lewis, one of the harshest critics of three studies North Carolina ordered on its award-winning CCNC program.

CCNC is the nonprofit organization and administrative heart of the state’s Medicaid program, which provides health insurance for the poor and disabled. It contracted with three consultants to determine cost trends.

Consultant Milliman found $984 million in savings from 2007 through 2010. Treo Solutions said the savings totaled $1.5 billion from 2007 through 2009. Mercer said savings ranged between $708 million and $758 million from 2005 through 2009.

While CCNC “provides terrific access” to Medicaid participants, Lewis, an expert in care management outcomes and instructor of health policy at Brandeis University, believes all those numbers are wildly exaggerated.

“I think the studies that they’ve done, every single one of them, is fatally flawed and that the state legislature is wasting taxpayer money,” said Michael Cousins of Clayton, a Ph.D. and nationally known care management and medical home methodologist.

“When clients are happy, they typically renew their contracts. It is a dirty little secret in our field,” Cousins said. “They’re giving the client what they want, so everybody is happy.”

In its study, Treo found by far the largest savings in CCNC. Treo has an ongoing support contract with CCNC, and currently is conducting another study for it.

Rich Keller, senior health care consultant at Treo, shrugged off the different findings by consultants.

“Any study will address different questions, using different methods, and possibly covering slightly different time periods and population groups; differences in the scope of the study will lead to differences in the results,” Keller said.

“The bottom line is that three different studies showed that costs were avoided by CCNC — this is something taxpayers should know,” Keller said.

But Marty Rowan, a Wilmington attorney and former president and CEO of Health Management Corp., a disease management company, called the Treo study “professionally embarrassing.”

Cousins said “the way they evaluated that program … would be laughed off the stage at any credible forum. They are trying to convince us their comparison was credible, yet they compared to apples and oranges.”

Nicole Fisher, senior policy director for Health Systems Innovations in Washington, D.C., says a healthy skepticism is justified both of cost savings in the CCNC studies, and as a general rule.

“Given the politicization of health care right now, I think it’s safe to assume if you’re hearing something touted strongly by one side or the other, it’s their version of the truth,” said Fisher, a former health policy analyst at the John Locke Foundation.

Lewis, Cousins, and Fisher agree that North Carolina needs to remove the responsibility for hiring consultants from Medicaid and place it with an independent, third-party organization with no skin in the game.

Lewis said that might be the Office of State Controller. Fisher and Cousins believe a multidisciplinary, blue-ribbon panel of experts makes the best sense, and a transparent review process should be a priority. That panel could partner with the legislature’s Program Evaluation Department, Cousins said.

Fisher said it is not uncommon for an agency to tell a vendor before hiring, “We need it to say this, and can you make that happen?” There are many methodologies to choose from, and each has “hundreds and hundreds” of numbers that can be put into evaluation models and manipulated to achieve a preordained outcome.

Such sweeping flexibility is tantamount to a guesswork system and not the best approach to crafting public policy, Fisher said.

“It might as well be the Wild West,” Cousins said. Problems associated with so many models using so many moving parts to reach conclusions acceptable to the agency doing the contracting can mean “almost anything goes.”

Moreover, recent advances in the fields of critical outcomes reporting analysis and population health management and epidemiology have many observers concluding that the old ways of crunching numbers may not be relevant when dealing with publicly funded health care programs.

“Applying classic epidemiological and health services research to these programs is relatively new,” Cousins said. “Although there isn’t a textbook on how to do these things, there are principles that we use to make judgments about what evaluations are credible and which aren’t.”

The Milliman and Mercer studies were done by actuaries who may lack expertise in population health management, Cousins said.

Fisher said she is putting together a summit in Washington, D.C., next year to bring together multidisciplinary participants — “everyone who’s a stakeholder in health care.”

She wants to “have a real discussion” about the national urgency to create an honest approach to analyzing what works best in health care and how best to analyze results to control ever-escalating costs.

It is easy for the uninitiated to get lost in the weeds when the opposing factions argue over numbers, massive sets of data, probabilities, corroborating evidence, and testing mechanisms used to assess programs.

For his part, Lewis said he has “systematically challenged a lot of what goes on in this field.” He said his book Why Nobody Believes the Numbers disproves the conclusions reached by CCNC in North Carolina and studies in other states.

Officials at CCNC defend the studies. In a written statement provided in lieu of an interview, they said Lewis relied on “a dubious ‘back of the envelope’ approach to statistical analysis.”

Bob Cosway, one of the Milliman study principals, said the authors did “a reasonable, accurate analysis,” and used differing methodologies to cross-test their findings.

“None were counter to the results that we published,” Cosway said. However, he allowed, “I’m sure somebody else doing the study could come up with some different results.”

Yet Rowan said Lewis’s criticisms “are mainstream, professional opinions” based on appropriate methodologies to measure CCNC.

The state’s consultants devised their studies “to come up with some cockamamie savings,” Rowan said. And none of them “is putting out credible data to say people are healthier, they’re being treated how they’re supposed to be treated, and we’re getting these cost savings.”

“As a taxpayer, I find it appalling,” she said. “This is taxpayer dollars at a time we don’t have taxpayer dollars to waste.”

Dan Way (@danway_carolina) is an associate editor of Carolina Journal.