National Academies School the Trump Administration on Gold-standard Science

Report on Effects of Human-Caused GHG Emissions on US Climate, Health, and Welfare shows how actual science is done.

It’s not news that the Trump administration has little interest in getting the facts right. But facts often matter for crafting policy that serves our societal goals. And it’s not rare for the law to require that specific factual findings underpin government decisions. In both cases, we need to assemble, understand, and apply the best available scientific information. This week the National Academies of Sciences, Engineering and Medicine did just that, responding to a profoundly unscientific report issued by the Department of Energy in July.

The two reports come to dramatically different conclusions. The Academies committee finds that “[m]ultiple lines of evidence show that greenhouse gas emissions from human activities are the primary driver of the observed long-term warming trend,” and that EPA’s 2009 conclusion that GHG emissions from motor vehicles endanger both the public health and the public welfare “continue[s] to be true and [is] supported by improved scientific evidence.” The conclusions of the report commissioned by the Department of Energy are much less crisply stated. It’s largely a catalog of individual complaints about the series of IPCC reports on GHG impacts. (By all means read it if you’re inclined to think I’m mischaracterizing.) But its gist, as best I can follow it, is: CO2 is good for plants; the earth is not actually warming, but if it were that could be from non-anthropogenic causes; the economic impacts might not be as bad as others are saying; and in any case no US policy measures can make any detectable difference.

Who is right on the substance matters for the fate of EPA’s proposal to rescind the 2009 endangerment finding and to halt all federal regulation of GHG emissions from mobile sources. But that’s not my point here. I want to focus on the way the reports were assembled, what that says for how their conclusions should be evaluated by the courts and the public, and what it says about the administration’s commitment to “restoring a gold standard for science” as set out in Executive Order 14303.

Research science is an iterative process of gathering information, interpreting that information to generate ideas about the way the world works, encouraging others to test and critique those ideas, and re-evaluating as new observations are gathered. Ideally, scientists do their best to put aside their inevitable personal biases and commitment to their prior ideas. At a minimum, those are disclosed to the scientific community. Over time, the decentralized work and ruminations of that community advance a consensus understanding of the topic under study. Of course it doesn’t always work perfectly, but most of the time it works pretty well. There has to be room in the scientific process for questioning orthodoxy, but that doesn’t mean that contrarianism should be viewed as itself a marker of scientific quality or integrity.

Regulatory science like that underpinning the endangerment finding is a bit different. Typically, the government does not generate new data for regulatory purposes, although in some contexts it may require that industry do so. The essence of regulatory science is the synthesis, interpretation, and application of data gathered by others. That should be done by experts in the field who undertake and document a literature review that is as exhaustive as possible given time and resource constraints, who do not start with a preconceived conclusion in mind, who invite and are prepared to respond to broad peer review, who explain their conclusions in a manner intended to allow non-experts to understand and critique them, and who explicitly note uncertainties and how they have been dealt with.

The National Academies study meets those standards far better than the DOE report does.

The Academies are private non-profit organizations founded during the Civil War to provide independent scientific advice on policy questions. The membership is elected by their peers in recognition of their outstanding contributions to their field. Since 1916, the Academies have done much of their advisory work through committees constituted by the National Research Council; those committees include members of the Academies as well as non-members recruited for their expertise in the specific topic. The Academies have explicit standards for choosing committee members to ensure the right expertise, a range of perspectives, and the ability to “be objective and open-minded in addressing the issues.” Committee members must disclose financial conflicts of interest, relevant publications, and any other information that “could have a significant impact on public perception of the objectivity and value of the committee’s work.” The Committee that prepared this report included 16 members, with a range of expertise and backgrounds, including academia, government laboratories, and two members retired from work in the oil industry.

The Committee’s membership, their biographies, and their charge (to review evidence gathered since the 2009 endangerment finding to determine whether that finding’s conclusion that GHG emissions from motor vehicles endanger public health and welfare is still supported by the available evidence) were made public before it began work.

Most Committees have one or more public meetings at which they receive and discuss evidence. This one, working on a tight timeline to issue its report before the close of the comment period on EPA’s proposal to rescind the endangerment finding, did not. It did, however, solicit public input prior to beginning its deliberations on literature it should consider, generating pointers to more than 600 peer-reviewed articles since 2009.  As required by the Academies committee process, 15 peer reviewers provided comments on the draft before it was finalized.

The Committee Report itself carefully sticks to evaluating the evidence that has become available since 2009, and explains how it reached its conclusions (“the committee relied upon multiple sources of evidence and considered conclusions to be stronger if there is broad agreement among independent lines of evidence”). It reviewed both individual studies and large scale assessments prepared by large teams of scientists, incorporating broad public input, and subject to peer review. Although it considered climate models, “it weighed most heavily observational evidence.” The Report includes an extended discussion of evidence since 2009 of climate change impacts on human health.

The DOE report, by contrast, was drafted by a Climate Working Group consisting of 5 members selected personally by the Secretary of Energy. Four of the five have impressive scientific credentials (the fifth is an economist), but all appear to have been chosen because they share the same perspective — they are all prominent climate skeptics who have long been critical of the scientific consensus on anthropogenic global warming and of the value of potential policy responses. It did all its work behind closed doors, did not solicit input on data to consider, and did not invite pre-publication outside peer review. On the same day that the Academies Committee issued its report a federal court ruled that the DOE working group was subject to the Federal Advisory Committees Act, which requires greater transparency.

Perhaps unsurprisingly, given its small and deliberately contrarian membership, the Working Group relied heavily in its citations on the members’ own publications. It devotes most of its space to nitpicking consensus documents such as the IPCC reports. It reads as a series of disconnected complaints about the scientific consensus, designed primarily to sow doubt about the strength of the evidence supporting that consensus. The Working Group does not explain if or how it obtained a broad view of the current literature, or how it evaluated conflicting or reinforcing studies. It does not discuss the human health effects of a warming climate.

This is a useful case study in how members of the public can approach scientific controversies. It can certainly be difficult for a non-expert to pick up a document on a highly technical subject like climate change, read it, and evaluate it. It can be easy to evaluate solely on the basis of the result — if your priors are confirmed, that’s the assessment you prefer. But there is a better approach. Where there are dueling documents, it makes sense to look at: the degree of transparency about the task and how it was approached; the degree to which broad input on the task, relevant studies to consider, and the draft document were solicited and considered; the background of the authors, how they were selected, the breadth of perspectives represented, any financial conflicts of interest, and how entrenched their points of view were at the outset; and the care and accessibility of the explanation, including how the authors evaluated the strength of the evidence.

The shortcomings of the DOE Working Group report, issued with much fanfare and relied upon as a key support for rescinding the endangerment finding, also reveal the lack of commitment at the highest levels of this administration to the principles set out in the Executive Order on Restoring Gold Standard Science. In case there were any doubt, this story demonstrates that the President is not actually interested in transparency, interdisciplinarity, self-skepticism or unbiased review. He (and his obsequious cabinet) is interested only in creating a record that highlights any available evidence supporting his preferred policy outcome and buries any contrary evidence. That is anything but “restoring gold standard science.”

Department of Energy, EPA endangerment finding