In science, as in most disciplines, the process is as important as the product. The recent email/data release (aka Climategate) has exposed the process of scientific peer-review as failing. If the process is failing, it is reasonable to wonder what this implies about the product.
Several scientists have come forward to express their view on what light Climategate has shed on these issues. Judith Curry has some insightful views here and here, along with associated comments and replies. Roger Pielke Jr. has an opinion, as no doubt do many others.
Certainly a perfect process does not guarantee perfect results, and a flawed process does not guarantee flawed results, but the chances of a good result are much greater with the former than the latter. That’s why the process was developed in the first place.
Briefly, the peer-review process is this; before results are published in the scientific literature and documented for posterity, they are reviewed by one or more scientists who have some working knowledge of the topic but who are not directly associated with the work under consideration. The reviewers are typically anonymous and basically read the paper to determine if it generally seems like a reasonable addition to the scientific knowledge base, and that the results seem reproducible given the described data and methodology.
Generally, reviewers do not “audit” the results—that is, spend a lot of effort untangling the details of the data and or methodologies to see if they are appropriate, or to try to reproduce the results for themselves. How much time and effort is put into a peer review varies greatly from case to case and reviewer to reviewer. On most occasions, the reviewers try to include constructive criticism that will help the authors improve their work—that is, the reviewers serve as another set of eyes and minds to look over and consider the research, eyes that are more removed from the research than the co-authors and can perhaps offer different insights and suggestions.
Science most often moves forwards in small increments (with a few notable exceptions) and the peer-review process is designed to keep it moving efficiently, with as little back-sliding or veering off course as possible.
It is not a perfect system, nor, do I think, was it ever intended to be.
The guys over at RealClimate like to call peer-review a “necessary but not sufficient condition.”
Certainly is it not sufficient. But increasingly, there are indications that its necessity is slipping—and the contents of the released Climategate emails are hastening that slide.
Personally, I am not applauding this decline. I think that the scientific literature (as populated through peer-review) provides an unparalleled documentation of the advance of science and that it should not be abandoned lightly. Thus, I am distressed by the general picture of a broken system that is portrayed in the Climategate emails.
Certainly there are improvements that could make the current peer-review system better, but many of these would be difficult to impose on a purely voluntary system.
Full audits of the research would make for better published results, but such a requirement is too burdensome on the reviewers, who generally are involved in their own research (among other activities) and would frown upon having to spend a lot of time to delve too deeply into the nitty-gritty details of someone else’s research topic.
An easier improvement to implement would be a double-blind review process in which both the reviewers and the authors were unknown to each other. A few journals incorporate this double-blind review process, but the large majority does not. I am not sure why not. Such a process would go at least part of the way to avoiding pre-existing biases against some authors by some reviewers.
Another way around this would be to have a fully open review process, in which the reviewers and author responses were freely available and open for all to see, and perhaps contribute. A few journals in fact have instituted this type of system, but not the majority.
Nature magazine a few years ago hosted a web debate on the state of scientific peer-review and possible ways of improving it. It is worth looking at to see the wide range of views and reviews assembled there.
As it now stands, a bias can exist in the current system. That it does exist is evident in the Climategate emails. By all appearances, it seems that some scientists are interested in keeping certain research (and particular researchers) out of the peer-review literature (and national and international assessments derived there from). While undoubtedly these scientists feel that they are acting in the best interest of science by trying to prevent too much backsliding and thereby keeping things moving forward efficiently, the way that they are apparently going about it is far from acceptable.
Instead of improving the process, it has nearly destroyed it.
If the practitioners of peer-review begin to act like members of an exclusive club controlling who and what gets published, the risk is run that the true course of science gets sidetracked. Even folks with the best intentions can be wrong. Having the process too tightly controlled can end up setting things back much further than a more loosely controlled process which is better at being self-correcting.
Certainly as a scientist, you want to see your particular branch of science move forward as quickly as possible, but pushing it forward, rather than letting it move on its own accord, can oftentimes prove embarrassing.
As it was meant to be, peer-review is a necessary, but not sufficient condition. As it has become, however, the necessity has been eroded. And blogs have arisen to fill this need.
In my opinion, blogs should serve as discussion places where ideas get worked out. The final results of which, should then be submitted to the peer-reviewed literature. To me, blogs are a 21st-century post-seminar beer outing, lunch discussion, or maybe even scientific conference. But they should not be an alternative to the scientific literature—a permanent documentation of the development of scientific ideas.
But, the rise of blogs as repositories of scientific knowledge will continue if the scientific literature becomes guarded and exclusive. I can only anticipate this as throwing the state of science and the quest for scientific understanding into disarray as we struggle to figure out how to incorporate blog content into the tested scientific knowledgebase. This seems a messy endeavor.
Instead, I think that the current peer-review system either needs to be re-established or redefined.
The single-blind review system seems to be an outdated one. With today’s technology, a totally open process seems preferable and superior—as long as it can be constrained within reason. At the very least, double-blind reviews should be the default. Maybe even some type of an audit system could be considered by some journals or some organizations.
Perhaps some good will yet come out of this whole Climategate mess—a fairer system for the consideration of scientific contribution, one that could less easily be manipulated by a small group of influential, but perhaps misguided, individuals.
We can only hope.
Chip – You have made some excellent points. As someone who has participated in both the peer review process and in open discussions on blogs, I can see that both have benefits and weaknesses. There are some scientific and engineering journal publishers that seem to be testing ways to use techniques from both interactive media and paper published journals but they are struggling with the revenue model. Even non-profit professional societies that publish peer reviewed journals have to cover the cost of that activity – many of them do so by charging researchers a fee to publish and review their work.
Somehow there has to be a recognition that openness is important, but also a recognition that there is some need for gate keeping – not everyone’s opinion is worth hearing on complex technical issues. Really smart, ethical scientists and engineers are reluctant to comment or publish on topics outside of their field of expertise, but that does not stop people with agendas from taking advantage of open systems to dominate the conversation and skew the results.
Any blog owner out there will recognize that trolls and spammers exist and need to be controlled in order to keep conversations useful.
Rod Adams
Publisher, Atomic Insights
@ Rod Adams,
About your comment, with the deepest respect.
“Somehow there has to be a recognition that openness is important, but also a recognition that there is some need for gate keeping – not everyone’s opinion is worth hearing on complex technical issues. Really smart, ethical scientists and engineers are reluctant to comment or publish on topics outside of their field of expertise, but that does not stop people with agendas from taking advantage of open systems to dominate the conversation and skew the results.
Any blog owner out there will recognize that trolls and spammers exist and need to be controlled in order to keep conversations useful.”
I agree – to a point – that there needs to be some gate keeping but I am disturbed that people who have degrees in the area of conversation and have a point of view that has been deemed “non-mainstream” are often shut out of the conversation by being called or identified as “trolls.” This happens even when they make very legitimate points – just ones that are not popular on the particular site or journal they are submitting to. In this case there were only a few people who had degrees in the area of climate change who were willing to continue to try to publish in the face of strong opposition. So they became “trolls” because they would not be silent.
I am concerned that we seem to be leaving behind the capacity for honest disagreement. When everyone can write a blog the natural tendency is to read those you agree with and ignore or not look for those that you disagree with – conversations become quickly stilted and lacking in breadth. In the same way, climate change has become a fairly on sided conversation with people labeling anyone with questions as “right wing” or as “republicans” or as “deniers.” I have heard very strong questions and a great deal of careful research dismissed with “oh you are right wing.” When large amounts of money are getting ready to change hands – as in cap and trade or when deeply held philosophical and religious views are held as outside the preview of “science” (as in any kind of appeal to or recognition of design in the universe) one begins to see “science” not as a search for truth but as a political force aiming at shaping the thoughts and opinions of the majority of people who do not have the time, energy, and background to assess what is happening.
So, yes, not everyone should be heard on complex technical issues. But those who have a limited ability or those whose ideas simple do not stand up with eliminate themselves by having their ideas ignored by others when using an open system in a peer review journal. Yes, some quacks will write. But very few people actually know what these journals are or how to submit to them. It takes a pretty fair background to even get to that point. Some of those submitting will be annoying but this is far better than being led down the primrose path by those who stand to gain from our tax dollars by suppressing opposing points of view.
I have enjoyed your efforts to bring balance to the conversation about nuclear and I have noticed that at times sites block you from the conversation. While permissible it makes me strongly question those who want to shut others out.
So, on climate change are we being sold a bill of goods? Are there more hard drives that should be revealed? As you can see, this incident has added deeply to my suspicion. It will take a great deal of change in the processes for me to begin to trust those scientists who claim the sky is warming.
[…] Climategate: Is peer review in need of change? (Master […]
[…] following is from the blog Master Resource. Run by Robert Bradley. There are many interesting and well written essays there, and WUWt readers […]
Under the open access philosophy, Redalyc looks forward to contribute to the scientific editorial work produced in and about Iberoamerica, making available for the students and researchers the content of more then 550 magazines from different knowledge areas. You can access Redalyc through http://redalyc.uaemex.mx
Chip, thanks for clarifying what peer review is and isn’t. It is not an audit. The audit properly occurs after peer review and publication. To me what’s even more scandalous than the effort to bias the peer review process is the effort to obstruct and delay post-publication audit. Willis Eschenbach has written brilliantly on this (http://wattsupwiththat.com/2009/11/24/the-people-vs-the-cru-freedom-of-information-my-okole%e2%80%a6/). Here’s how I’d put the matter. Skeptics have been accused of waging a “war on science” because they frequently question the IPCC’s interpretation of the rapidly growing field of climate change research. But science is not a set of dogmas certified by government-funded bodies. Rather, science is fundamentally an adversarial process whereby competing scientists attempt to reproduce – that is, invalidate – each other’s results. This process absolutely depends on each combatant allowing the others to examine his data and methods. Researchers who hide their data and computer codes to prevent others from replicating or invalidating their work attack the heart of science. Such behavior is unethical at best. Worse, the Jones gang colluded to evade the UK’s Freedom of Information Act via various ruses and excuses including deleting emails, which is very likely illegal.
What do we want: Open Science!
When do we want it: Now!
Chip:
Good post.
In my view (as a physicist), the peer-review concept has not only been long dead, but it has instead become a tool to promote political agendas, not science.
It has (my opinion) been promoted and controlled by academic elitists, often as a way to not only display their “accomplishments,” but also to keep out the “unclean.”
To see a paramount example of this today, simply examine what is going on at the National Academy of Sciences — a once esteemed institution that has become infested with lobbyists. I wrote this article about one of their reports, but it is an indictment of their whole system (http://www.northnet.org/brvmug/WindPower/NationalAcademiesCorruption.pdf).
For several months now I have submitted FOIAs to the NAS to get some basic information on an important energy report. They have aggressively stonewalled every request.
Needless to say, an essential element of science is transparency.
It amuses me that the same people who religiously worship the “peer-review” process, find transparency anathema to their situation.
Another contradiction is that these same academics also have no clue as to what is the Scientific Method.
The most fundamental question about AGW is this: has this hypothesis been subjected to the rigors of the Scientific Method?
If YES: please provide such data forthwith.
If NO: please go back to the lab before foisting “solutions” on us concerning your opinions.
The only problem I have with peer-review is when a field or topic is small or has only a small number of established researchers in the reviewer pool. I was in lipid research years ago and had problems with one reviewer. All of our papers were sent to him because he was the best there was outside of our lab; he was also our major competition, or that was the way he saw it. Generally, criticism was not a problem, but he would sit on our papers for months and slow down publication to a crawl. A few times we had to appeal to the journal involved for them to demand the return of the paper with the reviewer’s comments or publish anyhow without that individual’s input. It was rather frustrating, but out of our control – as it should be. I would not change this system, but the journals were generally willing to listen to well couched complaints from authors.
Chip,
Very thoughtful post you’ve written. I think the peer review process can benefit from the demonstrated effectiveness of two phenomena: crowd-sourcing and the way honey bees make decisions (see http://www.ece.osu.edu/~passino/PapersToPost/GrpDecMakHoneyBees-AmSci.pdf). Both use the collective wisdom of many individuals who participate to their level of ability and the value of the information they produce (rather than the producer’s “credentials”) is judged by others. The process is both egalitarian and efficient.
Of course there are lots of details to be worked out in terms of professional credit, priority of discovery, and other concerns now handled by the slow process of publication, but these don’t seem insurmountable and the professional societies might be the logical starting place for establishing the new paradigm in “peer” review. The whole topic is worthy of scholarly effort. It’s clear that the current process can be co-opted rather easily and needless scandal seems to be the only way to correct it.
I thought that Michaels and Balling’s suggestion in the last chapter of Climate of Extremes sounded like a good idea.
[…] be trusted. Quite a few blogs, such as Chip Knappenberger at Master Resource blog, suggest that the whole peer review process might be at fault. Certainly a large proportion of people are suspicious of mainstream science – […]
Begin by making data open and available: All the technology standards (See Open Geospatial Consortium: http://www.opengeospatial.org) are in place to build open systems that enable live and stored geospatial data (data with a location component) to be efficiently discovered, assessed, accessed, reused, etc. Then provide consensus-derived documentation of studies for the scientific record.
” Climategate: Is Peer-Review in Need of Change?”
Indeed, the Climate Science has been thoroughly reviewed and by that British Peer of the Realm, Lord Monckton – but unfortunately that wasn’t part of the peer-review-process – and the process of keeping him and like minded inquisitors, and among them some very emminent scientists, out of it – is what’s brought it to this.
Scientific Peer-Review is a Lightweight Process
Posted by Shannon Love on November 30th, 2009
http://chicagoboyz.net/archives/10481.html
Excellent post. Shannon concludes:
Why then do we use peer review? Simple, publishing glaringly flawed papers or being seen as taking sides in a scientific dispute destroys journals both professionally and economically. Peer review protects the journals and the careers of the people who staff them.
***
Replication and proven predictive power, not the opinions of peers, test science assertions. Those iron objective tests separate science from all other disciplines. In the long run, scientific peers are always wrong. Science does not advance by consensus, it advances by the repeated destruction of each generation’s existing consensus.
If the strongest defense someone can muster for a paper’s conclusions is that “it has been peer reviewed,” that is a dead giveaway that the paper itself is very, very weak. Basing policy on scientific studies that have been merely peer reviewed is just flat insane.
The polar ice caps are melting for several reasons. The
most evident and still easiest to stop, is the huge
amount of kinetic energy sent into those waters by
the Ice Breaker Ships, Tankers, and Freighters we
allow to disturb those waters. Like a glass of ice
cubes being swirled around inside a glass of water,
the kinetic energy is melting the ice. Until we put
the glass back into the freezer, solidification will
not occur in our next ten generations (300 years).
The bad habits of our shipping and government
operations is the problem at it’s heart. The simple
corruption of the spirit of man will be all our down-
falls, and the decline of most species on planet
earth as we know it, including Homo Sapien sapien.
There is only a few other options, not worth trying
and too expensive to push at this late hour in the
timeline. The challenge is to cut through the egos
and pride, the paperwork and politics, and come
down on anyone transporting through Polar Seas.
These huge egg-beaters stir up the H2O so much,
the wakes go out for many miles to melt the ice.
Allowing them to continue is a method of madness.
Apply all the scientific theory you want, but open
your eyes first and apply common sense to the most
obvious lesson in high school science class: Apply
energy, raise the temperature, change the mass to
a different form.
Are you paying attention?
Go and see for yourself. The wakes are self evident,
if you’ll only “test the waters”.
[…] Chip Knappenberger As it now stands, a bias can exist in the current system. That it does exist is evident in the Climategate emails. By all appearances, it seems that some scientists are interested in keeping certain research (and particular researchers) out of the peer-review literature (and national and international assessments derived there from). While undoubtedly these scientists feel that they are acting in the best interest of science by trying to prevent too much backsliding and thereby keeping things moving forward efficiently, the way that they are apparently going about it is far from acceptable. […]
[…] is more remarkable to see Paul “Chip” Knappenberger join in. Knappenberger is a colleague of Patrick J. Michaels, the libertarian climate scientist […]