A fresh round of climate science emails were hacked and released to the public last week. With the debate over secrecy in science back in the headlines, science writer Fred Pearce makes the argument for open access

Steve McIntyre is a pernickety Canadian. A retired mining geologist, trained mathematician and amateur climatologist, he has for the past eight years locked horns with the Climatic Research Unit (CRU) at the University of East Anglia, trying to gain access to their data on the history of global temperatures.

He is not (repeat: not) paid by, beholden to or in regular contact with fossil fuel companies or lobby groups trying to undermine climate change science. He is not even a climate sceptic. For years, McIntyre has been asking for CRU’s “crown jewels”, raw data assembled from weather stations round the world that it says proves how much the world has warmed in the past 160 years.

He does not believe this conclusion is a big lie. But he does want to see for himself. And in particular to look at how the data had been “manipulated” — a perfectly honourable process in which, for instance, some weather stations are made to count for more than others because they represent large areas with few weather stations, while others are discounted because their rural locations have been invaded by growing cities.

It’s not what everybody wants to do on a Saturday night, but surely he is exactly the kind of citizen investigator the 2000 Freedom of Information Act was intended to help.

Of course, his persistence has not made him a friend of CRU’s director Phil Jones. The crown jewels are his life’s work. For years Jones held out, with the backing of his university’s Freedom of Information (FOI) officers, from releasing the data to McIntyre. Jones said it was commercially valuable. He said it was his intellectual property. He said revealing it would damage international relations. CRU has been congenitally hostile to FOI requests from McIntyre and others. At the end of 2009, 105 FOI requests had been submitted to UEA for CRU data, of which only ten had been acceded to in full.

The crown jewels and data libertarians

The battle between the two men for the crown jewels was the backdrop — and very possibly the motive — for the still-mysterious hacking of CRU’s emails and their publication online at the end of 2009. Much of the world’s science community sided with Jones in the resulting “climategate” saga, condemning what they regarded as politically and commercially motivated attacks on their research.

But others took McIntyre’s side, seeing him as a data libertarian. And last June, following a new request for the data from Jonathan Jones, an Oxford physicist and “climate agnostic”, the FOI’s commissioner, Christopher Graham, finally ruled that the crown jewels should be handed over. And they were, a month later. The world did not fall in.

If CRU had been more open with its data from the start, a great deal of time and angst on the part of its scientists — and a great deal of public uttering of paranoid nonsense from climate deniers — would have been avoided. And if, in the months before the hack, Jones and his colleagues had not spent ever more time bitching about McIntyre and scheming to keep their data and working methods secret, then the emails would have contained little of outside interest.

Graham’s decision unlocks some four million temperature readings taken at 4,000 weather stations over the past 160 years. But as the journalist Jonathan Jones put it, “the most significant features of this decision are the precedents that have been set”. It could open the door to thousands of other British researchers being required to share their data with the public. Good.

Under the 2000 Freedom of Information Act, universities, like other public institutions, must share their data unless there are good reasons not to. It is now clear that the good reasons have to be just that — not excuses. Graham, who is the final arbiter in FOI requests, was scathing in his ruling that CRU claims that sharing data would harm international relations were “highly speculative”. And on commercial considerations, he noted acidly: “it is not clear how UEA might have planned to commercially exploit the information”.

But should all publicly funded data be free, and all publicly funded researchers required to hand it over? In an age when data distribution is so easy, it is hard to make a case that sharing data is just too hard. After the military, scientific researchers were the first people to use the internet, precisely so they could share large data sets among themselves. So why not let us all join in? But what about emails and research notes and the data from failed experiments? Some believe requests for such stuff would both damage research and overwhelm researchers. And some think your access should depend on who you are.

At the same time as the climategate FOI requests began building up at CRU, the giant tobacco company Philip Morris began — initially anonymously — asking for data from Scottish researchers who had interviewed thousands of teenage smokers on what they thought about tobacco marketing. Not only was this expensive research – paid for by a cancer charity — it was also, as the head of the Stirling University research unit Gerard Hastings put it, “the sort of research that would get a tobacco company into trouble if it did it itself”.

The researchers have held out — and went public with their disgust in the Independent in September. But here’s the bottom line. FOI legislation is “applicant blind”, as Maurice Frankel of the UK Campaign for Freedom of Information puts it. It does not matter if the thoughts of smoking teenagers are of interest to Philip Morris or the National Heart Foundation or someone who wants to stop their child from starting to smoke. They are, and should be, all the same. Otherwise Friends of the Earth would never get pollution data.

In this case, researchers may be able to argue that disclosing the information could jeopardise future planned research, for instance by drying up funds from cancer charities. But Graham’s tough line with CRU suggests that argument is not guaranteed to succeed.

Data sharing and a climate of fear

One reason scientists have such a problem with FOI is that virtually none of them realised that it would apply to them. Certainly, the science community failed to consider the consequences or lobby for the drafting of laws that might make sense for them. Only now is the Royal Society trying to catch up by forming a working group to discuss openness in science.

It is also true that there is little consistency among scientists about what the rules on data sharing and confidentiality should be. Some peer-reviewed journals have tough rules requiring access to that data underpinning research papers, but others do not, including some academic institutions. But there is a growing move to more openness that should surely be welcomed. Cameron Neylon, a biophysicist at the Rutherford Appleton Laboratory in Oxfordshire, writing in New Scientist in September, said the aim should be for “anyone, anywhere to contribute to science”. You can hear the shudders in the labs across the land. But to those who fear an avalanche of ill-informed nonsense arising from data sharing, he said: “If you care about the place of science in society or are worried about the quality of information on the web, then openness offers massive potential to engage people more deeply, educate them about how science works and increase the store of quality information on the web.”

In the months after climategate there was much discussion in the science community about the need for greater openness. But outside those in the open access movement, it has faltered. The message in the labs is that the inquiries into the affair absolved the scientists of any wrongdoing.

That is not quite true. The inquiries decided, rightly, that there was no grand conspiracy, although they felt they were not in a position to judge the finer points about the conduct of the science. The main enquiry, under Sir Muir Russell, seemed particularly confused about FOI. It noted damagingly that CRU had shown a “consistent pattern of failing to display the proper degree of openness”. But on the detail it showed a sometimes breathtaking lack of attention. It concluded that “there was no attempt to delete information with respect to an [FOI] request already made”, when the emails published online revealed quite clearly that one round-robin requesting the deletion of an email correspondence was sent two days after an FOI request for precisely that information.

Much of science has “closed ranks” behind the idea that those demanding access to their data are troublemakers. Nobel laureate and Royal Society president Sir Paul Nurse says “some researchers … are getting lots of requests for, among other things, all drafts of scientific papers prior to their publication in journals, with annotations, explaining why changes were made between successive versions. If it is true, it will consume a huge amount of time. And it’s intimidating.” Maybe, but the current law allows vexatious requests to be rejected. So that is a straw man.

A fine line between the crackpot and the sublime

In any event, the whole point of research is that it should be open to maximum scrutiny. And the scientific priesthood can no longer claim that scrutiny should only be among their specialist fellows.

And there is sometimes a fine line between the crackpot and the sublime. Earth science guru Jim Lovelock — a doyen for many modern climate researchers — left institutional academia in frustration at his ideas being ostracised. The Independent began its report of the 2011 winner of the Nobel prize for chemistry, Daniel Shechtman, thus: “An Israeli scientist who was once asked to resign his research post because his discovery of a new class of solid material was too unbelievable has won this year’s Nobel Prize in Chemistry – for that same discovery.”

The charge of sloppiness in the way science often portrays its findings to the wider public is also a warning against allowing too much self-policing. In June, the Intergovernmental Panel on Climate Change issued what it said was a summary of the findings of a detailed study of renewable energy. It headlined the claim that 77 per cent of the world’s energy needs could be met from green power by 2050. In fact, the “77 per cent” finding was the most optimistic of hundreds of academic studies reviewed in the report itself. Moreover, that particular study was conducted by one of the report’s own lead authors, who was also a Greenpeace campaigner. Curious. But most damagingly of all, this highly relevant information only emerged a month after the press release and subsequent media coverage, when IPCC got round to publishing the report itself. This was shameful spin.

The fuss over climategate showed that the world is increasingly unwilling to accept the message that “we are scientists; trust us”. Other people want to join the scientific conversation. Good scientists, interested in finding truth, should want to encourage them, not put up the shutters. The wider world instinctively knows to distrust those in all walks of life who reject openness. As McIntyre put it recently, “probably no single issue damages the reputation of the climate science community more than the refusal to show the data that supports their work”. There should, for the good of science as well as public discourse, be a presumption in favour of open access.

McIntyre, meanwhile, is still hunting. He believes CRU researchers using tree rings to unpick temperatures in past eras may have been cherry-picking their Siberian logs to help sustain the argument that recent decades are warmer than anything in the past 2,000 years. He cannot be sure, because they are still refusing to hand over their full data sets. CRU’s justifications have a familiar ring. Disclosure could do “financial harm” to the university by reducing its “ability to attract research funding”. Really?

If McIntyre eventually gets the data, could it undermine the case that man is warming the world? Certainly not; that is independent of past natural variability. Could it change our ideas about past natural climate change? Conceivably, yes. Is it a scandal that McIntyre cannot get to see the data to review CRU’s work and do his own science? I believe it is.

Fred Pearce is author of The Climate Files: The Battle for the Truth about Global Warming (Guardian Books)

This article appears in Dark Matter the winter 2011 edition of Index on Censorship magazine, which explores science and censorship.

Click here for subscription options and more