Joel Bernstein writes:
I just finished reading your oped article on reproducibility in science. As an experimental scientist – more precisely a chemical crystallographer – I have had to deal with this kind of situation a number of times, and at least two examples may serve as the possible exceptions to your rules.
One of the beauties of x-ray crystallography is the internal self-consistent confirmation of the result of an x-ray crystal structure determination. X-ray crystal structure determinations are today used (increasing required by journals) as essentially absolute proof of the molecular structure of a new compound. With the result of a determination there is no doubt that the crystal from which the data were obtained contained molecules with the composition and structure that can be represented by a three dimensional ball and stick model. There occasionally are some problems with measurements and refinements of structures, but they are a very, very small fraction of the mass of crystal structures (now nearing one million in the Cambridge Structural Database, the archive for such results) and those problems are almost always readily recognizable by a skilled practitioner.
Having said that, in my own experience now approaching 50 years I have had two situations in which a chemical reaction yielded a totally unexpected result with the production of one single crystal. In both cases the structure of the single crystal was not only unexpected but the structures were polymorphs of materials for which other, totally difference structures were already known. Thus reactions “gone awry” yielded unpredicted and unexpected results, and the veracity of those results is essentially incontestable, and has not been challenged in any way by referees of the resulting publication.
However – and here is where I come to the point of your oped piece – while I have total confidence in the results of those experiments I do have serious doubts about our ability to reproduce the experiments that led to the formation of those crystals. In the interest of full disclosure we have admitted this “failing” in our papers, but that does not detract from the significance of the results of the crystal structure determination. Thus, there may be cases, at least in chemistry, where the conditions of the experiment may be difficult – but certainly not impossible! – to reproduce, but the result is still valid.
Bernstein adds:
I am reviewing some old work and just recalled a 1972 paper we wrote on a crystal structure determination of one single crystal that took an entire year to grow (while the chemist was away on sabbatical). Until then the material had resisted all attempts to crystallize it. I am attaching a copy of that paper (on muconic acid). Our best example is the second polymorph of tetrathiafulvalene, which we published in 1994. I am attaching reprint of that paper as well. The third example is ammonium hydrogen succinate which actually is in press, so I don’t have a reprint to send you (yet!).
I don’t know anything about crystallography except that James Watson didn’t think his colleague had good fashion sense, but I can respect the general idea of hard-to-replicate results, as these arise all the time in observational social science, where you can’t just spin the wheel and get data on N more elections, or recessions, or wars, or whatever.
What strikes me immediately about the difference between this experiment and the deconstruction of elections is that in both cases you can’t replicate but in the former you have clean data (the crystal) and a universally trusted way to analyze the data (x-ray crystallography) that has its own history of many successful replications with essentially no failures.
In the latter your data is anything but clean and the tools for data analysis are anything but universally trusted with a long history of successful replications.
I don’t see a difference here. In most cases these unreproducible social science examples are also “valid” in the sense that under some unknown set of circumstances (that may never be repeated) those results are what you get.
The invalid part is trying to extrapolate from that to more general circumstances.
One difference I can think of is that typically you wouldn’t get a different x-ray pattern if you put the same crystal in multiple times or used a different detector or whatever, whereas in social sciences you can easily completely change the outcome of the experiment by rewording the questionnaire, or asking the subjects the same questions after they’ve discovered what the purpose of your experiment is, or the like.
Measurement in physical sciences is typically much more reliable. In biology, less so, in social sciences it can be questionable whether you should even call something a measurement.
I just don’t see a difference between “somehow we got how people respond on a political survey to be surprisingly correlated with how many smiley faces they saw” with “somehow we got this crystal to form that we can see no particular use for”.
Anon:
Amazingly enough, there was no striking correlation in that smiley-face study! The famous result that purportedly “punched a hole in democratic theory” arose only after the data were processed to within an inch of their life using path analysis.
“somehow we got how people respond on a political survey to be surprisingly correlated”
“somehow we got this crystal to form that we can see no particular use for”
The first case refers to the success of the analytical method; the second case refers to the creation of the thing to be analyzed. It’s not the analysis of the crystal that can’t be reproduced – the analysis is known to be accurate because it has internal checks; it’s the subject of the analysis – the crystal itself – that can’t be reproduced, probably because the conditions that created the crystal aren’t known precisely or are difficult to reproduce.