Since the initial publication of the “hockey sticks” by Michael Mann and members of his self-described “team,” there has been controversy over the methodology used in the studies and the reliability of the results. The subsequent history is well known.
Papers were published by Steve McIntyre and Ross McKitrick showing that the centering methods used in the Mann papers produced statistically biased results. Questions were raised about the proxies used and it became evident that one could produce similar “temperature” series by utilizing certain types of artificially generated random series rather than physical proxies which purportedly were related to the temperatures. This eventually led to a congressional hearing to establish whether the M & M criticism was valid. Prof. Edward Wegman was commissioned to produce a report examining the various claims.
For further discussion , it is important to understand the makeup of the contents of this report:
Executive summary (5 pages)
Introduction (3 pages)
Background on Paleoclimate Temperature Reconstruction (13 pages)
- Includes Paleo Info, PCs, Social networks
Literature Review of Global Climate Change Research (5 pages)
Reconstructions And Exploration Of Principal Component Methodologies (10 pages)
Social Network Analysis Of Authorships In Temperature Reconstructions (10 pages)
Findings (3 pages)
Conclusions and Recommendations (2 pages)
Bibliography (7 pages)
Appendix (33 pages)
- Includes PCA math, Summaries of papers and Sundry
The three topics which have been most discussed on the web include the background explanatory material on proxies (~6 pages), the analysis of the Mann applications of PCA (~16 pages) and the social network analysis (~15 pages which include 9 figures, each of which occupies a major portion of a page).
It should also be understood that these three topics are stand-alone. The material discussed within a topic along with any results obtained is independent of that in each of the others. Therefore, criticism of some aspect of a particular topic will have no bearing on the accuracy or correctness of any portion of the other topics.
The material on social networks was then used in preparing a publication using social networking procedures to relationships among authors in the paleo-climate community. The resulting paper was published in the Journal, Computational Statistics and Data Analysis, about a year later under the authorship of Yasmin H. Said, Edward J.Wegman, Walid K. Sharabati, and John T. Rigsby.
The original report has been a thorn in the side of advocates of Global Warming since it was presented. Many attempts have been made to discredit the report including accusations of plagiarism of parts of a work by R. S. Bradley who made the rather extraordinary demand that the entire report be withdrawn even though, as I point out above, any such use of the material from his work had no impact on other portions of the report.
The concerted effort by the ankle-biters to discredit Prof. Wegman continued until it seems to have resulted in the withdrawal of the social networking paper due to charges that some material in the paper was not properly referenced. The quality of the paper was further discussed in USA Today in an email interview with a “well-established expert in network analysis”, Kathleen Carley of Carnegie Mellon. It is this latter news article that I wish to discuss.
Q: Would you have recommended publication of this paper if you were asked to review it for regular publication — not as an opinion piece — in a standard peer-reviewed network analysis journal?
A: No – I would have given it a major revision needed.
Over the past thirty or so years, there has been a move in the statistical community to present a greater number of papers illustrating applications of newer statistical techniques. These are not papers intended to “move the science forward” as much as to inform other statisticians about the use of the techniques and to often highlight an innovative application of the methodology. As such, they would certainly not be the type of paper submitted to a journal which specialized in the subject matter. This sort of paper may sometimes written by students with their supervisor’s cooperation.
Note the letter Prof. Wegman sent to the editor of the journal:
Yasmin Said and I along with student colleagues are submitting a manuscript entitled ―Social Network Analysis of Author-Coauthor Relationships.This was motivated in part by our experience with Congressional Testimony last summer. We introduce the idea of allegiance as a way of clustering these networks. We apply these methods to the coauthor social networks of some prominent scholars and distinguish several fundamentally different modes of co-authorship relations. We also speculate on how these might affect peer review.
The indication is clear that the paper is intended to present a simple application of the methodology to the wider statistical audience.
Q: (How would you assess the data in this study?)
Data[sic]: Compared to many journal articles in the network area the description of the data is quite poor. That is the way the data was collected, the total number of papers, the time span, the method used for selecting articles and so on is not well described.
I agree that the data is not described in depth in this paper. However, a better description of the data was given in the earlier report in which it was initially used. That report was referenced in the submitted paper, but it is quite possible that it was not read by Prof. Carley.
It should be noted that the authors had decided not to mention any names of the subjects whose author network was analyzed. This would reduce both the type and the amount of information that could be included without violating that anonymity.
Q: (So is what is said in the study wrong?)
A: Is what is said wrong? As an opinion piece – not really.
Is what is said new results? Not really. Perhaps the main “novelty claim” are the definitions of the 4 co-authorship styles. But they haven’t shown what fraction of the data these four account for.
As I mention above, this is an expository presentation of the use of the methodology, NOT an “opinion piece” as characterized by Prof. Carley. No “new results” as such are needed. Furthermore, she does not indicate that the results in the presentation could be incorrect in any way.
There was one other paragraph in the article which caught my attention:
Carley is a well-established expert in network analysis. She even taught the one-week course that one of Wegman’s students took before 2006, making the student the “most knowledgeable” person about such analyses on Wegman’s team, according to a note that Wegman sent to CSDA in March.
Social network methodology is not rocket science. The average statistician could become reasonably proficient in applying the methodology in a relatively short period of time. Understanding the methodology sufficiently to “advance the science” would indeed require considerably more study and time to develop the skills needed. It is unfortunate that the journalist exposed his uninformed biases with a negative comment such as this.
There have been questions about the short review period before the paper was accepted. Dr. Azen indicated in the USA Today article that he personally reviewed and accepted the paper, not a surprise if you take into account my earlier comments about the expository nature of the paper. However, It is unfortunate that he did not demand a more comprehensive bibliography since it is indeed much too sparse. If he had done so, we would likely not even be discussing this subject.
Nonetheless, nobody has demonstrated that the science in the paper has been faulty and, regardless of its demise, the fact stands that the original report and its conclusions regarding the flawed hockey stick cannot be impacted by this is any way.