Since the initial publication of the “hockey sticks” by Michael Mann and members of his self-described “team,” there has been controversy over the methodology used in the studies and the reliability of the results. The subsequent history is well known.
Papers were published by Steve McIntyre and Ross McKitrick showing that the centering methods used in the Mann papers produced statistically biased results. Questions were raised about the proxies used and it became evident that one could produce similar “temperature” series by utilizing certain types of artificially generated random series rather than physical proxies which purportedly were related to the temperatures. This eventually led to a congressional hearing to establish whether the M & M criticism was valid. Prof. Edward Wegman was commissioned to produce a report examining the various claims.
For further discussion , it is important to understand the makeup of the contents of this report:
Executive summary (5 pages)
Introduction (3 pages)
Background on Paleoclimate Temperature Reconstruction (13 pages)
– Includes Paleo Info, PCs, Social networks
Literature Review of Global Climate Change Research (5 pages)
Reconstructions And Exploration Of Principal Component Methodologies (10 pages)
Social Network Analysis Of Authorships In Temperature Reconstructions (10 pages)
Findings (3 pages)
Conclusions and Recommendations (2 pages)
Bibliography (7 pages)
Appendix (33 pages)
– Includes PCA math, Summaries of papers and Sundry
The three topics which have been most discussed on the web include the background explanatory material on proxies (~6 pages), the analysis of the Mann applications of PCA (~16 pages) and the social network analysis (~15 pages which include 9 figures, each of which occupies a major portion of a page).
It should also be understood that these three topics are stand-alone. The material discussed within a topic along with any results obtained is independent of that in each of the others. Therefore, criticism of some aspect of a particular topic will have no bearing on the accuracy or correctness of any portion of the other topics.
The material on social networks was then used in preparing a publication using social networking procedures to relationships among authors in the paleo-climate community. The resulting paper was published in the Journal, Computational Statistics and Data Analysis, about a year later under the authorship of Yasmin H. Said, Edward J.Wegman, Walid K. Sharabati, and John T. Rigsby.
The original report has been a thorn in the side of advocates of Global Warming since it was presented. Many attempts have been made to discredit the report including accusations of plagiarism of parts of a work by R. S. Bradley who made the rather extraordinary demand that the entire report be withdrawn even though, as I point out above, any such use of the material from his work had no impact on other portions of the report.
The concerted effort by the ankle-biters to discredit Prof. Wegman continued until it seems to have resulted in the withdrawal of the social networking paper due to charges that some material in the paper was not properly referenced. The quality of the paper was further discussed in USA Today in an email interview with a “well-established expert in network analysis”, Kathleen Carley of Carnegie Mellon. It is this latter news article that I wish to discuss.
Q: Would you have recommended publication of this paper if you were asked to review it for regular publication — not as an opinion piece — in a standard peer-reviewed network analysis journal?
A: No – I would have given it a major revision needed.
Over the past thirty or so years, there has been a move in the statistical community to present a greater number of papers illustrating applications of newer statistical techniques. These are not papers intended to “move the science forward” as much as to inform other statisticians about the use of the techniques and to often highlight an innovative application of the methodology. As such, they would certainly not be the type of paper submitted to a journal which specialized in the subject matter. This sort of paper may sometimes written by students with their supervisor’s cooperation.
Note the letter Prof. Wegman sent to the editor of the journal:
Yasmin Said and I along with student colleagues are submitting a manuscript entitled ―Social Network Analysis of Author-Coauthor Relationships.This was motivated in part by our experience with Congressional Testimony last summer. We introduce the idea of allegiance as a way of clustering these networks. We apply these methods to the coauthor social networks of some prominent scholars and distinguish several fundamentally different modes of co-authorship relations. We also speculate on how these might affect peer review.
The indication is clear that the paper is intended to present a simple application of the methodology to the wider statistical audience.
Q: (How would you assess the data in this study?)
Data[sic]: Compared to many journal articles in the network area the description of the data is quite poor. That is the way the data was collected, the total number of papers, the time span, the method used for selecting articles and so on is not well described.
I agree that the data is not described in depth in this paper. However, a better description of the data was given in the earlier report in which it was initially used. That report was referenced in the submitted paper, but it is quite possible that it was not read by Prof. Carley.
It should be noted that the authors had decided not to mention any names of the subjects whose author network was analyzed. This would reduce both the type and the amount of information that could be included without violating that anonymity.
Q: (So is what is said in the study wrong?)
A: Is what is said wrong? As an opinion piece – not really.
Is what is said new results? Not really. Perhaps the main “novelty claim” are the definitions of the 4 co-authorship styles. But they haven’t shown what fraction of the data these four account for.
As I mention above, this is an expository presentation of the use of the methodology, NOT an “opinion piece” as characterized by Prof. Carley. No “new results” as such are needed. Furthermore, she does not indicate that the results in the presentation could be incorrect in any way.
There was one other paragraph in the article which caught my attention:
Carley is a well-established expert in network analysis. She even taught the one-week course that one of Wegman’s students took before 2006, making the student the “most knowledgeable” person about such analyses on Wegman’s team, according to a note that Wegman sent to CSDA in March.
Social network methodology is not rocket science. The average statistician could become reasonably proficient in applying the methodology in a relatively short period of time. Understanding the methodology sufficiently to “advance the science” would indeed require considerably more study and time to develop the skills needed. It is unfortunate that the journalist exposed his uninformed biases with a negative comment such as this.
There have been questions about the short review period before the paper was accepted. Dr. Azen indicated in the USA Today article that he personally reviewed and accepted the paper, not a surprise if you take into account my earlier comments about the expository nature of the paper. However, It is unfortunate that he did not demand a more comprehensive bibliography since it is indeed much too sparse. If he had done so, we would likely not even be discussing this subject.
Nonetheless, nobody has demonstrated that the science in the paper has been faulty and, regardless of its demise, the fact stands that the original report and its conclusions regarding the flawed hockey stick cannot be impacted by this is any way.
Statistics and Other Things 
Pingback: Wegman and the ? « the Air Vent
Thanks for the additional detail. My understanding has been that the content of the citations was and is pretty much boilerplate cliché material in the field, and was used just to set up the real discussion and analysis.
But I think the attacks on Wegman have failed, in the sense that much more attention has been drawn to his report than would have otherwise happened. The Believers are always stuck defending the indefensible.
The indication is clear that the paper is intended to present a simple application of the methodology to the wider statistical audience.
That indication was not at all apparent in the Journal, where it was published as just a regular research paper listed under:
Section II: Statistical Methodology for Data Analysis.
“Dr. Azen indicated in the USA Today article that he personally reviewed and accepted the paper, not a surprise if you take into account my earlier comments about the expository nature of the paper.”
Well, not quite.
“Azen says he must have overseen an earlier, more extensive review of the paper involving outside reviewers. But he says he has no records of this earlier review, because his records were destroyed in an office move. “I would never have done just a personal review,” he says.”
Sounds like Azen can’t defend a personal review. But I noted the paper itself, in the abstract, solemnly warned that:
“We conjecture that certain styles of co-authorship lead to the possibility of group-think, reduced creativity, and the possibility of less rigorous reviewing processes.”
Perish the thought! I wonder who they are talking about?
So??? Why do you expect that the journal needs to segregate such such papers? I would suggest that most journals do not use identical format. Some papers have such sectional partitioning and others do not. Would you not think that this material qualifies as being “Statistical Methodology for Data Analysis?”
With regard to Dr. Azen, I used the quote from the email written at the time the paper was accepted. I guess you would of course prefer that I use the ad hoc “adjusted” quote from years later as the accurate one. The latter could be a CYA statement or he might well have actually forgotten what had been done with one small paper years earlier. However, I don’t see what that has to do with Prof. Wegman’s culpability in any of this.
I don’t think that anyone will deny that such things happen in virtually every journal and every effort should be made to minimize such incidents.
In this particular case, there does not seem like any harm was done to the integrity of the scientific content of the paper. However, I would compare this with the harm that could occur when such treatment of the review process occurs with developmental papers which receive preferential treatment.
Do you really believe that the early Mann hockey stick papers received diligent reviews for methodology, for proxy choices, for proper calculation of estimates and error bounds? I don’t.
I have read numerous climate papers for which proper evaluation would have not been possible from the vague or even nonexistent descriptions given. Nobody should receive a free pass on this stuff.
“Some papers have such sectional partitioning and others do not.”
And this journal does, as described here:
“The journal consists of three refereed sections, and a fourth section dedicated to news on statistical computing.” (my bold)
MBH98 appeared in Nature. I’ve no reason to believe that Nature’s editors were part of Mann’s alleged tight circle. He was a junior scientist then. And I’m sure they appointed the usual five or so reviewers, and would not have chosen them all to be Mann’s pals. Nature’s review processes are famously rigorous.
What’s your point? That the type of application note I have described would not be peer reviewed? That it would be classified as “news”??? As far as I can see, it was NOT in a separate section.
The depth of the peer review process for might not be as stringent as required by papers which involve mathematical derivations and the content not earth-shaking, but you are seriously grasping at straws here!
I hadn’t realized that Mann’s was the only name on those papers. 😉
You would think that (as in many areas of focused research activity) cliques may form and a sense of community emerge among the participants because of their common interests. This did not start with the existence of Mann so pal reviews have predated his entry into the arena. The co-authors were not unknowns in the field.
Contrast that with the situation where someone from outside of the field might attempt to publish a paper in the same area which would have far-reaching implications for changing currently held ideas. Do you think that these would be reviewed in an identical fashion (or in many cases, not even accepted for consideration)?
One of the problems with too strong a sense of community in an area of research is the tendency of researchers to not accept the limitations of their expertise when working with methods that have traditionally been the bailiwick of another field of science. This has been the case with climate science and statistics. It relates to pal reviews through the fact that when ad hoc innovations (whose effects have not been evaluated in a meaningful theoretical fashion) are introduced, the reviewers (who may not even be closely “related” to the authors) will still be tempted to trust the use of these methods even though they do not have a sufficient understanding of their effects .
Nature’s review processes may have a reputation for rigor, but it does not necessarily guarantee that it works in every case. For papers from a given field of research, the responsibility for the validity of the reviews falls almost entirely on the individuals from that area who form the pool of researchers from which the reviewers are selected. As I point out above, for some cases, unless the reviewers admit their inability to properly review the subject matter and qualified specialists from outside are included, it will still be a pal review .
“Social network methodology is not rocket science. ”
Indeed! I am a long retired state school maths teacher. Twenty years ago I used to do this with thirteen year old school children, analysing friendship according to how often children visited each others houses etc.
What the warmists cannot abide is that Wegman showed that their professional relationships are so incestuous that they fall under the saying “self praise is no reccomendation”.
Such “network” analysis has been done informally for years. It has become more formalized and advanced its capabilities tremendously with regard to analysis and graphical representations.
I would not use terms as harsh as “incestuous”, however, it is certainly reasonable that if the researchers in an area of endeavor become a tightly knit exclusive group, then the individuals within the group could indeed resort to taking unethical actions against outsiders to the group.
IMHO, it was exactly this type of result that was very much evident in many of the Climategate emails.
Regarding the irony of weak review for Wegman, I doubt very much that even the O’Donnell submission was checked by reviewers for citation error and that microscope was not underpowered.
The paper was an easy stone skip with correctness yet little import. They swung at the underhand pitch and didn’t pay enough attention to where the wording for the background info came from.
To explain further, I can easily imagine missing the citation from quoting a previous work. Were that previous work a submission to congress, I think my own citations would be accurate but I don’t live the life of a big-shot professor. Most, in fact all of the professors whom I’ve worked with would have caught this kind of mistake, but there is simply no motive here. If Wegman or Said were ’embarrassed’ to cite Wiki then reword it, elsewise cite it — who cares.
The whole thing is so stupid it hurts.
I don’t know the snip line for politics at Roman’s blog so I’ll just snip myself.
Jeff:
I am pretty certain that this almost never gets done in practice.
In my view, it’s the reviewer’s responsibility to determine whether the content of their paper is accurate and consistent, not to verify they are following ethical guidelines for the conduct of responsible research. It’s my view if they aren’t, and this becomes known, it is egg on the face of the authors for failing to e.g. properly cite their work.
Of course if the results are not novel and the authors don’t cite the appropriate research publications that previously reported these results, this is something we do look for and will ding them for. But who has time to go the authors’ narrative and make sure they’ve properly cited/block quoted etc text from other papers that is merely descriptive in nature? The onus is on them to get that right. to.
In the case of Wegman and Said it looks like something like this should have been done: Wegman is fully accountable for everything that is in the paper, especially since he choose first author status. There simply is no excuse for him either not reviewing the papers contents for proper citation of other work, or having somebody else review it for him.
If you put the grad student’s name first, at least you can legitimately say he is primarily responsible for the papers content. As things stand here, there is no way back from sloppy research that is abusive of the ethical responsibilities of a senior researcher for his graduate students.
(In the case of students, come on… you should be familiar with their style and recognize the abrupt transition when they are copy and pasting wikipedia entries. Did you even read fully read the paper that you put your name on as first author, leaving one of the coauthors off?)
Carrick, the paper has four authors listed in the order: Yasmin H. Said, Edward J.Wegman,Walid K. Sharabati, John T. Rigsby. Wegman is identified as the “Corresponding author”. I agree with you, however, that the real responsibility for the paper clearly rested with Prof. Wegman and not Said. Such negligence is easy to do, particularly if you have many graduate students working with you, but in the long run that should not be an excuse.
With regard to the other two authors,Walid Sharabati was a Ph.D graduate student at George Mason and worked in Network theory. There are two interesting Powerpoint presentations I found using Google that indicate the authors might have been sufficiently “expert” at this material, here and here. Dr. Sharabati is currently at Purdue University.
The final author was John T. Rigsby from the Naval Surface Warfare Center who received his Master’s degree in Computational Statistics in 2005 from George mason University. I was able to find several references to technical work that he was involved in:
From 2006, a publication in the Journal of Information Science, Brief Communication Adjacency and proximity searching in the Science Citation Index and Google , and a possibly unpublished paper Data Cleansing & Preparation at the Gates: A Data-Streaming Perspective by a number of people including both Rigsby and Wegman.
Although the withdrawn paper was somewhat weak, I think that these authors were reasonably capable of handling the material they were working with.
RomanM, I was thinking of the grad student who wasn’t even a coauthor who Wegman blames for the plagiarism. Wegman basically said the student copied and pasted the material into the congressional report, and then the material got reused in this manuscript.
That’s pretty lame.
Her version is a bit different again:
“I was Dr. Wegman’s graduate student when I provided him with the overview of social network analysis, at his request. My draft overview was later incorporated by Dr. Wegman and his coauthors into the 2006 report. I was not an author of the report.”
Nick, the “difference” comes from reading two separate articles by the same author, Dan Vergano.
In the one dated May 15th, he writes.
I believe that Carrick would have been referring to this statement. You will notice that the words “into the congressional report” are not in quotation marks in the article so they appear to have been paraphrased by Vergano.
However, you are referring to the article that he wrote on May 16th (and updated the following day). The full quote from the email from Ms. Reeves clearly states that she contributed a draft document to Prof. Wegman’s request which he later included in the report:
Her justification for excluding references was that the material “followed the format of the report” and in this she appears to be partly correct. From my own experience, requisitioned reports outside of academic scholarly work (whether written by academics or not) do not usually provide line-by-line referencing. If it is important to relate a specific statement to its source, then an exception would be made. Otherwise, all of the references would be given either at the end of each section or in a single location within the report document.
However, the problem appears to be that she may likely have neglected to give any references at all leading Wegman to believe that the “overview” was completely her own work. You will notice that Prof. Wegman states that “We would never knowingly publish plagiarized material”.
As far as “author status” goes, it has not been my experience of any reason (ethical or otherwise) that everyone contributing to a report need necessarily be acknowledged as a contributor. However, as Carrick states in his comment, this traditionally does not carry over to published academic work where the ownership of intellectual content plays a much more central role.
I think the difference is classic he said…she said…
She says that she didn’t write part of the report – she wrote an overview which others included in the report. She describes her version as a draft.
It seems unclear whether she wrote it as something that she expected to be included in the report. The first para suggests no. The reference to the format suggests yes, but isn’t conclusive.
Nick, your hair-splitting obfuscation is incredible.
She states that she is not an author. That is a well-defined term meaning that her name is not on the report listing her as a person who is responsible for the end product. It does not in any way suggest that she had no expectation that the material would be included in the final document.
Look at the facts: She produced her portion in response to someone’s request – it was not just “volunteered” content. In your own words, “She describes her version as a draft” – for what purpose would changes might be made later to that material if NOT the expectation that it would be used in print form? . She specifically follows the format which is used in the report – a format she is well aware of before she hands in the draft.
Yet, you can still say with a straight face: It seems unclear whether she wrote it as something that she expected to be included in the report.
!!! Appropriate words elude me …
Roman,
Yes, it is unclear in the first para. She says the draft was later incorporated. The fact that she spelt that out suggests to me that she didn’t write it with that destination in mind. And the fact that she emphasises that she wasn’t an author reinforces that.
But in any case, as Carrick said, placing the blame on someone you weren’t prepared to credit with authorship is pretty feeble.
Nick, it certainly could NOT have been incorporated earlier. Here, the word “later” is an uninformative filler word , not a word intended to stress some specific aspect of the situation ( as for example, much later might do). The rest of her statement just doesn’t make sense if it had been unintended for inclusion in the report.
Equus est mortuus…
Pingback: Climategate Documents Confirm Wegman’s Hypothesis « Climate Audit
“uninformative filler word “
How do you know?
I think she meant it.
Reblogged this on Climate Ponderings.
The point was not to blame the grad student. It was to explain that what happened was an honest error of judgment. While on principle copying was always an issue, there was no enforcement in the absense of a complaint. There was no complaint AT THE TIME as there was no thief of ideas, just recycling standard phases. The attacks were selective enforcement. If it were standard to enforce at the time of publication, you would have expected the journal would have picked it up at the spell checking stage, before publication. This is standard to running tools to check for this in student works now. Withdrawing the paper politically motivated. The appropriate response was an correction. .