Mario: I am afraid you are right in almost all respects. Being an old-timer, I have spent much time trying to educate users, students, etc. on what good flow data is and how to present it. However, I think the problems goes much deeper in the sense that it is not just flow cytometry that is being battered and abused in the literature. There are a lot of other areas both new and old. One sees the same lack of understanding or scientific rigor in the presentation of Western blots, fluorescence microscopic images [i.e., the concept that green and red stained molecules when they give you yellow fluorescence means that actually interact], genomics [failure to provide information that errors exist at every level of the analysis], etc. Yes, there should be documents available to journals and authors that help both to decide how best to present data. Those in the position to should continue to teach new users or students how to present their data. Those on editorial boards or reviewers for all journals should be very demanding of authors when their manuscripts contain cytometric data (as one should with any data). Yet, as you pointed out, to the extent that this is already going on it has had only a small impact on this issue (e.g., many manuscripts don't go through the hands of "experts" on cytometry before they are published). And it is also true that many investigators will fail to ask or follow the advice of flow people in core facilities. How to respond. One way is to moan and groan as we have been doing lo these many years. Another is to provide post-publication review of articles that have especially egregious examples of bad flow data. As you say this may be a bit ugly. I submit, however, that if one is willing to go that far why not take it one step further. Maybe it is time to start publicly embarrassing scientists who misrepresent or misinterpret data, i.e., through questions following talks/presentations, letter to the editor, etc. I am not advocating embarrassing people over poorly labeled axes on their dot or contour plots. I am more concerned with those people who use misunderstood, misinterpreted data for support of a scientific conclusion. This does all science a disservice and is one of the few areas where the damage is serious enough to require this type of action. (By the way, in my experience, this is not as infrequent as it should be). There will be lots of examples where the flow is bad but does not change the bottom line - was the hypothesis proved. Those examples are annoying but not sufficiently serious for public comment. While scientists who published misunderstood or misinterpreted data are not guilty of knowingly misrepresenting the data, in some instances, the effect is the same. Should we as individuals or as a group of scientists with a particular expertise take such a proactive stance? Might some of us not be guilty of butchering some other disciplines data? I do not know the answers. Maybe its time that some researchers that have, what might be called a cavalier attitude about these things, realize that sometimes such behavior may not go unpunished. Respectively Frank Traganos -----Original Message----- From: Mario Roederer [SMTP:roederer@drmr.com] Sent: Thursday, November 01, 2001 1:23 PM To: Cytometry Mailing List Subject: Bad FACS data presentation -- more discussion points. I must respectfully disagree with the recent prevailing sentiment expressed by "old-timers" on this board. (Since I'm not an old-timer, and I've been in the field for almost 20 years, then I guess the definition of old-timers are those who were publishing FACS data in the 70's). While my comments below may rankle a few people--nothing new for me--please be assured that I mean no disrespect to anyone. This is a very serious topic, one that has obviously raised a lot of interest if not emotions in the community. It is one about which I have a very central interest, have studied extensively, and have even taught about. Hence, I have much to say on the topic. (What a surprise.) First of all, I strongly support an endeavor to produce a document (or two) to help people present their flow data. This has been strongly needed for a long time, and I will be more than happy to help in any way I can. But to think that such a document will make a dent in bad data publications is just plain fantasy. Face it, the "experts" have been around for 30 years trying to explain to scientists how to present data, and it hasn't really helped. A document such as is proposed isn't going to be considered outside of our little group of cytometrists. Why? Well, for example, how many people at whom this is targeted have been bothered to read Howard's or Alice's or anyone else's book about Flow Cytometry? If they haven't read those excellent texts (or even parts of them), I don't think that presenting them with even a brief missive is going to budge them. The fact is, scientists don't have the time to assimilate any more "helpful" documents. The problem is not that they don't want to, it is that they don't believe that they have a problem. They don't understand what's wrong with what they are doing, especially since they've been doing it for decades. And no amount of our going to them and saying "Hey, here's a better way to present your data" is going to change them. Scientists are the ultimate curmudgeons. I've heard from dozens of Operators or facility Managers that they try very hard to be a part of the review process for data presentation out of their facilities. However, most scientists at these facilities simply disdain the effort. What happens? Our facility people go to them to help them, but the scientists think they already know how. They simply won't accord the facility people with enough respect, and admitting that maybe the flowjocks might actually know something the vaunted immunologist doesn't. Giving the flowjock a piece of paper to hand the scientist isn't going to be of much help in this regard. And to those of you who suggested that we need to do our part by reviewing the documents that come out of our facilities or our departments and to provide presentation help to the scientists, you are 100% correct. But unfortunately this too is fantasy. The reality is, I have enough trouble in my own institute "educating" people on flow data presentation. And I have the advantage of being one of their peers (i.e., "faculty"). They wouldn't give me the time of day if I were "only" an Operator, despite the fact that most of the Operators around know far more about flow cytometry than they do. This is why there has to be an effort to demonstrate what's wrong with the existing data presentations. Among others, Zbigniew, Guenter, Paul, and Howard have been vocal about this being the wrong approach. The tenet underlying their opposition is that we don't want to be negative, we want to be positive about things. Of course, this presumes that scientists are a sensitive lot that can't take criticism and will respond poorly to such criticism. (Well, OK, many of them are. But the majority, I believe, are not. In fact, I believe that the majority of scientists would take the criticism in stride. To say otherwise would be to accuse them of insecurity.) The only way to convince scientists to turn to the Operators, to the Facility Managers, to the flowjocks, is to prove to them that there are serious problems with what the scientists are publishing. I don't like being negative, but I'm willing to bet that for the time being, it's going to be the only way to get the attention of the people outside our community. Of course, once we have their attention, once we've demonstrated that there are serious flaws in most presentations, then they will turn to the Facility people and ask for advice. Then they will be receptive to documents that illustrate good presentation practices. Then they will turn to "only" Operators and discover that the facility people actually do know what they're talking about. And maybe they'll start listening to the Operators, to the Managers. I am convinced that it is only by empowering the people who actually run the facilities to educate the scientists, that we will we be able to change the prevailing presentation techniques. Only by empowering the Operators and the Managers to advise the scientists in a meaningful way will result in long-term success. This requires a two-pronged approach. First, we must indeed generate the appropriate educational material -- documents -- that illustrate what the right way is to present flow data, and why, and how, etc. This document can have the brief form useful as a checklist for publication, and as a more complete form that not only gives the checklist and guidelines, but explains WHY those guidelines are important. But second, we must prove that the existing literature is replete with awful FACS data. In my opinion, the best way to do this is to convince editorial boards to solicit a separate specialized review of all FACS data in tentatively-accepted papers to ensure that the data is presented properly (and, if there are problems pointed out in the FACS review, to give the original reviewers a chance to re-evaluate based on what they missed in the first go-around). However, I am sure that editorial boards will be loathe to load up the review process any more than it already is. Should we encounter resistance to this kind of review assistance, then I will stand by my original suggestion of post-publication review. Sure, it's ugly, and will likely be viewed as arrogant. So what! Everyone (including scientists) already thinks that scientists are arrogant. But it just might have a positive impact: imagine your local immunologist finding out that the data presentation atrocities he or she just published in Nature have been pointed out for the world to see. Do you think said immunologist will not think once or twice about asking for advice with the next paper? And the bottom line is that most scientists will admit that the artifacts inherent to the technology are too many, and often too subtle, for them to be facile with those problems. Hence, a forum for those who are familiar with the problems to illustrate them can be very valuable. This is nothing unique to flow cytometry; the recent burgeoning field of gene-chip analysis has many problems that could be similarly addressed. I will admit that this is a radical approach, one that is fraught with danger because of the perception that it is criticism only for the point of being critical. In fact, it is what I consider a last-ditch effort. But we've now had 25+ years of bad data. Just when is it that we've come to the last ditch, if not now? mr (PS: I found an old "questions" list that I gave my class some years ago--questions to ask when you are reading a FACS-based paper--and to kick things off, I put it at <http://www.drmr.com/facsquestions.html>. It's very short, it hasn't been updated in a long time, but it's still valid...)
This archive was generated by hypermail 2b29 : Sun Jan 05 2003 - 19:01:37 EST