I think the main problem is not people in the field but scientists
who buy a machine, plop it out on the desk top and start doing flow- isn't
it as easy as running a gel. Recently a member of my department presented
data supposedly showing coexpression of 2 antigens in a cell line. It was a
nice diagonal and looked just like very poor compensation for one and
positivity for the other. I told him later, in private, that his flow work
did not demonstrate expression of one of the antigens- it was just poorly
compensated. I offered to help (he probably though it was like me offering
to show him how to extract DNA). He said it wasn't necessary as the paper
was already accepted to EMBO and it has since been published. People read
his paper and think cells transfected with FBPHA express increased c-myc.
You or I look at the figure and see bad compensation. This is the main
problem and as long as people think flow is easy- a concept promoted by the
companies selling machines- they will continue to do awful work.
Maryalice Stetler-Stevenson, M.D., Ph.D.
Chief, Flow Cytometry Unit
Laboratory of Pathology, NCI, NIH
If there is no justice then there is no peace.
If the people know justice, they will know peace.
> ----------
> From: Ray Hicks
> Sent: Wednesday, October 17, 2001 8:42 PM
> To: Cytometry Mailing List
> Subject: Re: Bad Flow Data & reviewing -- What can we do?
>
>
> Many good points Mario, but I'm going to take you back a few years to our
> discussion on dot plot versus contour, and how misleading contours are.
> I'd
> reverse your logic in " remember that contour plots are also
> histograms (2D histograms), and they have no numbers on the "Z" axis
> corresponding to event frequency. Why should univariate histograms have
> them?", and suggest that contour plots need even more annotation.
>
> I'm sorely tempted to attach a few figures to this e-mail, but I've
> restrained myself, and made them available at:
> http://www.fcspress.com/seeWhatIMean.gif
> and
> http://www.fcspress.com/512AlongTheAxis.gif
>
> The first <http://www.fcspress.com/seeWhatIMean.gif> shows how strikingly
> different contour plots of the same data can be (the data is from the
> FlowJo
> tutorial set, the figures are made in FlowJo 3.2 and FCSPress 1.3). The
> top
> left dotplot is from FlowJo, and shows the crowding you object to, the
> upper
> central plot is FlowJo's default contour plot of SSCvFSC with ten thousand
> cells, the upper right plot is a plot of 1600 cells gated from the same
> file
> - doesn't look like fewer cells does it?
> The lower left plot is a log 50% contour plot of the data in the top left
> and top centre plots, what is one to make of those contours based on four
> cells that jump out in the lower left? The lower central plot is a dot
> plot
> from FCSPress, plotting data at 512 points along the axis (the data has a
> range of 512 "channels"), FCSPress has dithered the plot to represent how
> it
> would (and does) print on a printer which isn't limited to screen
> resolution (using the "clarify option), you'll notice that using higher
> resolution avoids much of the coalescing to a black blob that you object
> to
> in dot plots (the second figure,
> <http://www.fcspress.com/512AlongTheAxis.gif>, shows this graph at full
> size
> with no dithering) . The lower right plot shows a density plot from
> FlowJo,
> the smoothing belies the sparsity of the data.
>
> What's an expert to do when presented with this kind of thing? Would
> labelling the upper left and lower left plots as having the same number of
> cells be enough to make you see them as representing the exact same data
> set? The dot plot of 1600 cells (not shown for brevity) clearly has fewer
> cells than that of 10000, and does a better job of warning the viewer,
> expert or not, of how confident they should be in making conclusions based
> on the plot than numbering the events on the two contour plots (upper left
> and upper right).
>
> Oh, alright then, I've put a further figure up with two dot plots and two
> contour plots with paired numbers of events at:
>
> http://www.fcspress.com/nowDoYouSee.gif
>
> The other issue I take is; how is the collective going to select the
> experts? Surely the people who are publishing this stuff ARE people "with
> a
> modicum of experience in flow". Putting the responsibility on editorial
> boards is probably going to end up in a status quo. How about pressuring
> your lab-fellows to sling the FACS aspect of papers, that they're
> reviewing
> anyway, in your direction?
>
> Ray
>
> ps as an aside, there's something freaky happening on the axes of these
> graphs - they're 512 channel data, but the linear FSC axis runs out just
> past 200, and one of the events exceeds the maximum for side scatter (ie
> the
> one that juumps above the red line in the left hand plots - has this been
> fixed in later versions of FlowJo? Would this be something an expert
> could
> criticise/reject a paper for?
>
>
> > From: "Roederer, Mario (VRC)" <MarioR@mail.nih.gov>
> > Date: Tue, 16 Oct 2001 13:00:05 -0400
> > To: Cytometry Mailing List <cytometry@flowcyt.cyto.purdue.edu>
> > Subject: Bad Flow Data & reviewing -- What can we do?
> >
> >
> > This topic strikes a nerve with many of us. Indeed, ISAC did at one
> point
> > have the decent notion to have a committee on "data presentation
> standards"
> > or something like that. I remember seeing something at Montpellier--a
> > pamphlet on presentation, I think. Since then, I haven't heard about
> the
> > progress of this committee. I made a number of suggestions on the
> > committee's effort, as it was a reasonable start, but don't know if that
> had
> > any affect. Indeed, even this pamphlet had a number of mistaken
> notions,
> > showing how ingrained things can get even within the community.
> >
> > For example, there was the suggestion that we should always put numbers
> on
> > the Y axis of a univariate histogram ("# of cells"). In reality, these
> > numbers are meaningless--they depend on the resolution with which the
> data
> > is binned, which can vary from program to program and instrument to
> > instrument. The reasoning was that the only way to compare histograms
> was
> > to have these numbers to ensure that the data was interpreted properly.
> > However, this is a misconception--in reality, the peak height in a
> histogram
> > is rarely meaningful; it is the peak area which carries meaning. What
> is
> > necessary in a histogram presentation is to identify how many cells were
> > collected (and displayed in the histogram), and, if any peak in the
> > histogram is cut off, to identify what fraction of the vertical scale is
> > shown. I.e., the only thing worth putting on the Y axis label is "%
> max",
> > where "max" is the maximum peak height. Admittedly, many of my papers
> have
> > the meaningless numbers on the axis... but I'm still learning...
> >
> > I am sure that even this little discussion may set off a minor
> > firestorm--and that's probably good: it will be educational, which is
> the
> > main point of this list! (By the way, remember that contour plots are
> also
> > histograms (2D histograms), and they have no numbers on the "Z" axis
> > corresponding to event frequency. Why should univariate histograms have
> > them?)
> >
> > Jim Houston asks about the needed information for histograms or dot
> > plots--always, the minimum information is the number of events
> displayed.
> > (And yes, I am guilty of not always putting that information in my own
> > publications.) I still strongly advocate against dot plots; there are
> much
> > more informative displays available.
> >
> > But the point of this email is not to address the specific defects in
> data
> > presentation, nor even to start to lay them out. That, in fact, would
> be
> > better done in a book.
> >
> > Both Jim and Robert Zucker bring up the lack of the Community's
> involvement
> > in peer review. It is worth noting that JAMA requires every paper to be
> > reviewed by a statistician, outside of the normal review. Why not have
> the
> > same thing for every flow paper? It seems that the major publications
> > should require an expert to review papers containing FACS
> > presentations/analyses for appropriateness. But it won't happen: if we
> > can't even police our own Journals to ensure appropriate data
> presentation,
> > then what makes anyone think we have the competence to do so for other
> > Journals?
> >
> > Some years ago, a few of us bantied around an idea of "post-publication"
> > review of articles that would be placed online. The concept was as
> follows:
> > each major journal would be assigned to one or two expert reviewers.
> Each
> > issue would be examined for articles that had flow cytometry in them,
> and
> > then the reviewer would go over the paper with a predefined list of
> > criteria. The review would explicitly avoid any judgment about the
> paper's
> > conclusions; it would only address whether the flow cytometric analyses
> were
> > properly presented, interpreted, and then to note what additional
> > information is required, what possible artifacts need to be eliminated,
> etc.
> > The review process would be fundamentally based on a checklist (e.g.,
> "was
> > cell viability assessed?", "what staining controls were performed?", "is
> the
> > data properly compensated?", "did the authors note how many events were
> > displayed?", "are the statistical intreprations of low event counts
> > appropriate?" etc. etc.... I could envision a 100-item list). There
> would
> > be "sub-lists" for different types of flow, like "cell cycle",
> > "immunophenotyping", "intracellular detection", and "it's obvious I
> dropped
> > my samples off at my local core facility, didn't tell them what was in
> each
> > tube, forgot my controls anway, had them generate a few graphs for me,
> and
> > then xeroxed them until the dots I didn't like went away, so don't blame
> me
> > because I can't understand the difference between a contour plot and a
> > photomultiplier tube." The reviews would be posted on-line.
> >
> > The idea of the online post-publication review is that the general
> > scientific community, when reviewing an article, could turn to the web
> site
> > and quickly see if there are major problems with the technology that
> they
> > might not appreciate because of the subtleties. Since the criteria
> would
> > all be published online as well, the goal would be that authors would
> start
> > turning to this site before publication in order to better present data,
> > rather than seeing criticisms of their papers show up afterwards.
> Authors
> > might be allowed to appeal aspects of a review that they feel are
> > inappropriate, thereby providing an ongoing evolution of the evaluation
> > process. There might even be a manuscript pre-review service where
> authors
> > could ensure appropriateness before submitting for review.
> >
> > What would this require? No more than a one or two dozen FACS-savvy
> people
> > to volunteer for this public service. Anyone with a modicum of
> experience in
> > flow would be excellent for this; in fact, it's probably better to
> recruit
> > younger (less jaundiced) people for the process. In reality, the review
> > process would be very rapid, since these are not detailed reviews aimed
> at
> > the science of the paper, but only at the data presentation. I was so
> hot
> > on this idea (now 2 years old) that I even registered a domain for its
> use
> > (http://www.sciwatch.org)--a registration I renew in the hopes that
> > something might actually come of it.
> >
> > In my idealistic vision, eventually journals would turn to the Flow
> > community to do this as a standard of practice rather than have it go on
> > post-publication. Journals might even adopt the standard data
> presentation
> > requirements. People might actually publish FACS data that we can
> believe.
> >
> > But maybe we need to start at home first. I'd like to suggest that
> > Cytometry and Clinical Communications in Cytometry both make an
> editorial
> > decision to require all published papers to come up to some minimum
> > acceptable standard. If these journals make the commitment, then
> perhaps
> > there will be enough motivation for a document outlining these
> procedures to
> > be put together. However much it makes sense, I do not suggest that
> this be
> > done by a committee under the auspices of ISAC, since that effort has
> > essentially failed, principally through inaction. Rather, I think the
> > Editorial Boards should empower a group to put such a document together.
> If
> > such an effort works, it can serve as a model for other journals to
> adopt.
> >
> > mr
> >
>
>
> > From: "Roederer, Mario (VRC)" <MarioR@mail.nih.gov>
> > Date: Tue, 16 Oct 2001 13:00:05 -0400
> > To: Cytometry Mailing List <cytometry@flowcyt.cyto.purdue.edu>
> > Subject: Bad Flow Data & reviewing -- What can we do?
> >
> >
> > This topic strikes a nerve with many of us. Indeed, ISAC did at one
> point
> > have the decent notion to have a committee on "data presentation
> standards"
> > or something like that. I remember seeing something at Montpellier--a
> > pamphlet on presentation, I think. Since then, I haven't heard about
> the
> > progress of this committee. I made a number of suggestions on the
> > committee's effort, as it was a reasonable start, but don't know if that
> had
> > any affect. Indeed, even this pamphlet had a number of mistaken
> notions,
> > showing how ingrained things can get even within the community.
> >
> > For example, there was the suggestion that we should always put numbers
> on
> > the Y axis of a univariate histogram ("# of cells"). In reality, these
> > numbers are meaningless--they depend on the resolution with which the
> data
> > is binned, which can vary from program to program and instrument to
> > instrument. The reasoning was that the only way to compare histograms
> was
> > to have these numbers to ensure that the data was interpreted properly.
> > However, this is a misconception--in reality, the peak height in a
> histogram
> > is rarely meaningful; it is the peak area which carries meaning. What
> is
> > necessary in a histogram presentation is to identify how many cells were
> > collected (and displayed in the histogram), and, if any peak in the
> > histogram is cut off, to identify what fraction of the vertical scale is
> > shown. I.e., the only thing worth putting on the Y axis label is "%
> max",
> > where "max" is the maximum peak height. Admittedly, many of my papers
> have
> > the meaningless numbers on the axis... but I'm still learning...
> >
> > I am sure that even this little discussion may set off a minor
> > firestorm--and that's probably good: it will be educational, which is
> the
> > main point of this list! (By the way, remember that contour plots are
> also
> > histograms (2D histograms), and they have no numbers on the "Z" axis
> > corresponding to event frequency. Why should univariate histograms have
> > them?)
> >
> > Jim Houston asks about the needed information for histograms or dot
> > plots--always, the minimum information is the number of events
> displayed.
> > (And yes, I am guilty of not always putting that information in my own
> > publications.) I still strongly advocate against dot plots; there are
> much
> > more informative displays available.
> >
> > But the point of this email is not to address the specific defects in
> data
> > presentation, nor even to start to lay them out. That, in fact, would
> be
> > better done in a book.
> >
> > Both Jim and Robert Zucker bring up the lack of the Community's
> involvement
> > in peer review. It is worth noting that JAMA requires every paper to be
> > reviewed by a statistician, outside of the normal review. Why not have
> the
> > same thing for every flow paper? It seems that the major publications
> > should require an expert to review papers containing FACS
> > presentations/analyses for appropriateness. But it won't happen: if we
> > can't even police our own Journals to ensure appropriate data
> presentation,
> > then what makes anyone think we have the competence to do so for other
> > Journals?
> >
> > Some years ago, a few of us bantied around an idea of "post-publication"
> > review of articles that would be placed online. The concept was as
> follows:
> > each major journal would be assigned to one or two expert reviewers.
> Each
> > issue would be examined for articles that had flow cytometry in them,
> and
> > then the reviewer would go over the paper with a predefined list of
> > criteria. The review would explicitly avoid any judgment about the
> paper's
> > conclusions; it would only address whether the flow cytometric analyses
> were
> > properly presented, interpreted, and then to note what additional
> > information is required, what possible artifacts need to be eliminated,
> etc.
> > The review process would be fundamentally based on a checklist (e.g.,
> "was
> > cell viability assessed?", "what staining controls were performed?", "is
> the
> > data properly compensated?", "did the authors note how many events were
> > displayed?", "are the statistical intreprations of low event counts
> > appropriate?" etc. etc.... I could envision a 100-item list). There
> would
> > be "sub-lists" for different types of flow, like "cell cycle",
> > "immunophenotyping", "intracellular detection", and "it's obvious I
> dropped
> > my samples off at my local core facility, didn't tell them what was in
> each
> > tube, forgot my controls anway, had them generate a few graphs for me,
> and
> > then xeroxed them until the dots I didn't like went away, so don't blame
> me
> > because I can't understand the difference between a contour plot and a
> > photomultiplier tube." The reviews would be posted on-line.
> >
> > The idea of the online post-publication review is that the general
> > scientific community, when reviewing an article, could turn to the web
> site
> > and quickly see if there are major problems with the technology that
> they
> > might not appreciate because of the subtleties. Since the criteria
> would
> > all be published online as well, the goal would be that authors would
> start
> > turning to this site before publication in order to better present data,
> > rather than seeing criticisms of their papers show up afterwards.
> Authors
> > might be allowed to appeal aspects of a review that they feel are
> > inappropriate, thereby providing an ongoing evolution of the evaluation
> > process. There might even be a manuscript pre-review service where
> authors
> > could ensure appropriateness before submitting for review.
> >
> > What would this require? No more than a one or two dozen FACS-savvy
> people
> > to volunteer for this public service. Anyone with a modicum of
> experience in
> > flow would be excellent for this; in fact, it's probably better to
> recruit
> > younger (less jaundiced) people for the process. In reality, the review
> > process would be very rapid, since these are not detailed reviews aimed
> at
> > the science of the paper, but only at the data presentation. I was so
> hot
> > on this idea (now 2 years old) that I even registered a domain for its
> use
> > (http://www.sciwatch.org)--a registration I renew in the hopes that
> > something might actually come of it.
> >
> > In my idealistic vision, eventually journals would turn to the Flow
> > community to do this as a standard of practice rather than have it go on
> > post-publication. Journals might even adopt the standard data
> presentation
> > requirements. People might actually publish FACS data that we can
> believe.
> >
> > But maybe we need to start at home first. I'd like to suggest that
> > Cytometry and Clinical Communications in Cytometry both make an
> editorial
> > decision to require all published papers to come up to some minimum
> > acceptable standard. If these journals make the commitment, then
> perhaps
> > there will be enough motivation for a document outlining these
> procedures to
> > be put together. However much it makes sense, I do not suggest that
> this be
> > done by a committee under the auspices of ISAC, since that effort has
> > essentially failed, principally through inaction. Rather, I think the
> > Editorial Boards should empower a group to put such a document together.
> If
> > such an effort works, it can serve as a model for other journals to
> adopt.
> >
> > mr
> >
>
>
This archive was generated by hypermail 2b29 : Wed Apr 03 2002 - 11:57:58 EST