Flow cytometry is one of the few areas that the primary data are recorded in
a machine-readable format, along with much information regarding the means
in which the data are gathered. As well as setting up guidelines for the
presentation of aspects of that data, it wouldn't be unreasonable to expect
the primary data to be archived by journals and made accessible to reviewers
and readers so they can analyse it themselves and draw their own inferences,
would it? Not all data would lend itself so well as cytometric does, but it
would be a good start.
Direct access to the primary data would certainly have allowed you to settle
some of the doubts that you observed in this paper and decide one way or the
other. You presume that there is a typographical error (repeated in the
body text of the article and the figure legend), could it not be that they
actually did gate on medium to high scatter, but that they put the wrong
figure into the box ? (I'm not asserting that they did, but it can happen),
this would probably have some impact on interpretation.
I find it remarkable that you made point 14:
> 14. Data plots
> show different numbers of events for each plot (i.e. dot
> plots). Why is this? All presumably should have the same number of events.
> The problem is that for panel G, for example, the double positive events
> appear to be 20% (or more) of the collected events; but from panel E, this
> should be impossible. Panel H could have an enormous number of events that
> are double-negative; the dot plot is saturated and we cannot know how many
> events were actually plotted. In part, the authors could overcome the
> deficiencies of dot plot displays by simply telling us the fraction of
> events in each quadrant of their data. Better yet would be to present this
> information as absolute numbers of cells, i.e., the absolute number of B
> cells in the bone marrow, of T cells, etc.
in light of your accompanying letter "Contour plots and data display" in
which you state:
> the common mistake that bivariate plots (or even univariate histograms) should
> be used to estimate absolute cell counts. This is nonsense. If you want to
> know how many events are in a graph or a particular region, then you should
> express it in a table (or an annotation on the graphic). One does NOT use
> graphs to tell people how many events are in a population. For example, in
> how many publications have you counted the dots in the entire graph to know
> exactly how many events were collected?
(I've excised your erroneous attribution of this misconception to myself!)
You obviously got an inkling from the dot plots that the sample size varied
(useful, eh?). Would you even have noticed this problem if you were
presented with smooth contour plots? I think not! But you're absolutely
right that plots without numbers are Bad Things.
For examples of contour plots with different numbers of cells, see that gif
on my website:
http://www.fcspress.com/seeWhatIMean.gif
and a comparison of dot plot and contour plots with low cell number:
http://www.fcspress.com/overconfident.GIF
Ray
> From: "Roederer, Mario (VRC)" <MarioR@mail.nih.gov>
> Date: Tue, 16 Oct 2001 12:54:18 -0400
> To: Cytometry Mailing List <cytometry@flowcyt.cyto.purdue.edu>
> Subject: Example of Bad Flow Data
>
>
> A number of people have asked me about the paper. I went back to my notes
> and found the critique, which I decided to include below. Please note that
> this review passes no judgment on the conclusions of the paper. I give it
> simply to illustrate how even an auspicious journal can let highly
> problematic FACS analysis through to publication.
>
> Nature Medicine, September 1998. pp 1041 and 1042. Figure 4 on p. 1042 and
> its description was all that we addressed in the class lecture; attached
> below is my "answer" to the problem ("What's wrong with this picture?").mr
>
>
> Problems (in parentheses, I rate how significant the problem is: Minor,
> Major, or Gut-wrenching)1. The text states that the lymphocyte gate is
> medium-to-high
> forward and
> side scatter. That's obviously incorrect; it is ver low forward scatter.
> (Minor, probably typographical error, even though it is repeated in the main
> text and the figure legend).2. Panel B claims to be an isotype control for
> FITC and PE.
> However, there
> is only one curve shown; is it FITC or PE? It can't be both. (Major, it is
> important to be able to ascertain the background staining in the experiment,
> as will be seen below).3. Where is the isotype control marker for the third
> color
> (the one they use
> for CD45)? (Major, as for part 2).4. How was Panel B used to determine the
> isotype
> gate settings? Different
> settings are used for PE and FITC (Y and X axes, I presume), but we need to
> know what fraction of events are above these gates for the isotype control
> (Major).5. How was region R2 defined in Panel C? There appear to be many
> "background" events (probably murine cells) that fall into this gate (based
> on the observation from Panel B of how many events are above the isotype
> control marker). This means that many events in subsequent panels may not
> be of human origin. (Gut-wrenching; throws intrepretation of Panels D, F,
> G, and H into question).6. Panel D is purported to show CD15 and CD33 positive
> events,
> or cells of
> myeloid lineage. Unfortunately, the data has been gated (Panel A) for cells
> of lymphoid lineage; therefore, no myeloid cells should have been observed
> in this panel irrespective of the ability of the stems cells to
> reconstitute! Indeed, I take this figure to show the level of nonspecific
> staining that is observed in their experiment (the events on the diagonal in
> the double-positive quadrant are most likely nonspecifically stained or
> highly-autofluorescent cells). (Gut-wrenching; this panel provides no data
> that can be used to substantiate authors' claims, and, in fact, provides
> ammunition to disregard the intepretations of other figures).7. Panel E is the
> only
> panel I believe (i.e., that nearly all of the cells
> are B cells). However, there is a problem here: Panel E shows essentially
> 100% of the cells are CD19+ B cells. If that's the case, then how can there
> also be T cells (CD3+CD2+ which must be CD19-), myeloid cells (also CD19-),
> progenitor cells (CD19-)? If you were to add up the percentages of T cells,
> myeloid cells, progenitor cells, and B cells in panels D, E, F, and G, you
> would easily surpass 150%. (Gut-wrenching; indicates that in their system,
> ONLY B cells can be reconstituted).8. Panel F shows that the data is
> overcompensated
> mildly (The CD38+ cells are
> against the axis; showing that the PE into FITC compensation setting is too
> high). This can also be seen in Panel H. (Minor; the slight
> over-compensation does not affect the analysis they are performing).9. Panels
> F,
> G and H show similar numbers of events on the diagonal just
> above "isotype" controls. This is very consistent with background staining
> (or highly autofluorescent cells) rather than any specific staining of CD34,
> CD2, or CD3. The authors need to show us these exact plots for cells
> treated identically (staining conditions) except, for instance, for the
> addition of CD2 and CD3 (for panel G). If those cells are still there, then
> they would be autofluorescent contaminants. (Gut-wrenching; I am completely
> unconvinced of the presence in these animals of T, myeloid, or progenitor
> cells).10. The stains should be compared to positive controls. This would aid
> in
> knowing where CD3 and CD2 (for example) positive cells should appear. The
> cells purported to be CD3+ are actually very dim for CD3; this is yet
> another indication that the actual cells are artefact and not truly CD3+
> (Major; when the staining pattern for antigens is different than expected,
> then positive and negative controls MUST be shown in order to convince us
> that the staining is real).11. The authors could address this by using a
> staining
> combination such as
> CD3 vs. CD19 (i.e., mutually exclusive reagents). This lets them eliminate
> from consideration any events that are nonspecifically binding reagents or
> highly autofluorescent that appear on the diagonal and double-positive.
> (Major; this would have allayed suspicions about Panesl D, F, G, and H.
> Lesson: don't always generate staining combinations that would result in
> your desired cell population being positive for all stains!)12. There is no
> indication
> of the live/dead status of the cells. At least a
> viability stain (like propidium iodide) should have been included in one of
> the stains. (Major; it is possible that all of the "double positive" events
> in Panels D, F, G, and H are actually dead cells that nonspecifically bind
> reagents).13. Basic FACS information such as which fluorophore is used with
> which
> antibody is missing. Which antibodies (clones) are used; are they all of
> the same isotype such that the isotype controls are actually meaningful?
> (Minor; typically, because of space limitations, such material is dropped.
> This is unfortunate, because, as we see above, such information becomes more
> important to be able to rescue the interpretations of this figure!)14. Data
> plots
> show different numbers of events for each plot (i.e. dot
> plots). Why is this? All presumably should have the same number of events.
> The problem is that for panel G, for example, the double positive events
> appear to be 20% (or more) of the collected events; but from panel E, this
> should be impossible. Panel H could have an enormous number of events that
> are double-negative; the dot plot is saturated and we cannot know how many
> events were actually plotted. In part, the authors could overcome the
> deficiencies of dot plot displays by simply telling us the fraction of
> events in each quadrant of their data. Better yet would be to present this
> information as absolute numbers of cells, i.e., the absolute number of B
> cells in the bone marrow, of T cells, etc. Had the authors done this, they
> would have recognized that much of their analysis was inappropriate, when
> their totals added up to much more than 100% (Gut-wrenching).In summary, the
> major
> problems are:(1) Bad isotype control use. How were isotype gates defined?
> How were
> "nonspecific" events over the isotype gate setting dealt with?(2) Incorrect
> interpretation of "double positive" events. These are not
> specifically stained, but are either autofluorescent or nonspecfically
> stained (perhaps even dead cells).Unfortunately, these major problems cause
> the
> necessitate that the
> intepretation of this Figure be completely changed. The only population
> that appears to be reconstituted in these animals are B cells; there is no
> presented evidence for T cells or myeloid cells, or, for that matter,
> progenitor cells.
>
>
This archive was generated by hypermail 2b29 : Sun Jan 05 2003 - 19:01:35 EST