Since we are casting stones, note the 6 typos in Mario's comments.
Denoted by "****"
-Phil
A number of people have asked me about the paper. I went back to my notes
and found the critique, which I decided to include below. Please note that
this review passes no judgment on the conclusions of the paper. I give it
simply to illustrate how even an auspicious journal can let highly
problematic FACS analysis through to publication.
Nature Medicine, September 1998. pp 1041 and 1042. Figure 4 on p. 1042 and
its description was all that we addressed in the class lecture; attached
below is my "answer" to the problem ("What's wrong with this picture?").mr
Problems (in parentheses, I rate how significant the problem is: Minor,
Major, or Gut-wrenching)1. The text states that the lymphocyte gate is
medium-to-high
forward and
side scatter. That's obviously incorrect; it is **** ver **** low forward
scatter.
(Minor, probably typographical error, even though it is repeated in the main
text and the figure legend).2. Panel B claims to be an isotype control for
FITC and PE.
However, there
is only one curve shown; is it FITC or PE? It can't be both. (Major, it is
important to be able to ascertain the background staining in the experiment,
as will be seen below).3. Where is the isotype control marker for the third
color
(the one they use
for CD45)? (Major, as for part 2).4. How was Panel B used to determine the
isotype
gate settings? Different
settings are used for PE and FITC (Y and X axes, I presume), but we need to
know what fraction of events are above these gates for the isotype control
(Major).5. How was region R2 defined in Panel C? There appear to be many
"background" events (probably murine cells) that fall into this gate (based
on the observation from Panel B of how many events are above the isotype
control marker). This means that many events in subsequent panels may not
be of human origin. (Gut-wrenching; throws **** intrepretation**** of
Panels D, F,
G, and H into question).6. Panel D is purported to show CD15 and CD33
positive events,
or cells of
myeloid lineage. Unfortunately, the data has been gated (Panel A) for cells
of lymphoid lineage; therefore, no myeloid cells should have been observed
in this panel irrespective of the ability of the stems cells to
reconstitute! Indeed, I take this figure to show the level of nonspecific
staining that is observed in their experiment (the events on the diagonal in
the double-positive quadrant are most likely nonspecifically stained or
highly-autofluorescent cells). (Gut-wrenching; this panel provides no data
that can be used to substantiate authors' claims, and, in fact, provides
ammunition to disregard the **** intepretations **** of other figures).7.
Panel E is the only
panel I believe (i.e., that nearly all of the cells
are B cells). However, there is a problem here: Panel E shows essentially
100% of the cells are CD19+ B cells. If that's the case, then how can there
also be T cells (CD3+CD2+ which must be CD19-), myeloid cells (also CD19-),
progenitor cells (CD19-)? If you were to add up the percentages of T cells,
myeloid cells, progenitor cells, and B cells in panels D, E, F, and G, you
would easily surpass 150%. (Gut-wrenching; indicates that in their system,
ONLY B cells can be reconstituted).8. Panel F shows that the data is
overcompensated
mildly (The CD38+ cells are
against the axis; showing that the PE into FITC compensation setting is too
high). This can also be seen in Panel H. (Minor; the slight
over-compensation does not affect the analysis they are performing).9.
Panels F,
G and H show similar numbers of events on the diagonal just
above "isotype" controls. This is very consistent with background staining
(or highly autofluorescent cells) rather than any specific staining of CD34,
CD2, or CD3. The authors need to show us these exact plots for cells
treated identically (staining conditions) except, for instance, for the
addition of CD2 and CD3 (for panel G). If those cells are still there, then
they would be autofluorescent contaminants. (Gut-wrenching; I am completely
unconvinced of the presence in these animals of T, myeloid, or progenitor
cells).10. The stains should be compared to positive controls. This would
aid in
knowing where CD3 and CD2 (for example) positive cells should appear. The
cells purported to be CD3+ are actually very dim for CD3; this is yet
another indication that the actual cells are artefact and not truly CD3+
(Major; when the staining pattern for antigens is different than expected,
then positive and negative controls MUST be shown in order to convince us
that the staining is real).11. The authors could address this by using a
staining
combination such as
CD3 vs. CD19 (i.e., mutually exclusive reagents). This lets them eliminate
from consideration any events that are nonspecifically binding reagents or
highly autofluorescent that appear on the diagonal and double-positive.
(Major; this would have allayed suspicions about **** Panesl **** D, F, G,
and H.
Lesson: don't always generate staining combinations that would result in
your desired cell population being positive for all stains!)12. There is no
indication
of the live/dead status of the cells. At least a
viability stain (like propidium iodide) should have been included in one of
the stains. (Major; it is possible that all of the "double positive" events
in Panels D, F, G, and H are actually dead cells that nonspecifically bind
reagents).13. Basic FACS information such as which fluorophore is used with
which
antibody is missing. Which antibodies (clones) are used; are they all of
the same isotype such that the isotype controls are actually meaningful?
(Minor; typically, because of space limitations, such material is dropped.
This is unfortunate, because, as we see above, such information becomes more
important to be able to rescue the interpretations of this figure!)14. Data
plots
show different numbers of events for each plot (i.e. dot
plots). Why is this? All presumably should have the same number of events.
The problem is that for panel G, for example, the double positive events
appear to be 20% (or more) of the collected events; but from panel E, this
should be impossible. Panel H could have an enormous number of events that
are double-negative; the dot plot is saturated and we cannot know how many
events were actually plotted. In part, the authors could overcome the
deficiencies of dot plot displays by simply telling us the fraction of
events in each quadrant of their data. Better yet would be to present this
information as absolute numbers of cells, i.e., the absolute number of B
cells in the bone marrow, of T cells, etc. Had the authors done this, they
would have recognized that much of their analysis was inappropriate, when
their totals added up to much more than 100% (Gut-wrenching).In summary, the
major
problems are:(1) Bad isotype control use. How were isotype gates defined?
How were
"nonspecific" events over the isotype gate setting dealt with?(2) Incorrect
interpretation of "double positive" events. These are not
specifically stained, but are either autofluorescent or **** nonspecfically
****
stained (perhaps even dead cells).Unfortunately, these major problems cause
the
necessitate that the
**** intepretation **** of this Figure be completely changed. The only
population
that appears to be reconstituted in these animals are B cells; there is no
presented evidence for T cells or myeloid cells, or, for that matter,
progenitor cells.
attached mail follows:
This archive was generated by hypermail 2b29 : Sun Jan 05 2003 - 19:01:35 EST