I still say, academic publishing needs a 4th party, an auditor, to review
the work of the author after it has been peer reviewed and before it is
published.
In order to audit peer reviewed articles a body of standards are needed to
audit compliance by and to audit fairness of presentation.
The CPA profession created from the financial mess of the 1920's a body of
standards which every financial statement tries to comply with. The audit
function developed inorder to allow investors and others to follow the flow
of progress of a specific company or industry and to allow for
intercompany comparison, on a fair basis. To do this the presentation of
financial information had to be standardized and the data which was used to
create it had to be made the completely reliable and traceable from its
creation to the final presentation of the data in the form of financial
statements.
One of the basic principles of auditing is that "the body of people who do
auditing should not be involved in research and should not be academics,
nor should they have a vested interest in a vendor, publisher, government
or other related entity. In short, they should be totally independent.
The opinion should read something like this
Addressed to the author:
In our opinion, the accompanying "peer reviewed" scientific presentation of
"subject of the paper including 22000 words, 8 figures and 9 drawings
submitted for publication on date" is in conformity with principles of
scientific presentations generally accepted in the United States. The
scientific presentation is the responsibility of the author(s) and the
reviewers; our responsibility is to express an opinion on these scientific
presentations in accordance with auditing standards for presentation of
"peer reviewed" scientific research which is generally accepted in the
United States. These standards generally require that we plan and perform
the audit to obtain resonable assurance about whether the peer reviewed
scientific presentation (article) is without material misstatement. An
audit includes examining, on a test basis, evidence supporting a formulated
hypothesis that has been properly tested in the work performed, assessing
the literature reviewed, the assumptions made, the sample procedures used,
the instrumentation used, and evaluating overall the fairness of the
presentation of "peer reviewed" scientific research. We believe the audit
completed herein provides a resonable basis for the opinion expressed above.
signed: the auditor
dated: on the date the field work was completed
At 11:54 AM 10/24/01 +0100, Gerhard Nebe-von-Caron wrote:
>Content-Type: TEXT/PLAIN; CHARSET=US-ASCII
>Content-Language: en-GBR
>
>A disclaimer to start with:
>This comment does not claim to be correct in any way. Please feel free to
enjoy
>all typo's, spelling errors, omissions and inconsistencies with previous
>comments. If I made them before my last coffee break I would have forgotten
>anyhow.
>
>
>I liked Ray's and Mario's approach to show a few examples. It shows the
>artistic freedom one can get on how to display the data. And they only
used one
>data set and two software packages and far from every possible iteration of
>scaling smoothing and colour. And they happily accepted Fluor:CD8 as an axis
>label. I would have thought it to be fluorescent if that is what Fluor
means. I
>tend to write for example CD8*FITC 525nm log which still doesn't give you
>bandwidth etc but tells you at least it is green fluorescence. I actually
love
>density plots (please don't call them colour dot plots as this tends to be
used
>for colour gated dot plots) over contours and actually most of all a 3-D
>display where I can see mountains or clouds. Unfortunately we can not yet
print
>the moving pictures described in Harry Potter but that is when the fun really
>starts. However, it can be done in on-line publications.
>
>The confusion created by our beloved contour or dot plots can be enormous.
Not
>for no reason one of Marc Abraham's AIR-logo's shows a dot plot under the
>stinker (http://www.improbable.com/bookstore/bookstore-top.html). Actually
>nicely shown in Mario's example of smoothed contour plots is how you can make
>cells vanish. Are these small populations between the single and double
>positive cells important? or perhaps a compensation error?... If one
wants to
>show those cells to make a point, one would not use the smoothed contour
plot.
>If one was to sort them, the contours would lead nowhere. From that point the
>old Ortho 50H software with it's grey level dot plots was far advanced at
it's
>time. With 16 grey levels it allowed a huge dynamic range for density plots
>(e.g. 1=2^0 to 32768=2^15 per channel) w/o compromising the resolution at
the
>bottom - unless you do not like the log-2 z-scale.
>
>Let's step back a bit
>
>Why do we publish? Normally to convince someone else about the importance and
>values of our work.
>Why would we add those funny little pictures? To convince the reader with a
>visual aid about our data, the population we have seen doing something, e.g.
>once you seen a cluster - you shall believe. They are descriptive
sketches and
>nothing else as they don't give is any statistics themselves. In particular
>with low event numbers it becomes very important to reflect on confidence
>intervals of your data as nicely presented by Terry Hoy on the Royal
>Microscopical Society Flow Course is Sheffield a couple of weeks ago.
However,
>the plots have something in common with statistic:
>
>Statistics are like a bikini - what they reveal is suggestive, but what they
>conceal is vital. (Aaron Levenstein)
>
>
>Now if we argue the use of plots for illustration only but the use of "solid
>numbers" to substantiate our work with "sound statistics" we have to keep in
>our mind that all this number crunching only characterises / validate the
>position of the dots on the screen, nothing else. We can only test the
numerics
>of our measurements and not the measurement itself. Apart from the
requirement
>for correct instrument set-up this is subject to scientific reasoning; about
>how the measurement was performed and what controls were done, if saturation
>was reached etc. in order to interpret the indirect measures we have
>undertaken. This includes detailed description of clones and
concentrations and
>control data and "unmassaged" pictures of raw data and gated data.
>
>The question remaining is how to validate the quality of the data and who
is to
>do it.
>For validation in house for example we display / print all fluorescence
>channels versus log side scatter ungated. For whole blood this helps as it
>gives you already a differentiation of lymph's, mono's and neutrophils (and
>eosinophils if for example using FACSlyse). Thus if you suddenly spot CD8
>positive neutrophils you know that there is trouble. For cultured cell lines
>log forward scatter vs. fluorescence seems to be better as in a lot of cases
>the cells with lower forward scatter are the ones on their way out. This type
>of display also indicates the presence of fluorescent aggregates that can
lead
>to problems by sticking to or coinciding with cells (how do we want to report
>coincidence factors?) and shows if the antigen expression is scatter / volume
>related or not. But this is all a matter of personal taste.
>
>I do not believe we can eradicate poor publications. On one side there is
>always human error. On the other side there is an incredible inflation of
>science and in particular the numbers of publications and journals,
inevitably
>leading to a decay in quality. We can only try to educate people in their
>ability to set up experiments and analyse data. Attempts like Mike's CD-ROM
>approach or Paul's virtual laboratory where you can even do the virtual
>pipetting of antibody are leading in the right direction. The latter could
even
>be spiked with pitfalls like changing the buffer batch in an experiment,
>picking one made up with water contaminated with red fluorescent algae.
>
>Now in a proper clinical setting, as I know from my brother, all the results
>have to be signed off and interpreted by him or a colleague as a specialist
>before they leave the lab for the physicians out in the University. Unless
you
>run your samples in a core facility with an experienced cytometrist, such
gate
>keepers may not exist. For political financial and logistic reasons a lot of
>people have bought their own instruments independent from an already existing
>expertise. The time and effort put into the proper planning and analysis of
>experiments by someone who knows his cytometry would cost (not money
>necessarily). If the gate keeper is to be the reviewer of an article then he
>should not only be skilled enough to understand the flow data but he also
would
>require some validation information even if not published as it is not
possible
>to judge complex work from one or two pictures. In that context I think that
>the suggested discussion board for post publishing review offered via the
>publisher to their subscribers speaks in favour of the publisher, as this
type
>of feedback does allow him to improve the journal quality.
>
>I am sure we can come up with a suggested list of information that should be
>included on cytometry experiments if applicable and make it easier not to
>forget the necessary controls and I would ask James Watson
>(http://www.cyto.purdue.edu/flowcyt/books/bookl.htm) to join the advisory
party
>that might be set up for that.
>But there will always be a lot of poor data published, for numerous reasons,
>and the only one to blame is the reader that believes information to be true
>because it was printed in a newspaper or a publication to be correct just
>because it was published in a scientific journal (see www.improbable.com for
>further enjoyment on that).
>
>Gerhard
>
>Content-Type: TEXT/x-cdsi-msrtf; CHARSET=US-ASCII
>Content-ID: 0
>Content-Language: en-GBR
>
>{\rtf1\ansi\ansicpg1252\deff0{\fonttbl{\f0\fscript\fcharset0 Comic Sans MS;}}
>{\colortbl ;\red0\green128\blue128;}
>\viewkind4\uc1\pard\cf1\lang1033\f0\fs20 A disclaimer to start with:\par
>This comment does not claim to be correct in any way. Please feel free to
enjoy all
>typo's, spelling errors, omissions and inconsistencies with previous
comments. If I
>made them before my last coffee break I would have forgotten anyhow.\par
>\par
>\par
>I liked Ray's and Mario's approach to show a few examples. It shows the
artistic
>freedom one can get on how to display the data. And they only used one
data set and two
>software packages and far from every possible iteration of scaling
smoothing and colour.
>And they happily accepted Fluor:CD8 as an axis label. I would have thought
it to be
>fluorescent if that is what Fluor means. I tend to write for example
CD8*FITC 525nm
>log which still doesn't give you bandwidth etc but tells you at least it
is green
>fluorescence. I actually love density plots (please don't call them colour
dot plots
>as this tends to be used for colour gated dot plots) over contours and
actually most
>of all a 3-D display where I can see mountains or clouds. Unfortunately we
can not yet
>print the moving pictures described in Harry Potter but that is when the
fun really
>starts. However, it can be done in on-line publications. \par
>\par
>The confusion created by our beloved contour or dot plots can be enormous.
Not
>for no reason one of Marc Abraham's AIR-logo's shows a dot plot under the
stinker
>(http://www.improbable.com/bookstore/bookstore-top.html). Actually nicely
shown in
>Mario's example of smoothed contour plots is how you can make cells
vanish. Are these
>small populations between the single and double positive cells important?
or perhaps
>a compensation error?... If one wants to show those cells to make a
point, one would
>not use the smoothed contour plot. If one was to sort them, the contours
would lead
>nowhere. From that point the old Ortho 50H software with it's grey level
dot plots
>was far advanced at it's time. With 16 grey levels it allowed a huge
dynamic range for
>density plots (e.g. 1=2^0 to 32768=2^15 per channel) w/o compromising the
resolution
>at the bottom - unless you do not like the log-2 z-scale.\par
>\par
>Let's step back a bit\par
>\par
>Why do we publish? Normally to convince someone else about the importance
and values
>of our work.\par
>Why would we add those funny little pictures? To convince the reader with
a visual
>aid about our data, the population we have seen doing something, e.g. once
you seen
>a cluster - you shall believe. They are descriptive sketches and nothing
else as
>they don't give is any statistics themselves. In particular with low event
numbers
>it becomes very important to reflect on confidence intervals of your data
as nicely
>presented by Terry Hoy on the Royal Microscopical Society Flow Course is
Sheffield a
>couple of weeks ago. However, the plots have something in common with
statistic:\par
>\par
>Statistics are like a bikini - what they reveal is suggestive, but what
they conceal
>is vital. (Aaron Levenstein)\par
>\par
>\par
>Now if we argue the use of plots for illustration only but the use of
"solid numbers"
>to substantiate our work with "sound statistics" we have to keep in our
mind that
>all this number crunching only characterises / validate the position of
the dots on
>the screen, nothing else. We can only test the numerics of our
measurements and not
>the measurement itself. Apart from the requirement for correct instrument
set-up
>this is subject to scientific reasoning; about how the measurement was
performed and
>what controls were done, if saturation was reached etc. in order to
interpret the
>indirect measures we have undertaken. This includes detailed description
of clones and
>concentrations and control data and "unmassaged" pictures of raw data and
gated data.\par
> \par
>The question remaining is how to validate the quality of the data and who
is to do
>it. \par
>For validation in house for example we display / print all fluorescence
channels
>versus log side scatter ungated. For whole blood this helps as it gives
you already
>a differentiation of lymph's, mono's and neutrophils (and eosinophils if
for example
>using FACSlyse). Thus if you suddenly spot CD8 positive neutrophils you
know that there
>is trouble. For cultured cell lines log forward scatter vs. fluorescence
seems to be
>better as in a lot of cases the cells with lower forward scatter are the
ones on their
>way out. This type of display also indicates the presence of fluorescent
aggregates
>that can lead to problems by sticking to or coinciding with cells (how do
we want to
>report coincidence factors?) and shows if the antigen expression is
scatter / volume
>related or not. But this is all a matter of personal taste.\par
>\par
>I do not believe we can eradicate poor publications. On one side there is
always human
>error. On the other side there is an incredible inflation of science and
in particular
>the numbers of publications and journals, inevitably leading to a decay in
quality.
>We can only try to educate people in their ability to set up experiments
and analyse
>data. Attempts like Mike's CD-ROM approach or Paul's virtual laboratory
where you can
>even do the virtual pipetting of antibody are leading in the right
direction. The latter
>could even be spiked with pitfalls like changing the buffer batch in an
experiment,
>picking one made up with water contaminated with red fluorescent algae.\par
>\par
>Now in a proper clinical setting, as I know from my brother, all the
results have to
>be signed off and interpreted by him or a colleague as a specialist before
they leave
>the lab for the physicians out in the University. Unless you run your
samples in a
>core facility with an experienced cytometrist, such gate keepers may not
exist. For
>political financial and logistic reasons a lot of people have bought their
own
>instruments independent from an already existing expertise. The time and
effort put
>into the proper planning and analysis of experiments by someone who knows
his cytometry
>would cost (not money necessarily). If the gate keeper is to be the
reviewer of an
>article then he should not only be skilled enough to understand the flow
data but
>he also would require some validation information even if not published as
it is not
>possible to judge complex work from one or two pictures. In that context I
think that
>the suggested discussion board for post publishing review offered via the
publisher
>to their subscribers speaks in favour of the publisher, as this type of
feedback does
>allow him to improve the journal quality.\par
>\par
>I am sure we can come up with a suggested list of information that
>should be included on cytometry experiments if applicable and make it
>easier not to forget the necessary controls and I would ask James Watson
>(http://www.cyto.purdue.edu/flowcyt/books/bookl.htm) to join the advisory
party that
>might be set up for that. \par
>But there will always be a lot of poor data published, for numerous
reasons, and the
>only one to blame is the reader that believes information to be true
because it was
>printed in a newspaper or a publication to be correct just because it was
published
>in a scientific journal (see www.improbable.com for further enjoyment on
that). \par
>\par
>Gerhard\par
>\par
>\par
>}
>
Computer Aided Cell and Molecular Biology (CACMB), not medicine, will find
the cure for cancer and other diseases. There will always be a need for
the trained clinician (MD/RN) but, advanced diagnostic and treatment option
selection has become gene based, has moved from the physician's practice to
the computerized cell and molecular biology laboratory, and appropriate
treatment options should now be based on the personal biology of the
patient.
This archive was generated by hypermail 2b29 : Sun Jan 05 2003 - 19:01:36 EST