Exploit Interactive Issue 5: The Invisible Hand of Peer Review
COLLECTED BY
Organization:
Alexa Crawls
Starting in 1996,
Alexa Internet
has been donating their crawl data to the Internet Archive. Flowing in every day, these data are added to the
Wayback Machine
after an embargo period.
this data is currently not publicly accessible.
The Wayback Machine - https://web.archive.org/web/20090710045102/http://www.exploit-lib.org:80/issue5/peer-review/
The Invisible Hand of Peer Review
The refereed journal literature needs to be freed from both
paper and its costs, but not from peer review, whose "invisible hand" is what
maintains its quality. The residual cost of online-only peer review is low enough
to be recovered from author-institution-end page charges, covered from institutional
subscription savings, thereby vouchsafing a toll-free literature for everyone,
everywhere, forever.
Introduction
Human nature being what it is, it cannot be altogether relied upon to police
itself. Individual exceptions there may be, but to treat them as the rule
would be to underestimate the degree to which our potential unruliness
is vetted by collective constraints, implemented formally.
So it is in civic matters, and it is no different in the world of Learned
Inquiry. The "
quis custodiet
" problem among scholars has traditionally
been solved by means of a quality-control and certification (QC/C) system
called "peer review"
[1]
: The work of specialists is submitted
to a qualified adjudicator, an editor, who in turn sends it to fellow-specialists,
referees, to seek their advice about whether the paper is potentially publishable,
and if so, what further work is required to make it acceptable. The paper
is not published until and unless the requisite revision can be and is
done to the satisfaction of the editor and referees.
Pitfalls of peer policing
Neither the editor nor the referees is infallible. Editors can err in the
choice of specialists (indeed, it is well-known among editors that a
deliberate bad choice of referees can always ensure that a paper is either
accepted or rejected, as preferred); or editors can misinterpret or misapply
referees' advice. The referees themselves can fail to be sufficiently
expert, informed, conscientious or fair.
Nor are authors always conscientious in accepting the dictates of peer
review. (It is likewise well-known among editors that virtually every paper
is eventually published, somewhere
[2]
[3]
:
There is a quality hierarchy among journals, based on the rigour of their peer
review, all the way down to an unrefereed vanity press at the bottom.
Persistent authors can work their way down until their paper finds its
own level, not without considerable wasting of time and resources along the way,
including the editorial office budgets of the journals and the freely given time
of the referees, who might find themselves called upon more than once to
review the same paper, sometimes unchanged, for several different journals.)
The system is not perfect, but it is what has vouchsafed us our refereed
journal literature to date, such as it is, and so far no one has demonstrated
any viable alternative to having experts judge the work of their peers,
let alone one that is at least as effective in maintaining the quality
of the literature as the present imperfect one is
[4]
.
Self Policing?
Alternatives have of course been proposed, but to propose is not to demonstrate
viability. Most proposals have envisioned weakening the constraints of classical
peer review in one way or other. The most radical way being to do away with it
altogether: Let authors police themselves; let every submission be published, and
let the reader decide what is to be taken seriously. This would amount to discarding
the current hierarchical filter -- both its active influence, in directing revision,
and its ranking of quality and reliability to guide the reader trying to navigate
the ever-swelling literature
[5]
.
There is a way to test our intuitions about the merits of this sort of proposal
a priori, using a specialist domain that is somewhat more urgent and immediate than
abstract "learned inquiry"; if we are not prepared to generalise this intuitive
test's verdict to scholarly/scientific research in general, we really need to ask
ourselves how seriously we take the acquisition of knowledge: If someone near and
dear to you were ill with a serious but potentially treatable disease, would you
prefer to have them treated on the basis of the refereed medical literature or on
the basis of an unfiltered free-for-all where the distinction between reliable
expertise and ignorance, incompetence or charlatanism is left entirely to the
reader, on a paper by paper basis?
A variant on this scenario is currently being tested by the British Medical
Journal
[6]
, but instead of entrusting entirely to the reader
the quality control function performed by the referee in classical peer review,
this variant, taking a cue from some of the developments and goings-on on both the
Internet and Network TV chat-shows, plans to publicly post submitted papers
unrefereed on the Web and to invite any reader to submit a commentary; these
commentaries will then be used in lieu of referee reports as a basis for deciding
on formal publication.
Expert Opinion or Opinion Poll?
Is this peer review? Well, it is not clear whether the self-appointed
commentators will be qualified specialists (or how that is to be ascertained).
The expert population in any given speciality is a scarce resource, already
overharvested by classical peer review, so one wonders who would have the
time or inclination to add journeyman commentary services to this load on their own
initiative, particularly once it is no longer a rare novelty, and the entire raw,
unpoliced literature is routinely appearing in this form first. Are those who have
nothing more pressing to do with their time than this really the ones we want to
trust to perform such a critical QC/C function for us all?
And is the remedy for the possibility of bias or incompetence in
referee-selection on the part of editors really to throw selectivity to the winds,
and let referees pick themselves? Considering all that hangs on being published
in refereed journals, it does not take much imagination to think of ways
authors could manipulate such a public-polling system to their own advantage,
human nature being what it is.
Peer Commentary vs. Peer Review
And is peer commentary (even if we can settle the vexed "peer" question)
really peer review? Will I say publicly about someone who might be refereeing
my next grant application or tenure review what I really think are the
flaws of his latest raw manuscript? (Should we then be publishing our names
alongside our votes in civic elections too, without fear or favour?) Will
I put into a public commentary -- alongside who knows how many other such
commentaries, to be put to who knows what use by who knows whom -- the
time and effort that I would put into a referee report for an editor I
know to be turning specifically to me and a few other specialists for our
expertise on a specific paper?
If there is anyone on this planet who is in a position to attest to the
functional difference between peer review and peer commentary
[4]
[7]
, it is surely the author of
the present article, who has been umpiring a peer-reviewed paper journal of Open
Peer Commentary,
Behavioral and Brain Sciences
[BBS]
[8]
published by Cambridge University Press
[9]
for over two
decades
[10]
, as well as a brave new online-only journal
of Open Peer Commentary, likewise peer-reviewed
(
Psycoloquy
[11]
, sponsored by the American Psychological
Association
[12]
), which entered its second decade with the
millennium.
Both journals are rigorously refereed; only those papers that have successfully
passed through the peer review filter go on to run the gauntlet of open peer
commentary, an extremely powerful and important
supplement
to peer
review, but certainly no
substitute
for it. Indeed, no one but the
editor sees [or should have to see] the population of raw, unrefereed submissions,
consisting of some manuscripts that are eventually destined to be revised
and accepted after peer review, but also (with a journal like BBS, having
a 75% rejection rate) many manuscripts not destined to appear in that particular
journal at all. Referee reports, some written for my eyes only, all written
for at most the author and fellow referees, are nothing like public commentaries
for the eyes of the entire learned community, and vice versa. Nor do 75% of the
submissions justify soliciting public commentary, or at least not commentary at the
BBS level of the hierarchy.
It has been suggested that in fields such as Physics, where the rejection
rate is lower (perhaps in part because the authors are more disciplined
and realistic in their initial choice of target journal, rather than trying
their luck from the top down), the difference between the unrefereed preprint
literature and the refereed reprint literature may not be that great; hence
one is fairly safe using the unrefereed drafts, and perhaps the refereeing
could be jettisoned altogether.
Successful Test-Site in Los Alamos
Support for this possibility has been adduced from the remarkable success of the
NSF/DOE-supported Los Alamos Physics Archive
[13]
, a free,
public repository for a growing proportion of the current physics literature, with
over 25,000 new papers annually and 35,000 users daily. Most papers are initially
deposited as unrefereed preprints, and for some (no one knows how many), their
authors never bother replacing them with the final revised draft that is accepted
for publication
[5]
and
[14]
.
Yet Los Alamos is actively used and cited by the physics community
[15]
[16]
[17]
.
Is this really evidence that peer review is not indispensable after all?
Hardly, for the "Invisible Hand" of peer review is still there, exerting
its civilising influence: Just about every paper deposited in Los Alamos
is also destined for a peer reviewed journal; the author knows it will
be answerable to the editors and referees. That certainly constrains how
it is written in the first place. Remove that invisible constraint -- let
the authors be answerable to no one but the general users of the Archive
(or even its self-appointed "commentators") -- and watch human nature take
its natural course, standards eroding as the Archive devolves toward the
canonical state of unconstrained postings: the free-for-all chat-groups
of Usenet
[18]
, that Global Graffiti Board for Trivial Pursuit
-- until someone re-invents peer review and quality control.
A subversive proposal
Now it is no secret that I am a strong advocate of a free literature along the
lines of Los Alamos
[19]
. How are we to reconcile the
conservative things I've said about QC/C here with the radical things I've
advocated elsewhere about public author archiving
[20]
[21]
[22]
?
The answer is very simple. The current cost of the refereed paper journal
literature is paid for by Subscription, Site License and Pay-Per-View (S/L/P).
Both the medium (paper) and the method of cost-recovery (S/L/P) share the
feature that they block access to the refereed literature, whereas the authors,
who contribute their papers for free, would infinitely prefer free, universal
access to their work.
The optimal (and inevitable) solution is an online-only refereed journal
literature, which will be much less costly (less than 1/3 of the current price per
page) once it is paper-free
[23]
and resides in
open archives
[24]
but still not entirely cost-free, because
the peer review (and editing) still needs to be paid for
[25]
.
If those residual QC/C costs are paid at the author-institution-end (not out of the
author's pocket, of course, but out of institutional publication funds redirected
from 1/3 of the 3/3 annual institutional savings from serial S/L/P cancellations),
the dividend will be that the papers are all accessible for free for all (via
interoperable open archives such as CogPrints
[26]
-- integrated seamlessly into a single global "virtual" archive,
mirrored worldwide, which will then have an unrefereed preprint sector
and a refereed, published, reprint sector, tagged by journal name). Journal
publishers will continue to provide and be paid for their QC/C while the
public archive will serve as the "front end" for both journal submissions
(tagged "unrefereed prepints") and published articles (tagged "refereed
reprints [plus journal name, etc.]"
[27]
.
Streamlining peer review for the airwaves
Peer review is medium-independent, but the online-only medium will make
it possible for journals to implement it not only more cheaply and efficiently,
but also more equitably and effectively than was possible in paper, through
subtle variants of the very means I have criticised above
[28]
and
[29]
: Papers will be submitted in electronic form, and
archived on the Web (in hidden referee-only sites, or publicly, in open-archive
preprint sectors, depending on the author's preferences). Referees need no longer
be mailed hard copies; they will access the submissions from
the Web
[30]
.
To distribute the load among referees more equitably (and perhaps also
to protect editors from themselves), the journal editor can formally approach
a much larger population of selected, qualified experts about relevant papers they
are invited to referee if they have the time and inclination. Referee reports can
be emailed or deposited directly through a password-controlled Web interface.
Accepted final drafts can be edited and marked up online, and the final draft can
then be deposited in the public Archive for all, superseding the preprint.
Galactic hitch-hiking, PostGutenberg
Referee reports can be revised, published and linked to the published
article as commentaries if the referee wishes; so can author rebuttals.
And further commentaries, both refereed and unrefereed, can be archived
and linked to the published article, along with author responses. Nor is
there any reason to rule out postpublication author updates and revisions
of the original article -- 2nd and 3rd editions, both unrefereed and refereed.
Learned Inquiry, as I have had occasion to write before
[31]
is a continuum; reports of its findings -- informal and formal, unrefereed and
refereed -- are milestones, not gravestones; as such, they need only be reliably
sign-posted. The discerning hitch-hiker in the PostGutenberg Galaxy can take care
of the rest
[32]
.
Overall, the dissemination of learned research, once we have attained the
optimal and inevitable state described here, will be substantially accelerated,
universally accessible, and incomparably more interactive in the age of Scholarly
Skywriting than it was in our own pedestrian, papyrocentric one; Learned Inquiry
itself -- and hence all of society -- will be the chief beneficiary.
References
- Rational disagreement in peer review
,
Harnad, S., Science, Technology and Human Values 10: 55 - 62
- A difficult balance : editorial peer review in
medicine
, London : Nuffield Provincial Hospitals Trust.
- Policing the Paper Chase. (Review of S. Lock, A difficult
balance: Peer review in biomedical publication.)
, Nature 322: 24 - 5, 1986
- Peer commentary on peer review: A case study in scientific
quality control
, New York: Cambridge University Press, 1982, Harnad, S. (ed.)
- Developing services for open eprint archives: globalisation,
integration and the impact of links
, Proceedings of the 5th ACM Conference
on Digital Libraries. San Antonio Texas June 2000,
Hitchcock, S. Carr, L., Jiao, Z., Bergmark, D., Hall, W., Lagoze, C. & Harnad, S. (2000)
URL: <
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad00.acm.htm
>
- British Medical Journal
, web site
URL: <
http://www.bmj.com/cgi/shtml/misc/peer/index.shtml
>
- Commentaries, opinions and the growth of scientific
knowledge
, American Psychologist 39: 1497 - 1498, (1984) Harnad, S.
- Behavioral and Brain Sciences (BBS)
URL: <
http://www.cogsci.soton.ac.uk/bbs/
>
- Cambridge University Press
URL: <
http://www.journals.cup.org/
>
- Creative disagreement
,
The Sciences 19: 18 - 20, Harnad, S. (1979)
- Psyoloquy
URL: <
http://www.cogsci.soton.ac.uk/cgi/psyc/newpsy
>
- American Psychological Association
URL: <
http://www.apa.org/
>
- Los Alamos Physics Archive
URL: <
http://xxx.lanl.gov/
>
- A usage based analysis of CoRR [A commentary on "CoRR: a
Computing Research Repository" by Joseph Y. Halpern]
,
ACM SIGDOC Journal of Computer Documentation (May 2000),
Carr, L., Hitchcock, S., Hall, W. & Harnad, S.
URL: <
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad00.halpern.htm
>
- First Steps Towards Electronic Research Communication
,
Computers in Physics. (August, American Institute of Physics) 8(4): 390-396,
Ginsparg, P
URL: <
http://xxx.lanl.gov/blurb/
>
- Winners and Losers in the Global research Village
,
Invited contribution, UNESCO Conference HQ, Paris, 19-23 Feb 1996, Ginsparg, P.
URL: <
http://xxx.lanl.gov/blurb/pg96unesco.html
>
- Citation patterns to traditional and electronic preprints
in the published literature
, College and Research Libraries 59(5):
448-456, Youngen G.K. (1998)
URL: <
http://www.physics.uiuc.edu/library/preprint.html
>
- The Complete Reference to Usenet Newsgroups
,
TILE.NET/NEWS
URL: <
http://tile.net/news/listed.html
>
- Scholarly Journals at the Crossroads; A Subversive Proposal
for Electronic Publishing
,
Washington, DC., Association of Research Libraries, June 1995,
Okerson A. & O'Donnell, J. (Eds.)
URL: <
http://www.arl.org/scomm/subversive/
>
- For Whom the Gate Tolls? Free the Online-Only Refereed Literature
,
American Scientist Forum, Harnad, S.
URL: <
http://www.cogsci.soton.ac.uk/~harnad/amlet.html
>
- On-Line Journals and Financial Fire-Walls
,
Nature 395(6698): 127-128,(1998) Harnad, S.
URL: <
http://www.cogsci.soton.ac.uk/~harnad/nature.html
>
- Free at Last: The Future of Peer-Reviewed Journals
,
D-Lib Magazine 5(12) December 1999, Harnad, S.
URL: <
http://www.dlib.org/dlib/december99/12harnad.html
>
- Archives of SEPTEMBER98-FORUM@LISTSERVER.SIGMAXI
,
September 1998 American Scientist Forum
URL: <
http://amsci-forum.amsci.org/archives/september98-forum.html
>
- The Open Archives initiative
URL: <
http://www.openarchives.org/
>
- The economics of electronic journals
,
In: Ekman R. and Quandt, R. (Eds) Technology and Scholarly Communication Univ. Calif.
Press, (1998) Odlyzko, A.M.
URL: <
http://www.research.att.com/~amo/doc/economics.journals.txt
>
- CogPrints
URL: <
http://cogprints.soton.ac.uk/
>
- EPrints.org
URL: <
http://www.eprints.org/
>
- Implementing Peer Review on the Net: Scientific Quality
Control in Scholarly Electronic Journals
, In: Peek, R. & Newby, G.
(Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT
Press. Pp. 103-118. (1996) Harnad, S.
URL: <
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
>
- Learned Inquiry and the Net: The Role of Peer Review,
Peer Commentary and Copyright
, Learned Publishing 11(4) 283-292. (1997)
URL: <
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad98.toronto.learnedpub.html
>
A short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared
in the University of Toronto Bulletin: 51(6) P. 12.
URL: <
http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html
>
- Using the web for peer review and publication
of scientific journals
URL: <
http://www.consecol.org/Journal/consortium.html
>
- Scholarly Skywriting and the Prepublication Continuum of
Scientific Inquiry
, Psychological Science 1: 342 - 343 (1990),
Harnad, S.
URL: <
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad90.skywriting.html
>
- Post-Gutenberg Galaxy: The Fourth Revolution in the Means
of Production of Knowledge
, Public-Access Computer Systems Review 2 (1):
39 - 53 (1991), Harnad, S.
URL: <
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad91.postgutenberg.html
>
Author Details
Stevan Harnad
Intelligence, Agents, Multimedia
Electronics and Computer Science Department
Southampton University
Highfield
Southampton
SO17 1BJ
United Kingdom
Email:
<
harnad@soton.ac.uk
> or
<
harnad@princeton.edu
>
URL:
<
http://www.princeton.edu/~harnad/intpub.html
>
<
http://cogsci.soton.ac.uk/~harnad/intpub.html
>
For citation purposes
:
Stevan Harnad, "The Invisible Hand of Peer Review",
Exploit Interactive, issue 5, April 2000
URL: <http://www.exploit-lib.org/issue5/peer-review/>
[HTML Validation]
-
[Accessibility check]