Consensus Journals: Invitational journals based upon peer consensus
David S. Stodolsky
dsstodol at daimi.aau.dk
Tue Nov 13 12:26:09 EST 1990
Invitational journals based upon peer consensus
David S. Stodolsky
Roskilde University Centre
david at ruc.dk
Computer networks open new possibilities for
scientific communication in terms of
quality, efficiency, and rapidity. Consensus
journals have the economy of invitational
journals and the objectivity of journals
based upon the peer review. That is, all
articles are published and the reader
benefits from article selection based upon
impartial refereeing. An additional benefit
of consensus journals is that the
negotiation process, that typically occurs
prior to publication, is automated, thus
saving efforts of participants.
Readers submit reviews that evaluate
articles on agreed dimensions. A statistical
procedure is used to identify the most
knowledgeable representative of each
consensus position and these persons are
invited to submit articles that justify the
review judgments they have submitted. A
major advantage of this approach is the
ability to develop reputation without
The approach includes a protection mechanism
based upon pseudonyms, that substitutes for
the protection of anonymity typical with
scientific journals. This reduces the
potential for irresponsible behavior and
facilitates reputation development. The
level of quality enhancement is superior to
that achievable with anonymous peer review.
Eliminating the editor and the delay
associated with conventional refereeing
makes message quality enhancement available
in message systems for educational and
This document has been prepared for electronic publication.
Underscore characters indicate the start and end of italicized
character sequences. Figures and tables assume a monospace font.
Citation: Stodolsky, D. S. (1990). Consensus Journals: Invitational
journals based upon peer consensus. _Datalogiske Skrifter_ (Writings
on Computer Science). No. 29 / 1990. Roskilde University Centre,
Institute of Geography, Socioeconomic Analysis, and Computer
Science. (ISSN 0109-9779-29)
Invitational journals can be distinguished from typical scholarly
journals by the sequence of events that results in publication of an
article. The sequence of events with a typical journal starts with
the writing of an article. The article is then transmitted to an
editor and refereed. After a successful review, often contingent
upon negotiated revisions, the article is published and read. With
invitational journals, however, events are reversed. The tentative
decision to publish an author is made first, often based upon the
reading of previous work by that author. Then negotiation between
the editor and author occurs, or there is informal refereeing of a
proposal, which if successful, results in the writing of an article.
The great advantage of this second sequence -- read, negotiate,
write -- is that almost every article written gets published. The
disadvantage is that selection of authors is somewhat arbitrary and
there is no way an unknown author can get published. The objective
of this article is to outline a method of scientific communication
that has the economy of invitational journals and the objectivity of
journals based upon the peer review. These self-edited journals will
be called _consensus journals_ in order to distinguish them from
conventional invitational journals.
Any reader of an article in a consensus journal can act as a
referee. Assume, for simplicity, that referees send reviews to a
mediator. At a deadline, the mediator performs calculations and
issues invitations to the referees who have been selected as new
authors (Figure 1). These calculations are implicit negotiations,
that is, they predict which persons would have been selected to
respond to the reviewed article if referees has actually negotiated
and reached a consensus. One benefit of consensus journals is that
the negotiation process is automated, thus saving participant
------ R: Review ------- M: Invitation -------
| Read |----------->| Calc. |--------------->| Write |----------->
------ ------- ------- R: Article
| R: Renege
R = Referee Calc. = Calculate consensus |
M = Mediator V
Figure 1. Simplified cycle of operation for a consensus journal
The simplified cycle of operation for a consensus journal shows
actions in boxes and messages as arrows. In this simplified cycle,
referees invited to publish (and justify the reviews they have
submitted) have a choice of submitting their article by a deadline
or reneging on the promise implied by their review. This simplified
cycle of operation assumes, additionally, that consensus positions
can be calculated and that published articles are retained
indefinitely. Eliminating these assumptions requires a more
articulated cycle of operation and additional message types.
Before considering a more articulated cycle of operation, however,
it is necessary to note an important feature of peer review that
contributes to impartial judgment. This feature is a protection
mechanism, typically anonymity, that shields referees from pressures
that might be associated with evaluation of a colleague. Further,
names and affiliations of authors are often hidden from referees to
ensure that only article content is the basis for evaluation.
Protection can be alternatively be provided by a pseudonym system.
This has the advantage of reducing opportunities for irresponsible
behavior as compared to systems based upon anonymity (Stodolsky,
1990). It has a further advantage of permitting reputation
development through the refereeing alone, thereby making it possible
to establish a reputation without contributing articles.
When there are multiple referees, it is important that their
judgments are independent, so referee reports must not be made
available until all have been submitted. This last requirement can
be met by ensuring that reviews transmitted to the mediator are
hidden until the deadline. The dynamics and implementation of
protection systems are beyond the scope of this article, so only the
necessity for the simultaneous release of information is addressed
Definition of message types
While in the simplest case, messages in the consensus journal
environment consist of only articles and reviews (Stodolsky, 1990),
considerations of effective negotiation and of storage management
suggest defining additional message types. There are five types of
messages transmitted in the consensus journal environment.
Articles, while shorter than those usually seen in conventional
journals, will most often play the same role. However, it is quite
possible to have an article in a consensus journal that is only a
few lines long, and that can only be understood in connection with
the review message it follows and its target article.
Review messages must be distinguished from conventional reviews
because they are characterized by a vector of numbers that summarize
a reader's reaction to an article. If we think of articles as nodes
in a graph or pages in a hypertext network, then review messages are
the labels on arcs or links that connect the articles. Reviews can
go beyond merely evaluating an article, by offering to provide new
information that may be essential to support the target article's
position. Review messages also serve as a commitment to deliver a
justification of the reader's judgment, if invited.
Invitations are public, and therefore, impossible to refuse without
some loss of reputation. This makes them somewhat different than
invitations from an editor of a journal. In effect, the invitation
says, "We offer you storage space for an article." Also, a person
may post an invitation for themselves during the negotiation stage
of review, if they feel confident they can support the position
claimed in their review message.
It is possible for an author to cancel an article, thereby releasing
the associated storage space. The article then goes off-line (i.e.,
"out of print") along with its reviews and the articles that were
dependent upon it for their place in storage. This would typically
occur during explicit negotiation after the author had seen the
article's reviews. It could, however, occur much later, when a new
criticism was delivered.
Finally, during explicit negotiations, a review may be withdrawn.
This eliminates the referee from those prepared to respond to an
Cycle of operation
The sequence of events with a consensus journal is the same as with
an invitational journal. The review method, however, involves the
entire readership, or at least those who offer a judgment. New
authors are then selected based upon the review judgements. While
most articles will follow from reviews and be connected to their
target articles, independent articles are also permitted. However,
articles posted without consensus based invitations are less likely
to be read and can not be assumed to have support of other referees.
If we assume that a consensus journal is already functioning, we can
follow the events through a cycle of operation that starts with
reading of an article. While it is not essential for smaller
readerships, we assume that participants exchange information
All readers are presented with a target article at the same time. A
reader offers a review judgment in order to be considered for future
authorship. The review message must be received before a certain
deadline, say one week later. The review message consists of scores
along several preselected dimensions. For instance, a scientific
article is expected to be relevant, correct, and original. A more
conversational approach might include the dimensions completeness,
clarity, and appropriateness.
At the deadline, the mediator runs a statistical procedure to
determine if there are consensus positions among the referees. The
most central referee from each of these positions is invited to
submit a new article. These most central referees are also
considered most knowledgeable, within the framework of cultural
consensus theory. D'Andrade (1987) discusses the evidence supporting
Cultural consensus theory is based on the assumptions of common
truth (i. e., there is a fixed answer pattern "applicable" to all
referees), of local independence (i.e., the referee-dimension
response variables satisfy conditional independence), and of
homogeneity of items (i.e., each respondent has a fixed "cultural
competence" over all dimensions) (Romney, Weller & Batchelder,
1986). Results can be obtained with as few as three respondents, but
four are required if the significance of the results are to be
calculated (i. e., a degree of freedom is then available in the
statistical model) (Batchelder & Romney, 1988). A recent development
in the model is the ability to identify two consensual groupings
within the population of respondents (Romney, Weller & Batchelder,
1987) This is extremely helpful since it permits a minority to
publicize their viewpoint under the same conditions as a majority.
Cultural consensus theory assumes that we have no _a priori_
knowledge about referees, that is, they have no reputations. This is
extremely valuable when a new topic comes up or when there are
violation of assumptions required for calculations concerning a
current article based upon previous information (Stodolsky, 1984b).
Given that reputations have developed and assumptions are satisfied,
however, the theory requires elaboration to be applied most
effectively. Cultural consensus theory provides, in effect, a cross
sectional estimation of competence. That is, given a sample of
responses at a given moment, relative competence is estimated. On
the other hand, given a performance history, Bayesian estimation can
be used to assess the relative importance of different persons'
judgments. That is, there are reputations that give information
about relative competence independent of the current responses. This
assumes stationarity, that is, that the same area of competence is
required for correct response, and that responses are generated in
the same manner (e. g., respondents continue to give honest
answers). Both methods are based upon likelihood estimation,
therefore, a combined theory should be achievable. The combined
sources of information would likely make achieving an implicit
consensus more frequent.
The mediator issues an invitation report showing submitted
judgments, the degree of consensus achieved, the number of consensus
positions identified, degree of knowledge of each referee, and so
on. If consensus has been reached, invited referees are expected to
Negotiation must proceed explicitly if no consensus can be
identified (Figure 2). In that case, referees may look at the
judgments submitted and decide if their positions have sufficient
support. If not, they could reconsider their review judgments, and
either revise them or withdraw from the review process. The author
of an article might, on the basis of these judgments, cancel an
article, thus avoiding potential reputation damaging criticism.
------ R: Review ------- M: Invitation -------
| Read |----------->| Calc. |--------------->| Write |----------->
------ ------- ------- R: Article
^ | |
R: Review | | M: No consensus | R: Renege
| | |
| V V
----------- R: Withdrawal
| A: Cancelation
A = Author
M = Mediator Calc. = Calculate consensus
R = Referee
Figure 2. Cycle of operation for a consensus journal
(A referee becomes an author only after a submitted article has been
published by the mediator [not shown in figure]).
Assuming that the article was not cancelled, the combined effects of
withdrawal by referees with most deviant judgments and
reconsideration by others would likely lead to consensus,
particularly if the requirements for consensus were successively
relaxed. This assumes that revision of judgments would be in the
direction of dominant view points, a common finding. The result
would be an invitation issued by the mediator to selected referees.
Subsequent submission of an articles by selected referees and their
publication by the mediator would complete the cycle of operation.
The invitation report can guide negotiation when a consensus can not
be identified. Individual invitation staging could proceed along
with a relaxation of requirements for consensus. For instance, if
the first round of reviews did not generate a consensus, referees
could issue invitations to others (Table 1). If the second round of
reviews did not generate a consensus, referees could issue self
invitations (these would be acceptances for those who had received
invitations), or perhaps, direct the invitations they had already
received to others. Failure on the third round would permit these
previously issued invitations to serve as a coordination mechanism.
That is, certain referees would have indicated a readiness to
respond and others would have rejected the option of authorship
unsupported by a consensus. Thus, duplication of effort could be
avoided by examining the ranking of persons in terms of the
invitations received and accepted, and responding accordingly.
Message Level and (Receiver):
Stage of Process:
Reconsideration Review(2), Withdraw, Cancel,
Reconsideration Review(3), Withdraw, Cancel,
Table 1. Message level, receiver, and type, by stage of processing
(assumes no consensus reached during negotiation)
New articles are requested either by the mediator or by referees, if
an author and referees follow negotiations to completion. New
articles must be submitted before a deadline. At the deadline, the
new articles received are published. This makes them available to
the readership and completes the cycle of operation for a consensus
Thus in the simplest case, articles are read, reviews are
transmitted, invitations are issued, and new articles are submitted
in a timely manner. In the extended cycle, at least a single
reconsideration or negotiation stage occurs during which a target
article can be cancelled by its author and during which referees can
withdraw. A failure to achieve consensus leads to explicit
negotiation and options such as nonconsensus invitation. The
extended negotiation option makes the consensus journal more similar
to a conventional journal, because there is explicit negotiation
prior to the writing of an article.
The quality of a consensus journal can be assessed by the degree of
consensus achieved. Readers might select only those articles
resulting from a consensus-based invitation, thereby controlling the
quality of articles they see.
Rules of dialogue
The rules of operation of a consensus journal can be thought of as
specifying an action system, or language game, where the actions
relate to the placement of articles in a network of interconnected
nodes. Participants in the game try to maximize their influence.
Reputation is a crucial resource in scientific argumentation
(Smolensky, Fox, King, Lewis, 1988). Participants are expected to
maximize this resource. While there may be other payoffs available
within a given system, such as royalty payments, this discussion
assumes only reputation maximization as an individual objective.
There are several opportunities for reputation enhancement in the
cycle of operation. Selection as an author is a major opportunity
for reputation enhancement. However, refereeing also offers
significant opportunities that are not available with conventional
journals. Referees can commit themselves to delivering a rebuttal to
an article and thereby improve their reputation (assuming they make
good on their commitment given an opportunity). If an author
examines the reviews an article receives and decides to cancel it
before a rebuttal is written, the referees offering rebuttals would
have their reputations enhanced, without any further risk or effort.
With a consensus journal, the review message can be thought of as an
offer to deliver a certain type of article before the deadline.
Obviously, a review message that claims a target article is
erroneous, and thereby offers to deliver a rebuttal, plays a
different structural role in a debate than one that criticizes an
article for not being original. Thus, reviews can have a great deal
of structural impact and can express a level of commitment, which
would not be relevant in an environment that limits referees to a
With electronically published documents, it is very desirable to
structure interconnections so that retrieval is facilitated and the
relevance of statements becomes clear (Smolensky, Fox, King, Lewis,
1988). Thus, review messages can deal not only with the quality of
an article, but also its relationship to its target article.
Explicit relationships among articles becomes more necessary as the
size of articles decrease and number of articles increases.
With conventional journals, reviews are used to determine whether or
not an article should be published. The publication decision is not
dominant with electronic media, however, since distribution
constraints are greatly relaxed (Quarterman, 1990, p. 259;
Stodolsky, in press). Because of this, the period during which an
article remains on-line assumes importance, because storage is
limited. It is in this connection that the reviews of articles and
the relations between articles becomes critical. In the simplest
case, an article that is found incorrect by an overwhelming
consensus is cancelled by its author. Failure to cancel an the
article results in a continuing devaluation of the author's
reputation as more and more readers come to agree with the majority.
In the case of conflicting consensus positions, a rebuttal claiming
that a target article is flawed is explicitly linked to the target.
Failure to rebut that claim in turn has much the same effect as an
overwhelming consensus that the target article is incorrect. Most
interactions, therefore, take place at the knowledge frontier, as
various positions are argued. These interactions generate very
"bushy" argument trees, that require sophisticated navigation
strategies, if large amounts of effort are not to be expended
unnecessarily (Stodolsky, 1984a). The trees are thinned in the
process of argumentation. Positions that are sustained remain on-
line until they are thoroughly integrated into summaries or
A central mediator has been assumed in this description to simplify
explanation. There is no reason why the calculations necessary to
select new authors could not be performed decentrally. In fact, this
would be necessary if readers preferred different methods of
calculation for author selection. Then coordination in the selection
of new authors would be shifted from consensus calculation to
collection of invitations. Various types of voting rules could be
applied. Authors receiving the most invitations would then be
expected to submit articles. Thus, decentralization leads to an
integration of the two types of invitations (consensus and
individual) already discussed.
The task of protecting review judgments until the deadline is
reached is another required function. It is necessary, for example,
because analysis of earlier submitted judgments could permit a
referee submitting at the last moment to simulate a competence that
did not exist, thus violating assumptions of the model. Protection
can, however, be achieved decentrally using cryptography, assuming a
"beacon" that emits enciphering and deciphering keys at fixed
intervals (Rabin, 1983). Use of cryptography would be necessary, in
any case, to ensure the authenticity of messages.
A consensus journal requires mechanisms for both coordination and
protection. In the simplest case, a mediator can provide these. This
assumes protected channels of communication and a trusted mediator.
Coordination is necessary to identify consensus positions and avoid
duplication of effort. Protection of reviews is necessary to ensure
that assumptions of models for evaluating expertise are not
violated. This protection allows valid reputation development by
both authors and referees. Such reputations can then be used to
ensure effective allocation of expertise. The extension of review
opportunities to the entire readership vastly extends the available
field of expertise. This, combined with the effective allocation of
expertise and coordination that eliminates duplication of effort,
provide consensus journals with a significant advantage over current
mechanisms for enhancement of message quality.
Batchelder, W. H. & Romney, A. K. (1988). Test theory without an
answer key. _Psychometrika_, _53_(1), 71-92.
D'Andrade, R. G. (1987). Modal response and cultural expertise.
_American Behavioral Scientist_, _31_(2), 194-202.
Quarterman, J. S. (1990). _The matrix: Computer networks and
conferencing systems worldwide_. Bedford, MA: Digital Press.
Rabin, M. (1983). Transaction protection by beacons. _Journal of
Computer and Systems Science_, _27_(2), 256-267.
Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1986). Culture
as consensus: A theory of culture and informant accuracy. _American
Anthropologist_, _88_(2), 313-338.
Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1987). Recent
applications of cultural consensus Theory. _American Behavioral
Scientist_, _31_(2), 163-177.
Smolensky, P., Fox, B., King, R., & Lewis, C. (1988). Computer-aided
reasoned discourse or, how to argue with a computer. In R. Guindon
(Ed.), _Cognitive science and its applications for human-computer
interaction_. Hillsdale, NJ: Lawrence Erlbaum Associates.
Stodolsky, D. (1984a). Commonalities amoung [sic] conferencing
systems and their implication for marketing strategy.
_Organisatoriske Fragmenter 1984_, _12_, 43-58.
Stodolsky, D. (1984b, December). _Self-management of criticism in
dialog: Dynamic regulation through automatic mediation_. Paper
presented at the symposium Communicating and Contracts between
people in the Computerized Society, Gothenburg University, Sweden.
Stodolsky, D. S. (1990). Protecting expression in teleconferencing:
Pseudonym-based peer review journals. _Canadian Journal of
Educational Communication_, 19, 41-51. ([1989, May 9].
_Communication Research and Theory Network [CRTNET]_, No. 175 [Semi-
final draft available by electronic mail from LISTSERV at PSUVM.BITNET
at University Park, PA: The Pennsylvania State University,
Department of Speech Communication and COMSERVE at Vm.ecs.rpi.edu at
Troy, NY: Rensselaer Polytechnic Institute, Department of Language,
Literature, and Communication])
Stodolsky, D. S. (in press). Archiving secure interactions [Letter].
_Psychological Science_. ([1990, May 25]. Comments on Gardner's
Electronic Archive by Stodolsky. _Psycoloquy_, _1_.)
David S. Stodolsky Office: + 45 46 75 77 11 x 21 38
Department of Computer Science Home: + 45 31 55 53 50
Bldg. 20.2, Roskilde University Center Internet: david at ruc.dk
Post Box 260, DK-4000 Roskilde, Denmark Fax: + 45 46 75 74 01
More information about the Bioforum