Consensus Journals: Invitational journals based upon peer consensus

David S. Stodolsky dsstodol at daimi.aau.dk
Tue Nov 13 12:26:09 EST 1990


Consensus Journals:
Invitational journals based upon peer consensus


David  S. Stodolsky

Roskilde University Centre
DK-4000 Roskilde
david at ruc.dk


Abstract


Computer networks open new possibilities for 
scientific communication in terms of 
quality, efficiency, and rapidity. Consensus 
journals have the economy of invitational 
journals and the objectivity of journals 
based upon the peer review. That is, all 
articles are published and the reader 
benefits from article selection based upon 
impartial refereeing. An additional benefit 
of consensus journals is that the 
negotiation process, that typically occurs 
prior to publication, is automated, thus 
saving efforts of participants. 

Readers submit reviews that evaluate 
articles on agreed dimensions. A statistical 
procedure is used to identify the most 
knowledgeable representative of each 
consensus position and these persons are 
invited to submit articles that justify the 
review judgments they have submitted. A 
major advantage of this approach is the 
ability to develop reputation without 
article publication. 

The approach includes a protection mechanism 
based upon pseudonyms, that substitutes for 
the protection of anonymity typical with 
scientific journals. This reduces the 
potential for irresponsible behavior and 
facilitates reputation development. The 
level of quality enhancement is superior to 
that achievable with anonymous peer review.

Eliminating the editor and the delay 
associated with conventional refereeing 
makes message quality enhancement available 
in message systems for educational and 
business environments.

____________________________________________________________________

This document has been prepared for electronic publication. 
Underscore characters indicate the start and end of italicized 
character sequences. Figures and tables assume a monospace font. 
Citation: Stodolsky, D. S. (1990). Consensus Journals: Invitational 
journals based upon peer consensus. _Datalogiske Skrifter_ (Writings 
on Computer Science). No. 29 / 1990. Roskilde University Centre, 
Institute of Geography, Socioeconomic Analysis, and Computer 
Science. (ISSN 0109-9779-29)
____________________________________________________________________


Invitational journals can be distinguished from typical scholarly 
journals by the sequence of events that results in publication of an 
article. The sequence of events with a typical journal starts with 
the writing of an article. The article is then transmitted to an 
editor and refereed. After a successful review, often contingent 
upon negotiated revisions, the article is published and read. With 
invitational journals, however, events are reversed. The tentative 
decision to publish an author is made first, often based upon the 
reading of previous work by that author. Then negotiation between 
the editor and author occurs, or there is informal refereeing of a 
proposal, which if successful, results in the writing of an article. 
The great advantage of this second sequence -- read, negotiate, 
write -- is that almost every article written gets published. The 
disadvantage is that selection of authors is somewhat arbitrary and 
there is no way an unknown author can get published. The objective 
of this article is to outline a method of scientific communication 
that has the economy of invitational journals and the objectivity of 
journals based upon the peer review. These self-edited journals will 
be called _consensus journals_ in order to distinguish them from 
conventional invitational journals. 

Any reader of an article in a consensus journal can act as a 
referee. Assume, for simplicity, that referees send reviews to a 
mediator. At a deadline, the mediator performs calculations and 
issues invitations to the referees who have been selected as new 
authors (Figure 1). These calculations are implicit negotiations, 
that is, they predict which persons would have been selected to 
respond to the reviewed article if referees has actually negotiated 
and reached a consensus. One benefit of consensus journals is that 
the negotiation process is automated, thus saving participant 
effort.


--------------------------------------------------------------------

 ------  R: Review   -------  M: Invitation   -------
| Read |----------->| Calc. |--------------->| Write |----------->
 ------              -------                  -------  R: Article
                                                 |
                                                 | R: Renege
 R = Referee    Calc. = Calculate consensus      |
 M = Mediator                                    V

--------------------------------------------------------------------
Figure 1. Simplified cycle of operation for a consensus journal


The simplified cycle of operation for a consensus journal shows 
actions in boxes and messages as arrows. In this simplified cycle, 
referees invited to publish (and justify the reviews they have 
submitted) have a choice of submitting their article by a deadline 
or reneging on the promise implied by their review. This simplified 
cycle of operation assumes, additionally, that consensus positions 
can be calculated and that published articles are retained 
indefinitely. Eliminating these assumptions requires a more 
articulated cycle of operation and additional message types.

Before considering a more articulated cycle of operation, however, 
it is necessary to note an important feature of peer review that 
contributes to impartial judgment. This feature is a protection 
mechanism, typically anonymity, that shields referees from pressures 
that might be associated with evaluation of a colleague. Further, 
names and affiliations of authors are often hidden from referees to 
ensure that only article content is the basis for evaluation. 
Protection can be alternatively be provided by a pseudonym system. 
This has the advantage of reducing opportunities for irresponsible 
behavior as compared to systems based upon anonymity (Stodolsky, 
1990). It has a further advantage of permitting reputation 
development through the refereeing alone, thereby making it possible 
to establish a reputation without contributing articles. 

When there are multiple referees, it is important that their 
judgments are independent, so referee reports must not be made 
available until all have been submitted. This last requirement can 
be met by ensuring that reviews transmitted to the mediator are 
hidden until the deadline. The dynamics and implementation of 
protection systems are beyond the scope of this article, so only the 
necessity for the simultaneous release of information is addressed 
here.

Definition of message types

While in the simplest case, messages in the consensus journal 
environment consist of only articles and reviews (Stodolsky, 1990), 
considerations of effective negotiation and of storage management 
suggest defining additional message types. There are five types of 
messages transmitted in the consensus journal environment. 

_Article

Articles, while shorter than those usually seen in conventional 
journals, will most often play the same role. However, it is quite 
possible to have an article in a consensus journal that is only a 
few lines long, and that can only be understood in connection with 
the review message it follows and its target article. 

_ Review

Review messages must be distinguished from conventional reviews 
because they are characterized by a vector of numbers that summarize 
a reader's reaction to an article. If we think of articles as nodes 
in a graph or pages in a hypertext network, then review messages are 
the labels on arcs or links that connect the articles. Reviews can 
go beyond merely evaluating an article, by offering to provide new 
information that may be essential to support the target article's 
position. Review messages also serve as a commitment to deliver a 
justification of the reader's judgment, if invited.

_ Invitation

Invitations are public, and therefore, impossible to refuse without 
some loss of reputation. This makes them somewhat different than 
invitations from an editor of a journal. In effect, the invitation 
says, "We offer you storage space for an article." Also, a person 
may post an invitation for themselves during the negotiation stage 
of review, if they feel confident they can support the position 
claimed in their review message. 

_ Cancellation

It is possible for an author to cancel an article, thereby releasing 
the associated storage space. The article then goes off-line (i.e., 
"out of print") along with its reviews and the articles that were 
dependent upon it for their place in storage. This would typically 
occur during explicit negotiation after the author had seen the 
article's reviews. It could, however, occur much later, when a new 
criticism was delivered.

_Withdrawal

Finally, during explicit negotiations, a review may be withdrawn. 
This eliminates the referee from those prepared to respond to an 
invitation.

Cycle of operation

The sequence of events with a consensus journal is the same as with 
an invitational journal. The review method, however, involves the 
entire readership, or at least those who offer a judgment. New 
authors are then selected based upon the review judgements. While 
most articles will follow from reviews and be connected to their 
target articles, independent articles are also permitted. However, 
articles posted without consensus based invitations are less likely 
to be read and can not be assumed to have support of other referees.

If we assume that a consensus journal is already functioning, we can 
follow the events through a cycle of operation that starts with 
reading of an article. While it is not essential for smaller 
readerships, we assume that participants exchange information 
electronically.

All readers are presented with a target article at the same time. A 
reader offers a review judgment in order to be considered for future 
authorship. The review message must be received before a certain 
deadline, say one week later. The review message consists of scores 
along several preselected dimensions. For instance, a scientific 
article is expected to be relevant, correct, and original. A more 
conversational approach might include the dimensions completeness, 
clarity, and appropriateness. 

At the deadline, the mediator runs a statistical procedure to 
determine if there are consensus positions among the referees. The 
most central referee from each of these positions is invited to 
submit a new article. These most central referees are also 
considered most knowledgeable, within the framework of cultural 
consensus theory. D'Andrade (1987) discusses the evidence supporting 
this view. 

Cultural consensus theory is based on the assumptions of common 
truth (i. e., there is a fixed answer pattern "applicable" to all 
referees), of local independence (i.e., the referee-dimension 
response variables satisfy conditional independence), and of 
homogeneity of items (i.e., each respondent has a fixed "cultural 
competence" over all dimensions) (Romney, Weller & Batchelder, 
1986). Results can be obtained with as few as three respondents, but 
four are required if the significance of the results are to be 
calculated (i. e., a degree of freedom is then available in the 
statistical model) (Batchelder & Romney, 1988). A recent development 
in the model is the ability to identify two consensual groupings 
within the population of respondents (Romney, Weller & Batchelder, 
1987) This is extremely helpful since it permits a minority to 
publicize their viewpoint under the same conditions as a majority.

Cultural consensus theory assumes that we have no _a priori_ 
knowledge about referees, that is, they have no reputations. This is 
extremely valuable when a new topic comes up or when there are 
violation of assumptions required for calculations concerning a 
current article based upon previous information (Stodolsky, 1984b). 
Given that reputations have developed and assumptions are satisfied, 
however, the theory requires elaboration to be applied most 
effectively. Cultural consensus theory provides, in effect, a cross 
sectional estimation of competence. That is, given a sample of 
responses at a given moment, relative competence is estimated. On 
the other hand, given a performance history, Bayesian estimation can 
be used to assess the relative importance of different persons' 
judgments. That is, there are reputations that give information 
about relative competence independent of the current responses. This 
assumes stationarity, that is, that the same area of competence is 
required for correct response, and that responses are generated in 
the same manner (e. g., respondents continue to give honest 
answers). Both methods are based upon likelihood estimation, 
therefore, a combined theory should be achievable. The combined 
sources of information would likely make achieving an implicit 
consensus more frequent.

The mediator issues an invitation report showing submitted 
judgments, the degree of consensus achieved, the number of consensus 
positions identified, degree of knowledge of each referee, and so 
on. If consensus has been reached, invited referees are expected to 
submit articles.

Negotiation must proceed explicitly if no consensus can be 
identified (Figure 2). In that case, referees may look at the 
judgments submitted and decide if their positions have sufficient 
support. If not, they could reconsider their review judgments, and 
either revise them or withdraw from the review process. The author 
of an article might, on the basis of these judgments, cancel an 
article, thus avoiding potential reputation damaging criticism. 

--------------------------------------------------------------------

 ------  R: Review   -------  M: Invitation   -------
| Read |----------->| Calc. |--------------->| Write |----------->
 ------              -------                  -------  R: Article
                      ^ |                        |
           R: Review  | | M: No consensus        | R: Renege
                      | |                        |
                      | V                        V
                  -----------  R: Withdrawal    
                 | Reconsider|-------------->           
                  ----------- 
                       |
                       | A: Cancelation
                       |
                       V

Key

A = Author              
M = Mediator          Calc.  = Calculate consensus
R = Referee
--------------------------------------------------------------------

Figure 2. Cycle of operation for a consensus journal

(A referee becomes an author only after a submitted article has been 
published by the mediator [not shown in figure]).


Assuming that the article was not cancelled, the combined effects of 
withdrawal by referees with most deviant judgments and 
reconsideration by others would likely lead to consensus, 
particularly if the requirements for consensus were successively 
relaxed. This assumes that revision of judgments would be in the 
direction of dominant view points, a common finding. The result 
would be an invitation issued by the mediator to selected referees. 
Subsequent submission of an articles by selected referees and their 
publication by the mediator would complete the cycle of operation.

The invitation report can guide negotiation when a consensus can not 
be identified. Individual invitation staging could proceed along 
with a relaxation of requirements for consensus. For instance, if 
the first round of reviews did not generate a consensus, referees 
could issue invitations to others (Table 1). If the second round of 
reviews did not generate a consensus, referees could issue self 
invitations (these would be acceptances for those who had received 
invitations), or perhaps, direct the invitations they had already 
received to others. Failure on the third round would permit these 
previously issued invitations to serve as a coordination mechanism. 
That is, certain referees would have indicated a readiness to 
respond and others would have rejected the option of authorship 
unsupported by a consensus. Thus, duplication of effort could be 
avoided by examining the ranking of persons in terms of the 
invitations received and accepted, and responding accordingly. 


--------------------------------------------------------------------

                          Message Level and (Receiver):

                           Public     Private
                          (Readers)  (Mediator)

Stage of Process:

Publication                Article

Refereeing                            Review(1)

Invitation                 Report

Reconsideration                       Review(2), Withdraw, Cancel, 
Invite

Invitation                 Report

Reconsideration                       Review(3), Withdraw, Cancel, 
Self invitation

Invitation                 Report

Submit                                Article

--------------------------------------------------------------------

Table 1. Message level, receiver, and type, by stage of processing
         (assumes no consensus reached during negotiation)


New articles are requested either by the mediator or by referees, if 
an author and referees follow negotiations to completion. New 
articles must be submitted before a deadline. At the deadline, the 
new articles received are published. This makes them available to 
the readership and completes the cycle of operation for a consensus 
journal. 

Thus in the simplest case, articles are read, reviews are 
transmitted, invitations are issued, and new articles are submitted 
in a timely manner. In the extended cycle, at least a single 
reconsideration or negotiation stage occurs during which a target 
article can be cancelled by its author and during which referees can 
withdraw. A failure to achieve consensus leads to explicit 
negotiation and options such as nonconsensus invitation. The 
extended negotiation option makes the consensus journal more similar 
to a conventional journal, because there is explicit negotiation 
prior to the writing of an article. 

The quality of a consensus journal can be assessed by the degree of 
consensus achieved. Readers might select only those articles 
resulting from a consensus-based invitation, thereby controlling the 
quality of articles they see. 

Rules of dialogue

The rules of operation of a consensus journal can be thought of as 
specifying an action system, or language game, where the actions 
relate to the placement of articles in a network of interconnected 
nodes. Participants in the game try to maximize their influence. 
Reputation is a crucial resource in scientific argumentation 
(Smolensky, Fox, King, Lewis, 1988). Participants are expected to 
maximize this resource. While there may be other payoffs available 
within a given system, such as royalty payments, this discussion 
assumes only reputation maximization as an individual objective. 

There are several opportunities for reputation enhancement in the 
cycle of operation. Selection as an author is a major opportunity 
for reputation enhancement. However, refereeing also offers 
significant opportunities that are not available with conventional 
journals. Referees can commit themselves to delivering a rebuttal to 
an article and thereby improve their reputation (assuming they make 
good on their commitment given an opportunity). If an author 
examines the reviews an article receives and decides to cancel it 
before a rebuttal is written, the referees offering rebuttals would 
have their reputations enhanced, without any further risk or effort. 

With a consensus journal, the review message can be thought of as an 
offer to deliver a certain type of article before the deadline. 
Obviously, a review message that claims a target article is 
erroneous, and thereby offers to deliver a rebuttal, plays a 
different structural role in a debate than one that criticizes an 
article for not being original. Thus, reviews can have a great deal 
of structural impact and can express a level of commitment, which 
would not be relevant in an environment that limits referees to a 
gatekeeping role.

Structural Aspects

With electronically published documents, it is very desirable to 
structure interconnections so that retrieval is facilitated and the 
relevance of statements becomes clear (Smolensky, Fox, King, Lewis, 
1988). Thus, review messages can deal not only with the quality of 
an article, but also its relationship to its target article. 
Explicit relationships among articles becomes more necessary as the 
size of articles decrease and number of articles increases.

With conventional journals, reviews are used to determine whether or 
not an article should be published. The publication decision is not 
dominant with electronic media, however, since distribution 
constraints are greatly relaxed (Quarterman, 1990, p. 259; 
Stodolsky, in press). Because of this, the period during which an 
article remains on-line assumes importance, because storage is 
limited. It is in this connection that the reviews of articles and 
the relations between articles becomes critical. In the simplest 
case, an article that is found incorrect by an overwhelming 
consensus is cancelled by its author. Failure to cancel an the 
article results in a continuing devaluation of the author's 
reputation as more and more readers come to agree with the majority. 
In the case of conflicting consensus positions, a rebuttal claiming 
that a target article is flawed is explicitly linked to the target. 
Failure to rebut that claim in turn has much the same effect as an 
overwhelming consensus that the target article is incorrect. Most 
interactions, therefore, take place at the knowledge frontier, as 
various positions are argued. These interactions generate very 
"bushy" argument trees, that require sophisticated navigation 
strategies, if large amounts of effort are not to be expended 
unnecessarily (Stodolsky, 1984a). The trees are thinned in the 
process of argumentation. Positions that are sustained remain on-
line until they are thoroughly integrated into summaries or 
overarching theories.

Decentralization

A central mediator has been assumed in this description to simplify 
explanation. There is no reason why the calculations necessary to 
select new authors could not be performed decentrally. In fact, this 
would be necessary if readers preferred different methods of 
calculation for author selection. Then coordination in the selection 
of new authors would be shifted from consensus calculation to 
collection of invitations. Various types of voting rules could be 
applied. Authors receiving the most invitations would then be 
expected to submit articles. Thus, decentralization leads to an 
integration of the two types of invitations (consensus and 
individual) already discussed.

The task of protecting review judgments until the deadline is 
reached is another required function. It is necessary, for example, 
because analysis of earlier submitted judgments could permit a 
referee submitting at the last moment to simulate a competence that 
did not exist, thus violating assumptions of the model. Protection 
can, however, be achieved decentrally using cryptography, assuming a 
"beacon" that emits enciphering and deciphering keys at fixed 
intervals (Rabin, 1983). Use of cryptography would be necessary, in 
any case, to ensure the authenticity of messages.

Summary

A consensus journal requires mechanisms for both coordination and 
protection. In the simplest case, a mediator can provide these. This 
assumes protected channels of communication and a trusted mediator. 
Coordination is necessary to identify consensus positions and avoid 
duplication of effort. Protection of reviews is necessary to ensure 
that assumptions of models for evaluating expertise are not 
violated. This protection allows valid reputation development by 
both authors and referees. Such reputations can then be used to 
ensure effective allocation of expertise. The extension of review 
opportunities to the entire readership vastly extends the available 
field of expertise. This, combined with the effective allocation of 
expertise and coordination that eliminates duplication of effort, 
provide consensus journals with a significant advantage over current 
mechanisms for enhancement of message quality.


References

Batchelder, W. H. & Romney, A. K. (1988). Test theory without an 
answer key. _Psychometrika_, _53_(1), 71-92. 

D'Andrade, R. G. (1987). Modal response and cultural expertise. 
_American Behavioral Scientist_, _31_(2), 194-202.

Quarterman, J. S. (1990). _The matrix: Computer networks and 
conferencing systems worldwide_. Bedford, MA: Digital Press.

Rabin, M. (1983). Transaction protection by beacons. _Journal of 
Computer and Systems Science_, _27_(2), 256-267.

Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1986). Culture 
as consensus: A theory of culture and informant accuracy. _American 
Anthropologist_, _88_(2), 313-338.

Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1987). Recent 
applications of cultural consensus Theory. _American Behavioral 
Scientist_, _31_(2), 163-177.

Smolensky, P., Fox, B., King, R., & Lewis, C. (1988). Computer-aided 
reasoned discourse or, how to argue with a computer. In R. Guindon 
(Ed.), _Cognitive science and its applications for human-computer 
interaction_. Hillsdale, NJ: Lawrence Erlbaum Associates.

Stodolsky, D. (1984a). Commonalities amoung [sic] conferencing 
systems and their implication for marketing strategy. 
_Organisatoriske Fragmenter 1984_, _12_, 43-58.

Stodolsky, D. (1984b, December). _Self-management of criticism in 
dialog: Dynamic regulation through automatic mediation_. Paper 
presented at the symposium Communicating and Contracts between 
people in the Computerized Society, Gothenburg University, Sweden.

Stodolsky, D. S. (1990). Protecting expression in teleconferencing: 
Pseudonym-based peer review journals. _Canadian Journal of 
Educational Communication_, 19, 41-51. ([1989, May 9]. 
_Communication Research and Theory Network [CRTNET]_, No. 175 [Semi-
final draft available by electronic mail from LISTSERV at PSUVM.BITNET 
at University Park, PA: The Pennsylvania State University, 
Department of Speech Communication and COMSERVE at Vm.ecs.rpi.edu at 
Troy, NY: Rensselaer Polytechnic Institute, Department of Language, 
Literature, and Communication])

Stodolsky, D. S. (in press). Archiving secure interactions [Letter]. 
_Psychological Science_. ([1990, May 25]. Comments on Gardner's 
Electronic Archive by Stodolsky. _Psycoloquy_, _1_[8].)

--
David S. Stodolsky                  Office: + 45 46 75 77 11 x 21 38
Department of Computer Science                Home: + 45 31 55 53 50
Bldg. 20.2, Roskilde University Center        Internet: david at ruc.dk
Post Box 260, DK-4000 Roskilde, Denmark        Fax: + 45 46 75 74 01



More information about the Bioforum mailing list