dfischer at indigo.cs.bgu.ac.il
Wed Nov 17 15:44:45 EST 1999
We announce the Second Critical Assessment of Fully Automated
Structure Prediction Experiment (CAFASP2).
The goal of CAFASP2 is to evaluate the performance of fully automatic
structure prediction servers available to the community. In contrast
to the normal CASP procedure, CAFASP2 will answer the question of how
well servers do without any intervention of experts, i.e. how well
ANY user can predict protein structure.
As in CAFASP1, CAFASP2 assesses the performance of methods without
the user intervention allowed in CASP submissions.
This may aid users in choosing which programs they wish to use,
and in evaluating the reliability of the programs when applied to
their specific prediction targets. In addition, evaluation of fully
automated methods is particularly important to assess their applicability
at genomic scales.
CAFASP2 will take place as an official section of CASP4, which will be
held by mid-2000. We invite developers of automatic prediction servers
to participate in CAFASP2. The rules, formats, assessment procedures
and other details will be published later as soon as they are determined
by the CAFASP2 committee in coordination with the CASP4 committee.
Interested parties are invited to send an e-mail to dfischer at cs.bgu.ac.il
so they will be included in the CAFASP2 mailing list.
In what follows a preliminary set of rules for CAFASP2 is described.
A "server" is the automated structure prediction server that participates
in CAFASP. A server will have an identification name and its corresponding
A "server's person" is the person in charge of the maintainance and function
of a server.
A "raw server output" is the output obtained by a server.
A "server's submission" is the prediction that a server's person files to CAFASP
using the raw server output.
REGISTRATION TO CAFASP2.
A server's person willing to include his/her server to CAFASP2 must register
as a predictor to CASP4, and state that this registration corresponds to his/her
server. The server's person will receive a participant's indentification.
A server's person can also register as a regular CASP4 participant.
Servers that run different methods should register each of the methods
SUBMISSION OF PREDICTIONS TO CAFASP2.
By the expiry date of each target the server's person is responsible to submit
the prediction to CASP4, using the regular CASP4 procedure. This submission must
be identical in content to the output produced by his/her server, but may vary
in format. Because it may be difficult to enforce that a server produces valid
CASP4 formats, we allow a server's person to take the raw server output and
transform it to valid CASP4 format, as long as it is identical in content.
To verify that the content is the same, the CAFASP committee will submit
each target to each of the registered servers by the time each target is
released. The raw server's output will be made available to all in the
CAFASP2 web-site. The CAFASP committee will verify that the raw server output
corresponds exactly (in content) to the submission filed by the server's person.
If differences are found, the submission will be disquailfied.
CATEGORIES FOR CAFASP2.
Any automated server can participate in CAFASP2, including Homology Modeling,
Threading, Ab-Initio and Secondary Structure Prediction.
ASSESSMENT OF PREDICTIONS.
Predictions submitted to CAFASP will undergo the exact same evaluation procedure
as the normal CASP submission. The CAFASP committe may decide to apply additional
evaluation procedures. The latter will include only automatic evaluators and
the measures used be them will be made public as soon as possible.
More details about the additional cafasp evaluation will be announced later
on. A comparison of the performance of CAFASP2 versus the regular CASP4
submission will also be carried out.
Currently the people involved in the CAFASP committee are Leszek Rychlewski,
Arne Elofsson, Burkhard Rost and Daniel Fischer.
Sub-committees for homology modeling, threading, ab-initio and secondary structure
servers will be assigned. The Sub-committees will be coordinated by the CAFASP committee.
The Sub-committees will be in charge of the compilation of the servers' raw
output, the additional evaluation (if required) and the comparative analysis
of the results.
As of today these are the sub-committees and people assigned to them:
Threading sub-committee: Leszek Rychlewski, Arne Elofsson and Daniel Fischer.
Secondary Structure sub-committee: Burkhard Rost.
We invite interested parties to be involved in the variuos sub-committees.
CAFASP2 url: http://www.cs.bgu.ac.il/~dfischer/CAFASP2
Dr. Daniel Fischer
Dept. of Math and Computer Science
Ben Gurion University, Beer-Sheva 84015, Israel
e-mail: dfischer at cs.bgu.ac.il Fax +972- 7-6477648 or 6472910
More information about the Molmodel