From owner-genbankb@hgmp.mrc.ac.uk Tue Apr 20 08:22:41 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 1A8ED7D1AA for ; Tue, 20 Apr 2004 08:22:41 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id A7A8A7D1FC for ; Tue, 20 Apr 2004 08:22:40 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id E50067D1AA; Tue, 20 Apr 2004 08:22:39 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 20 Apr 2004 00:39:22 +0100 From: Mark Cavanaugh Subject: GenBank 141.0 Close-Of-Data Message-Id: <20040420072239.E50067D1AA@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Greetings GenBank Users, Close-of-data for the upcoming GenBank Release 141.0 is planned for tomorrow at approximately 1:30am EDT, Tuesday April 20. The subsequently generated incremental GenBank Update files nc0420.aso, nc0420.flat (etc) will contain data through the close. Mark Cavanaugh GenBank NCBI/NLM/NIH/DHHS --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Sun Apr 25 03:17:47 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 12A327D25D for ; Sun, 25 Apr 2004 03:17:47 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id A0F8A7D200 for ; Sun, 25 Apr 2004 03:17:46 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 676B67D25D; Sun, 25 Apr 2004 03:17:44 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 25 Apr 2004 03:11:30 +0100 From: Mark Cavanaugh Subject: GenBank Release 141.0 Now Available Message-Id: <20040425021744.676B67D25D@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Greetings GenBank Users, GenBank Release 141.0 is now available via ftp from the National Center for Biotechnology Information (NCBI): Ftp Site Directory Contents ---------------- --------- --------------------------------------- ftp.ncbi.nih.gov genbank GenBank Release 141.0 flatfiles ncbi-asn1 ASN.1 data used to create Release 141.0 Uncompressed, the Release 141.0 flatfiles require approximately 131 GB (sequence files only) or 146 GB (including the 'short directory' and 'index' files). The ASN.1 version requires approximately 115 GB. From the release notes: Release Date Base Pairs Entries 140 Feb 2004 37893844733 32549400 141 Apr 2004 38989342565 33676218 Close-of-data was 04/20/2004. Five working days were required to prepare this release. In the eight week period between the close dates for GenBank releases 140.0 and 141.0, the non-WGS portion of GenBank grew by 1,095,497,832 basepairs and by 1,126,818 sequence records. During that same period, 556,412 records were updated. Combined, this yields an average of about 28,500 new and/or updated records per day. Between releases 140.0 and 141.0, the WGS component of GenBank grew by 1,954,410,330 basepairs and by 923,778 sequence records. *NOTE* Problems were encountered during release processing which prevented the generation of the gbacc.idx and gbjou.idx 'index' files for GenBank 141.0. Please see Section 1.3.1 of the release notes for further details. We would like to remind our users that GenBank mirrors are available at ftp://genbank.sdsc.edu/pub and ftp://bio-mirror.net/biomirror/genbank. Those who experience slow FTP transfers due to a high volume of traffic at NCBI might realize an improvement in transfer rates from these alternate sites. For additional release information, see the README files in either of the directories mentioned above, and the release notes (gbrel.txt) in the genbank directory. Sections 1.3 and 1.4 of the release notes (Changes in Release 141.0 and Upcoming Changes) have been appended below. *NOTE* Section 1.4.1 discusses a very important change : the removal of sequence length limits for all classes of GenBank sequence records, as of June 2004. We strongly encourage all users to review this information. Release 141.0 data, and subsequent updates, are available now via NCBI's Entrez and Blast services. If you encounter problems while ftp'ing or uncompressing Release 141.0, please send email outlining your difficulties to info@ncbi.nlm.nih.gov . Mark Cavanaugh, Vladimir Alekseyev, Anton Butanaev, Michael Kimelman GenBank NCBI/NLM/NIH 1.3 Important Changes in Release 141.0 1.3.1 Two 'index' files unavailable for GenBank 141.0 An unexpected software problem prevented the generation of two of the 'index' files which normally accompany GenBank releases: gbacc.idx gbjou.idx Resolving the problem would have delayed release processing by three days, with a concommitant delay in the indexing of recently processed records for Entrez. So it was decided to make GenBank 141.0 available without these index files. Our apologies for any inconvenience that this might cause. 1.3.2 Organizational changes The total number of sequence data files increased by 16 with this release: - the BCT division is now comprised of 9 files (+1) - the EST division is now comprised of 305 files (+9) - the GSS division is now comprised of 104 files (+2) - the PLN division is now comprised of 11 files (+1) - the ROD division is now comprised of 12 files (+1) - the STS division is now comprised of 4 files (+1) - the VRT division is now comprised of 6 files (+1) 1.3.3 GSS File Header Problem GSS sequences at GenBank are maintained in one of two different systems, depending on their origin. One recent change to release processing involves the parallelization of the dumps from those systems. Because the second dump (for example) has no prior knowledge of exactly how many GSS files will be dumped from the first, it doesn't know how to number it's own output files. There is thus a discrepancy between the filenames and file headers for thirteen GSS flatfiles in Release 141.0. Consider the gbgss92.seq file: GBGSS1.SEQ Genetic Sequence Data Bank April 15 2004 NCBI-GenBank Flat File Release 141.0 GSS Sequences (Part 1) 88249 loci, 65635323 bases, from 88249 reported sequences Here, the filename and part number in the header is "1", though the file has been renamed as "92" based on the files dumped from the other system. We will work to resolve this discrepancy in future releases, but the priority is certainly much lower than many other tasks. 1.4 Upcoming Changes 1.4.1 **Sequence Length Limitation To Be Removed In June 2004** At the May 2003 collaborative meeting among representatives of GenBank, EMBL, and DDBJ, it was decided that the 350 kilobase limit on the sequence length of database records will be removed as of June 2004. Individual, complete sequences are currently expected to be a maximum of 350 kbp in length. One major reason for the existence of this limit is as an aid to users of sequence analysis software, some of which might not be capable of processing megabase-scale sequences. However, very significant exceptions to the 350 kbp limit have existed for several years; Phase 1 (unordered, unoriented) and Phase 2 (ordered, oriented) high-throughput genomic sequences (HTGS) generated by efforts such as the Human Genome Project; large dispersed eukaryotic genes with an intron/exon structure that spans more than 350 kbp; and sequences which result from assemblies of Whole Genome Shotgun (WGS) project data. Given these exceptions, and the technological advances which have made large-scale sequencing practical for an increasing number of researchers, the collaboration has decided that the 350 kbp limit must be removed. As of June 2004, the length of database sequences will be limited only by the natural structures of an organism's genome. For example, a single record might be used to represent all of human chromosome 1, which is approximately 245 Mbp in length. Software developers for some of the larger commercial sequence analysis packages were recently asked what timeframe would be appropriate for this change. Answers ranged from "immediately", to "several months", to "one year". So the one-year timeframe was selected, to provide ample time to implement changes which megabase-scale sequences may require. Some sample records with very large sequences have been made available so that developers can begin to test their software modifications: ftp://ftp.ncbi.nih.gov/genbank/LargeSeqs Many changes are expected after the removal of the length limit. For example, complete bacterial genomes (typically on the order of several megabases) will be re-assembled into single sequence records. The submission process for such genomes will become much more streamlined, since database staff will no longer have to split the genomes into pieces. BLAST services will be enchanced, so that hits reported within very large sequences will be presented in a meaningful context. All such changes will be discussed more fully in future release notes, the NCBI newsletter, and the GenBank newsgroup. 1.4.2 Rename of File 'Last.Release' and Deletion of /daily Subdirectories The files named Last.Release which are located at: ftp://ftp.ncbi.nih.gov/genbank/daily/Last.Release ftp://ftp.ncbi.nih.gov/ncbi-asn1/daily/Last.Release contain the number of the GenBank release which is currently installed on the NCBI FTP site. As of Release 142.0 in June 2004, these files will be moved and renamed as: ftp://ftp.ncbi.nih.gov/genbank/GB_Release_Number ftp://ftp.ncbi.nih.gov/ncbi-asn1/GB_Release_Number The /daily subdirectories, which had been used for cumulative update products that are no longer supported, will be deleted at that time. --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Mon Apr 26 03:52:20 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 145027D166 for ; Mon, 26 Apr 2004 03:52:20 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id AD6727D2A7 for ; Mon, 26 Apr 2004 03:52:19 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 4E8587D166; Mon, 26 Apr 2004 03:52:18 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 26 Apr 2004 03:50:36 +0100 From: Francis Ouellette Subject: Re: GenBank Release 141.0 Now Available Message-Id: <20040426025218.4E8587D166@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Dear Mark, a question via the GenBank newsgroup: As per these anouncements over the last year, the next release of GenBank will have unlimited lengths in GenBank record. Do you expext this will have much of an impact in the next release? Or do you anticipate these longer records to trickle in over time? For example, does NCBI plan to re-release all bacterial genomes by next release of GenBank in this unified single record? Will there be a public release of BLAST ahead of the next GenBank release? (so that those of us who maintain local BLAST servers for our communities can have a BLAST server readdy for this new data?) cheers, and thanks for the wonderful work your #1 fan, f. -- BF Francis Ouellette http://bioinformatics.ubc.ca/ouellette On Sun, 25 Apr 2004, Mark Cavanaugh wrote: > Greetings GenBank Users, > > GenBank Release 141.0 is now available via ftp from the National > Center for Biotechnology Information (NCBI): [...] > Release Date Base Pairs Entries > > 140 Feb 2004 37893844733 32549400 > 141 Apr 2004 38989342565 33676218 > 1.4 Upcoming Changes > > 1.4.1 **Sequence Length Limitation To Be Removed In June 2004** > > At the May 2003 collaborative meeting among representatives of GenBank, > EMBL, and DDBJ, it was decided that the 350 kilobase limit on the sequence > length of database records will be removed as of June 2004. > > Individual, complete sequences are currently expected to be a maximum > of 350 kbp in length. One major reason for the existence of this limit is > as an aid to users of sequence analysis software, some of which might not > be capable of processing megabase-scale sequences. > > However, very significant exceptions to the 350 kbp limit have existed > for several years; Phase 1 (unordered, unoriented) and Phase 2 (ordered, > oriented) high-throughput genomic sequences (HTGS) generated by efforts > such as the Human Genome Project; large dispersed eukaryotic genes with > an intron/exon structure that spans more than 350 kbp; and sequences > which result from assemblies of Whole Genome Shotgun (WGS) project data. > > Given these exceptions, and the technological advances which have made > large-scale sequencing practical for an increasing number of researchers, > the collaboration has decided that the 350 kbp limit must be removed. > > As of June 2004, the length of database sequences will be limited only > by the natural structures of an organism's genome. For example, a single > record might be used to represent all of human chromosome 1, which is > approximately 245 Mbp in length. > > Software developers for some of the larger commercial sequence analysis > packages were recently asked what timeframe would be appropriate for this > change. Answers ranged from "immediately", to "several months", to "one year". > So the one-year timeframe was selected, to provide ample time to implement > changes which megabase-scale sequences may require. > > Some sample records with very large sequences have been made available > so that developers can begin to test their software modifications: > > ftp://ftp.ncbi.nih.gov/genbank/LargeSeqs > > Many changes are expected after the removal of the length limit. For > example, complete bacterial genomes (typically on the order of several > megabases) will be re-assembled into single sequence records. The submission > process for such genomes will become much more streamlined, since database > staff will no longer have to split the genomes into pieces. BLAST services > will be enchanced, so that hits reported within very large sequences will > be presented in a meaningful context. > > All such changes will be discussed more fully in future release notes, > the NCBI newsletter, and the GenBank newsgroup. --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Mon Apr 26 03:52:42 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 3F9167D2AD for ; Mon, 26 Apr 2004 03:52:42 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id DC6257D155 for ; Mon, 26 Apr 2004 03:52:41 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 851D27D275; Mon, 26 Apr 2004 03:52:40 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 26 Apr 2004 03:50:36 +0100 From: Francis Ouellette Subject: Re: GenBank Release 141.0 Now Available Message-Id: <20040426025240.851D27D275@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Dear Mark, a question via the GenBank newsgroup: As per these anouncements over the last year, the next release of GenBank will have unlimited lengths in GenBank record. Do you expext this will have much of an impact in the next release? Or do you anticipate these longer records to trickle in over time? For example, does NCBI plan to re-release all bacterial genomes by next release of GenBank in this unified single record? Will there be a public release of BLAST ahead of the next GenBank release? (so that those of us who maintain local BLAST servers for our communities can have a BLAST server readdy for this new data?) cheers, and thanks for the wonderful work your #1 fan, f. -- BF Francis Ouellette http://bioinformatics.ubc.ca/ouellette On Sun, 25 Apr 2004, Mark Cavanaugh wrote: > Greetings GenBank Users, > > GenBank Release 141.0 is now available via ftp from the National > Center for Biotechnology Information (NCBI): [...] > Release Date Base Pairs Entries > > 140 Feb 2004 37893844733 32549400 > 141 Apr 2004 38989342565 33676218 > 1.4 Upcoming Changes > > 1.4.1 **Sequence Length Limitation To Be Removed In June 2004** > > At the May 2003 collaborative meeting among representatives of GenBank, > EMBL, and DDBJ, it was decided that the 350 kilobase limit on the sequence > length of database records will be removed as of June 2004. > > Individual, complete sequences are currently expected to be a maximum > of 350 kbp in length. One major reason for the existence of this limit is > as an aid to users of sequence analysis software, some of which might not > be capable of processing megabase-scale sequences. > > However, very significant exceptions to the 350 kbp limit have existed > for several years; Phase 1 (unordered, unoriented) and Phase 2 (ordered, > oriented) high-throughput genomic sequences (HTGS) generated by efforts > such as the Human Genome Project; large dispersed eukaryotic genes with > an intron/exon structure that spans more than 350 kbp; and sequences > which result from assemblies of Whole Genome Shotgun (WGS) project data. > > Given these exceptions, and the technological advances which have made > large-scale sequencing practical for an increasing number of researchers, > the collaboration has decided that the 350 kbp limit must be removed. > > As of June 2004, the length of database sequences will be limited only > by the natural structures of an organism's genome. For example, a single > record might be used to represent all of human chromosome 1, which is > approximately 245 Mbp in length. > > Software developers for some of the larger commercial sequence analysis > packages were recently asked what timeframe would be appropriate for this > change. Answers ranged from "immediately", to "several months", to "one year". > So the one-year timeframe was selected, to provide ample time to implement > changes which megabase-scale sequences may require. > > Some sample records with very large sequences have been made available > so that developers can begin to test their software modifications: > > ftp://ftp.ncbi.nih.gov/genbank/LargeSeqs > > Many changes are expected after the removal of the length limit. For > example, complete bacterial genomes (typically on the order of several > megabases) will be re-assembled into single sequence records. The submission > process for such genomes will become much more streamlined, since database > staff will no longer have to split the genomes into pieces. BLAST services > will be enchanced, so that hits reported within very large sequences will > be presented in a meaningful context. > > All such changes will be discussed more fully in future release notes, > the NCBI newsletter, and the GenBank newsgroup. --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Mon Apr 26 17:35:20 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 092B97D30F for ; Mon, 26 Apr 2004 17:35:20 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id DD7017D30C for ; Mon, 26 Apr 2004 17:35:18 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id F2E787D313; Mon, 26 Apr 2004 17:35:00 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 26 Apr 2004 16:49:43 +0100 From: Mark Cavanaugh Subject: Removal of sequence length limits (WAS: GenBank Release 141.0 Now Message-Id: <20040426163500.F2E787D313@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk We do not expect a huge rush to instantiate very large records for genomes and chromosomes which are currently split into multiple pieces. For example, the Drosophila genome is comprised of on the order of 1200 records. Single records for each chromosome (or chromosome arm) will not replace these until the submitters of the genome, and NCBI's processing pipeline, are ready to accomodate that change. My guess is that the first post-June update of the Drosophila genome, possibly by Fall 2004, might result in single records for each chromosome/arm (assuming that the submitter desires this). If so, then the six current CON-division records for Drosophila: AE014134 AE013599 AE014296 AE014297 AE014135 AE014298 would become 'normal' (non-CON) records, containing sequence data of up to 28 Mbp . The area that will see the most immediate change after June 2004 regards bacterial genomes. New complete bacterial genomes will no longer be broken into pieces, so GenBank Update and Release products will contain records in the 1 to 5 Mbp range quite soon. Older bacterial genomes split into pieces over the past several years will eventually be replaced by single records, but a timetable for this has not yet been established. Note that several classes of large sequences already exist: HTGS Phase 0, 1, and 2; Whole Genome Shotgun contigs; and very large, dispersed, eukaryotic genes. As far as I know, BLAST already handles cases like these (up to several Mbp). But I will inquire with the BLAST group to determine if any software changes are anticipated for sequences of 100s Mbp. If so, then yes, we will certainly distribute a BLAST release prior to the distribution of sequences of that size, and announce that release here and via other channels. Reminder: Some sample large records are available at the NCBI FTP site: ftp://ftp.ncbi.nih.gov/genbank/LargeSeqs Thanks very much for your inquiry Francis; we encourage any others with concerns about the length limit removal to ask questions via this group. Regards, Mark Cavanaugh GenBank NCBI/NLM/NIH/DHHS > X-Original-To: genbankb-list@hgmp.mrc.ac.uk > To: genbank@net.bio.net > Date: 26 Apr 2004 03:50:36 +0100 > From: Francis Ouellette > Subject: Re: GenBank Release 141.0 Now Available > X-Scanned-By: MIMEDefang 2.36 > > > > Dear Mark, > > a question via the GenBank newsgroup: > > As per these anouncements over the last year, the next release of > GenBank will have unlimited lengths in GenBank record. Do you > expext this will have much of an impact in the next release? Or do > you anticipate these longer records to trickle in over time? > > For example, does NCBI plan to re-release all bacterial genomes by > next release of GenBank in this unified single record? Will there > be a public release of BLAST ahead of the next GenBank release? > (so that those of us who maintain local BLAST servers for our > communities can have a BLAST server readdy for this new data?) > > cheers, and thanks for the wonderful work > > f. --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Mon Apr 26 17:35:22 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 93F257D148 for ; Mon, 26 Apr 2004 17:35:22 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id B53AD7D307 for ; Mon, 26 Apr 2004 17:35:21 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 8EF927D148; Mon, 26 Apr 2004 17:35:07 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 26 Apr 2004 17:03:43 +0100 From: Mark Cavanaugh Subject: RE: GenBank Release 141.0 Now Available Message-Id: <20040426163507.8EF927D148@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk > Subject: RE: GenBank Release 141.0 Now Available > Date: Mon, 26 Apr 2004 07:29:07 -0400 > Thread-Topic: GenBank Release 141.0 Now Available > thread-index: AcQqa3GdHe2q9ATVQMm/6x604ICmhABFeNTM > From: "Roels, Steven" > To: "Mark Cavanaugh" , > > Hi Mark, Regarding the two missing index files for GenBank 141.0: > gbacc.idx > gbjou.idx > >Will they be release in a few days, or are they simply not going to be >released this time around? Thanks, -Steve Neither of these files will be available for GenBank 141.0 I'm afraid. We generate them from the 'replicant' side of our database system, to which dataflow is suspended during release generation. (This means that we are working from a static dataset, guaranteed to be consistent.) Unfortunately, Entrez uses that same replicant side. So when we try to fix an unexpected problem (one day for the change, then another day and a half to redump data), we begin to inconvenience *many* users. Recently processed records aren't visible to the public when replication is suspended. If we turned replication back on, resolved the problem after GB 141 had been made available, and *then* tried to generate the missing index files... they will contain data that is *not* in GB 141 itself. A bit of a catch-22 situation I'm afraid. Our apologies for the inconvenience. Regards, Mark Cavanaugh GenBank NCBI/NLM/NIH/DHHS --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Mon Apr 26 17:35:31 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id D5DE47D307 for ; Mon, 26 Apr 2004 17:35:30 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id D0E777D30B for ; Mon, 26 Apr 2004 17:35:29 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 4C3607D311; Mon, 26 Apr 2004 17:35:20 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 26 Apr 2004 12:30:13 +0100 From: "Roels, Steven" Subject: RE: GenBank Release 141.0 Now Available Message-Id: <20040426163520.4C3607D311@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Hi Mark, =A0 Regarding the two missing index files for GenBank 141.0: =A0 =A0=A0=A0=A0=A0=A0=A0 gbacc.idx =A0=A0=A0=A0=A0=A0=A0 gbjou.idx =A0 Will they be release in a few days, or are they simply not going to be rele= ased this time around? =A0 Thanks, =A0 -Steve -----Original Message----- From: owner-genbankb@hgmp.mrc.ac.uk=A0on behalf of=A0Mark Cavanaugh Sent: Sat 4/24/2004 10:11 PM To: genbank@net.bio.net Cc: Subject: GenBank Release 141.0 Now Available Greetings GenBank Users, =A0 GenBank Release 141.0 is now available via ftp from the National Center for Biotechnology Information (NCBI): =A0 Ftp Site=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Directory=A0=A0 Contents =A0 ----------------=A0=A0 ---------=A0=A0 --------------------------= ------------- =A0 ftp.ncbi.nih.gov=A0=A0 genbank=A0=A0=A0=A0 GenBank Release 141.0 = flatfiles =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 ncbi-asn= 1=A0=A0 ASN.1 data used to create Release 141.0 =A0 Uncompressed, the Release 141.0 flatfiles require approximately 1= 31 GB (sequence files only) or 146 GB (including the 'short directory' and 'index' files).=A0 The ASN.1 version requires approximately 115 GB. F= rom the release notes: =A0=A0 Release=A0 Date=A0=A0=A0=A0=A0=A0 Base Pairs=A0=A0 Entries =A0=A0 140=A0=A0=A0=A0=A0 Feb 2004=A0=A0 37893844733=A0 32549400 =A0=A0 141=A0=A0=A0=A0=A0 Apr 2004=A0=A0 38989342565=A0 33676218 =A0 Close-of-data was 04/20/2004. Five working days were required to = prepare this release. In the eight week period between the close dates for Ge= nBank releases 140.0 and 141.0, the non-WGS portion of GenBank grew by 1,09= 5,497,832 basepairs and by 1,126,818 sequence records. During that same period,= 556,412 records were updated. Combined, this yields an average of about 28,50= 0 new and/or updated records per day. =A0 Between releases 140.0 and 141.0, the WGS component of GenBank gr= ew by 1,954,410,330 basepairs and by 923,778 sequence records. =A0 *NOTE* Problems were encountered during release processing which prevented the generation of the gbacc.idx and gbjou.idx 'index' files for GenBank 141.0. Please see Section 1.3.1 of the release notes for further details. =A0 We would like to remind our users that GenBank mirrors are availa= ble at ftp://genbank.sdsc.edu/pub and ftp://bio-mirror.net/biomirror/genb= ank. Those who experience slow FTP transfers due to a high volume of traff= ic at NCBI might realize an improvement in transfer rates from these altern= ate sites. =A0 For additional release information, see the README files in eithe= r of the directories mentioned above, and the release notes (gbrel.txt) in the genbank directory. Sections 1.3 and 1.4 of the release notes (Changes in Release 141.0 and Upcoming Changes) have been appended below. =A0 *NOTE* Section 1.4.1 discusses a very important change : the remo= val of sequence length limits for all classes of GenBank sequence records= , as of June 2004. We strongly encourage all users to review this information. =A0 Release 141.0 data, and subsequent updates, are available now via NCBI's Entrez and Blast services. =A0 If you encounter problems while ftp'ing or uncompressing Release 141.0, please send email outlining your difficulties to info@ncbi.nlm.nih.gov . Mark Cavanaugh, Vladimir Alekseyev, Anton Butanaev, Michael Kimelman GenBank NCBI/NLM/NIH 1.3 Important Changes in Release 141.0 1.3.1 Two 'index' files unavailable for GenBank 141.0 =A0=A0=A0 An unexpected software problem prevented the generation of = two of =A0 the 'index' files which normally accompany GenBank releases: =A0=A0=A0=A0=A0=A0=A0 gbacc.idx =A0=A0=A0=A0=A0=A0=A0 gbjou.idx =A0=A0=A0 Resolving the problem would have delayed release processing= by three =A0 days, with a concommitant delay in the indexing of recently proce= ssed =A0 records for Entrez. So it was decided to make GenBank 141.0 avail= able =A0 without these index files. Our apologies for any inconvenience th= at this =A0 might cause. 1.3.2 Organizational changes =A0 The total number of sequence data files increased by 16 with this= release: =A0 - the BCT division is now comprised of=A0=A0 9 files (+1) =A0 - the EST division is now comprised of 305 files (+9) =A0 - the GSS division is now comprised of 104 files (+2) =A0 - the PLN division is now comprised of=A0 11 files (+1) =A0 - the ROD division is now comprised of=A0 12 files (+1) =A0 - the STS division is now comprised of=A0=A0 4 files (+1) =A0 - the VRT division is now comprised of=A0=A0 6 files (+1) 1.3.3 GSS File Header Problem =A0 GSS sequences at GenBank are maintained in one of two different s= ystems, depending on their origin. One recent change to release processing in= volves the parallelization of the dumps from those systems. Because the seco= nd dump (for example) has no prior knowledge of exactly how many GSS files wi= ll be dumped from the first, it doesn't know how to number it's own output = files. =A0 There is thus a discrepancy between the filenames and file header= s for thirteen GSS flatfiles in Release 141.0. Consider the gbgss92.seq fil= e: GBGSS1.SEQ=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Genetic Sequence Data Bank =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 April 15 2004 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 NCBI-GenBank Flat File = Release 141.0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0 GSS Sequences (Part 1) =A0=A0 88249 loci,=A0=A0=A0 65635323 bases, from=A0=A0=A0 88249 repor= ted sequences =A0 Here, the filename and part number in the header is "1", though t= he file has been renamed as "92" based on the files dumped from the other sys= tem. =A0 We will work to resolve this discrepancy in future releases, but = the priority is certainly much lower than many other tasks. 1.4 Upcoming Changes 1.4.1 **Sequence Length Limitation To Be Removed In June 2004** =A0 At the May 2003 collaborative meeting among representatives of Ge= nBank, EMBL, and DDBJ, it was decided that the 350 kilobase limit on the seq= uence length of database records will be removed as of June 2004. =A0 Individual, complete sequences are currently expected to be a max= imum of 350 kbp in length. One major reason for the existence of this limi= t is as an aid to users of sequence analysis software, some of which might= not be capable of processing megabase-scale sequences. =A0 However, very significant exceptions to the 350 kbp limit have ex= isted for several years; Phase 1 (unordered, unoriented) and Phase 2 (order= ed, oriented) high-throughput genomic sequences (HTGS) generated by effor= ts such as the Human Genome Project; large dispersed eukaryotic genes wi= th an intron/exon structure that spans more than 350 kbp; and sequences which result from assemblies of Whole Genome Shotgun (WGS) project da= ta. =A0 Given these exceptions, and the technological advances which have= made large-scale sequencing practical for an increasing number of research= ers, the collaboration has decided that the 350 kbp limit must be removed. =A0 As of June 2004, the length of database sequences will be limited= only by the natural structures of an organism's genome. For example, a sin= gle record might be used to represent all of human chromosome 1, which is approximately 245 Mbp in length. =A0 Software developers for some of the larger commercial sequence an= alysis packages were recently asked what timeframe would be appropriate for = this change. Answers ranged from "immediately", to "several months", to "o= ne year". So the one-year timeframe was selected, to provide ample time to impl= ement changes which megabase-scale sequences may require. =A0 Some sample records with very large sequences have been made avai= lable so that developers can begin to test their software modifications: =A0=A0=A0=A0=A0=A0=A0 ftp://ftp.ncbi.nih.gov/genbank/LargeSeqs =A0 Many changes are expected after the removal of the length limit. = For example, complete bacterial genomes (typically on the order of severa= l megabases) will be re-assembled into single sequence records. The sub= mission process for such genomes will become much more streamlined, since dat= abase staff will no longer have to split the genomes into pieces. BLAST ser= vices will be enchanced, so that hits reported within very large sequences = will be presented in a meaningful context. =A0 All such changes will be discussed more fully in future release n= otes, the NCBI newsletter, and the GenBank newsgroup. 1.4.2 Rename of File 'Last.Release' and Deletion of /daily Subdirecto= ries The files named Last.Release which are located at: =A0=A0=A0=A0=A0=A0=A0 ftp://ftp.ncbi.nih.gov/genbank/daily/Last.Relea= se =A0=A0=A0=A0=A0=A0=A0 ftp://ftp.ncbi.nih.gov/ncbi-asn1/daily/Last.Rel= ease contain the number of the GenBank release which is currently installe= d on the NCBI FTP site. As of Release 142.0 in June 2004, these files w= ill be moved and renamed as: =A0=A0=A0=A0=A0=A0=A0 ftp://ftp.ncbi.nih.gov/genbank/GB_Release_Numbe= r =A0=A0=A0=A0=A0=A0=A0 ftp://ftp.ncbi.nih.gov/ncbi-asn1/GB_Release_Num= ber The /daily subdirectories, which had been used for cumulative update products that are no longer supported, will be deleted at that time. --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/=A0=A0= =A0=A0=A0=A0 - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbank= b - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb= =A0=A0=A0=A0=A0 - GenBank on the WWW, see:=A0 http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ub= c.ca=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 This e-mail, including any attachments, is a confidential business communic= ation, and may contain information that is confidential, proprietary and/or= privileged. This e-mail is intended only for the individual(s) to whom it= is addressed, and may not be saved, copied, printed, disclosed or used by = anyone else. If you are not the(an) intended recipient, please immediately= delete this e-mail from your computer system and notify the sender. Thank= you. - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ =20 - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb = =20 - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca = =20 From owner-genbankb@hgmp.mrc.ac.uk Mon Apr 26 22:41:06 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 5AD1C7D126 for ; Mon, 26 Apr 2004 22:41:06 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id E84167D321 for ; Mon, 26 Apr 2004 22:41:05 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 0FC9E7D126; Mon, 26 Apr 2004 22:41:05 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 26 Apr 2004 20:23:41 +0100 From: Mark Cavanaugh Subject: GB Release 141.0 Problem : rel141.fsa_aa Message-Id: <20040426214105.0FC9E7D126@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Greetings GenBank Users, The FASTA file for proteins that are annotated as coding regions on GenBank records which was installed on Saturday, April 24th: -r--r--r-- 1 cavanaug gbrel 301055803 Apr 24 22:01 rel141.fsa_aa.gz contained 13,814 duplicated sequences as a result of incorrectly processing one of the classes of data that contributes to the CON division of Release 141.0. This problem was fixed on Monday, April 26th and a new version of the file was installed at about 2:00pm EDT: -r--r--r-- 1 cavanaug gbrel 299081010 Apr 26 14:01 rel141.fsa_aa.gz Our thanks to a user at Novartis for pointing out the problem. The scrutiny of data products provided by GenBank users is greatly appreciated by NCBI. Mark Cavanaugh GenBank NCBI/NLM/NIH/DHHS --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca From owner-genbankb@hgmp.mrc.ac.uk Wed Apr 28 21:04:41 2004 Return-Path: X-Original-To: genbankb-outgoing Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with SMTP id 4AC037D1AD for ; Wed, 28 Apr 2004 21:04:41 +0100 (BST) X-Original-To: genbankb-list@hgmp.mrc.ac.uk Received: from localhost (localhost [127.0.0.1]) by mercury.hgmp.mrc.ac.uk (Postfix) with ESMTP id D98F97D36F for ; Wed, 28 Apr 2004 21:04:40 +0100 (BST) Received: by mercury.hgmp.mrc.ac.uk (Postfix, from userid 60001) id 118417D1AD; Wed, 28 Apr 2004 21:04:40 +0100 (BST) To: genbank@net.bio.net Newsgroups: bionet.molbio.genbank Date: 27 Apr 2004 20:57:25 +0100 From: Mark Cavanaugh Subject: GenBank 141.0 : Release Notes Updated Message-Id: <20040428200440.118417D1AD@mercury.hgmp.mrc.ac.uk> Sender: owner-genbankb@hgmp.mrc.ac.uk Precedence: bulk Greetings GenBank Users, A developer at Accelrys pointed out that the stats in Section 2.2.6 of the 141.0 Release Notes had not been updated since the previous release. This has been corrected, and a new version of gbrel.txt was made available on April 27th at approximately 3:45pm EDT : -r--r--r-- 1 cavanaug gbrel 163450 Apr 27 15:45 gbrel.txt Our apologies for any inconvenience. Mark Cavanaugh GenBank NCBI/NLM/NIH/DHHS --- - gttaacaattaaagagtgtttatcgaaattcattatatagtggtttatatagaccacttc - - GenBank newsgroup see: http://www.bio.net/hypermail/genbankb/ - GENBANKB e-mail: messages sent to genbankb@net.bio.net - subscribe: e-mail biosci-server@net.bio.net with: subscribe genbankb - unsub: e-mail biosci-server@net.bio.net with: unsubscribe genbankb - GenBank on the WWW, see: http://www.ncbi.nlm.nih.gov/Genbank/ - problems with GENBANKB? E-mail moderator: francis@bioinformatics.ubc.ca