[TE] WWW Mirrors, proxies and caches

pwoollar at hgmp.mrc.ac.uk pwoollar at hgmp.mrc.ac.uk
Thu Feb 29 04:46:13 EST 1996

The network between many sites is overloaded. There is currently a big
problem for example between the UK and the US. Overloading means that
it takes a long time to view WWW pages or transfer files. There are
several complementary approaches to reducing the problems.

This document is written very much from a UK perspective, and the
detailed information will be most useful to users in the UK. Users
everywhere will, however, also be able to reap great benefits if they
are able to identify and use cache and mirror servers local to them.

WWW Caches

A World-Wide Web proxy cache acts on behalf of a number of Web browsing
clients. Instead of a client (e.g. Netscape) having to fetch documents
itself, it asks the proxy. The proxy fetches the document and returns it
to the client. The proxy may keep a copy of the document and if it is
requested again (by the same or a different client) the proxy can
satisfy that request without having to return to the remote site. This
is of greatest benefit when there are many clients all making use of
the same proxy as the chance that two clients will request the same
document is greatly increased.

The advantage of a WWW cache is that it will save you and your
colleagues lots of time in accessing WWW sites. Frequently accessed
pages stored by the cache will load almost instantaneously instead of
taking minutes (or hours!).

Using a proxy cache will only benefit you if the proxy server is
significantly closer to you than the distant site. Many institutes and
universities already operate a cache service, and we encourage you to
make use of such services if you do not do so already. A related point
is that you don't want to go through a proxy for pages that are nearby,
so you should configure your browser to not make proxy requests for
sites on your own network.

For example, our users have their browsers configured to use our WWW
proxy cache (which others in the UK interested in genomics/molecular
biology are welcome to use)
  e.g. If you have Netscape:
         HTML proxy wwwcache.hgmp.mrc.ac.uk		8080
  e.g. If you have Mosaic/Lynx on Unix:
	 setenv http_proxy http://wwwcache.hgmp.mrc.ac.uk:8080/
Note that users outside the UK will not benefit from using our cache -
you should use one as local as possible. If there is not a suitable
local proxy then you  should point to a national cache,

Our browsers also have "No Proxy" set to ac.uk so that accesses on
SuperJanet go directly, as they are pretty quick anyway. Again, this is
something that will have to be adjusted appropriately.

There is lots more information about WWW Caching at:

You should read this for a more detailed description and how to
configure your setup.

WAIS Proxy 

You may well need one of these as many browsers can't handle WAIS at
all.  We have a WAIS proxy/gateway 

e.g. Netscape:
      WAIS Proxy      wwwcache.hgmp.mrc.ac.uk      8001
e.g. If you have Mosaic/Lynx on Unix:
      setenv wais_proxy http://wwwcache.hgmp.mrc.ac.uk:8001/

WWW Mirror Sites

A WWW mirror site is one which duplicates a WWW service offered
elsewhere. For databases this is very important as a WWW proxy cache
will not (usually) cache database queries or searches, so the only way
to speed those up is to install a mirror server locally. Mirror servers
also act to spread the load so that the primary server doesn't drown
handling all the requests.

There are many WWW mirror sites for important applications and
databases. These will save you even more time than caches. These will
usually be as up to date as the "main" site, although this may vary
after major releases e.g. GDB 6 was available at JHU well before being
available at the mirror sites. The mirrors will also often have links
to local versions of WWW resources referenced within the database.

The intial setting up of these mirrors is often not trivial and routine
updating and maintenance needs to be carried out. So please make use of
them! It staggers me how many sites near mirrors do not make use of

(Some sites will not allow themselves to have services mirrored for
technical or political reasons.)

Below are some of the examples I know about.

1) GDB   

   Original: U.S. John Hopkins http://gdbwww.gdb.org/
      United Kingdom - HGMP Resource Centre  http://www.hgmp.mrc.ac.uk/gdb/docs/gdbhome.html
      The Netherlands - CAOS/CAMM Center  http://www-gdb.caos.kun.nl/gdb/docs/gdbhome.html
      France - INFOBIOGEN  http://www.infobiogen.fr/gdbwww/
      Germany - DKFZ  http://gdbwww.dkfz-heidelberg.de/
      Sweden - Biomedical Center  http://gdb.embnet.se:443/gdb/docs/gdbhome.html
      Israel - Weizmann Institute  http://inherit1.weizmann.ac.il/gdb/docs/gdbhome.html
      Australia - ANGIS http://morgan.angis.su.oz.au/gdb/docs/gdbhome.html
      Australia - WEHI  http://www.wehi.edu.au/gdb/docs/gdbhome.html
      Japan - JICST  http://gdb.gdbnet.ad.jp/

2) Sequence Retrieval System (SRS)

   This contains over 100 different databases at 20 different sites 
   world wide. 
   The actual databases included vary site to site, reflecting local
   priorities, but most of the sites have a core of common databases.

   The following will give you a list of sites supporting WWW-SRS:
   Original:           http://www.embl-heidelberg.de/srs/srsc
   Mirrors:            lots of :-)
     including ours:   http://srs.hgmp.mrc.ac.uk/srs/srsc

3) MGD   (also GXD)

   Original: http://WWW.informatics.jax.org/mgd.html
   Mirror:   http://mgd.hgmp.mrc.ac.uk/

4) The dysmorphic human and mouse homology database

   Original: http://aix-150.ion.bpmf.ac.uk/~cevans/dysmorph.html
   Mirror:   http://www.hgmp.mrc.ac.uk/DHMHD/dysmorph.html

5) Fungal Genetics Stock Center

   Original:  http://www.sky.net/~kcom/kccom/kcomhome.html
   Mirror:    http://www.seqnet.dl.ac.uk/research/fgsc/main.html

6) The Whitehead Institute's STS-Based Map of the Human Genome

   Original: http://www-genome.wi.mit.edu/cgi-bin/contig/phys_map
   Mirror:   http://www.hgmp.mrc.ac.uk/cgi-bin/contig/phys_map

7) Sheepbase

   Original: http://dirk.invermay.cri.nz/
   Mirror:   http://www.ri.bbsrc.ac.uk/sheepmap

8) UNIXhelp

   Original: http://www.ed.ac.uk/~unixhelp/ 
   Mirror:   http://www.hgmp.mrc.ac.uk/Documentation/Unixhelp/TOP_.html
           +many more 

There are lots of other mirrors or equivalent services out there. If
you know of any not listed here, let us know.

Other Mirrors

There are are many software archive mirrors throughout the world,
so if you have difficulty check a local national archive.

There are even some for biological software http://www.ebi.ac.uk/

Peter Tribble  (p.tribble at hgmp.mrc.ac.uk) 
Peter Woollard (p.woollard at hgmp.mrc.ac.uk)

More information about the Bio-www mailing list