ARGOS SYSTEM MANAGEMENT alpha 0.7 release; Don Gilbert; March 2004 This is a brief introduction to the unix system functions we use to maintain Argos web/database servers. These are still in development, and need to be used and adjusted to suit local servers. They are located from the @ARGOS_ROOT@ folder. Available documents for installing FlyBase-NG, euGenes, and related packages are at http://flybase.net:8081/ OR http://eugenes.org:8081/ These documents have details of installation and operation install/Readme-current.txt install/argos-system-management.txt These have background information on Argos docs/gmod-argos-sep03.pdf (slide show) docs/argos-ms2.doc.pdf (draft report) Read first the ARGOS install/README.txt for INSTALLATION NOTES For a simple server, which is locally used, several of the below options can be skipped. The main need is for periodic data updates. For heavily used servers, checking and restarting as needed are necessary. SERVER PACKAGES The Argos system consists of several independent software packages, each with particular configuration files that say what file system paths, ports, environ variables are important to it, and which need to be changed to accomodate each local computer enviroment. For average use, these are the current services which need to run as stand-alone processes: one Apache web server for main web pages, one Tomcat servlet server, with a planned addition of a PostgreSQL server. Each of these servers has one or more Unix port numbers, usually in the medium unused range (7000-8000) which listen for incoming client calls. The Tomcat and Postgres ports will only be used locally and should not be a security concern. The Apache web server ports are open to public access by default configuration; local policy can change this. CONFIGURATION The install/install.cgi script is the best starting point. It shows basic steps, where you pick those packages desired (e.g. FlyBase) and a root file path for installation. The forms will generate two primary configuration files: (1) the install.conf.local configuration file, with values of paths, ports, and other variables (2) a crontab entry for running software and automating updates The primary configuration file is install/install.conf.local, which is the localized version of install.conf. This has settings for the major configurations that need path and port edits, including Apache web server, Tomcat web server, with Postgres server to be added. Other software at this time will work without additional local configurations. Main service configurations are now located in ROOT/conf/ folder, but there are also service-specific configurations in each 'service'/conf folder, such as flybase/conf/apache.conf, tomcat-server.xml, gmod.conf, etc. As of Feb 2004, the primary install/installargos.pl script has a -reconfigure option, but this is not fully functional. Major configuration changes still need to be handled by a person. The replication from reference server will not overwrite local configurations, but neither will it update them as needed. Management and replication of these configurations is an ongoing development issue. Checks of available multi-platform, multi-service configuration management methods, in particular the GridConfig system for NSF Middleware Initiative, point to helpful directions but not to implementable answers yet. We would like to have a configuration management system where the reference services provide and update configurations, while local instantiations maintain necessary paths, ports and policy configurations. This is complicated because of the muliple software configuration formats needed to support a system build on available components. SERVER OPERATION The scripts in ROOT/bin/ are the primary location for management tools. These check configurations and run the various server software programs, located usually in common/servers/ The ROOT/bin/ run-apache2, run-tomcat2 and run-postgres2 scripts start these, reading port numbers and paths from conf/ files [Version 6 note: install/run- scripts will be replaced by these]. Other functions, such as data search and reporting, BLAST, Gbrowse, are handled as CGI or Servlet functions from these three servers. As of March 2004, the ROOT/bin/run-postgres2 script is working and can be used with GMOD Chado database. A sample installation of this is in the gmod/ package. For high-load use, we have several replicated Argos servers, and use the Apache mod_backhand configurations to proxy web calls from main server to worker servers. This requires local configuration in the ROOT/conf/apache* files, with IP addresses and extra port numbers. The heavily used FlyBase.net server has required tuning of various kinds to keep run-away web clients, misbehaving robots and others from causing outages. In course of this, the single main apache server was split into two main/root apache instances, each with its own configurations. For this and for the Tomcat servlet server, the Argos run scripts now support multiple instances and configurations depending on need. PROBLEMS WITH CONFIGURATION This is still an area where automation is a moving target, and likely will need human attention. The main install step 'install/install -install -v' is supposed to generate proper local configuration files, after fetching all the software and data packages. If this step is aborted or otherwise failed, that won't be done, and even if done, may need hand-tuning of some config. files. This command will list out the automated steps for reconfiguring and the config. files to be edited: install/install -reconfig -v -n You can run this without '-n' to do the edits, using primary configuration information from install/ files: install.conf.local == values of path, ports for local system reconfigure.conf == what files and parts to edit Automated updates are set to exclude the local configuration files, however updating these to match changes in the reference configurations is still problematic (and being worked on). There is a set of 'conf.default' files which are replicated, and can be used as reference for local updates. ENVIRON Many Argos scripts use Unix ENV variables to find paths, port numbers, etc. This script is useful for setting environ variables in a shell (csh/tcsh/sh/bash) for working with other scripts. It is also used by below 'ROOT/bin/run-xxxx' scripts. source `ROOT/bin/argos-env -set` [csh/tcsh] . `ROOT/bin/argos-env -set` [sh/bash] There is also a module in progress for Perl (CGI, database) functions that will likewise load Argos configurations (common/perl/lib/Argos/Config.pm) for each service as needed. It does essentially the same a argos-env shell script. SYSTEM STARTUP This script can be adapted to start the Argos servers at system restarts usually by placing (after editting paths) in system /etc/init.d daemon start folder dont forget to 'ln -s ../init.d/argosweb /etc/rc3.d/S50argosweb' or equivalent ROOT/bin/argosweb-start.sh DATA UPDATES and SERVER RESTARTS Using the 'install/install' script is currently the only way to properly update an Argos mirror server against reference copies. It uses configurations in the install/ folder for package sources and local paths. Crontab entries such as these will handle daily Argos/FlyBase-NG updates and server restarts. A better server restart/check mechanism is in development still and will change as likely will the update mechanism. Please edit @ARGOS_ROOT@ below to the real path on your system. #---------------- # argos - update packages that have been installed, do before restarts below 18 2 * * * cd @ARGOS_ROOT@; [ -x install/install ] && install/install -update -rsopts=-v -v > logs/upd.log 2>&1 # argos - restart apache web server (theraputic, usually not needed) 38 3 * * * cd @ARGOS_ROOT@; [ -x install/run-apache ] && install/run-apache restart > /dev/null # argos - restart tomcat web server (theraputic) 38 3 * * * cd @ARGOS_ROOT@; [ -x install/run-tomcat ] && install/run-tomcat restart > /dev/null # clean out temporary files regularly for each service you have installed 10 2 * * * cd @ARGOS_ROOT@; find flybase/tmp/ -mtime 1 -type f -exec /bin/rm {} \; > /dev/null 2>&1 ## these start variants are still being tested: # 30 3 * * * cd @ARGOS_ROOT@; [ -x ROOT/bin/run-apache2 ] && ROOT/bin/run-apache2 restart > /dev/null # 35 3 * * * cd @ARGOS_ROOT@; [ -x ROOT/bin/run-tomcat2 ] && ROOT/bin/run-tomcat2 restart > /dev/null ## postgres server replication is still in test phase, not deployed ## 40 3 * * * cd @ARGOS_ROOT@; [ -x ROOT/bin/run-postgres2 ] && ROOT/bin/run-postgres2 restart > /dev/null #------------------- SERVER WATCHDOG This script can be run as a continual background watchdog to restart apache server in cases of failures and system/web overloads, which we get on occasion at the very busy flybase.net. It basically uses '/bin/ps', netstat and a few other system calls to check server stability and load level. We have only tested it at flybase.net so far. It is run as root, as the apache server at flybase.net is run so. It should be tested at each system, configured as needed. ROOT/bin/argos-checker.pl [help is printed without options] # an example use is ROOT/bin/argos-checker.pl --max=10800 [ run for 7 days; use really big value to run longer ] --sleep=60 [ check each 1 minute ] --overload=20 --notice=6 [ system load levels to respond to ] --mail=root [ who to email about problems] --logfile=/var/tmp/argos-checker.log [ where to write problem log ]