Operations

*** RETIRE THIS PAGE -- IT'S OLD ***

Quick daily operation commands here, or read further down for details:

VPSs and Domains

OCL has 3 server environments setup: production, staging, and dev. Each environment has a web application and an API subdomain that can be accessed at either of the parent domains openconceptlab.com and openconceptlab.org.

Environment	Web Application URL	API URL
production	www.openconceptlab.com www.openconceptlab.org	api.openconceptlab.com
staging	staging.openconceptlab.com staging.openconceptlab.org	api.staging.openconceptlab.com
dev	dev.openconceptlab.com dev.openconceptlab.org	api.dev.openconceptlab.com

The subdomain mappings are processed by the nginx/sites-enabled/ocl files on each server. These files are created via the fabric scripts, and the actual values are defined in the fabric file, via the env.web_domain and env.api_domain entries for dev, staging and production.

Operating Environment

The OCL operating environment is as follow:

• All outside requests are handled by nginx; nginx uses named virtualhost to distinguish types of request
• Both API and WEB Django applications run internally on specific ports responding only to localhost request
• Both applications use gunicorn and wsgi
• Supervisor is used to control the Django applications
• Both applications run under the deploy user account

Using fabric to control the server

• Your local machine's public key should be on the server's configuration
• in your environment (typically set in your [virtual_env_project_dir]/bin/postactivate) the following:

export FAB_USER='deploy'
export FAB_PASSWORD=''
export AWS_ACCESS_KEY_ID='blah'
export AWS_SECRET_ACCESS_KEY='blah'

Normal Code Release Process

OCL has 3 operating environments:

• production
• staging
• dev

OCL has two release scripts, one for the Web Application and one for the API:

• release_web_app
• release_api_app

For a normal software release, navigate to the root of the ocl_web repo where the fabfiles live to do the release.

workon ocl_web
fab [:env] [:release_script]

For example:

workon ocl_web
fab dev release_web_app
fab dev release_api_app

The fab file in the oclapi repo still works as well for the api app:

workon oclapi
cd oclapi/src/django-nonrel/ocl
fab dev deploy

Normal Restart using fabric

fab [env] restart_api
fab [env] restart_web

Normal rebuild index using fabric

The following command will trigger a full index rebuild. Note that this will wipe the entire index! Also note that this is a long running command (30+ hours for the full CIEL dictionary). An SSH session will time-out and automatically terminate the process, so this should instead be run locally on the specified server.

fab [env] rebuild_index

Starting/Stopping

Django applications

• To start and stop the Django applications, login to the server and simply use the supervisor control commands using the deployment user account:

supervisorctl stop
supervisorctl start

# or

supervisorctl restart

Note that supervisorctl also has an interactive mode. Just enter supervisorctl. For example:

celery                           RUNNING    pid 27532, uptime 4 days, 16:12:02
ocl_api                          RUNNING    pid 27518, uptime 4 days, 16:12:09
ocl_web                          RUNNING    pid 28651, uptime 4 days, 13:24:22
supervisor> restart celery
celery: stopped
celery: started
supervisor> restart ocl_api
ocl_api: stopped
ocl_api: started
supervisor> exit

Normally, you can just use the fabric commands to start/stop the servers remotely.

Supervisor

The supervisor daemon starts via system startup normally. To start/stop/restart it manually (as root):

/etc/init.d/supervisord start
/etc/init.d/supervisord stop
/etc/init.d/supervisord restart

nginx

The nginx light weight web server starts via system startup normally. To start/stop/restart it manually (as root):

/etc/init.d/nginx start
/etc/init.d/nginx stop
/etc/init.d/nginx restart

Log files

All the log files are currently written to the /var/log/ocl and /var/log directories. See:

• /var/log/ocl/web_app.log for web application normal logging (via Django logger)
• /var/log/ocl/web_debug.log for web application debug logging of API calls to api server
• /var/log/ocl/ocl_api.log for API application normal logging (via Django logger)
• /var/log/ocl/api_supervisor.log for supervisor output for the API gunicorn, sys err, outs basically
• /var/log/ocl/web_supervisor.log for supervisor output for the WEB gunicorn, sys err, outs basically
• /var/log/ocl/ocl_api_batch.log for import process outputs
• /var/log/ocl/celery_supervisor.log for celery processes
• /var/log/mongodb/mongodb.log for record of every query performed

Configuration Files

See:

• nginx
  • /etc/nginx/nginx.conf
  • /etc/nginx/conf.d/*
  • /etc/nginx/sites-enabled/*
• supervisor
  • /etc/supervisor/conf.d/*.conf
• Solr
  • /opt/deploy/solr/collection1/conf/*

Environment Variable/Configurations

Settings for the Django applications are set in the appropriate supervisor configuration files:

• On servers, initial environment variables are set in shell_prep.sh in the home directory of the deploy user

Normally you never need to set these because you operate the server via remote/fabric. However if you are running management commands on the server itself, make sure you source the shell_prep.sh file first.

source ~\shell_prep.sh
# these two commands are just aliases (see .bash_aliases)
activate_web
cd_web
# then:
manage.py run-something