Skip to content
This repository has been archived by the owner on Mar 30, 2023. It is now read-only.

Latest commit

 

History

History
130 lines (128 loc) · 11.6 KB

INSTALL_UBUNTU18.md

File metadata and controls

130 lines (128 loc) · 11.6 KB
Raw

devstats installation on Ubuntu

Prerequisites:

  • Ubuntu 18.04.
  • golang.
    • apt-get update
    • sudo locale-gen "en_US.UTF-8"
    • Add to /etc/environment:
    LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
    • Check if language/locale settings are ok: perl -e exit.
    • apt install golang - this installs Go 1.10.1.
    • apt install git psmisc jsonlint yamllint gcc
    • mkdir /data; mkdir /data/dev
  1. Configure Go:
    • For example add to ~/.bash_profile:
    GOPATH=/data/dev; export GOPATH
PATH=$PATH:$GOPATH/bin; export PATH
    • Logout and login again.
    • golint: go get -u golang.org/x/lint/golint
    • goimports: go get -u golang.org/x/tools/cmd/goimports
    • goconst: go get -u github.com/jgautheron/goconst/cmd/goconst
    • usedexports: go get -u github.com/jgautheron/usedexports
    • errcheck: go get -u github.com/kisielk/errcheck
    • cors: go get -u github.com/rs/cors
    • jsoniter: go get -u github.com/json-iterator/go
    • If you want to use ElasticSearch output: elastic: go get -u github.com/olivere/elastic.
  2. Go to $GOPATH/src/github.com/cncf and clone devstats and devstatscode there:
    • git clone https://github.com/cncf/devstats.git.
    • git clone https://github.com/cncf/devstatscode.git.
    • cd devstats.
    • Set reuse TCP connections: ./cron/sysctl_config.sh
  3. If you want to make changes and PRs, please clone devstats and devstatscode from GitHub UI, and clone your forked version instead, like this:
    • git clone https://github.com/your_github_username/devstats.git
    • git clone https://github.com/your_github_username/devstatscode.git
  4. Go to devstatscode directory, so you are in ~/dev/go/src/github.com/cncf/devstatscode directory and compile binaries:
    • make
  5. If compiled sucessfully then execute test coverage that doesn't need databases:
    • make test
    • Tests should pass.
  6. Install binaries:
    • sudo mkdir /etc/gha2db
    • sudo chmod 777 /etc/gha2db
    • sudo make install
  7. Go to devstats directory, so you are in ~/dev/go/src/github.com/cncf/devstats and install scripts, metrics and other config/data files:
    • make install
  8. Install Postgres database (link):
    • Make sure you database uses en_US.UTF-8 locale and collation.
    • apt install postgresql (you can use specific version, for example postgresql-9.6)
    • devstats repo directory must be available for postgres user. chmod -R ugo+r /data/.
    • sudo -i -u postgres, psql and as root sudo -u postgres psql to test installation.
    • Postgres only allows local connections by default so it is secure, we don't need to disable external connections:
    • Config file is: /etc/postgresql/10/main/pg_hba.conf, instructions to enable external connections (not recommended): http://www.thegeekstuff.com/2014/02/enable-remote-postgresql-connection/?utm_source=tuicool
    • You need to set something like host all all 1.2.3.4/32 md5 in this file and also listen_addresses = '*' in postgresql.conf.
    • Set bigger maximum number of connections, at least 4x number of your CPU cores or more: /etc/postgresql/10/main/postgresql.conf. Default is 100. max_connections = 300.
    • You can also set shared_buffers = ... to something like 25% of your RAM. This is optional.
    • service postgresql restart or systemctl restart postgresql@10-main.
    • ./devel/set_psql_password.sh to set postgres user password.
    • chown -R postgres /var/log/postgresql.
  9. Setup GitHub OAuth
    • You need to have GitHub OAuth token, either put this token in /etc/github/oauth file or specify token value via GHA2DB_GITHUB_OAUTH=deadbeef654...10a0 (here you token value)
    • If you want to use multiple tokens, create /etc/github/oauths file that contain list of comma separated OAuth keys or specify token values via GHA2DB_GITHUB_OAUTH=key1,key2,...,keyN
    • If you really don't want to use GitHub OAuth2 token, specify GHA2DB_GITHUB_OAUTH=- - this will force tokenless operation (via public API), it is a lot more rate limited than OAuth2 which gives 5000 API points/h
    • If you set GHA2DB_GHAPISKIP=1 all GitHub API calls will be skipped. You can set GHA2DB_GHAPISKIP=1 then, because artificial events cleanup is not needed when GitHub API is not needed. If both those variables are set, ghapi2db won't be called at all.
    • If your project(s) use icons (some of them has value other than ICON="-", then you need to clone CNCF artwork repo into ~/dev/cncf/artwork: cd ~/dev/cncf/, git clone https://github.com/cncf/artwork.git.
    • You can have artwork elsewhere, then you must use ARTWORK=/path/to/artwork/repo. If all projects use ICON="-" artwork is not needed.
    • You also need ImageMagick's convert utility: apt install imagemagick.
    • You need to have /var/www/html/img directory available for deploy user.
  10. Install Grafana.
    • Go to: https://grafana.com/grafana/download.
    • wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.x.x_amd64.deb.
    • sudo dpkg -i grafana_5.x.x_amd64.deb
  11. Install Apache & SSL (You need to have a working DNS name for this). This is optional.
    • apt-get install apache2.
    • Create "web" directory: mkdir /var/www/html/ (it will hold gha databases dumps and other static info on the main domain.)
    • chmod -R ugo+rwx /var/www.
    • Enable mod proxy and mod rewrite:
    • ln /etc/apache2/mods-available/proxy.load /etc/apache2/mods-enabled/
    • ln /etc/apache2/mods-available/proxy.conf /etc/apache2/mods-enabled/
    • ln /etc/apache2/mods-available/proxy_http.load /etc/apache2/mods-enabled/
    • ln /etc/apache2/mods-available/rewrite.load /etc/apache2/mods-enabled/
    • ln /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled/
    • You can enable SSL, to do so You need to follow SSL instruction in SSL (that requires domain name).
    • service apache2 restart
    • Test via: apt install links, links 127.0.0.1
    • If you don't use Apache/SSL proxy, you should add EXTERNAL=1 when deploying - this will expose grafana to outside world by binding to 0.0.0.0 instead of local only 127.0.0.1. See devel/create_grafana.sh.
  12. Automatic deploy of first database (few examples)
    • You should probably use last two command examples or tweak any of commands listed below.
    • Top deploy one of CNCF projects see projects.yaml chose one projectname and specify to deploy only this project: ONLY=projectname.
    • Use INIT=1 to specify that you want to initialize logs database devstats, you need to provide passowrd for admin PG_PASS, ro_user (PG_PASS_RO used by Grafana for read-only access) and devstats_team (PG_PASS_TEAM - this is just another role to allow readonly access from the bastion ssh server).
    • You must use INIT=1 when running for the first time, this creates shared logs database and postgres users.
    • This will generate all data, without fetching anything from backups: INIT=1 ONLY=projectname PG_PASS=... PG_PASS_RO=... PG_PASS_TEAM=... ./devel/deploy_all.sh
    • You shoud add SETPASS=1 on the first run to set main postgres database user password. Thsi requires user interaction, so it shouldn't be used from automatic scripts.
    • If you do not specify ONLY="project1 project2 ... projectN" it will deploy all projects defined in projects.yaml.
    • Use SKIPWWW=1 to skip Apache/SSL config (which requires DNS name for a server), the final result will be Grafana via HTTP on port 3xxx on accessible by server IP or name.
    • Example with skipping GitHub API and custom ARTWORK: ARTWORK=/data/artwork SKIPWWW=1 GHA2DB_GITHUB_OAUTH=- GHA2DB_GHAPISKIP=1 IGET=1 GET=1 ONLY=someproj PG_PASS=... ./devel/deploy_all.sh
    • Other possible env variables used for automatic deploy are in the comments section in a first lines of deploy files: devel/deploy_all.sh devel/deploy_proj.sh devel/create_databases.sh devel/init_database.sh devel/create_grafana.sh devel/create_www.sh all/add_project.sh.
    • Example command that worked on a clean Ubuntu 18 without domain name: EXTERNAL=1 SKIPTEMP=1 ARTWORK=/data/artwork INIT=1 PG_PASS=p PG_PASS_RO=p PG_PASS_TEAM=p ONLY=cncf GET=1 SKIPWWW=1 GHA2DB_GITHUB_OAUTH=- GHA2DB_GHAPISKIP=1 GHA2DB_PROJECTS_OVERRIDE="+cncf" ./devel/deploy_all.sh.
    • Assuming you have GitHub OAuth file and artwork is in the default localtion and there is no domain name: EXTERNAL=1 SKIPTEMP=1 INIT=1 PG_PASS=p PG_PASS_RO=p PG_PASS_TEAM=p ONLY=cncf GET=1 SKIPWWW=1 GHA2DB_PROJECTS_OVERRIDE="+cncf" ./devel/deploy_all.sh.
    • You can also take a look at ADDING_NEW_PROJECT.md file for more info about setting up new projects.
    • Assuming that you deployed cncf project (like in the last example), you should see response using links 127.0.0.1:3255. You should albo be able to access your Grafana from outside world via: http://ip:3255
    • Now when postgres users are created, you test all stuff that require databases (on cncf/devstats repo): PG_PASS=p make.
    • Tests should pass.
  13. To enable Continuous deployment using Travis, please follow instructions here.
  14. We need to setup cron job that will call sync every hour (10 minutes after 1:00, 2:00, ...)
    • Setup devstats, webhook (to handle CI/CD deployments) and cron_db_backup_all.sh cron jobs, similar to this:
    7 * * * * PATH=$PATH:/path/to/GOPATH/bin PG_PASS="..." devstats 2>> /tmp/gha2db_sync.err 1>> /tmp/gha2db_sync.log
30 3 * * * PATH=$PATH:/path/to/GOPATH/bin cron_db_backup_all.sh 2>> /tmp/gha2db_backup.err 1>> /tmp/gha2db_backup.log
*/5 * * * * PATH=$PATH:/path/to/GOPATH/bin GOPATH=/go/path GHA2DB_CMDDEBUG=1 GHA2DB_PROJECT_ROOT=/path/to/repo GHA2DB_DEPLOY_BRANCHES="production,master" PG_PASS=... GHA2DB_SKIP_FULL_DEPLOY=1 webhook 2>> /tmp/gha2db_webhook.err 1>> /tmp/gha2db_webhook.log
    • First crontab entry is for automatic GHA sync.
    • Second crontab entry is for automatic daily backup of postgres databases.
    • Third crontab entry is for Continuous Deployment - this a Travis Web Hook listener server, it deploys project when specific conditions are met, details here.
    • You need to change "..." PG_PASS to the real postgres password value and copy this line.
    • You need to change "/path/to/your/GOPATH/bin" to the value of "$GOPATH/bin", you cannot use $GOPATH in crontab directly.
    • Run crontab -e and put this line at the end of file and save.
    • Cron job will update Postgres database at 0:08, 1:08, ... 23:08 every day.
    • It outputs logs to /tmp/gha2db_sync.log and /tmp/gha2db_sync.err and also to gha Postgres devstats database: into table gha_logs.
    • Check database values and logs about 25 minutes after full hours, like 14:25:
    • Check max event created date: select max(created_at) from gha_events and logs select * from gha_logs order by dt desc limit 20.

More details