Post-EMI deployment.

As of June 2017, the EMI repositories will no longer be available.
This release adds a new submodule for the UMD role and updates the playbooks to use the new role. Several playbooks for grid middleware deployment are now able to be run against containers.

Final Shibboleth-v2 Identity Provider Release

This release contains several contributions, but focusses on the stabilisation of the playbook and roles providing the Shibboleth IdP (V2) functionality. Several fixes have been made to ensure that the Ansible code in the roles is updated to Ansible-2.x syntax. Work has also been done to put in place commerical certificates if they are present, right from the start, and deploying self-signed certificates as a default fall-back approach.

Some work has been done to move the sensitive ldap role variables to the vault-encrypted passwords file for the group (site) that the services fall under.

The release is not "containerised", meaning that there is no use yet of Ansible-Container.

NGI Argus release

This release contains the playbook and role needed to deploy an NGI ARGUS central banning service.

AAROC DevOps pre-release v0.1.0

AAROC DevOps : Executable Infrastructure for Africa and Arabia.

Audience and content

This repository contains the ansible playbooks that are used to deploy Grid services for the
sites in the Africa-Arabia Regional Operations Centre. The playbooks contained in this repository are meant for once-off site deployment, as well as continued site maintenance and upgrades.

This repository is meant for site administrators of sites in the Africa-Arabia Regional Operations Centre. Anyone can use, fork or contribute to it though.

Site Deployment

These playbooks are meant to be run by the site administrator against a site in an initial state defined roughly as the following :

• An inventory of services reflecting that which is typically provided by a site is defined. These are:
  • One Information Service: Site-BDII (Site information index)
  • At least 1 Compute Service: CREAM-CE (Front-end to HPC cluster)
  • At least one Worker Node
  • Recommended also one Data Service: DPM-based SE (Storage Element) for local persistent storage
• Each of the hosts which will run these services must have an Operating System installed. Supported Operating Systems are dictated by the chosen release and version of middleware (see below where the Operating Level Agreement is discussed). Supported OS depend on the include Scientific Linux 5, Scientific Linux 6 and Debian 6.

Important Notes

• Architecture and distribution considerations
  • Only x86_64 architecture is supported.
  • Other RHEL clones, such as CEntOS should work, as long as official repositories are used.
• Networking considerations
  • All hosts, except WN must have public IPv4 addresses and must have reverse DNS resolution
  • The South African National Research Network SANREN runs a network monitor and testing service based on PerfSONAR PS. You can optionally include a perfSONAR endpoint at your site to enable network debuging and performance testing.
  • One of the biggest issues when deploying a grid site is the institutional or local firewall which does not allow communication on required ports. An effort is being made to document these ports in the ansible variables, however if problems arise, the site admin is required to check the service reference card of the respective produtcts.
• Host certificates
  • All services except the BDII and User Interface required a host certificate.
    • During the testing phase, it's possible to use non-accredited certificates from a national CA developed by the ei4Africa project in Kenya, Tanzania, Nigeria, and South Africa. Other CA's may be added in due course.
      • It is, however, recommended to create a Registration Authority (RA) for the site, under the EGI Catch-All Certificate Authority. See http://see-grid-ca.hellasgrid.gr/ra-setup.html

What if I am starting from bare metal ?

These playbooks currently only provide functionality 'from the OS up', and make some assumptions about the initial state of the servers. We understand that there is a need as well for 'bare metal' deployment. For now, we make no playbooks available for that, and it's up to the site administrator to install the base OS (whether on a physical or virtual machine). Use the discussion forum at http://discourse.sci-gaia.eu

Supported Middleware

The default middleware expected to be deployed on AAROC sites is the Monte Bianco release of the European Middleware Initiative. The Unified Middleware Distribution is also supported.

Certification and OLA

In order to be included in production infrastructure, at any level basic critiera must be met by the site. These are referred to in the Operating Level Agreement (OLA).

Participation to external infrastructure

If your site is planning to become part of the production infrastructure for AAROC and is considering exposing resources to international Virtual Organisations supported by EGI.eu then it has to undergo the site certification procedure. This implies that the site signs and accepts the site Operating Level Agreement which describes the minimum number, type and level of services required by an EGI-certified site.

Local, National or Regional participation

If your site is only serving local communities and does not need to be visible to EGI, it can forgo the EGI certification procedure and adopt a slightly-less stringent AAROC certification procedure (under development) with consquent OLA.

Assumptions taken in building these templates:

• Network configurations of all of the nodes is done beforehand. Firewall should be setup to deny all inbound connections except on port 22 - ssh and ntp client ports
• a user named ansible (with a home directory preferably on /ansible) exists on all nodes. The user ansible has sudo rights configured on all nodes
• on all of your nodes the packages redhat-lsb, yum-utils, vim screen, ruby are installed. A further set of required packages will be installed on a per-service basis - see the individual playbooks and handlers
• an ntp service is installed and operational within your region. All grid nodes have ntp clients configured properly
  These are now done in the bootstrap playbook

How to use these playbooks

Site registration - manual steps

These playbooks are developed to automate as far as possible site deployment. However, there are some manual steps which have to done first.

The EGI Resource Centre integration procedure is a good starting point. Once you have completed the initial procedure to register your site in the GOCDB, you will be ready to use this code.

Quickstart guide

The basic workflow for deploying a site in the ROC using Ansible is as follows :

1. Define your inventory by adding a file to the top-level directory (e.g. za-meraka.inventory)
2. Define the variables for your site by adding a file in the group_vars directory (e.g. za-meraka)
3. run the bootstrap and preflight test playbooks to see if you can prepare the machines properly.
4. Choose the services you want to configure at your site in the main playbook (playbook.yml)
5. run the playbook - this will install and configure everything necessary
6. let the operations coordinator know that your site is ready and needs to be included in the GOCDB
7. proceed to create the roles and tasks for your site.

A detailed guide is in each of the roles of the playbooks.

AAROC DevOps pre-release

This release contains the Ansible modules required for deploying small HPC clusters integrated with EMI middleware. Several bugs have been fixed and Puppet modules for CVMFS and APEL have been added by @kalvaer.

Ansible LDAP/Shibboleth Role Release

Pre-release v0.0.3

In this release, we have new functionality for deploying a fully-integrated Shibboleth identity provider, along with LDAP backend. The idp-ldap.yml playbook configures both services, and delpoys the web frontend for the Shibboleth IDP on the identity provider.

New Functionality

The playbooks support RedHat 6 clones (CentOS) and Debian 6 (including Ubuntu). Variables for these OSs can be found in group_vars/{{ ansible_os_family }}.yml

New functionality with respect to the previous version includes:

1. Shibboleth Identity provider deployment
2. LDAP integration with IdP
3. Web frontend (IDPPublic) deployment and integration

New Ansible Roles

New Ansible roles have been developed and included in this release

1. fmarco76.tomcat : provisions the tomcat instance for Shibboleth.
2. fmarco76.firewall : applies the correct iptables for the site services
3. fmarco76.IDPPublic : deployes the web mnanagement interface
4. shibboleth-idp : provisions the Shibboleth identity provider

Using this release

The idp-ldap.yml playbook will configure the services at your site, on hosts defined in your inventory. You need to specify certain site-specific variables along with the inventory :

---
server_country: 
server_state: 
server_location: 
server_organization: 
organisation: 
mail_contact:
useradmin_password: 
ldap_server:

These are used to configure the ldap and shibboleth integration.

Testing and Feedback

This has been tested against the dev site at INFN Catania and the ZAMREN site in Zambia. Please open tickets if there are any issues.

AAROC Ansible LDAP role release

This release sees the publication of the new LDAP role for Shibboleth Identity Providers.

AAROC DevOps Ansible

Pre-production release

This is the pre-production release of the AAROC DevOps Ansible. Most roles have been added, only missing DPM.

v0.0.1

this release is just to check Zenodo integration.