This is an ansible playbook that will configure your datacenter based on multi-OS-platform roles on Ubuntu/Centos/Debian linux and windows servers.
The bootstrap_vm_template.yml playbook is used by vm-templates repo to build VMware Ubuntu, Debian, and Centos templates.
The 'ansible' and 'vm template build' pipelines are both automated using the pipeline-automation-lib jenkins library.
Testing of the linux OS bootstrap playbooks is performed by molecule with platforms defined in 'molecule.yml' and the 'converge.yml'. Molecule scenarios have been set up to include the overall platform provisioning/orchestration role converge playbook for bootstrap_linux as well as multiple key roles invoked within the bootstrap_linux
orchestration role to allow/enable isolated/granular testing when and as needed.
Further details on running molecule tests from this repo can be found in the 'Run molecule tests' section of the molecule/README.md.
The molecule test pipeline is set up in the github actions ci.yml and the molecule converge test results for each platform can be viewed on github actions results page.
The systemd-python enabled docker images used by the molecule tests can be found on dockerhub. The corresponding dockerfile image definitions for the systemd-python enabled docker platform containers used in the molecule tests can be found here.
A companion ansible-developer repository here can be used to bootstrap/set-up an ansible development environment.
The installer shell script from this repo will:
- create the local developer repo directory under $HOME/repos/ansible
- clone the repo into the developer's local repo directory at $HOME/repos/ansible/ansible-developer
- setup/synchronize the developer's bash environment with source bash files located in
files/scripts/bashenv
- source the bash env
For install from public github:
$ INSTALL_REMOTE_SCRIPT="https://raw.githubusercontent.com/lj020326/ansible-developer/main/install.sh" && bash -c "$(curl -fsSL ${INSTALL_REMOTE_SCRIPT})"
The environment setup from the aforementioned repo is utilized to prepare the developer environment to:
- run playbooks
- run molecule testing
- Collection of Ansible roles, playbooks, plugins, and modules
- OS image build systems (packer, vsphere)
- Runtime environment software installs
- Runtime environment application deployments
- Runtime machine instance maintenance
Workflow for ansible provisioning integration with image build systems
graph TD;
A[Packer Build Spec] --> B{Virtual Machine}
B -->|yes| C["Install OS (Centos/Ubuntu/Debian)"]
B -->|no| D[Container Build]
C --> E[Post OS Install - VM Base VM Template Image]
D --> F[Post OS Install - OS Base Container Image]
E --> G["Ansible Provision Role + Harden Security Profile (VM/Cloud/Container)"]
F --> G["Ansible Provision Role + Harden Security Profile (VM/Cloud/Container)"]
G --> H["Ansible Post OS Install - Software Install"]
H --> I[Ansible Application Deploy]
I --> J[Ansible Maintenance]
- Clone this Ansible deployment playbook
git clone https://github.com/lj020326/ansible-datacenter.git
- Setup galaxy collections/roles to be used: *This is internally performed by script if using to run on remote ansible/control node
## install collections
ansible-galaxy collection install -r ./collections/requirements.yml
## install roles
ansible-galaxy install -r ./roles/requirements.yml
- Add host info to hosts.yml inventory and ping the nodes
ansible -i inventory/hosts.yml all -m ping -b -vvvv
-
Create the vault file used to protect important data in source control. For more information go here. Also, see here for an example of the vault file used for this project
The vault file used has to have the name vars/vault.yml.
# create private file ansible-vault create vars/vault.yml
Running the command above will ask you for a password to encrypt with, and open an editor. In that file set the variables highlighted in the vault.yml.example file.
An ansible role is used to setup all docker stack instances.
The bootstrap_docker_stack
ansible role used to stand up the docker stack can be found here.
The bootstrap_docker_stack
ansible role contains the jenkins config-as-code (jcac) yaml definition in template form used to setup the jenkins jcac instance.
The jcac definition can be found here.
The pipeline automation library used can be found here. The pipeline automation library defines the shared jenkins templates that are used throughout all of the jenkins ansible pipelines.
Configure the library in jenkins as seen below.
A root folder for the ansible-datacenter environment can be setup similar to the following.
Setup the ansible vault file credential to be used by the ansible playbook pipeline and passed into every play. If using the aforementioned pipeline-automation-library, make sure the credential ID is 'ansible-vault-pwd-file'.
Then setup folders for each environment defined in the inventory similar to the following.
Each job folder corresponds to a tag defined in the site.yml playbook.
To make setting up each folder consistent and simple as possible, the jobs all are exactly the same except the folder name. The job folders all use the same pipeline definition as seen below. Using this method, whenever a new ansible tag is created, adding a corresponding jenkins job folder is as easy as copying an existing one and naming it respectively to match the newly created ansible tag.
All jobs use the same 2 parameters for the limit hosts directive and debug.
The job history for the tag execution is readily/easily viewable.
Here is the bootstraps linux job history.
Specify host(s) or leave blank to run across all hosts for the group(s) defined for the play(s) associated with the tag.
See the job console for all ansible pipeline input values and play output.
The pipeline job console output.
Another job just created to bootstrap docker stacks onto machines.
Setup jenkins CICD pipeline folders for each repository.
The jenkins CICD branch strategy folder is used to automatically pick up the respective branches and merge strategy. We are using a clone of the public ansible repo here. Once the pipeline is configured with the repo, jenkins will scan the repo branches for the existance of the Jenkinsfile and then setup the corresponding branch folders used to run ansible for each branch.
echo "foobarpass" > ~/.vault_pass
chmod 600 ~/.vault_pass
ansible-playbook report-windows-facts.yml -i inventory/DEV/hosts.yml -t untagged,report-windows-facts --vault-password-file ~/.vault_pass
$ git clone https://github.com/lj020326/ansible-datacenter.git
$ cd ansible-datacenter
$ export MOLECULE_DISTRO=redhat7-systemd-python
$ molecule login
$ molecule --debug test -s bootstrap_linux_package
$ molecule destroy
$ MOLECULE_DISTRO=redhat8-systemd-python molecule --debug test -s bootstrap_linux_package
$ MOLECULE_DISTRO=redhat8-systemd-python molecule login
$ molecule destroy
$ MOLECULE_DISTRO=redhat8-systemd-python molecule converge
$ molecule destroy
$ MOLECULE_DISTRO=centos8-systemd-python molecule --debug converge
$ molecule destroy
$ MOLECULE_DISTRO=ubuntu2204-systemd-python molecule --debug converge
$ MOLECULE_DISTRO=redhat8-systemd-python molecule create
$ MOLECULE_DISTRO=redhat8-systemd-python molecule login
$ molecule destroy
# find the temp dir used for the ansible-local provisioner from the packer log
$ cd /tmp/packer-provisioner-ansible-local/63b193ab-d1c4-b355-f4cf-9e9153570896
$ ansible-playbook bootstrap_vm_template.yml --vault-password-file=~/.vault_pass -c local -i vm_template.yml
$ ansible-playbook site.yml --tags bootstrap --limit dc_os_Ubuntu
$ ansible-playbook site.yml --tags bootstrap-ansible --limit admin02
Note: this is performed from jenkins docker build pipeline and not performed directly using ansible unless necessary The docker image build pipeline source is located here here.
$ ansible-playbook site.yml --tags bootstrap-docker-images --limit admin02
Note: We now use the samba docker container to run the samba server and no longer build on the VM.
$ ansible-playbook site.yml --tags docker-samba-node
$ ansible-playbook site.yml --tags bootstrap-samba-client
$ ansible-playbook site.yml --tags bootstrap-user --vault-password-file ~/.vault_pass
$ run-playbook.sh site.yml -t bootstrap-docker-stack -l admin01
$ run-playbook.sh bootstrap-ntp.yml -l testgroup_lnx
$ ansible-playbook site.yml --tags iscsi-client
ansible -i inventory/hosts.yml openstack -m ping
ansible -i inventory/hosts-openstack.ini openstack -m ping
ansible-playbook site.yml --tags bootstrap-linux --limit os_linux
ansible-playbook site.yml --tags bootstrap-linux-firewalld --limit vmub2201
ansible-playbook site.yml --tags bootstrap-docker --limit docker
ansible-playbook site.yml --tags bootstrap-network --limit node01
ansible-playbook site.yml --tags bootstrap-openstack
ansible-playbook site.yml --tags bootstrap-openstack-deploy-node
ansible-playbook site.yml --tags bootstrap-user --limit ubuntu18
ansible-playbook site.yml --tags openstack-deploy-node
ansible-playbook site.yml --tags openstack-osclient
kolla-ansible -v -i inventory/hosts-openstack.ini bootstrap-servers
kolla-ansible -v -i inventory/hosts-openstack.ini prechecks
kolla-ansible -v -i inventory/hosts-openstack.ini deploy
kolla-ansible -v -i inventory/hosts-openstack.ini post-deploy
working with openstack node cleanup/destroy/reset
kolla-ansible -v -i inventory/hosts-openstack.ini destroy
kolla-ansible -v -i inventory/hosts-openstack.ini destroy --yes-i-really-really-mean-it
working with openstack env setup
scripts/kolla-ansible/init-runonce.sh
ansible-playbook site.yml --tags bootstrap-openstack-cloud
openstack server create --image cirros --flavor m1.tiny --key-name mykey --network demo-net demo1
Other useful plays
ansible-playbook site.yml --tags bootstrap-linux-mounts --limit os_linux
ansible-playbook site.yml --tags bootstrap-linux-mounts --limit dc_os_centos_7
ansible-playbook site.yml --tags bootstrap-linux-mounts --limit postgres
ansible-playbook site.yml --tags bootstrap-linux-mounts --limit media
ansible-playbook site.yml --tags bootstrap-openstack
ansible-playbook site.yml --tags bootstrap-openstack-cloud
ansible-playbook site.yml --tags bootstrap-openstack-deploy-node
Openstack plays
#ansible-playbook site.yml --tags openstack-deploy-node
ansible-playbook site.yml --tags bootstrap-openstack
kolla-ansible -i inventory/hosts-openstack.ini bootstrap-servers
kolla-ansible -i inventory/hosts-openstack.ini prechecks
kolla-ansible -i inventory/hosts-openstack.ini deploy
## running post-deploy creates the /etc/kolla/openrc.sh
## ref: https://github.com/lj020326/kolla-ansible/blob/main/ansible/post-deploy.yml
kolla-ansible -i inventory/hosts-openstack.ini post-deploy
## setup osclient configs if necessary
## NOTE: not necessary to run this since it is included in bootstrap-openstack-cloud play
#ansible-playbook site.yml --tags openstack-osclient
openstack image list
openstack service list
openstack network list
openstack router list
openstack server list
openstack compute service list
openstack dns service list
openstack zone list
## if the above works - then can run custom cloud config
ansible-playbook site.yml --tags bootstrap-openstack-cloud
## to reconfigure kolla-ansible configure based on latest changes
kolla-ansible -i inventory/hosts-openstack.ini reconfigure
## to reconfigure a specific service, e.g., nova, neutron, etc
kolla-ansible -i inventory/hosts-openstack.ini reconfigure --tags nova
docker ps -f name=compute
kolla-ansible -i inventory/hosts-openstack.ini reconfigure --tags neutron
docker ps -f name=neutron
kolla-ansible -i inventory/hosts-openstack.ini reconfigure --tags designate
docker ps -f name=designate
#openstack zone create --email admin@openstack.example.int openstack.example.int.
openstack zone create --email admin@example.int openstack.example.int.
## or per (https://ask.openstack.org/en/question/113699/kolla-ansible-how-to-managemodify-configuration-files/)
kolla-ansible -i inventory/hosts-openstack.ini genconfig ## (and restart manually the containers)
./inventory/openstack_inventory.py --list
## to destroy/reset everything back to the beginning for the inventory:
kolla-ansible -i inventory/hosts-openstack.ini destroy --yes-i-really-really-mean-it
Other useful tests
ansible -u administrator -e ansible_password=${ANSIBLE_SSH_PASSWORD} -m ping ubuntu18
ansible -v -u administrator -e ansible_password=${ANSIBLE_SSH_PASSWORD} -e ansible_pyth/bin/python3 -i inventory/hosts.yml -m ping ubuntu18
ansible-playbook site.yml --tags display-vars -l control01
ansible-playbook site.yml --tags display-domain-vars -l os_linux
ansible-playbook site.yml --tags display-domain-vars -l nas02
ansible-playbook site.yml --tags display-domain-vars -l control01
ansible all -m debug -a var=groups['ca_domain']
ansible-playbook site.yml --tags bootstrap-bind
ansible-playbook site.yml --tags bootstrap-cacerts
ansible-playbook site.yml --tags deploy-cacerts
ansible-playbook site.yml --tags docker-control-node
ansible-playbook site.yml --tags docker-admin-node
ansible-playbook site.yml --tags docker-media-node
gethist | grep remote | uniq >> ./README.md
ansible-config dump
ansible-config dump |grep DEFAULT_MODULE_PATH
ansible-inventory --graph output -i inventory/
ansible-inventory --graph output -i inventory/ ntp
ansible-inventory --graph output -i inventory/ ntp_server
ansible-inventory --graph output -i inventory/DEV/
ansible-inventory --graph output -i inventory/PROD/ ntp
ansible-inventory -i inventory/ --graph ntp
ansible-inventory -i inventory/DEV/ --graph ntp
ansible-inventory -i inventory/QA/ --graph output
ansible-inventory -i inventory/PROD/ --graph output group
ansible-inventory -i inventory/PROD/ --graph output ntp
ansible-inventory -i inventory/PROD/ --list ntp
ansible-inventory -i inventory/PROD/ntp.yml --graph output
ansible-inventory -i inventory/DEV/site1.yml --graph output
ansible-playbook -i ./inventory display-ntp-servers.yml
ansible-playbook -i ./inventory/ display-ntp-servers.yml
ansible-playbook -i ./inventory/ playbook.yml
ansible-playbook -i ./inventory/dmz display-ntp-servers.yml
ansible-playbook -i ./inventory/internal display-ntp-servers.yml
ansible-playbook -i ./inventory/internal display-ntp-servers.yml
ansible all -m debug -a var=groups['ca_domain']
ansible -i inventory/DEV/hosts.yml windows -m debug -a var=ansible_port
ansible -i inventory/DEV/hosts.yml windows -m debug -a var=ansible_winrm_transport
ansible -i inventory/DEV/hosts.yml windows -m debug -a var=ansible_host,ansible_port
Run site-setup play:
ansible-playbook site.yml
Run site-setup play with a tag:
ansible-playbook site.yml --tags docker-media-node
Run plays for specific configuration needed
To run ansible commands from ansible/control node:
ansible -v -m ping
ansible -m ping ubuntu18
Run play for specific node:
ansible-playbook site.yml --tags display-hostvars --limit admin01
ansible-playbook site.yml --tags install-cacerts --limit media01
Run a play for a specific group of nodes:
ansible-playbook site.yml --tags install-cacerts --limit windows
ansible-playbook site.yml --tags install-cacerts --limit dc_os_ubuntu
ansible-playbook site.yml -t display-hostvars -l dc_os_centos
ansible-playbook site.yml -t display-hostvars -l docker
E.g., Run site setup play on control node with a tag from windows/msys shell.
ansible-playbook site.yml --tags bootstrap-ansible
ansible-playbook site.yml --tags bootstrap-bind
ansible-playbook site.yml --tags bootstrap-cacert
ansible-playbook site.yml --tags bootstrap-caroot
ansible-playbook site.yml --tags bootstrap-cicd
ansible-playbook site.yml --tags bootstrap-docker-stack
ansible-playbook site.yml --tags bootstrap-idrac
ansible-playbook site.yml --tags bootstrap-jenkins-agent
ansible-playbook site.yml --tags bootstrap-keyring
ansible-playbook site.yml --tags bootstrap-kvm
ansible-playbook site.yml --tags bootstrap-ldap-client
ansible-playbook site.yml --tags bootstrap-linux
ansible-playbook site.yml --tags bootstrap-linux-core
ansible-playbook site.yml --tags bootstrap-docker
ansible-playbook site.yml --tags bootstrap-linux-firewalld
ansible-playbook site.yml --tags configure-linux-firewall
ansible-playbook site.yml --tags bootstrap-mergerfs
ansible-playbook site.yml --tags bootstrap-ntp
ansible-playbook site.yml --tags bootstrap-openstack
ansible-playbook site.yml --tags bootstrap-openstack-cloud
ansible-playbook site.yml --tags bootstrap-postfix
ansible-playbook site.yml --tags bootstrap-proxmox
ansible-playbook site.yml --tags bootstrap-stepcli
ansible-playbook site.yml --tags bootstrap-user
ansible-playbook site.yml --tags bootstrap-vmware-esxi
ansible-playbook site.yml --tags build-docker-images
ansible-playbook site.yml --tags deploy-cacerts
ansible-playbook site.yml --tags deploy-vm
ansible-playbook site.yml --tags deploy-vsphere-dc
ansible-playbook site.yml --tags display-hostvars
ansible-playbook site.yml --tags docker-admin-node
ansible-playbook site.yml --tags docker-control-node
ansible-playbook site.yml --tags docker-media-node
ansible-playbook site.yml --tags docker-samba-node
ansible-playbook site.yml --tags deploy-nfs-service
ansible-playbook site.yml --tags vmware-remount-datastores
ansible-playbook site.yml --tags upgrade-vmware-esxi
Setup vsphere dc
ansible-playbook site.yml --tags deploy-vsphere-dc
Deploy VMs
ansible-playbook site.yml --tags deploy-vm
Bootstrap VM nodes if needed Note: This is not used any longer since this is now performed from the jenkins pipeline. The jenkins pipeline is responsible for building VM template images using packer. The vm image build pipeline source is located here here.
ansible-playbook site.yml --tags bootstrap-linux --limit admin02
Bootstrap node network config *should not be necessary since this is mostly done in deploy-vm
ansible-playbook site.yml --tags bootstrap-network --limit node01
Docker stack plays
ansible-playbook site.yml --tags docker-admin-node
ansible-playbook site.yml --tags docker-media-node
Useful site.yml
tag based plays to build/update/configure datacenter:
ansible-playbook site.yml --tags bootstrap-bind
ansible-playbook site.yml --tags bootstrap-docker
ansible-playbook site.yml --tags bootstrap-linux-core
ansible-playbook site.yml --tags bootstrap-docker
ansible-playbook site.yml --tags bootstrap-linux-firewalld
ansible-playbook site.yml --tags bootstrap-openstack
ansible-playbook site.yml --tags bootstrap-openstack-cloud
ansible-playbook site.yml --tags bootstrap-user
ansible-playbook site.yml --tags bootstrap-vmware-esxi
ansible-playbook site.yml --tags build-docker-images
ansible-playbook site.yml --tags cacerts-deploy
ansible-playbook site.yml --tags deploy-vm
ansible-playbook site.yml --tags deploy-vsphere-dc
ansible-playbook site.yml --tags display-hostvars
ansible-playbook site.yml --tags docker-admin-node
ansible-playbook site.yml --tags docker-media-node
ansible-playbook site.yml --tags fetch-osimages
ansible-playbook site.yml --tags iscsi-client
ansible-playbook site.yml --tags nfs-service
ansible-playbook site.yml --tags vmware-remount-datastores
ansible-playbook site.yml --tags upgrade-vmware-esxi
Example inventory checks
ansible -v all --list-hosts
ansible -i ./inventory/PROD/hosts.yml -m debug -a var=cacert_keystore_host admin03
ansible -i ./inventory/PROD/hosts.yml -m debug -a var=jenkins_swarm_agent_controller admin01
ansible -i ./inventory/PROD/hosts.yml -m debug -a var=jenkins_swarm_agent_labels admin01
ansible -i ./inventory/PROD/hosts.yml -m debug -a var=jenkins_swarm_agent_controller admin01
ansible -i inventory/ -m debug -a var=service_route_internal_root_domain vcontrol01
ansible -i inventory/ -m debug -a var=ca_domain vcontrol01
ansible -i inventory/ -m debug -a var=service_route_internal_root_domain vcontrol01
ansible -i inventory/hosts.yml -m debug -a var=ca_domain vcontrol01
ansible -i inventory/hosts.yml -m debug -a var=group_names admin01
ansible -i inventory/hosts.yml -m debug -a var=internal_root_domain vcontrol01
ansible -i inventory/hosts.yml -m debug -a var=jenkins_swarm_agent_controller admin01
ansible -i inventory/hosts.yml -m debug -a var=service_route_internal_root_domain vcontrol01
ansible -i inventory/PROD/ -m debug -a var=ansible_host vcenter7
ansible -i inventory/PROD/ -m debug -a var=bootstrap_docker__script_dirs admin03
ansible -i inventory/PROD/ -m debug -a var=bootstrap_docker__swarm_managers admin03
ansible -i inventory/PROD/ -m debug -a var=bootstrap_docker__swarm_remote_addrs admin03
ansible -i inventory/PROD/ -m debug -a var=ca_domain admin01
ansible -i inventory/PROD/ -m debug -a var=ca_domain vcontrol01
ansible -i inventory/PROD/ -m debug -a var=docker_stack_internal_domain admin03
ansible -i inventory/PROD/ -m debug -a var=docker_stack_internal_root_domain admin03
ansible -i inventory/PROD/ -m debug -a var=group_names admin01
ansible -i inventory/PROD/ -m debug -a var=group_names vcenter7
ansible -i inventory/PROD/ -m debug -a var=group_names vcontrol01
ansible -i inventory/PROD/ -m debug -a var=internal_domain vcenter7
ansible -i inventory/PROD/ -m debug -a var=internal_domain vcontrol01
ansible -i inventory/PROD/ -m debug -a var=internal_root_domain vcenter7
ansible -i inventory/PROD/ -m debug -a var=internal_subdomain vcontrol01
ansible -i inventory/PROD/ -m debug -a var=service_route_internal_root_domain admin01
ansible -i inventory/PROD/hosts.yml -m debug -a var=jenkins_swarm_agent_controller admin01
ansible-inventory --help
ansible-inventory -h
ansible-inventory -i ./inventory/PROD/hosts.yml --graph
ansible-inventory -i inventory/PROD/ --graph vmware_vcenter
Example test playbook runs
runme.sh site.yml -t deploy-cacerts -l admin01
runme.sh bootstrap-pip.yml -l admin01
runme.sh -vvv bootstrap-docker.yml -l admin01
runme.sh bootstrap-docker-stack.yml -l docker_stack_jenkins_jcac
runme.sh bootstrap-jenkins-agent.yml -l admin01
Using the run-ansible.sh script to automatically first install all dependencies then run the command
A run-ansible.sh script is available that will upon execution always (1) check and create a virtualenv named 'venv' if not already exists, (2) install pip library requirements, (3) install collection requirements, (4) install role requirements and (5) run the command specified. It also checks in the latest code via git Add/Commit/Push (ACP) before the steps just mentioned. Finally, it also allows specification of a control/jump host to run the playbook via ssh wrapper.
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-ansible-user -l control01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-ansible-user -l media01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-docker-stack -l media01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-linux -l control01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-linux -l media01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-mounts -l media01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-registry -l media01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-user -l control01
run-ansible.sh ansible-playbook -i inventory/PROD/hosts.yml site.yml --tags bootstrap-user -l media01