.. _development: Development =========== This section covers procedures and information related to development of *Majic Ansible Roles*. Preparing environment --------------------- The easiest way to get going with role development is to set-up a separate Python virtual environment with the necessary packages. This can be done by performing the following steps: 1. Ensure that the following minimum set of packages are installed via distribution package manager: - `Git `_ - `libffi `_ runtime and development package. - `OpenSSL `_ runtime and development package. - `pip `_ - `virtualenv `_ - `virtualenvwrapper `_ - Development packages for Python. On Debian this can be easily done with:: apt-get install virtualenv virtualenvwrapper git python-pip python-dev \ libffi-dev libssl-dev 2. In order to be able to run role tests, it is necessary to install `VirtualBox `_ and `Vagrant `_, using instructions outlined on their respective websites. It is recommended to use latest versions available. At time of this writing the role tests have been successfully run on *VirtualBox 5.2.12* and *Vagrant 2.0.4*. 3. Clone the git repository:: git clone https://code.majic.rs/majic-ansible-roles/ ~/projects/ 4. Create a separate Python virtual environment:: mkvirtualenv majic-ansible-roles -a ~/projects/majic-ansible-roles/ 5. Make sure the virtual environment has been activated, and install `pip-tools `_:: workon majic-ansible-roles pip install pip-tools 6. Synchronise Python virtual environment with requirements file using **pip-tools**:: workon majic-ansible-roles pip-sync Development conventions ----------------------- In order to maintain consistency across different roles and documentation, this section describes development conventions that should be followed while making modifications. Task specifications ~~~~~~~~~~~~~~~~~~~ When writing new and updating existing tasks, keep the following in mind: - Quote sensibly. If specifying paths (for example ``src``, ``dest``, ``path`` etc in various models), quote the string to make it stand-out better and to avoid breakages. - Avoid usage of ``set_facts`` task when same functionality can be achieved via ``defaults/main.yml``. - When specifying tasks, use the fully expanded form. Do not use single-line form with ``param=value``. - When specifying ``command`` or ``shell`` tasks, in case a ``creates`` parameter or such need to be used, specify them as part of task's ``args`` parameter. E.g.:: - name: Run command command: mycommand args: creates: "/etc/mycommand" - When sepcifying tasks, keep the following ordering between different task parameters: - ``name`` - Module and its parameters. - ``become`` - ``become_user`` - ``when`` - ``with_items`` / ``with_dict`` / ``with_nested`` - ``wait_for`` - ``register`` - ``changed_when`` - ``failed_when`` - ``no_log`` - ``notify`` - Task tags. Running role tests directly --------------------------- Role tests are implemented using `Molecule `_, `Testinfra `_, `VirtualBox `_ and `Vagrant `_. *Molecule* and *Testinfra* are installed inside of Pyhton virtual environment, while *VirtualBox* and *Vagrant* need to be installed distribution-wide, following instructions outlined on their corresponding websites. Tests can be run directly for a single role, or for one or more roles using a dedicated shell script (see below). The shell script can also be used for generating reports in an automated environment. In order to run tests for a specific role, perform the following steps: 1. Switch to Python virtual environment:: workon majic-ansible-roles 2. Change directory:: cd roles/ROLENAME/ 3. Run the default test scenario (this will normally test against multiple Debian versions if supported):: molecule test Running role tests via shell script ----------------------------------- In order to make it easier to run tests for all roles, and eventually produce reports of such runs, a dedicated shell script is provided for running the tests. In order to run tests, perform the following steps: 1. Switch to Python virtual environment:: workon majic-ansible-roles 2. Make sure you are within the root directory of Git repository. 3. Run tests for all roles and generate report:: ./scripts/run_tests.sh -r all .. note:: Optionally you can run tests for a specific set of roles, or without generating the report, for example ``./scripts/run_tests.sh web_server common`` 4. Check results either from script output, or within directory ``test_report-YYYY_MM_DD-hh_mm_ss``. For overview of what roles have failed, have a look at ``summary.txt``. For details have a look at each role's individual report. Porting to Molecule 2.4.x and Testinfra 1.7.1 --------------------------------------------- The following instructions were applicable during the porting to new Molecule and Testinfra version. The transition made was: - Molecule 1.25.0 to Molecule 2.4.0. - Testinfra 1.5.5 to Testinfra 1.7.1. Instructions hold only some historical value. For each role perform the following steps: 1. Go to role directory:: workon majic-ansible-roles cd roles/ROLE_NAME 2. Create default scenario:: molecule init scenario -s default -d vagrant -r ROLE_NAME 3. Remove unneded files:: rm molecule/default/INSTALL.rst 4. Move all tests into new location:: rm -rf molecule/default/tests/ git mv tests molecule/default/ 5. Port the old ``molecule.yml`` configuration into ``molecule/default/molecule.yml``. 1. A good starting template for new Molecule configuration file would be:: --- dependency: {} driver: name: vagrant provider: name: virtualbox lint: name: yamllint platforms: - name: helper box: debian/contrib-jessie64 memory: 512 cpus: 1 interfaces: - auto_config: true ip: 10.31.127.2 network_name: private_network type: static - name: parameters-mandatory-jessie64 groups: - parameters-mandatory box: debian/contrib-jessie64 memory: 256 cpus: 1 interfaces: - auto_config: true ip: 10.31.127.3 network_name: private_network type: static - name: parameters-optional-jessie64 groups: - parameters-optional box: debian/contrib-jessie64 memory: 256 cpus: 1 interfaces: - auto_config: true ip: 10.31.127.4 network_name: private_network type: static - name: parameters-mandatory-stretch64 groups: - parameters-mandatory box: debian/contrib-stretch64 memory: 256 cpus: 1 interfaces: - auto_config: true ip: 10.31.127.5 network_name: private_network type: static - name: parameters-optional-stretch64 groups: - parameters-optional box: debian/contrib-stretch64 memory: 256 cpus: 1 interfaces: - auto_config: true ip: 10.31.127.6 network_name: private_network type: static provisioner: name: ansible config_options: ssh_connection: pipelining: "True" lint: name: ansible-lint scenario: name: default verifier: name: testinfra lint: name: flake8 2. Ensure to define additional hosts for different platforms (as supported by the role). If only a single platform is supported, still perform the rename. 3. Make sure the IP addresses are not coliding. 4. Make sure to add hosts to correct groups that will correspond to groups within ``playbook.yml``. 5. Remove old Molecule configuration:: git rm molecule.yml 6. Move the old playbook into new location:: rm molecule/default/playbook.yml git mv playbook.yml molecule/default/playbook.yml 7. Update the playbook: 1. Move idempotent parts of ``molecule/default/playbook.yml`` into ``molecule/default/prepare.yml``. Only the role-related set-up should remain. .. warning:: Don't forget to add the ``become: yes`` to tasks. 2. Update IP address information where necessary. Especially important in case of multiple Debian versions. 8. Update tests to use correct fixtures and Ansible inventory:: sed -i -e 's/Ansible(/host.ansible(/g;s/File(/host.file(/g;s/Group(/host.group(/g;s/LocalCommand(/local.run(/g;s/Package(/host.package(/g;s/PipPackage(/host.pip_package(/g;s/Service(/host.service(/g;s/Socket(/host.socket(/g;s/Sudo(/host.sudo(/g;s/User(/host.user(/g;s/Command(/host.run(/g' molecule/default/tests/test_*.py sed -i -r -e 's/^(def test.*)\(.*\):/\1(host):/' molecule/default/tests/test_*.py sed -i -e "s#'.molecule/ansible_inventory'#'.molecule/ansible_inventory.yml'#" molecule/default/tests/test_*.py 9. Start fixing the tests: 1. Locate all instances of ``local.run``, and ensure the ``local`` variable is initialised once prior to use in each test. Commands can be found with:: grep 'local.run' molecule/default/tests/test_*.py Snippet to insert:: local = host.get_host("local://") 2. Run linting checks, fixing any ensuing issues along the way:: molecule lint 3. Bring-up the machines, fixing any ensuing issues along the way:: molecule converge 4. Run idempotency test, fixing any ensuing issues along the way:: molecule idempotence 5. Figure out how to fix IP parametrisation of IPs in case of multiple (old) platforms, e.g. when having same machines with multiple debian versions. Good way to identify such tests:: grep -ri 10.31 molecule/default/tests/test_*.py 6. Figure out how to fix hostname parametrisation in case of multiple (old) platforms, e.g. when having same machines with multiple debian versions. Good way to identify such tests:: grep -ri parameters- molecule/default/tests/test_*.py 7. Run tests, fixing any ensuing issues along the way:: molecule verify 8. Perform one more relatively fast verification that things work fine:: molecule lint && molecule idempotence && molecule verify 9. Check the number of defined tests against number of tests performed. Keep in mind some tests are run for multiple machines, and refer to each test file to see what hosts will get loaded:: grep -c 'def test_' molecule/default/tests/test_* 10. Run the full test suite to ensure everything works correctly:: molecule test 10. Verify and commit the changes. Template commit message:: MAR-128: Upgraded tests for ROLE_NAME role: - Switch to new Molecule configuration. - Updated set-up playbook to use become: yes. - Moved some preparatory steps outside of the main playbook (eases idempotence tests). - Updated tests to reference the yml inventory file. - Updated tests to use new fixture (host instead of individual ones). - Switched to extracting IP address instead of hard-coding it in a couple of tests. .. _testsite: Test Site --------- *Majic Ansible Roles* comes with a small sample test site configuration which demonstrates use of every role. This test site also serves as starting point for developing new roles etc, and can be used for testing regressions/breakages. The test site covers everything, starting from generating the Debian preseed files, through bootstrap process for new nodes, and onto deployment of all remaining roles. By default, the test site uses domain ``example.com``, but it has been designed so it is easy to set your own domain (see below in step-by-step instructions). Some changes may be necessary to listed commands in that case (i.e. replace every occurance of ``example.com`` with your own domain). All example commands listed within this section should be ran from within the ``testsite`` directory in order to have proper environment available for playbook runs. A number of playbooks is provided out of the box: bootstrap.yml (for bootstrapping fresh nodes) This playbook can be used for bootstrapping fresh nodes. By default, the entire test site will be included in the bootstrap. If you wish to limit bootstrap to a single server, just run the playbook with (for example): .. code-block:: shell ansible-playbook -l ldap.example.com playbooks/bootstrap.yml ldap.yml This playbook sets-up the LDAP servers. It is included in ``site.yml``. mail.yml This playbook sets-up the mail server. It is included in ``site.yml``. preseed.yml This playbook sets-up the Debian preseed files. It is included in ``site.yml``. site.yml This playbook sets-up all servers, including preseed files on local host. web.yml This playbook sets-up the web server. It is included in ``site.yml``. xmpp.yml This playbook sets-up the XMPP server. It is included in ``site.yml``. backup.yml This playbook sets-up the backup server. It is included in ``site.yml``. The playbooks and configurations for test site make a couple of assumptions: * Each server will be set-up with an operating system user ``admin``, capable of running the sudo commands. * The password for operating system user ``admin`` is hard-coded to ``admin``. * An SSH ``authorized_keys`` file is set-up for the operating system user ``admin``. The SSH key stored in it will be read from location ``~/.ssh/id_rsa.pub`` (i.e. from home directory of user running the Ansible commands). For more details on how the playbooks and configuration have been implemented, feel free to browse the test site files (in directory ``testsite``). Deploying test site ~~~~~~~~~~~~~~~~~~~ In order to deploy the test site, the following steps would normally be taken: 1. As mentioned in introduction, default domain used by test site is ``example.com``. To change it, perform the following steps (otherwise, just skip to step 2): a. Update the file ``hosts``. Simply replace all occurances of ``example.com`` with your chosen domain. b. Update the file ``group_vars/all.yml``, changing the value of variable ``testsite_domain``. This value will then be used to calculate some of derived values, like LDAP base DN (which will be set to something along the lines of ``dc=example,dc=com`` or ``dc=your,dc=domain,dc=components``). 2. If you do not wish to have the hassle of creating the private keys and issuing certificates, there is a small playbook that can help you with this. Just run the ``tls.yml`` playbook, and skip to step 6 (otherwise follow steps 3 through 5): .. code-block:: shell ansible-playbook playbooks/tls.yml 3. Create TLS private keys (relative to top level directory), making sure to change domain in filenames if necessary: - ``testsite/tls/mail.example.com_imap.key`` - ``testsite/tls/mail.example.com_smtp.key`` - ``testsite/tls/xmpp.example.com_xmpp.key`` - ``testsite/tls/ldap.example.com_ldap.key`` - ``testsite/tls/web.example.com_https.key`` - ``testsite/tls/phpfino.example.com_https.key`` - ``testsite/tls/wsgi.example.com_https.key`` - ``testsite/tls/wsgireq.example.com_https.key`` 4. Issue TLS certificates corresponding to the generated TLS private keys (correct FQDN for DNS subject alternative name **must** be used), making sure to change domain in filenames if necessary: - ``testsite/tls/mail.example.com_imap.pem`` (subject alternative name should be ``mail.example.com``) - ``testsite/tls/mail.example.com_smtp.pem`` (subject alternative name should be ``mail.example.com``) - ``testsite/tls/xmpp.example.com_xmpp.pem`` (subject alternative name should be ``xmpp.example.com``) - ``testsite/tls/ldap.example.com_ldap.pem`` (subject alternative name should be ``ldap.example.com``) - ``testsite/tls/web.example.com_https.pem`` (subject alternative name should be ``web.example.com``) - ``testsite/tls/web.example.com_https.pem`` (subject alternative name should be ``web.example.com``) - ``testsite/tls/phpinfo.example.com_https.pem`` (subject alternative name should be ``phpinfo.example.com``) - ``testsite/tls/wsgi.example.com_https.pem`` (subject alternative name should be ``wsgi.example.com``) - ``testsite/tls/wsgireq.example.com_https.pem`` (subject alternative name should be ``wsgireq.example.com``) 5. Create ``PEM`` truststore file which contains all CA certificates that form CA chain for the issued end entity certificates from previous step at location ``testsite/tls/ca.pem``. It is very important to include the full CA chain used for LDAP server. 6. Generate SSH keys to be used by the backup server and backup clients: .. code-block:: shell mkdir ssh ssh-keygen -f ssh/backup_server_dsa_key -N '' -t dsa ssh-keygen -f ssh/backup_server_rsa_key -N '' -t rsa ssh-keygen -f ssh/backup_server_ed25519_key -N '' -t ed25519 ssh-keygen -f ssh/backup_server_ecdsa_key -N '' -t ecdsa ssh-keygen -f ssh/mail.example.com -N '' ssh-keygen -f ssh/ldap.example.com -N '' ssh-keygen -f ssh/xmpp.example.com -N '' ssh-keygen -f ssh/web.example.com -N '' ssh-keygen -f ssh/backup.example.com -N '' ssh-keygen -f ssh/ws01.example.com -N '' 7. Set-up a local GnuPG keyring that will contain the necessary encryption and signing keys for the backup clients:: mkdir ./backup_keyring chmod 700 ./backup_keyring cat << EOF | gpg2 --homedir ./backup_keyring --batch --gen-key Key-Type:RSA Key-Length:1024 Name-Real:ldap.example.com Expire-Date:0 %no-protection %commit Key-Type:RSA Key-Length:1024 Name-Real:mail.example.com Expire-Date:0 %no-protection %commit Key-Type:RSA Key-Length:1024 Name-Real:web.example.com Expire-Date:0 %no-protection %commit Key-Type:RSA Key-Length:1024 Name-Real:xmpp.example.com Expire-Date:0 %no-protection %commit Key-Type:RSA Key-Length:1024 Name-Real:backup.example.com Expire-Date:0 %no-protection %commit Key-Type:RSA Key-Length:1024 Name-Real:ws01.example.com Expire-Date:0 %no-protection %commit EOF 8. Generate the preseed files: .. code-block:: shell ansible-playbook playbooks/preseed.yml 9. Install all servers using the generated preseed files. Use *Debian 8 Jessie* or *Debian 9 Stretch* in any combination desired. 10. Add the SSH host fingerprints to your ``known_hosts`` file (don't forget to remove old entries if you are redoing the process). You can easily obtain all the necessary fingerprints with command (don't forget to modify domain if you need to): .. code-block:: shell ssh-keyscan -t ed25519 mail.example.com ldap.example.com xmpp.example.com web.example.com backup.example.com ws01.example.com $(dig +short mail.example.com) $(dig +short ldap.example.com) $(dig +short xmpp.example.com) $(dig +short web.example.com) $(dig +short backup.example.com) $(dig +short ws01.example.com) 11. Invoke the ``bootstrap.yml`` playbook in order to set-up some basic environment for Ansible runs on all servers: .. code-block:: shell ansible-playbook playbooks/bootstrap.yml 12. Finally, apply configuration on all servers: .. code-block:: shell ansible-playbook playbooks/site.yml Trying the test site out ~~~~~~~~~~~~~~~~~~~~~~~~ The test site ships with configuration files for a number of CLI-based tools/applications that can be easily used for trying the services out (you can otherwise set-up your own). Configuration files assume the default domain (``example.com``) is used. The following tools come with out-of-the-box configuration: - `mcabber `_ (for XMPP) - `mutt `_ (for IMAP/SMTP) In order to use the configuration files, make sure that you are running all tools from within the ``testsite/`` directory. To start the XMPP CLI client (*mcabber*) with provided configuration, run (take note it is lower-caser ``f``):: mcabber -f mcabber/john.doe mcabber -f mcabber/jane.doe To start the IMAP/SMTP CLI client (*Mutt*) with provided configuration, run (take note it is upper-case ``F``):: mutt -F mutt/john.doe mutt -F mutt/jane.doe