Files
@ ea69b2719d8e
Branch filter:
Location: majic-ansible-roles/docs/development.rst
ea69b2719d8e
11.2 KiB
text/prs.fallenstein.rst
MAR-22: Implemented tests for the common role:
- Added missing documentation for pipreqcheck_uid and pipreqcheck_gid
parameters.
- Use static-hashed passwords for reproducibility during testing in test
playbook.
- Install Emacs and libmariadb-client-lgpl-dev-compat via test playbook on one
of the testing instances in order to test related tasks.
- Fixed parameter for connection limitting in test playbook.
- Added explicit parameters to test playbook for pipreqcheck_gid and
pipreqcheck_uid.
- Fixed deployment of ferm configuration file ot include setting user/group and
mode.
- Added tests covering common deployment, deployment when only mandatory
parameters are provided, and deployment when optional parameters are set as
well.
- Added missing documentation for pipreqcheck_uid and pipreqcheck_gid
parameters.
- Use static-hashed passwords for reproducibility during testing in test
playbook.
- Install Emacs and libmariadb-client-lgpl-dev-compat via test playbook on one
of the testing instances in order to test related tasks.
- Fixed parameter for connection limitting in test playbook.
- Added explicit parameters to test playbook for pipreqcheck_gid and
pipreqcheck_uid.
- Fixed deployment of ferm configuration file ot include setting user/group and
mode.
- Added tests covering common deployment, deployment when only mandatory
parameters are provided, and deployment when optional parameters are set as
well.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 | .. _development:
Development
===========
This section covers procedures and information related to development of *Majic
Ansible Roles*.
Preparing environment
---------------------
The easiest way to get going with role development is to set-up a separate
Python virtual environment with the necessary packages. This can be done by
performing the following steps:
1. Ensure that the following minimum set of packages are installed via
distribution package manager:
- `Git <https://git-scm.com/>`_
- `libffi <https://sourceware.org/libffi/>`_ runtime and development package.
- `OpenSSL <https://www.openssl.org/>`_ runtime and development package.
- `pip <https://pypi.python.org/pypi/pip/>`_
- `virtualenv <https://pypi.python.org/pypi/virtualenv>`_
- `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_
- Development packages for Python.
On Debian this can be easily done with::
apt-get install virtualenv virtualenvwrapper git python-pip python-dev \
libffi-dev libssl-dev
2. In order to be able to run role tests, it is necessary to install `VirtualBox
<https://www.virtualbox.org/>`_ and `Vagrant <https://www.vagrantup.com/>`_,
using instructions outlined on their respective websites. It is recommended
to use latest versions available. At time of this writing the role tests have
been successfully run on *VirtualBox 5.1.22* and *Vagrant 1.9.5*.
3. Clone the git repository::
git clone https://code.majic.rs/majic-ansible-roles/ ~/projects/
4. Create a separate Python virtual environment::
mkvirtualenv majic-ansible-roles -a ~/projects/majic-ansible-roles/
5. Make sure the virtual environment has been activated, and install `pip-tools
<https://github.com/jazzband/pip-tools>`_::
workon majic-ansible-roles
pip install pip-tools
6. Synchronise Python virtual environment with requirements file using
**pip-tools**::
workon majic-ansible-roles
pip-sync
Running role tests
------------------
Role tests are implemented using `Molecule <https://molecule.readthedocs.io/>`_,
`Testinfra <https://testinfra.readthedocs.io/en/latest/>`_, `VirtualBox
<https://www.virtualbox.org/>`_ and `Vagrant
<https://www.vagrantup.com/>`_. *Molecule* and *Testinfra* are installed inside
of Pyhton virtual environment, while *VirtualBox* and *Vagrant* need to be
installed distribution-wide, following instructions outlined on their
corresponding websites.
In order to run tests for a specific role, perform the following steps:
1. Switch to Python virtual environment::
workon majic-ansible-roles
2. Change directory::
cd roles/ROLENAME/
3. List available supported platforms::
molecule list
4. Run tests for default platform (Debian Jessie)::
molecule test
5. Run tests for Debian Stretch (if role tests support it)::
molecule test --platform debian-stretch64
.. _testsite:
Test Site
---------
*Majic Ansible Roles* comes with a small sample test site configuration which
demonstrates use of every role. This test site also serves as starting point for
developing new roles etc, and can be used for testing regressions/breakages.
The test site covers everything, starting from generating the Debian preseed
files, through bootstrap process for new nodes, and onto deployment of all
remaining roles.
By default, the test site uses domain ``example.com``, but it has been designed
so it is easy to set your own domain (see below in step-by-step
instructions). Some changes may be necessary to listed commands in that case
(i.e. replace every occurance of ``example.com`` with your own domain).
All example commands listed within this section should be ran from within the
``testsite`` directory in order to have proper environment available for
playbook runs.
A number of playbooks is provided out of the box:
bootstrap.yml (for bootstrapping fresh nodes)
This playbook can be used for bootstrapping fresh nodes. By default, the
entire test site will be included in the bootstrap. If you wish to limit
bootstrap to a single server, just run the playbook with (for example):
.. code-block:: shell
ansible-playbook -l ldap.example.com playbooks/bootstrap.yml
ldap.yml
This playbook sets-up the LDAP servers. It is included in ``site.yml``.
mail.yml
This playbook sets-up the mail server. It is included in ``site.yml``.
preseed.yml
This playbook sets-up the Debian preseed files. It is included in
``site.yml``.
site.yml
This playbook sets-up all servers, including preseed files on local host.
web.yml
This playbook sets-up the web server. It is included in ``site.yml``.
xmpp.yml
This playbook sets-up the XMPP server. It is included in ``site.yml``.
backup.yml
This playbook sets-up the backup server. It is included in ``site.yml``.
In order to deploy the test site, the following steps would normally be taken:
1. As mentioned in introduction, default domain used by test site is
``example.com``. To change it, perform the following steps (otherwise, just
skip to step 2):
a. Update the file ``hosts``. Simply replace all occurances of
``example.com`` with your chosen domain.
b. Update the file ``group_vars/all.yml``, changing the value of variable
``testsite_domain``. This value will then be used to calculate some of
derived values, like LDAP base DN (which will be set to something along
the lines of ``dc=example,dc=com`` or
``dc=your,dc=domain,dc=components``).
2. If you do not wish to have the hassle of creating the private keys and
issuing certificates, there is a small playbook that can help you with
this. Just run the ``tls.yml`` playbook, and skip to step 6 (otherwise follow
steps 3 through 5):
.. code-block:: shell
ansible-playbook playbooks/tls.yml
3. Create TLS private keys (relative to top level directory), making sure to
change domain in filenames if necessary:
- ``testsite/tls/mail.example.com_imap.key``
- ``testsite/tls/mail.example.com_smtp.key``
- ``testsite/tls/xmpp.example.com_xmpp.key``
- ``testsite/tls/ldap.example.com_ldap.key``
- ``testsite/tls/web.example.com_https.key``
- ``testsite/tls/phpfino.example.com_https.key``
- ``testsite/tls/wsgi.example.com_https.key``
- ``testsite/tls/wsgireq.example.com_https.key``
4. Issue TLS certificates corresponding to the generated TLS private keys
(correct FQDN for DNS subject alternative name **must** be used), making sure
to change domain in filenames if necessary:
- ``testsite/tls/mail.example.com_imap.pem`` (subject alternative name should
be ``mail.example.com``)
- ``testsite/tls/mail.example.com_smtp.pem`` (subject alternative name should
be ``mail.example.com``)
- ``testsite/tls/xmpp.example.com_xmpp.pem`` (subject alternative name should
be ``xmpp.example.com``)
- ``testsite/tls/ldap.example.com_ldap.pem`` (subject alternative name should
be ``ldap.example.com``)
- ``testsite/tls/web.example.com_https.pem`` (subject alternative name should
be ``web.example.com``)
- ``testsite/tls/web.example.com_https.pem`` (subject alternative name should
be ``web.example.com``)
- ``testsite/tls/phpinfo.example.com_https.pem`` (subject alternative name
should be ``phpinfo.example.com``)
- ``testsite/tls/wsgi.example.com_https.pem`` (subject alternative name
should be ``wsgi.example.com``)
- ``testsite/tls/wsgireq.example.com_https.pem`` (subject alternative name
should be ``wsgireq.example.com``)
5. Create ``PEM`` truststore file which contains all CA certificates that form
CA chain for the issued end entity certificates from previous step at
location ``testsite/tls/ca.pem``. It is very important to
include the full CA chain used for LDAP server.
6. Generate SSH keys to be used by the backup server and backup clients:
.. code-block:: shell
mkdir ssh
ssh-keygen -f ssh/backup_server_dsa_key -N '' -t dsa
ssh-keygen -f ssh/backup_server_rsa_key -N '' -t rsa
ssh-keygen -f ssh/backup_server_ed25519_key -N '' -t ed25519
ssh-keygen -f ssh/backup_server_ecdsa_key -N '' -t ecdsa
ssh-keygen -f ssh/mail.example.com -N ''
ssh-keygen -f ssh/ldap.example.com -N ''
ssh-keygen -f ssh/xmpp.example.com -N ''
ssh-keygen -f ssh/web.example.com -N ''
ssh-keygen -f ssh/backup.example.com -N ''
ssh-keygen -f ssh/ws01.example.com -N ''
7. Set-up a local GnuPG keyring that will contain the necessary encryption and
signing keys for the backup clients::
mkdir ./backup_keyring
chmod 700 ./backup_keyring
cat << EOF | gpg2 --homedir ./backup_keyring --batch --gen-key
Key-Type:RSA
Key-Length:1024
Name-Real:ldap.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:mail.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:web.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:xmpp.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:backup.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:ws01.example.com
Expire-Date:0
%no-protection
%commit
EOF
8. Generate the preseed files:
.. code-block:: shell
ansible-playbook playbooks/preseed.yml
9. Install all servers using the generated preseed files. All servers except
``ws01.example.com`` are supposed to be running *Debian 8 Jessie*. The server
``ws01.example.com`` is meant to run *Debian 9 Stretch* (althogh, Debian
Jessie should function as well).
10. Add the SSH host fingerprints to your ``known_hosts`` file (don't forget to
remove old entries if you are redoing the process). You can easily obtain all
the necessary fingerprints with command (don't forget to modify domain if you
need to):
.. code-block:: shell
ssh-keyscan -t ed25519 mail.example.com ldap.example.com xmpp.example.com web.example.com backup.example.com ws01.example.com $(resolveip -s mail.example.com) $(resolveip -s ldap.example.com) $(resolveip -s xmpp.example.com) $(resolveip -s web.example.com) $(resolveip -s backup.example.com) $(resolveip -s ws01.example.com)
11. Invoke the ``bootstrap.yml`` playbook in order to set-up some basic
environment for Ansible runs on all servers:
.. code-block:: shell
ansible-playbook playbooks/bootstrap.yml
12. Finally, apply configuration on all servers:
.. code-block:: shell
ansible-playbook playbooks/site.yml
The playbooks and configurations for test site make a couple of assumptions:
* Each server will be set-up with an operating system user ``admin``, capable of
running the sudo commands.
* The password for operating system user ``admin`` is hard-coded to ``admin``.
* An SSH ``authorized_keys`` file is set-up for the operating system user
``admin``. The SSH key stored in it will be read from location
``~/.ssh/id_rsa.pub`` (i.e. from home directory of user running the Ansible
commands).
For more details on how the playbooks and configuration have been implemented,
feel free to browse the test site files (in directory ``testsite``).
|