Files
@ 5bc6b7fb4cb5
Branch filter:
Location: majic-ansible-roles/docs/development.rst - annotation
5bc6b7fb4cb5
13.6 KiB
text/prs.fallenstein.rst
MAR-127: Implemented time synchronisation set-up in common role:
- Added new parameter ntp_servers for defining list of servers to use.
- Deploy ntp and ntpdate packages and relevant configuration files.
- Updated tests, fixing expected restriction lines in ntp
configuration, and adding an additional test to make sure the ntp
daemon has reread its configuration.
- Added new parameter ntp_servers for defining list of servers to use.
- Deploy ntp and ntpdate packages and relevant configuration files.
- Updated tests, fixing expected restriction lines in ntp
configuration, and adding an additional test to make sure the ntp
daemon has reread its configuration.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 | 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 83c953f83425 d885a66830da d885a66830da 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc d885a66830da d885a66830da d885a66830da d885a66830da 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 0f349663e20c 0f349663e20c 0f349663e20c 0f349663e20c 0f349663e20c 6b784a5da8bc 6b784a5da8bc 6b784a5da8bc 0f349663e20c 0f349663e20c 0f349663e20c 0f349663e20c 6b784a5da8bc d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da d885a66830da 237acd00f6a1 237acd00f6a1 237acd00f6a1 6b784a5da8bc 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee 778ff940ac19 778ff940ac19 778ff940ac19 778ff940ac19 237acd00f6a1 237acd00f6a1 8b88132d2576 8b88132d2576 8b88132d2576 8b88132d2576 237acd00f6a1 237acd00f6a1 237acd00f6a1 8b88132d2576 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 052eefc4fab0 052eefc4fab0 052eefc4fab0 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 237acd00f6a1 052eefc4fab0 052eefc4fab0 052eefc4fab0 052eefc4fab0 052eefc4fab0 052eefc4fab0 c161524058d5 c161524058d5 c161524058d5 237acd00f6a1 237acd00f6a1 c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee 052eefc4fab0 052eefc4fab0 052eefc4fab0 c56a03a698ee c56a03a698ee c56a03a698ee c56a03a698ee 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 052eefc4fab0 052eefc4fab0 052eefc4fab0 2b3af82bc50e 96e9f230a669 c56a03a698ee c56a03a698ee c56a03a698ee 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 96e9f230a669 052eefc4fab0 052eefc4fab0 052eefc4fab0 052eefc4fab0 884beb9a0e1d 052eefc4fab0 052eefc4fab0 052eefc4fab0 2b3af82bc50e 2b3af82bc50e 052eefc4fab0 c56a03a698ee 96e9f230a669 c56a03a698ee c56a03a698ee 96e9f230a669 500658358454 500658358454 500658358454 500658358454 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 500658358454 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 7d197a9ca71e 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 7d197a9ca71e 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 7d197a9ca71e 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 7d197a9ca71e 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 3686169e9565 7d197a9ca71e 3686169e9565 24c957d877de 24c957d877de 24c957d877de 24c957d877de 24c957d877de 7d197a9ca71e 24c957d877de 3686169e9565 3686169e9565 3686169e9565 237acd00f6a1 237acd00f6a1 237acd00f6a1 3686169e9565 237acd00f6a1 24c957d877de 24c957d877de 24c957d877de 24c957d877de 237acd00f6a1 3686169e9565 3686169e9565 3686169e9565 3686169e9565 61ddc6eab566 3686169e9565 61ddc6eab566 24c957d877de 61ddc6eab566 3686169e9565 500658358454 237acd00f6a1 3686169e9565 237acd00f6a1 3686169e9565 237acd00f6a1 3686169e9565 237acd00f6a1 3686169e9565 237acd00f6a1 3686169e9565 237acd00f6a1 de1d9aa13410 778ff940ac19 778ff940ac19 778ff940ac19 778ff940ac19 778ff940ac19 778ff940ac19 d90dce790417 778ff940ac19 57667a2c528b de1d9aa13410 de1d9aa13410 | .. _development:
Development
===========
This section covers procedures and information related to development of *Majic
Ansible Roles*.
Preparing environment
---------------------
The easiest way to get going with role development is to set-up a separate
Python virtual environment with the necessary packages. This can be done by
performing the following steps:
1. Ensure that the following minimum set of packages are installed via
distribution package manager:
- `Git <https://git-scm.com/>`_
- `libffi <https://sourceware.org/libffi/>`_ runtime and development package.
- `OpenSSL <https://www.openssl.org/>`_ runtime and development package.
- `pip <https://pypi.python.org/pypi/pip/>`_
- `virtualenv <https://pypi.python.org/pypi/virtualenv>`_
- `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_
- Development packages for Python.
On Debian this can be easily done with::
apt-get install virtualenv virtualenvwrapper git python-pip python-dev \
libffi-dev libssl-dev
2. In order to be able to run role tests, it is necessary to install `VirtualBox
<https://www.virtualbox.org/>`_ and `Vagrant <https://www.vagrantup.com/>`_,
using instructions outlined on their respective websites. It is recommended
to use latest versions available. At time of this writing the role tests have
been successfully run on *VirtualBox 5.1.22* and *Vagrant 1.9.5*.
3. Clone the git repository::
git clone https://code.majic.rs/majic-ansible-roles/ ~/projects/
4. Create a separate Python virtual environment::
mkvirtualenv majic-ansible-roles -a ~/projects/majic-ansible-roles/
5. Make sure the virtual environment has been activated, and install `pip-tools
<https://github.com/jazzband/pip-tools>`_::
workon majic-ansible-roles
pip install pip-tools
6. Synchronise Python virtual environment with requirements file using
**pip-tools**::
workon majic-ansible-roles
pip-sync
Development conventions
-----------------------
In order to maintain consistency across different roles and documentation, this
section describes development conventions that should be followed while making
modifications.
Task specifications
~~~~~~~~~~~~~~~~~~~
When writing new and updating existing tasks, keep the following in mind:
- Quote sensibly. If specifying paths (for example ``src``, ``dest``, ``path``
etc in various models), quote the string to make it stand-out better and to
avoid breakages.
- Avoid usage of ``set_facts`` task when same functionality can be achieved via
``defaults/main.yml``.
- When specifying tasks, use the fully expanded form. Do not use single-line
form with ``param=value``.
- When specifying ``command`` or ``shell`` tasks, in case a ``creates``
parameter or such need to be used, specify them as part of task's ``args``
parameter. E.g.::
- name: Run command
command: mycommand
args:
creates: "/etc/mycommand"
- When sepcifying tasks, keep the following ordering between different task
parameters:
- ``name``
- Module and its parameters.
- ``become``
- ``become_user``
- ``when``
- ``with_items`` / ``with_dict`` / ``with_nested``
- ``wait_for``
- ``register``
- ``changed_when``
- ``failed_when``
- ``no_log``
- ``notify``
- Task tags.
Running role tests directly
---------------------------
Role tests are implemented using `Molecule <https://molecule.readthedocs.io/>`_,
`Testinfra <https://testinfra.readthedocs.io/en/latest/>`_, `VirtualBox
<https://www.virtualbox.org/>`_ and `Vagrant
<https://www.vagrantup.com/>`_. *Molecule* and *Testinfra* are installed inside
of Pyhton virtual environment, while *VirtualBox* and *Vagrant* need to be
installed distribution-wide, following instructions outlined on their
corresponding websites.
Tests can be run directly for a single role, or for one or more roles using a
dedicated shell script (see below). The shell script can also be used for
generating reports in an automated environment.
In order to run tests for a specific role, perform the following steps:
1. Switch to Python virtual environment::
workon majic-ansible-roles
2. Change directory::
cd roles/ROLENAME/
3. List available supported platforms::
molecule list
4. Run tests for default platform (Debian Jessie)::
molecule test
5. Run tests for Debian Stretch (if role tests support it)::
molecule test --platform debian-stretch64
Running role tests via shell script
-----------------------------------
In order to make it easier to run tests for all roles, and eventually produce
reports of such runs, a dedicated shell script is provided for running the
tests.
In order to run tests, perform the following steps:
1. Switch to Python virtual environment::
workon majic-ansible-roles
2. Make sure you are within the root directory of Git repository.
3. Run tests for all roles and generate report::
./scripts/run_tests.sh -r all
.. note::
Optionally you can run tests for a specific set of roles, or without
generating the report, for example ``./scripts/run_tests.sh web_server
common``
4. Check results either from script output, or within directory
``test_report-YYYY_MM_DD-hh_mm_ss``. For overview of what roles have failed,
have a look at ``summary.txt``. For details have a look at each role's
individual report.
.. _testsite:
Test Site
---------
*Majic Ansible Roles* comes with a small sample test site configuration which
demonstrates use of every role. This test site also serves as starting point for
developing new roles etc, and can be used for testing regressions/breakages.
The test site covers everything, starting from generating the Debian preseed
files, through bootstrap process for new nodes, and onto deployment of all
remaining roles.
By default, the test site uses domain ``example.com``, but it has been designed
so it is easy to set your own domain (see below in step-by-step
instructions). Some changes may be necessary to listed commands in that case
(i.e. replace every occurance of ``example.com`` with your own domain).
All example commands listed within this section should be ran from within the
``testsite`` directory in order to have proper environment available for
playbook runs.
A number of playbooks is provided out of the box:
bootstrap.yml (for bootstrapping fresh nodes)
This playbook can be used for bootstrapping fresh nodes. By default, the
entire test site will be included in the bootstrap. If you wish to limit
bootstrap to a single server, just run the playbook with (for example):
.. code-block:: shell
ansible-playbook -l ldap.example.com playbooks/bootstrap.yml
ldap.yml
This playbook sets-up the LDAP servers. It is included in ``site.yml``.
mail.yml
This playbook sets-up the mail server. It is included in ``site.yml``.
preseed.yml
This playbook sets-up the Debian preseed files. It is included in
``site.yml``.
site.yml
This playbook sets-up all servers, including preseed files on local host.
web.yml
This playbook sets-up the web server. It is included in ``site.yml``.
xmpp.yml
This playbook sets-up the XMPP server. It is included in ``site.yml``.
backup.yml
This playbook sets-up the backup server. It is included in ``site.yml``.
In order to deploy the test site, the following steps would normally be taken:
1. As mentioned in introduction, default domain used by test site is
``example.com``. To change it, perform the following steps (otherwise, just
skip to step 2):
a. Update the file ``hosts``. Simply replace all occurances of
``example.com`` with your chosen domain.
b. Update the file ``group_vars/all.yml``, changing the value of variable
``testsite_domain``. This value will then be used to calculate some of
derived values, like LDAP base DN (which will be set to something along
the lines of ``dc=example,dc=com`` or
``dc=your,dc=domain,dc=components``).
2. If you do not wish to have the hassle of creating the private keys and
issuing certificates, there is a small playbook that can help you with
this. Just run the ``tls.yml`` playbook, and skip to step 6 (otherwise follow
steps 3 through 5):
.. code-block:: shell
ansible-playbook playbooks/tls.yml
3. Create TLS private keys (relative to top level directory), making sure to
change domain in filenames if necessary:
- ``testsite/tls/mail.example.com_imap.key``
- ``testsite/tls/mail.example.com_smtp.key``
- ``testsite/tls/xmpp.example.com_xmpp.key``
- ``testsite/tls/ldap.example.com_ldap.key``
- ``testsite/tls/web.example.com_https.key``
- ``testsite/tls/phpfino.example.com_https.key``
- ``testsite/tls/wsgi.example.com_https.key``
- ``testsite/tls/wsgireq.example.com_https.key``
4. Issue TLS certificates corresponding to the generated TLS private keys
(correct FQDN for DNS subject alternative name **must** be used), making sure
to change domain in filenames if necessary:
- ``testsite/tls/mail.example.com_imap.pem`` (subject alternative name should
be ``mail.example.com``)
- ``testsite/tls/mail.example.com_smtp.pem`` (subject alternative name should
be ``mail.example.com``)
- ``testsite/tls/xmpp.example.com_xmpp.pem`` (subject alternative name should
be ``xmpp.example.com``)
- ``testsite/tls/ldap.example.com_ldap.pem`` (subject alternative name should
be ``ldap.example.com``)
- ``testsite/tls/web.example.com_https.pem`` (subject alternative name should
be ``web.example.com``)
- ``testsite/tls/web.example.com_https.pem`` (subject alternative name should
be ``web.example.com``)
- ``testsite/tls/phpinfo.example.com_https.pem`` (subject alternative name
should be ``phpinfo.example.com``)
- ``testsite/tls/wsgi.example.com_https.pem`` (subject alternative name
should be ``wsgi.example.com``)
- ``testsite/tls/wsgireq.example.com_https.pem`` (subject alternative name
should be ``wsgireq.example.com``)
5. Create ``PEM`` truststore file which contains all CA certificates that form
CA chain for the issued end entity certificates from previous step at
location ``testsite/tls/ca.pem``. It is very important to
include the full CA chain used for LDAP server.
6. Generate SSH keys to be used by the backup server and backup clients:
.. code-block:: shell
mkdir ssh
ssh-keygen -f ssh/backup_server_dsa_key -N '' -t dsa
ssh-keygen -f ssh/backup_server_rsa_key -N '' -t rsa
ssh-keygen -f ssh/backup_server_ed25519_key -N '' -t ed25519
ssh-keygen -f ssh/backup_server_ecdsa_key -N '' -t ecdsa
ssh-keygen -f ssh/mail.example.com -N ''
ssh-keygen -f ssh/ldap.example.com -N ''
ssh-keygen -f ssh/xmpp.example.com -N ''
ssh-keygen -f ssh/web.example.com -N ''
ssh-keygen -f ssh/backup.example.com -N ''
ssh-keygen -f ssh/ws01.example.com -N ''
7. Set-up a local GnuPG keyring that will contain the necessary encryption and
signing keys for the backup clients::
mkdir ./backup_keyring
chmod 700 ./backup_keyring
cat << EOF | gpg2 --homedir ./backup_keyring --batch --gen-key
Key-Type:RSA
Key-Length:1024
Name-Real:ldap.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:mail.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:web.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:xmpp.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:backup.example.com
Expire-Date:0
%no-protection
%commit
Key-Type:RSA
Key-Length:1024
Name-Real:ws01.example.com
Expire-Date:0
%no-protection
%commit
EOF
8. Generate the preseed files:
.. code-block:: shell
ansible-playbook playbooks/preseed.yml
9. Install all servers using the generated preseed files. All servers except
``ws01.example.com`` are supposed to be running *Debian 8 Jessie*. The server
``ws01.example.com`` is meant to run *Debian 9 Stretch* (althogh, Debian
Jessie should function as well).
10. Add the SSH host fingerprints to your ``known_hosts`` file (don't forget to
remove old entries if you are redoing the process). You can easily obtain all
the necessary fingerprints with command (don't forget to modify domain if you
need to):
.. code-block:: shell
ssh-keyscan -t ed25519 mail.example.com ldap.example.com xmpp.example.com web.example.com backup.example.com ws01.example.com $(resolveip -s mail.example.com) $(resolveip -s ldap.example.com) $(resolveip -s xmpp.example.com) $(resolveip -s web.example.com) $(resolveip -s backup.example.com) $(resolveip -s ws01.example.com)
11. Invoke the ``bootstrap.yml`` playbook in order to set-up some basic
environment for Ansible runs on all servers:
.. code-block:: shell
ansible-playbook playbooks/bootstrap.yml
12. Finally, apply configuration on all servers:
.. code-block:: shell
ansible-playbook playbooks/site.yml
The playbooks and configurations for test site make a couple of assumptions:
* Each server will be set-up with an operating system user ``admin``, capable of
running the sudo commands.
* The password for operating system user ``admin`` is hard-coded to ``admin``.
* An SSH ``authorized_keys`` file is set-up for the operating system user
``admin``. The SSH key stored in it will be read from location
``~/.ssh/id_rsa.pub`` (i.e. from home directory of user running the Ansible
commands).
For more details on how the playbooks and configuration have been implemented,
feel free to browse the test site files (in directory ``testsite``).
|