Files @ 3e6eaaff335e
Branch filter:

Location: majic-ansible-roles/docs/development.rst

3e6eaaff335e 16.7 KiB text/prs.fallenstein.rst Show Annotation Show as Raw Download as Raw
branko
MAR-218: Update filter syntax for checking IPv4/IPv6 addreses.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
.. _development:

Development
===========

This section covers procedures and information related to development of *Majic
Ansible Roles*.


Preparing environment
---------------------

The easiest way to get going with role development is to set-up a separate
Python virtual environment with the necessary packages. This can be done by
performing the following steps:

1. Ensure that the following minimum set of packages are installed via
   distribution package manager:

   - `Git <https://git-scm.com/>`_
   - `libffi <https://sourceware.org/libffi/>`_ runtime and development package.
   - `OpenSSL <https://www.openssl.org/>`_ runtime and development package.
   - `pip <https://pypi.python.org/pypi/pip/>`_
   - `virtualenv <https://pypi.python.org/pypi/virtualenv>`_
   - `virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`_
   - Development packages for Python.

   On Debian this can be easily done with::

     apt-get install virtualenv virtualenvwrapper git python-pip python-dev \
     libffi-dev libssl-dev

2. In order to be able to run role tests, it is necessary to install `VirtualBox
   <https://www.virtualbox.org/>`_ and `Vagrant <https://www.vagrantup.com/>`_,
   using instructions outlined on their respective websites. It is recommended
   to use latest versions available. At time of this writing the role tests have
   been successfully run on *VirtualBox 5.2.12* and *Vagrant 2.0.4*.

3. In order to allow static IPv6 addresses to be allocated to virtual
   machines during testing, it is necessary to explicitly white-list
   the range used by the tests. Once the configuration file has been
   created, however, even the VirtualBox default allowed IPv4 subnet
   needs to be in the configuration explicitly as well.

   Update the VirtulBox configuration file (and make sure it can be
   read by the user running the tests):

   :file:`/etc/vbox/networks.conf`

   ::

     * 192.168.56.0/21
     * fd00::192:168:56:0/116

4. Clone the git repository::

     git clone https://code.majic.rs/majic-ansible-roles/ ~/projects/

5. Create a separate Python virtual environment::

     mkvirtualenv majic-ansible-roles -a ~/projects/majic-ansible-roles/

6. Make sure the virtual environment has been activated, and install `pip-tools
   <https://github.com/jazzband/pip-tools>`_::

     workon majic-ansible-roles
     pip install pip-tools

7. Synchronise Python virtual environment with requirements file using
   **pip-tools**::

     workon majic-ansible-roles
     pip-sync

8. Patch the installed version of test runner (Molecule) plugins until
   upstream merges the requried fixes for logging-in into Molecule
   Vagrant instances:

   .. note::
      For more details, see the following project links:

      - https://github.com/ansible-community/molecule-plugins/issues/239
      - https://github.com/ansible-community/molecule-plugins/pull/240
      - https://github.com/ansible-community/molecule-plugins/pull/260

   ::

      sed -i -e 's/{{/{/g;s/}}/}/g' "$VIRTUAL_ENV/lib/python3."*"/site-packages/molecule_plugins/vagrant/driver.py"


Development conventions
-----------------------

In order to maintain consistency across different roles and documentation, this
section describes development conventions that should be followed while making
modifications.


Task specifications
~~~~~~~~~~~~~~~~~~~

When writing new and updating existing tasks, keep the following in mind:

- Quote sensibly. If specifying paths (for example ``src``, ``dest``, ``path``
  etc in various models), quote the string to make it stand-out better and to
  avoid breakages.
- Avoid usage of ``set_facts`` task when same functionality can be achieved via
  ``defaults/main.yml``.
- When specifying tasks, use the fully expanded form. Do not use single-line
  form with ``param=value``.
- When specifying ``command`` or ``shell`` tasks, in case a ``creates``
  parameter or such need to be used, specify them as part of task's ``args``
  parameter. E.g.::

    - name: Run command
      command: mycommand
      args:
        creates: "/etc/mycommand"

- When sepcifying tasks, keep the following ordering between different task
  parameters:

  - ``name``
  - Module and its parameters.
  - ``become``
  - ``become_user``
  - ``when``
  - ``with_items`` / ``with_dict`` / ``with_nested``
  - ``wait_for``
  - ``register``
  - ``changed_when``
  - ``failed_when``
  - ``no_log``
  - ``notify``
  - Task tags.


Running role tests directly
---------------------------

Role tests are implemented using `Molecule <https://molecule.readthedocs.io/>`_,
`Testinfra <https://testinfra.readthedocs.io/en/latest/>`_, `VirtualBox
<https://www.virtualbox.org/>`_ and `Vagrant
<https://www.vagrantup.com/>`_. *Molecule* and *Testinfra* are installed inside
of Pyhton virtual environment, while *VirtualBox* and *Vagrant* need to be
installed distribution-wide, following instructions outlined on their
corresponding websites.

Tests can be run directly for a single role, or for one or more roles using a
dedicated shell script (see below). The shell script can also be used for
generating reports in an automated environment.

In order to run tests for a specific role, perform the following steps:

1. Switch to Python virtual environment::

     workon majic-ansible-roles

2. Change directory::

     cd roles/ROLENAME/

3. Run the default test scenario (this will normally test against
   multiple Debian versions if supported)::

     molecule test


Running role tests via shell script
-----------------------------------

In order to make it easier to run tests for all roles, and eventually produce
reports of such runs, a dedicated shell script is provided for running the
tests.

In order to run tests, perform the following steps:

1. Switch to Python virtual environment::

     workon majic-ansible-roles

2. Make sure you are within the root directory of Git repository.

3. Run tests for all roles and generate report::

     ./scripts/run_tests.sh -r all

   .. note::
      Optionally you can run tests for a specific set of roles, or without
      generating the report, for example ``./scripts/run_tests.sh web_server
      common``

4. Check results either from script output, or within directory
   ``test_report-YYYY_MM_DD-hh_mm_ss``. For overview of what roles have failed,
   have a look at ``summary.txt``. For details have a look at each role's
   individual report.


.. _testsite:

Test Site
---------

*Majic Ansible Roles* comes with a small sample test site configuration which
demonstrates use of every role. This test site also serves as starting point for
developing new roles etc, and can be used for testing regressions/breakages.

The test site covers everything, starting from generating the Debian preseed
files, through bootstrap process for new nodes, and onto deployment of all
remaining roles.

By default, the test site uses domain ``example.com``, but it has been designed
so it is easy to set your own domain (see below in step-by-step
instructions). Some changes may be necessary to listed commands in that case
(i.e. replace every occurance of ``example.com`` with your own domain).

By default, the following hosts are used in test site (listed names
should be resolvable towards the servers):

- ``ldap.example.com``, acting as LDAP server
- ``xmpp.example.com``, acting as XMPP server
- ``mail.example.com``, acting as mail server
- ``web.example.com``, acting as web server
- ``backup.example.com``, acting as backup server
- ``ws01.example.com``, acting as "workstation" (mainly demonstrating
  use of mail forwarder role)

The web server is also set-up with one sample PHP application and two
WSGI applications deployed via roles:

- ``phpinfo``, used to serve information about the local PHP
  installation.
- ``wsgihello``, small hello world application.
- ``wsgihello2``, small hello world application using a couple of
  additional ``wsgi_website`` role parameters.

To make the web applications reachable, make sure the web server
(``web.example.com``) IP is also resolvable via names:

- ``phpinfo.example.com`` (``phpinfo`` application/role)
- ``wsgi.example.com`` (``wsgihello`` application/role)
- ``wsgireq.example.com`` (``wsgihello2`` application/role)

All example commands listed within this section should be ran from within the
``testsite`` directory in order to have proper environment available for
playbook runs.

A number of playbooks are provided out of the box:

bootstrap.yml (for bootstrapping fresh nodes)
  This playbook can be used for bootstrapping fresh nodes. By default, the
  entire test site will be included in the bootstrap. If you wish to limit
  bootstrap to a single server, just run the playbook with (for example):

  .. code-block:: shell

    ansible-playbook -l ldap.example.com playbooks/bootstrap.yml

ldap.yml
  This playbook sets-up the LDAP servers. It is included in ``site.yml``.

mail.yml
  This playbook sets-up the mail server. It is included in ``site.yml``.

preseed.yml
  This playbook sets-up the Debian preseed files. It is included in
  ``site.yml``.

site.yml
  This playbook sets-up all servers, including preseed files on local host.

web.yml
  This playbook sets-up the web server. It is included in ``site.yml``.

xmpp.yml
  This playbook sets-up the XMPP server. It is included in ``site.yml``.

backup.yml
  This playbook sets-up the backup server. It is included in ``site.yml``.

The playbooks and configurations for test site make a couple of assumptions:

* Each server will be set-up with an operating system user ``admin``, capable of
  running the sudo commands.
* The password for operating system user ``admin`` is hard-coded to ``admin``.
* An SSH ``authorized_keys`` file is set-up for the operating system user
  ``admin``. The SSH key stored in it will be read from location
  ``~/.ssh/id_rsa.pub`` (i.e. from home directory of user running the Ansible
  commands).

For more details on how the playbooks and configuration have been implemented,
feel free to browse the test site files (in directory ``testsite``).


Deploying test site
~~~~~~~~~~~~~~~~~~~

In order to deploy the test site, the following steps would normally be taken:

1. As mentioned in introduction, default domain used by test site is
   ``example.com``. To change it, perform the following steps (otherwise, just
   skip to step 2):

   a. Update the file ``hosts``. Simply replace all occurances of
      ``example.com`` with your chosen domain.
   b. Update the file ``group_vars/all.yml``, changing the value of variable
      ``testsite_domain``. This value will then be used to calculate some of
      derived values, like LDAP base DN (which will be set to something along
      the lines of ``dc=example,dc=com`` or
      ``dc=your,dc=domain,dc=components``).

2. If you do not wish to have the hassle of creating the private keys and
   issuing certificates, there is a small playbook that can help you with
   this. Just run the ``tls.yml`` playbook, and skip to step 6 (otherwise follow
   steps 3 through 5):

   .. code-block:: shell

     ansible-playbook playbooks/tls.yml

3. Create TLS private keys (relative to top level directory), making sure to
   change domain in filenames if necessary:

   - ``testsite/tls/mail.example.com_imap.key``
   - ``testsite/tls/mail.example.com_smtp.key``
   - ``testsite/tls/xmpp.example.com_xmpp.key``
   - ``testsite/tls/ldap.example.com_ldap.key``
   - ``testsite/tls/web.example.com_https.key``
   - ``testsite/tls/phpfino.example.com_https.key``
   - ``testsite/tls/wsgi.example.com_https.key``
   - ``testsite/tls/wsgireq.example.com_https.key``

4. Issue TLS certificates corresponding to the generated TLS private keys
   (correct FQDN for DNS subject alternative name **must** be used), making sure
   to change domain in filenames if necessary:

   - ``testsite/tls/mail.example.com_imap.pem`` (subject alternative name should
     be ``mail.example.com``)
   - ``testsite/tls/mail.example.com_smtp.pem`` (subject alternative name should
     be ``mail.example.com``)
   - ``testsite/tls/xmpp.example.com_xmpp.pem`` (subject alternative name should
     be ``xmpp.example.com``)
   - ``testsite/tls/ldap.example.com_ldap.pem`` (subject alternative name should
     be ``ldap.example.com``)
   - ``testsite/tls/web.example.com_https.pem`` (subject alternative name should
     be ``web.example.com``)
   - ``testsite/tls/web.example.com_https.pem`` (subject alternative name should
     be ``web.example.com``)
   - ``testsite/tls/phpinfo.example.com_https.pem`` (subject alternative name
     should be ``phpinfo.example.com``)
   - ``testsite/tls/wsgi.example.com_https.pem`` (subject alternative name
     should be ``wsgi.example.com``)
   - ``testsite/tls/wsgireq.example.com_https.pem`` (subject alternative name
     should be ``wsgireq.example.com``)

5. Create ``PEM`` truststore file which contains all CA certificates that form
   CA chain for the issued end entity certificates from previous step at
   location ``testsite/tls/ca.pem``. It is very important to
   include the full CA chain used for LDAP server.

6. Generate SSH keys to be used by the backup server and backup clients:

  .. code-block:: shell

     mkdir ssh
     ssh-keygen -f ssh/backup_server_rsa_key -N '' -t rsa
     ssh-keygen -f ssh/backup_server_ed25519_key -N '' -t ed25519
     ssh-keygen -f ssh/backup_server_ecdsa_key -N '' -t ecdsa
     ssh-keygen -f ssh/mail.example.com -N ''
     ssh-keygen -f ssh/ldap.example.com -N ''
     ssh-keygen -f ssh/xmpp.example.com -N ''
     ssh-keygen -f ssh/web.example.com -N ''
     ssh-keygen -f ssh/backup.example.com -N ''
     ssh-keygen -f ssh/ws01.example.com -N ''

7. Set-up a local GnuPG keyring that will contain the necessary encryption and
   signing keys for the backup clients::

     mkdir ./backup_keyring
     chmod 700 ./backup_keyring
     cat << EOF | gpg2 --homedir ./backup_keyring --batch --gen-key
     Key-Type:RSA
     Key-Length:1024
     Name-Real:ldap.example.com
     Expire-Date:0
     %no-protection
     %commit

     Key-Type:RSA
     Key-Length:1024
     Name-Real:mail.example.com
     Expire-Date:0
     %no-protection
     %commit

     Key-Type:RSA
     Key-Length:1024
     Name-Real:web.example.com
     Expire-Date:0
     %no-protection
     %commit

     Key-Type:RSA
     Key-Length:1024
     Name-Real:xmpp.example.com
     Expire-Date:0
     %no-protection
     %commit

     Key-Type:RSA
     Key-Length:1024
     Name-Real:backup.example.com
     Expire-Date:0
     %no-protection
     %commit

     Key-Type:RSA
     Key-Length:1024
     Name-Real:ws01.example.com
     Expire-Date:0
     %no-protection
     %commit
     EOF

8. Generate the preseed files:

  .. code-block:: shell

     ansible-playbook playbooks/preseed.yml

9. Install all servers using the generated preseed files. Use *Debian
   12 Bookworm*.

10. Add the SSH host fingerprints to your ``known_hosts`` file (don't forget to
    remove old entries if you are redoing the process). You can easily obtain all
    the necessary fingerprints with command (don't forget to modify domain if you
    need to):

    .. code-block:: shell

      ssh-keyscan -t ed25519 mail.example.com ldap.example.com xmpp.example.com web.example.com backup.example.com ws01.example.com $(dig +short mail.example.com) $(dig +short ldap.example.com) $(dig +short xmpp.example.com) $(dig +short web.example.com) $(dig +short backup.example.com) $(dig +short ws01.example.com)

11. Invoke the ``bootstrap.yml`` playbook in order to set-up some basic
    environment for Ansible runs on all servers:

    .. code-block:: shell

       ansible-playbook playbooks/bootstrap.yml

12. Finally, apply configuration on all servers:

    .. code-block:: shell

       ansible-playbook playbooks/site.yml


Trying the test site out
~~~~~~~~~~~~~~~~~~~~~~~~

The test site ships with configuration files for a number of CLI-based
tools/applications that can be easily used for trying the services out
(you can otherwise set-up your own).

Configuration files assume the default domain (``example.com``) is
used.

The following tools come with out-of-the-box configuration:

- `mcabber <https://mcabber.com/>`_ (for XMPP)
- `mutt <http://www.mutt.org/>`_ (for IMAP/SMTP)

In order to use the configuration files, make sure that you are
running all tools from within the ``testsite/`` directory.

To start the XMPP CLI client (*mcabber*) with provided configuration,
run (take note it is lower-caser ``f``)::

  mcabber -f mcabber/john.doe
  mcabber -f mcabber/jane.doe

To start the IMAP/SMTP CLI client (*Mutt*) with provided
configuration, run (take note it is upper-case ``F``)::

  mutt -F mutt/john.doe
  mutt -F mutt/jane.doe

To test the provided web applications, use the following links in a
browser:

.. note::
   To avoid untrusted certificate errors, it is recommended to test
   the links with dedicated instance of Mozilla Firefox, and to import
   the CA certificate (``tls/ca.pem``) into it.

- https://phpinfo.example.com/
- https://wsgi.example.com/
- https://wsgireq.example.com/