majic-ansible-roles Changeset

Changeset - 734eb461396e

Parent rev.

Child rev.

[Not reviewed]

0 5 0

Branko Majic (branko) - 11 days ago 2024-09-09 14:24:49
branko@majic.rs

MAR-218: Document the passlib Python package requirement:

- The library is used for hashing the operating system user passwords
during account creation.

5 files changed with 12 insertions and 6 deletions:

docs/about.rst

docs/index.rst

docs/releasenotes.rst

docs/usage.rst

requirements.in

0 comments (0 inline, 0 general)

docs/about.rst

➞

Show inline comments

 About Majic Ansible Roles
 =========================
 Majic Ansible Roles is a collection of Ansible roles that are used on regular
 basis for deployment and maintenance of Majic infrastructure.
 The roles are kept as a separate project in hope of making them potentially
 useful to wider audience, and for reference purposes.
 Roles cover different aspects of infrastructure, such as mail servers, web
 servers, web applications etc. The roles are mainly well-suited for smaller
 installations.
 Roles are written for use with *Debian GNU/Linux*. For more details on
 supported releases, see :ref:`rolereference`.
 At the moment, the roles have been written for and tested against **Ansible
 .9.x**.
 The roles also utilise the ``ipv4/ipv6`` lookup plugins which require
 ``netaddr`` package to be installed. Make sure you have the packages
 available on controller machine.
 ``netaddr`` package to be installed. The ``passlib`` package is used
 for hashing the operating system passwords. Make sure you have the
 packages available on controller machine.
 Why were these roles created?
 -----------------------------
 For a long time I have had a couple of Internet-facing servers where I hosted
 all the IT infrastructure I needed for my day-to-day life.
 This started off with some basic services, like mail and XMPP server, and in
 time extended to include a web server, code repository etc.
 As the number of services I used grew, I found it more difficult to track
 updates and upgrades, let alone test them in reliable way. The biggest problem
 in particular was lack of time to properly document all the different things
 I've set-up.
 Being familiar with some Puppet-based deployments, I've started looking into the
 possibility of using a configuration management system. Ansible emerged as
 something that I thought would be easy to use, due to its agent-less nature.
 Once I passed some basic tutorials and got to know the system a bit, I decided
 to start my journey on implementing the different roles, in the way I want them,
 that would let me easily set-up my servers (and reinstall them, amongst other
 things).
 The roles you see within this repository are the fruit of this labour. I hope
 you find them useful.
 Features
 --------
 *Majic Ansible Roles* have the following features:
 .. warning::
    Of course, you may want to take some statements with a pinch of salt, and
    possibly attribute them to either delusions of grandeur, or bragging :)
 * Emphasis on small, self-hosted deployments.
 * Modular role design where possible and where necessary.
 * A number of roles covering common set-up of servers, databases, web server,
   XMPP server, mail server, and LDAP server.
 * Streamlined integration with LDAP server for most of the services.
 * Well-documented, with role reference documentation, examples, and test/sample
   site.
 * Balanced implementation allowing both configurability and ease of deployment.
 * Free Software, released under liberal BSD license.
 Available roles:
 * ``bootstrap`` (for setting-up servers for Ansible use)
 * ``common`` (for setting-up basic security, accounts, and configuration on
   servers)
 * ``database`` (for creating databases to be used for applications)
 * ``database_server`` (for deploying a database server, MariaDB)
 * ``ldap_client`` (for setting-up LDAP client configuration)
 * ``ldap_server`` (for deploying an LDAP server, OpenLDAP)
 * ``mail_forwarder`` (for setting-up forwarding of local mails to smart host,
   Postfix)
 * ``mail_server`` (for deploying Postfix, Dovecot, ClamAV)
 * ``php_website`` (for deploying PHP websites)
 * ``preseed`` (for preparing Debian preseed files)
 * ``prosody`` (for deploying XMPP server, Prosody)
 * ``web_server`` (for deploying web server, Nginx)
 * ``wsgi_website`` (for deploying WSGI/Python applications)
 Support
 -------
 In case of problems with the roles or provided code, please do not hestitate to
 contact the author at **majic-ansible-roles (at) majic.rs**. The project can be
 found at:
 * https://code.majic.rs/majic-ansible-roles
 * https://projects.majic.rs/majic-ansible-roles
 License
 -------
 Majic Ansible Roles is released under terms of *BSD (3-Clause) License*::
   Copyright (c) 2018, Branko Majic
   All rights reserved.
   Redistribution and use in source and binary forms, with or without modification,
   are permitted provided that the following conditions are met:
     Redistributions of source code must retain the above copyright notice, this
     list of conditions and the following disclaimer.
     Redistributions in binary form must reproduce the above copyright notice, this
     list of conditions and the following disclaimer in the documentation and/or
     other materials provided with the distribution.
     Neither the name of Branko Majic nor the names of any other
     contributors may be used to endorse or promote products derived from
     this software without specific prior written permission.
   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
   ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
   WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
   DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
   ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
   (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
   LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
   ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
   SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

docs/index.rst

➞

Show inline comments

 Majic Ansible Roles documentation
 =================================
 Majic Ansible Roles is a collection of Ansible roles that are used on regular
 basis for deployment and maintenance of Majic infrastructure.
 The roles are kept as a separate project in hope of making them potentially
 useful to wider audience, and for reference purposes.
 Roles cover different aspects of infrastructure, such as mail servers, web
 servers, web applications etc. The roles are mainly well-suited for smaller
 installations.
 Roles are written for use with *Debian GNU/Linux*. For more details on
 supported releases, see :ref:`rolereference`.
 At the moment, the roles have been written for and tested against **Ansible
 .9.x**.
 The roles also utilise the ``ipv4/ipv6`` lookup plugins which require
 ``netaddr`` package to be installed. Make sure you have the packages
 available on controller machine.
 ``netaddr`` package to be installed. The ``passlib`` package is used
 for hashing the operating system passwords. Make sure you have the
 packages available on controller machine.
 Contents
 ========
 .. toctree::
    :maxdepth: 2
    about
    usage
    rolereference
    development
    releaseprocedures
    releasenotes
 Indices and tables
 ==================
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`

docs/releasenotes.rst

➞

Show inline comments

 Release notes
 =============
 x.y.z
 -----
 **Breaking changes:**
 * All roles
   * Dropped support for Debian 11 (Bullseye).
   * ``passlib`` Python package is now required for using the roles.
 **New features/improvements**
 * ``backup_client`` role
   * Switched to using Paramiko + SFTP backend (instead of pexpect +
     SFTP), which should improve the backup performance.
 **Bug fixes:**
 * ``common`` role
   * Fixed permission errors with Python cache directories in the pip
     requirements upgrade checks virtual environment that can happen if
     the initial virtual environment set-up fails.
 .0.0
 -----
 Dropped support for Python 2.7 and Debian 10 Buster. Added support for
 Debian 12 Bookworm. Numerous minor improvements and fixes.
 **Breaking changes:**
 * All roles
   * Dropped support for Debian 10 (Buster).
   * Added support for Debian 12 (Bookworm).
   * ``netaddr`` Python package is now required for using the roles.
   * ``dnspython`` Python package is no longer required for using the
     roles.
 * ``backup_client`` role
   * Previously the backup would run even if pre-backup scripts would
     fail. This is no longer the case, and all pre-backup scripts must
     exit with non-zero exit code in order for backup process to
     kick-in.
   * Old backups are now automatically purged after successful
     backup. This could lead to longer runtimes for entire backup
     process, as well as higher CPU usage.
 * ``common`` role
   * Dropped support for Python 2.7 pip requirements upgrade
     checks. Only Python 3 is supported now.
     Requirements (input) files for Python 3 are now put under the
     ``/etc/pip_check_requirements_upgrades`` directory instead of
     ``/etc/pip_check_requirements_upgrades-py3``.
     The ``pip_check_requirements_py3`` /
     ``pip_check_requirements_py3_in`` role parameters have been
     renamed to ``pip_check_requirements`` /
     ``pip_check_requirements_in``.
   * Parameter ``maintenance_allowed_hosts`` has been dropped and
     replaced with parameter ``maintenance_allowed_sources``. The new
     parameter expects a list of IPv4 and IPv6 addresses (or
     subnets). Resolvable names can no longer be specified.
   * NTP server configuration is now based on use of pools instead of
     servers. Parameter ``ntp_servers`` has been deprecated and
     replaced with parameter ``ntp_pools``.
 * ``ldap_server`` role
   * Starting with Debian 12 Bookworm, the role no longer deploys
     *rsyslog* and *logrotate* configuration for writing and rotating
     the LDAP servers logs under ``/var/log/slapd.log``. Primary
     reason is that Debian 12 Bookworm no longer installs *rsyslog* by
     default, and it is considered to be deprecated at this point. The
     LDAP server logs can be read via ``journalctl -u slapd`` when
     necessary.
 * ``mail_forwarder`` role
   * Firewall rules for incoming connections from the SMTP relay server
     are now based on relay's IPv4 and IPv6 addresses as resolved on
     managed machine during deployment time.
     In case the SMTP relay server's IP addresses change, the role
     needs to get reapplied against managed machines for those changes
     to take place.
     This change in behaviour was introduced to avoid firewall-related
     errors due to inability to resolve names via DNS servers during
     boot time.
 * ``mail_server`` role
   * Parameter ``mail_server_tls_protocols`` has been dropped and
     replaced with parameter ``mail_server_minimum_tls_protocol``. Full
     list of TLS protocols can no longer be specified, only the minimum
     one.
 * ``wsgi_website`` role
   * Dropped support for Python 2.7. Only Python 3 is supported now.
     The ``python_version`` role parameter has been dropped. The
     ``python_interpreter`` parameter is still available, but it
     defaults to Python 3 binary.
     Python (input) requirements files are now placed under the
     ``/etc/pip_check_requirements_upgrades`` path, in accordance to
     changes made in this release to the ``common`` role.
   * Dropped the ``proxy_headers`` parameter, and replaced it with the
     ``http_header_overrides`` parameter. The new parameter has similar
     function, but the values should no longer include double
     quotes. Main goal is ease of use and consistency between the PHP
     and WSGI website roles.
 **New features/improvements**
 * ``php_website`` role
   * Environment indicator can now be collapsed by clicking on the
     arrows on the left side.
   * Added parameter ``http_header_overrides`` which can be used to
     set/override request's HTTP headers before passing it on to the
     PHP application.
 * ``mail_server`` role
   * Protection against forging of sender addresses has been
     implemented, preventing logged-in users from using arbitrary
     sender mail addresses, even if authenticated. Authenticated users
     can use either their own login as sender, or one of the aliases
     that are associated with their mail account.
 * ``web_server`` role
   * Added parameter ``environment_indicator`` which is used on the
     default page to indicate what environment the web server belongs
     to. Useful for distinguishing between test, staging, and
     production environments (similar to what is already available in
     the ``php_website`` and ``wsgi_website`` roles). Indicator can be
     collapsed by clicking on the arrows on the left side.
 * ``wsgi_website`` role
   * Environment indicator can now be collapsed by clicking on the
     arrows on the left side.
 **Bug fixes:**
 * ``common`` role
   * Fix deprecation warnings for Python requirements upgrade checks
     when using pip-tools 7.3.0. This would result in unnecessary
     notifications being sent out to server administrator.
 .1.0
 -----
 Added support for Debian 11 (Bullseye), with some smaller bug fixes.
 **New features/improvements**
 * All roles
   * Added support for Debian 11 (Bullseye).
 **Bug fixes:**
 * ``xmpp_server`` role
   * Make sure to take care of deprecation-related package and
     configuration removals prior to running the rest of the tasks to
     avoid errors related to deprecated elements being invalid (like
     repository URLs for Prosody).
 .0.0
 -----
 Dropped support for Debian 9 (Stretch), moved away from using
 non-Debian project repositories (like Prosody ones).
 **Breaking changes:**
 * All roles
   * Dropped support for Debian 9 (Stretch).
 * ``xmpp_server`` role
   * Parameter ``xmpp_prosody_package`` has been dropped.
 **New features/improvements**
 * ``common`` role
   * Added parameters ``pip_check_requirements_in`` and
     ``pip_check_requirements_py3_in`` that can be used for specifying
     input requirements when checking for available package upgrades
     for Python virtual environments that are used for the checks
     themselves. This is particularly helpful in cases where Python
     version gets deprecated and some packages do not correctly declare
     the minimum version required, allowing to be more specific to
     avoid unnecessary warning mails being sent out.
   * Updated default package pins for virtual environments used to
     check for available pip package upgrades.
 * ``mail_server`` role
   * Added parameter ``mail_server_smtp_additional_configuration`` that
     provides ability to include additional configuration directives
     for the SMTP server.
 * ``xmpp_server`` role
   * Drop dependency on the external (Prosody) package
     repository. Install everything using official Debian
     repositories. This should help avoid future issues with Prosody
     project removing older versions of packages or dropping entire
     repository archives for older Debian releases.
   * Prosody package and some of its dependencies are installed from
     Debian backports to get more featureful release installed.
   * Role no longer depends on fetching external Prosody modules from
     project code repository, and instead relies on the prosody-modules
     package for LDAP authentication module.
 .0.0
 -----
 Added support for Debian 10 (Buster), alongside a couple of minor
 changes and features/improvements.
 **Breaking changes:**
 * ``ldap_server`` role
   * Use 2048-bit Diffie-Hellman parameters for relevant TLS
     ciphers. This could introduce incompatibility with older
     clients/servers trying to connect to the LDAP server. This change
     is applicable only under Debian Buster.
 **New features/improvements:**
 * All roles
   * Added support for Debian 10 (Buster).
 * ``common`` role
   * Added parameters ``maintenance`` and ``maintenance_allowed_hosts``
     for enabling maintenance mode. In maintenance mode only the listed
     hosts are allowed to connect to the server.
 **Bug fixes:**
 * ``ldap_server`` role
   * Allow use of DHE TLS ciphers by generating the necessary
     Diffie-Hellman parameters. This bug fix is applicable only under
     Debian Buster.
 * ``wsgi_website_`` role
   * When the virtual environment is created, the ``setuptools`` and
     ``pip`` packages will not get pinned to any specific version,
     allowing roles that are based on ``wsgi_website`` to easily
     install preferred versions, and avoid idempotence problems in the
     process.
 .0.0
 -----
 Upgrade to Ansible 2.9.x, dropping support for Debian 8 Jessie,
 upgrade to Python 3.x, dropping support for Python 2.7. A number of
 parameters have been made mandatory or deprecated. Security has been
 slightly improved in a number of roles, and there is plenty of
 bug-fixes and minor improvements throughout as well.
 **Breaking changes:**
 * Switched to Ansible 2.9.x, removing support for older versions. All
   documentation has been updated.
 * Switched to using Python 3 on both controller and managed server
   side. Python 2.7 can no longer be used for this purpose. Support for
   WSGI applications running on Python 2.7 remains.
 * All roles
   * Support for Debian 8 Jessie has been dropped.
   * Common parameters ``tls_private_key_dir`` and
     ``tls_certificate_dir`` are no longer used.
   * TLS private key and certificate parameters are now mandatory.
 * ``bootstrap`` role
   * Parameter ``ansible_key`` is now mandatory.
 * ``common`` role``
   * Minimum version of ``pip-tools`` in the ``pip_check_requirements``
     and ``pip_check_requirements_py3`` is now 5.3.0. This change was
     required in order to fix the deprecation warnings being sent out
     when the ``pip_check_requirements_upgrades.sh`` script is run.
 * ``database_server`` role
   * Parameter ``db_root_password`` has been deprecated. The root user
     can now login into the database (as the root database user) via
     unix socket authentication.
   * Role will drop the use of Debian system maintenance user
     (``debian-sys-maint``) in favour of using the root account with
     UNIX socket authentication if the database server has not already
     been set-up in that manner. This is the default behaviour starting
     from Debian Stretch, and the ``debian-sys-main`` will be present
     only if the server has been upgraded from older releases.
 * ``ldap_server`` role
   * Parameter ``ldap_server_domain`` is now mandatory.
   * Updated default set of TLS ciphers used by server
     (``ldap_tls_ciphers`` parameter). All CBC ciphers have been
     dropped. This could introduce incompatibility with older clients
     trying to connect to the LDAP server.
 * ``mail_forwarder`` role
   * Use 2048-bit Diffie-Hellman parameters for relevant TLS
     ciphers. This could introduce incompatibility with older
     clients/servers trying to connect to the SMTP server.
 * ``mail_server`` role
   * Use 2048-bit Diffie-Hellman parameters for relevant TLS
     ciphers. This could introduce incompatibility with older
     clients/servers trying to connect to the SMTP/IMAP server.
   * Updated default set of TLS ciphers used by IMAP/SMTP servers
     (``mail_server_tls_ciphers`` parameter). All CBC ciphers have been
     dropped. This could introduce incompatibility with older clients
     trying to connect to the IMAP/SMTP server.
   * Dropped the use of ``procmail`` for local mail deliveries.
 * ``php_website`` role
   * Parameter ``enforce_https`` has been deprecated and
     removed. HTTPS is now mandatory in all cases.
 * ``preseed`` role
   * Parameter ``ansible_key`` is now mandatory.
   * Parameter ``preseed_directory`` is now mandatory.
 * ``web_server`` role
   * Use 2048-bit Diffie-Hellman parameters for relevant TLS
     ciphers. This could introduce incompatibility with older clients
     trying to connect to the web server.
   * Updated default set of TLS ciphers used by the server
     (``web_server_tls_ciphers`` parameter). All CBC ciphers have been
     dropped. This could introduce incompatibility with older clients
     trying to connect to the server.
   * Parameter ``default_enforce_https`` has been deprecated and
     removed. HTTPS is now mandatory in all cases.
 * ``wsgi_website`` role
   * Parameters ``gunicorn_version`` and ``futures_version`` have been
     deprecated and removed. Existing roles should be updated to
     utilise the ``wsgi_requirements`` parameter instead.
   * Parameter ``enforce_https`` has been deprecated and
     removed. HTTPS is now mandatory in all cases.
   * Added parameter ``wsgi_requirements_in`` for listing top-level
     packages for performing pip requirements upgrade checks for
     Gunicorn requirements (listed via existing ``wsgi_requirements``
     parameter).
 * ``xmpp_server`` role
   * Parameter ``xmpp_domains`` is now mandatory.
   * Use 2048-bit Diffie-Hellman parameters for relevant TLS
     ciphers. This could introduce incompatibility with older
     clients/servers trying to connect to the XMPP server.
   * TLS hardening is now applied to the *c2s* (client) connections on
     both the standard (``5222``) and legacy (``5223``) ports. Protocol
     version and ciphers are configurable via new
     ``xmpp_server_tls_protocol`` and ``xmpp_server_tls_ciphers``
     parameters with defaults enforcing TLSv1.2+ and PFS (perfect
     forward secrecy) ciphers.
   * Support for older Prosody versions (``0.9.x``) has been
     dropped. Only Prosody ``0.10.x`` is supported at the moment (due
     to missing Lua LDAP bindings in Debian 9 Stretch).
   * Support for running Prosody 0.11.x has been added. This is also
     the new default version of Prosody that gets deployed to the
     target system.
 **Bug fixes:**
 * ``common`` role
   * Run apticron at least once during initial installation to avoid
     accidental locking later on during the same playbook run.
 * ``mail_server`` role
   * Fixed the problem with the SMTP server (Postfix) not using TLS at
     all for outgoing SMTP connections. The server will now default to
     using opportunistic TLS (using TLS where available).
 * ``wsgi_website`` role
   * Deploy the requirement files used for upgrade checks to correct
     location when using Python 3. Previously the files would get
     deployed to directory dedicated to Python 2 version, which means
     the checks would be performed using Python 2 instead of Python 3.
 **New features/improvements:**
 * Tests have been updated to work with latest Molecule/Testinfra as
   part of the Ansible upgrade process.
 * X.509 artefacts used during testing are now generated on the fly
   using `Gimmecert <https://gimmecert.readthedocs.io/>`_.
 * ``mail_forwader`` role
   * The role now supports specifying the maximum mail message size
     limit for the SMTP server to accept via
     ``mail_message_size_limit`` role parameter.
   * Mail server configuration has been slightly updated to better
     match what is currently the defaults in Debian Stretch.
 * ``mail_server`` role
   * The role now supports specifying the maximum mail message size
     limit for the SMTP server to accept via
     ``mail_message_size_limit`` role parameter.
   * Mail server configuration has been slightly updated to better
     match what is currently the defaults in Debian Stretch.
 * ``xmpp_server`` role
   * Server now supports blocking users via `XEP-0191: Blocking Command
     <https://xmpp.org/extensions/xep-0191.html>`_.
   * Server now supports `XEP-0280: Message Carbons
     <http://xmpp.org/extensions/xep-0280.html>`_, letting multiple
     online XMPP clients receive/store the same message.
   * Server now supports `XEP-0313: Message Archive Management
     <https://xmpp.org/extensions/xep-0313.html>`_, storing copies of
     received messages server-side. Message expiration is configurable
     via parameter ``xmpp_server_archive_expiration``.
   * XMPP server certificate is checked on daily basis using the
     ``prosodyctl check certs`` command. This helps catch issues where
     issued certificate does not include all the necessary subject
     alternative names (this has also been documented in the role
     reference documentation).
 **Deprecations:**
 * ``backup_server`` and ``backup_client`` role
   * Officially dropped support for DSA keys (this was mainly remnant
     from Debian 8 Jessie support, on Debian 9 Stretch and upwards the
     DSA keys were not supported at all).
 .0.0
 -----
 A couple of smaller bug-fixes, and introduction of (minor) breaking
 change related to handling of pip requirements upgrade checks in the
 ``common`` role (see below).
 Breaking changes:
 * ``common`` role:
   * Added separate parameter (``pip_check_requirements_py3``) for
     specifying dedicated Python 3 virtual environment package
     requirements used for package upgrade checks on (other
     user-provided) Python 3 virtual environments. If the existing
     ``pip_check_requirements`` parameter has been overridden, the new
     parameter will most likely need to be overridden in your site
     configuration as well. Take note that the new requirements will
     differ between Debian Jessie and Debian Stretch due to differnece
     in Python 3 minor version releases.
 Bug fixes:
 * ``backup_client`` role
   * Avoid errors related to lack of ``tty`` when invoking the GnuPG
     utility by using the ``--no-tty`` option.
 * ``common`` role
   * Fixed problem with pip requirements upgrades checks outputting
     package list to stderr, causing the cron job to report outdated
     packages to administrator even though nothing is outdated (cron
     job treats anything output to stderr as worthy of notification).
 .1.0
 -----
 Minor improvements and fixes.
 Breaking changes:
 * ``common`` role:
    * Default values for the ``pip_check_requirements`` have changed to
      include ``pip`` and ``setuptools`` (and a couple more). It might
      be necessary to update any customised values of this parameter to
      match the default set of packages.
 Bug fixes:
 * ``common`` role
   * Use Python 3 in Python virtual environment used for checking if
     upgrades are available for Python requirements files. Fixes
     possibly incorrect package resolution due to wrong version of
     Python (for packages that have different dependencies based on
     Python version).
 New features/improvements:
 * ``common`` role
   * Changed how the packages are installed in Python virtual
     environments used for performing pip requirements upgrade checks,
     making the process more reliable. The packages in those dedicated
     environments are now fully pinned, including system packages such
     as ``setuptools`` and ``pip`` itself.
   * Changed the pip requirements checks to now take into account
     unsafe packages as well (such as ``setuptools`` and ``pip``) if
     listed.
 .0.0
 -----
 Upgrade to Ansible 2.7.x and full support for Debian 9 (Stretch).
 Breaking changes:
 * Switched to Ansible 2.7.x, removing support for older versions. All
   documentation has been updated.
 * All roles
   * In order to avoid collision with reserved name, the variable for
     running the handlers explicitly has been changed from ``handlers``
     to ``run_handlers``. E.g. to run all handlers a command similar to
     ``ansible-playbook -t handlers -e run_handlers=true
     playbooks/site.yml`` should be used.
 * ``ldap_server`` role
   * Custom LDAP module ``m_ldap_entry`` has been removed. Role uses
     the official ``ldap_entry`` and ``ldap_attr`` modules.
   * The ``ldap_entries`` parameter now supports only the states
     supported by ``ldap_entry`` module (e.g. ``append`` is not
     supported any longer - since it came with custom LDAP module).
 * ``xmpp_server`` role
   * Installs Prosody nightly builds (default version 0.10) instead of
     latest stable. Change made to improve reproducability, and to
     avoid future breakages after new major/minor releases of
     Prosody. Prosody package name can be specified through the
     ``xmpp_prosody_package`` variable.
     This should most likely not break anything, but is included as
     possibly breaking change nevertheless since it is a big change in
     how Prosody is installed.
     This change had to be done because LDAP integration broke with
     Prosody 0.11 due to missing LDAP bindings for Lua 5.2. See
     `MAR-137: XMPP server LDAP integration not working for Prosody
 .11.x
     <https://projects.majic.rs/majic-ansible-roles/issues/MAR-137>`_
     for more details.
 New features/improvements:
 * Tests have been updated to work with latest Molecule/Testinfra as
   part of the Ansible upgrade process.
 * All roles
   * Full support for Debian 9 (Stretch) has been added to all roles.
 * ``common`` role
   * The ``pip`` requirements upgrade checks are now performed once per
     day instead of once per hour.
   * The ``pip`` requirements upgrade checks now do not output warning
     in case deployed ``.in`` file does not have a matching ``.txt``
     file.
   * Certificate expiration check is less verbose. No mails are sent
     out any longer in case no certificates have been configured for
     checking, nor in cases where all certificates have passed the
     check. E.g. mails are sent out only in case some of the configured
     certificates will expire within next 30 days.
 * ``wsgi_website`` role
   * Support for specifying Python version for Python virtual
     environment.
 .0.0
 -----
 Upgrade to Ansible 2.3.x, minor bug fixes and updates needed for the upgrade.
 Breaking changes:
 * Switched to Ansible 2.3.x, removing support for Ansible 1.9.x. All
   documentation has been updated.
 * Due to switch to Ansible 2.x which is more restrictive when deploying code on
   remote server, it is now necessary to use one of the methods listed in
   `Ansible documentation
   <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_
   if connecting to remote server as user other than ``root``. Easiest fix is to
   enable ``pipelining``. Tests have been already updated to take advantage of
   this.
 * ``ldap_server`` role
    * Renamed ``ldap_entry`` module to ``m_ldap_entry`` to avoid collision with
      official module.
    * Renamed ``ldap_permissions`` module to ``m_ldap_permissions`` to be
      consistent and to avoid potential future collisions with official module
      names.
 * ``wsgi_website`` role
    * Removed handler with parametrised name used for restarting the web
      service. Dependent roles should instead define their own handlers from now
      on.
 New features/improvements:
 * Updated documentation to refer to Debian Jessie documentation where necessary.
 * ``ldap_server`` role
   * Updated tests to be more resilient to ordering changes.
 * ``mail_forwarder`` role
   * Updated tests to be more resilient to ordering changes and time races.
 * ``backup_client`` role
   * Switched to using the ``file`` module when cleaning-up GnuPG backup keyring,
     which should make it more robust..
 Bug-fixes:
 * Updated pip requirements and its input file to include ``python-vagrant``
   (needed for Molecule tests).
 * Updated handling of key ID extraction for OpenPGP keys in order for it to work
   with Ansible 2.x.
 * Updated usage instructions in order to be able to install The Bug Genie via
   script, and added missing instructions for creating one of the directories for
   the demo wiki role (``handlers`` directory).
 * All roles
   * Updated test playbooks to avoid idempotence test failures due to apt cache
     updates.
 * ``backup_client`` role
   * Fixed scenario in which backup keys could not be properly replaced on Debian
     Stretch machines (due to more up-to-date version of GnuPG compared to Debian
     Jessie).
 * ``common`` role
   * Fixed handling of complex version specifications when installing packages
     via pip. Needed for Ansible 2.x.
 * ``wsgi_website`` role
   * Fixed handling of complex version specifications when installing packages
     via pip. Needed for Ansible 2.x.
   * Fixed erroneous calculation of adminstrator username in internal
     defaults parameter.
 .7.0
 -----
 Minor improvements for mail-related roles, internal refactoring of task syntax,
 and improvements of tests.
 New features/improvements:
 * Documentation
   * Added new sub-section in development section describing some of the
     conventions used while developing the roles.
 * All roles
   * Switched to using expanded syntax in all roles and cleaned-up the tasks a
     bit (mainly internal change).
   * Minor cleanups within tests and tasks to accomodate the syntax changes
     (mainly internal change).
 * ``mail_forwarder`` role
   * Added parameter ``smtp_from_relay_allowed`` that controls if managed machine
     should accept incoming SMTP connections from the relay server. Useful for
     NAT'ed or laptop machines.
   * Added parameter ``smtp_relay_host_port`` that controls what port is used for
     connecting to the specified SMTP relay. Useful for machines behind
     restrictive ISPs.
 * ``mail_server`` role
   * Introduced firewall rules to redirect from TCP port 27 to TCP port 25,
     useful for machines behind restrictive ISPs.
 .6.0
 -----
 Implemented full test suite with a plethora of smaller bug-fixes, and some minor
 (internal) improvements.
 Breaking changes:
 * All roles
   * Previously a number of roles would modify permissions on the ``/srv``
     directory. This has now been fixed in order to prevent weird backup failures
     etc. Manual intervention is necessary on existing servers to fix the issue
     by changing the mode to ``0755`` (this is the usual default upon the OS
     installation).

docs/usage.rst

➞

Show inline comments

 .. _usage:
 Usage
 =====
 Majic Ansible Roles are targeted at sysadmins who wish to deploy services for
 their own, small-scale use. This chapter gives a simple tutorial-like set of
 instructions for using all of the roles available.
 .. contents:: :local:
 Overview
 --------
 There is a number of different roles that can prove useful for setting-up a
 small infrastructure of your own.
 Some roles are suited for one-off operations during installation, like the
 ``preseed`` and ``bootstrap``, while some are better suited for periodic runs
 for maintaining the users and integrity of the system.
 By the end of following the instructions, you will have the following:
 * Ansible server, used as controller for configuring and managing the
   remaining servers.
 * Communications server, providing the LDAP, mail, and XMPP services.
 * Web server, providing the web services.
 * Backup server, used for storing all of the backups.
 Pre-requisites
 --------------
 For the set-up outlined in this usage guide you'll need the following:
 * One server where Ansible will be installed at. Debian Bookworm will
   be installed on top of this server. The server will be set-up
   manually (this is currently out of scope for the *Majic Ansible
   Roles* automated set-up).
 * Three servers where the services will be set-up. All servers must be
   able to communicate over network with each-other, the Ansible
   servers, and with Internet. Debian Bookworm will be installed on top
   of these servers as part of the usage instructions.
 * Debian Bookworm network installation CD.
 * All servers should be on the same network.
 * IP addresses for all servers should be known.
 * Netmask for all servers should be known.
 * Gateway for all servers should be known.
 In case of the servers listed above, it might be safest to have them
 as virtual machines - this is cheapest thing to do, and simplest (who
 wants to deal with pesky hardware anyway?).
 Usage instructions assume the following:
 * Domain used for all servers is ``example.com``. If you wish to use a different
   domain, adjust the instructions accordingly.
 * Server hostnames are ``ansible``, ``comms``, ``www``, and ``bak`` (for Ansible
   server, communications server, web server, and backup server, respectively).
 Installing the OS on Ansible server
 -----------------------------------
 Start-off by installing the operating system on the Ansible server:
 . Fire-up the ``ansible`` server, and boot from the network installation CD.
 . Select the **Install** option.
 . Pick **English** as language.
 . Pick the country you are living in (or whatever else you want).
 . Pick the **en_US.UTF-8** locale.
 . Pick the **American English** keymap.
 . Configure the network if necessary.
 . Set the hostname to ``ansible``.
 . Set the domain to ``example.com``.
 . Set the root password.
 . Create a new user. For simplicity, call the user **Ansible user**, with
     username **ansible**.
 . Set-up partitioning in any way you want. You can go for **Guided - use
     entire disk** if you want to keep it simple and are just testing things.
 . Wait until the base system has been installed.
 . Pick whatever Debian archive mirror is closest to you.
 . If you have an HTTP proxy, provide its URL.
 . Pick if you want to participate in package survey or not.
 . Make sure that at least the **standard system utilities** and **SSH server**
     options are selected on task selection screen.
 . Wait for packages to be installed.
 . Install the GRUB boot loader on MBR.
 . Finalise the server install, and remove the installation media from server.
 Installing required packages
 ----------------------------
 With the operating system installed, it is necessary to install a couple of
 packages, and to prepare the environment a bit on the Ansible server:
 . Install the necessary system packages (using the ``root`` account)::
      apt-get install -y virtualenv virtualenvwrapper git python3-pip python3-dev libffi-dev libssl-dev
 . Set-up loading of ``virtualenvwrapper`` via Bash completions (using the ``root`` account)::
      ln -s /usr/share/bash-completion/completions/virtualenvwrapper /etc/bash_completion.d/virtualenvwrapper
 . Set-up the virtual environment (using the ``ansible`` account):
    .. warning::
       If you are already logged-in as user ``ansible`` in the server, you will
       need to log-out and log-in again in order to be able to use
       ``virtualenvwrapper`` commands!
    ::
      mkdir ~/mysite/
      mkvirtualenv -a ~/mysite/ mysite
      pip install -U pip setuptools
      pip install 'ansible~=10.3.0' netaddr
+     pip install 'ansible~=10.3.0' netaddr passlib
 .. warning::
    The ``netaddr`` package is needed for ``ipv4/ipv6`` lookup plugins
    which is used internally by some of the roles.
    which is used internally by some of the roles. The ``passlib``
    package is required in order to hash passwords when creating system
    users.
 Cloning the *Majic Ansible Roles*
 ---------------------------------
 With most of the software pieces in place, the only missing thing is the Majic
 Ansible Roles:
 . Clone the git repository::
      git clone https://code.majic.rs/majic-ansible-roles ~/majic-ansible-roles
 . Checkout the correct version of the roles::
      cd ~/majic-ansible-roles/
      git checkout -b 8.0-dev 8.0-dev
 Preparing the basic site configuration
 --------------------------------------
 Phew... Now that was a bit tedious and boring... But at least you are now ready
 to set-up your own site :)
 First of all, let's set-up some basic directory structure and configuration:
 . Create Ansible configuration file.
    .. warning::
       Since Ansible 2.x has introduced much stricter controls over security of
       deployed Python scripts, it is recommended (as in this example) to use the
       ``pipelining`` option (which should also improve performance). This is in
       particular necessary in cases where the SSH user connecting to remote
       machine is *not* ``root``, but there are tasks that use ``become`` with
       non-root ``become_user`` (which is the case in Majic Ansible Roles). See
       `official documentation
       <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_
       and other alternatives to this.
    :file:`~/mysite/ansible.cfg`
    ::
      [defaults]
      roles_path=/home/ansible/majic-ansible-roles/roles:/home/ansible/mysite/roles
      force_handlers = True
      inventory = /home/ansible/mysite/hosts
      interpreter_python = /usr/bin/python3
      [ssh_connection]
      pipelining = True
 . Create directory where retry files will be stored at (so they woudln't
    pollute your home directory)::
      mkdir ~/mysite/retry
 . Create the inventory file.
    :file:`~/mysite/hosts`
    ::
      [preseed]
      localhost ansible_connection=local
      [communications]
      comms.example.com
      [web]
      www.example.com
      [backup]
      bak.example.com
 . Create a number of directories for storing playbooks, group
    variables, SSH keys, X.509 artefacts (for TLS), and GnuPG keyring
    (we'll get to this later)::
      mkdir ~/mysite/playbooks/
      mkdir ~/mysite/group_vars/
      mkdir ~/mysite/ssh/
      mkdir ~/mysite/tls/
      mkdir ~/mysite/gnupg/
 . Create SSH private/public key pair that will be used by Ansible for
    connecting to destination servers, as well as for some roles::
      ssh-keygen -f ~/.ssh/id_rsa -N ''
 Protecting communications using TLS
 -----------------------------------
 In order to protect the communications between users and servers, as
 well as between servers themselves, it is important to set-up and
 properly configure TLS for each role.
 *Majic Ansible Roles* mandates use of TLS wherever possible. In other
 words, *you must* have TLS private keys and certificates issued by
 some CA for all servers in order to be able to use most of the
 roles. The private keys and certificates are primarily meant to be
 generated *per service*, and that is the approach we will pursue here
 as well.
 TLS private keys should be ideally generated locally and kept in a
 safe environment (possibly encrypted until needed), while the X.509
 certificates should be issued by a relevant certification
 authority. You can choose to roll-out your own CA, use one of the
 public CAs, or perhaps go for a mix of both.
 For the purpose of this guide, we'll set-up a small simple local CA to
 issue all the necessary certificates, and we'll generate the private
 keys and issue server certificates on the go as needed, storing them
 all under the ``~/mysite/tls/`` directory.
 So, let us make a slight detour to create a CA of our own:
 . First off, install a couple more tools on the Ansible server. We
    will be using ``certtool`` for our improvised CA needs (run this as
    ``root``)::
      apt-get install -y gnutls-bin
 . Create a template for the ``certtool`` so it would know what
    extensions and content to have in the CA certificate:
    :file:`~/mysite/tls/ca.cfg`
    ::
       organization = "Example Inc."
       country = "SE"
       cn = "Example Inc. Test Site CA"
       expiration_days = 1825
       ca
       cert_signing_key
       crl_signing_key
 . Almost there... Now let us generate the CA private key and
    self-signed certificate::
      certtool --sec-param high --generate-privkey --outfile ~/mysite/tls/ca.key
      certtool --template ~/mysite/tls/ca.cfg --generate-self-signed --load-privkey ~/mysite/tls/ca.key --outfile ~/mysite/tls/ca.pem
 . And just one more small tweak - we need to provide a truststore PEM
    file containing all CA certificates in the chain for services to be
    able to connect to each-other (where necessary). In this particular
    case we have a super-simple hierarchy (root CA is also issuing the
    end entity certificates), so simply make a copy of the ``ca.pem``::
      cp ~/mysite/tls/ca.pem ~/mysite/tls/truststore.pem
 .. note::
    A useful feature that all roles implement is a check to see if
    certificates will expire within the next 30 days. This check is
    performed via cronjob at midnight, and failing results will end-up
    being delivered to the ``root`` user on local server. Later on,
    once you have configured the mail server, you should be able to
    set-up the necessary aliases to have the mails delivered to
    non-local accounts too.
 Preseed files
 -------------
 The ``preseed`` role is useful for generating Debian preseed files. Preseed
 files can be used for automating the Debian installation process.
 Preseed files are created on the Ansible controller, and then supplied
 to Debian installer.
 So, let's set this up for start:
 . First of all, create the playbook for generating the preseed files locally.
    :file:`~/mysite/playbooks/preseed.yml`
    ::
       ---
       - hosts: preseed
         roles:
           - preseed
 . Now we need to configure the role. Two parameters are mandatory -
    one that specifies where the preseed files are to be stored, and
    one that specifies the public key that should be used to
    pre-populate the SSH authorized keys for the ``root`` account. This
    is required for the initial bootstrap of servers because Debian
    GNU/Linux does not by default allow the ``root`` user to
    authenticate via SSH using a password. We will use the SSH public
    key generated earlier via the ``ssh-keygen`` command. Create the
    configuration file:
    :file:`~/mysite/group_vars/preseed.yml`
    ::
       ---
       # Public key used to authenticate remote logins via SSH for the
       # root account.
       ansible_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
       # Directory where the preseed files will be output to.
       preseed_directory: "~/mysite/preseed_files/"
 . Now we can generate the pressed files::
      workon mysite && ansible-playbook playbooks/preseed.yml
 . If all went well, you should have the following files created:
    * :file:`~/mysite/preseed_files/comms.example.com.cfg`
    * :file:`~/mysite/preseed_files/www.example.com.cfg`
    * :file:`~/mysite/preseed_files/bak.example.com.cfg`
 . You can have a look at them, but you might notice the settings in the file
    might not be to your liking. In particular, it could be using wrong timezone,
    defaulting to DHCP for network configuration etc. Let's concentrate on making
    the network configuration changes - this is the main thing that will probably
    differ in your environment. Update the preseed configuration file:
    :file:`~/mysite/group_vars/preseed.yml`
    ::
       ---
       # Public key used to authenticate remote logins via SSH for the
       # root account.
       ansible_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
       # Directory where the preseed files will be output to.
       preseed_directory: "~/mysite/preseed_files/"
       # Set your default (initial) root password.
       preseed_root_password: changeit
       # Use manual network configuration (no DHCP).
       preseed_network_auto: no
       # Set the gateway for all servers.
       preseed_gateway: 10.32.64.1
       # Set the netmask for all servers.
       preseed_netmask: 255.255.255.0
       # Set the DNS for all servers.
       preseed_dns: 10.32.64.1
       # Set the domain for all servers.
       preseed_domain: example.com
       # Set the server-specific options.
       preseed_server_overrides:
         comms.example.com:
           hostname: comms
           ip: 10.32.64.19
         www.example.com:
           hostname: www
           ip: 10.32.64.20
         bak.example.com:
           hostname: bak
           ip: 10.32.64.23
 . Now re-run the preseed playbook::
      workon mysite && ansible-playbook playbooks/preseed.yml
 . The preseed files should have been updated now, and you should have the new
    customised configuration files in the ``preseed_files`` directory. You can
    now use these to install the servers.
 Installing the servers with preseed files
 -----------------------------------------
 You have your preseed files now, so you can go ahead and install the
 servers ``comms.example.com``, ``www.example.com``, and
 ``bak.example.com`` using them with network install CD. Have a look at
 `Debian instructions
 <https://www.debian.org/releases/bookworm/amd64/apb.en.html>`_ for
 more details.
 If you need to, you can easily serve the preseed files from the Ansible server
 with Python's built-in HTTP server::
   cd ~/mysite/preseed_files/
   python3 -m http.server 8000
 Then you can point installer to the preseed file selecting the
 ``Advanced options -> Automated install`` (don't press ``ENTER`` yet),
 then pressing ``TAB``, and appending the following at the end (just
 fill-in the correct hostname - ``comms``, ``www``, or ``bak``)::
   url=http://ansible.example.com:8000/HOSTNAME.example.com.cfg
 Bootstrapping servers for Ansible set-up
 ----------------------------------------
 In order to effectively use Ansible, a small initial bootstrap always has to be
 done for managed servers. This mainly involves set-up of Ansible users on the
 destination machine, and distributing the SSH public keys for authorisation.
 When you use the preseed configuration files to deploy a server, you get the
 benefit of having the authorized_keys set-up for the root operating system user,
 making it easier to bootstrap the machines subsequently via Ansible.
 Let's bootstrap our machines now:
 . For start, create a dedicated playbook for the bootstrap process.
    :file:`~/mysite/playbooks/bootstrap.yml`
    ::
       ---
       - hosts:
           - communications
           - web
           - backup
         remote_user: root
         roles:
           - bootstrap
 . The ``bootstrap`` role has only one parameter - an SSH key which
    should be deployed for the Ansible user on managed server (in the
    ``authorized_keys`` file). Since this role is applied against all
    servers, we will use the same value everywhere. Configure the role:
    :file:`~/mysite/group_vars/all.yml`
    ::
       ---
       ansible_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
 . SSH into all machines at least once from the Ansible server in order to
    store the SSH fingerprints into known hosts file::
      ssh root@comms.example.com date && \
      ssh root@www.example.com date && \
      ssh root@bak.example.com date
 . Now, simply run the bootstrap role against the servers::
      workon mysite && ansible-playbook playbooks/bootstrap.yml
 . At this point you won't be able to ssh into the machines with the
    ``root`` account anymore. You will be able to ssh into the machines
    using the ``ansible`` user (from the Ansible server). The
    ``ansible`` user will also be granted ability to run the ``sudo``
    commands without providing password.
 . Now you can finally move on to configuring what you really want -
    common configuration and services for your site.
 Common server configuration
 ---------------------------
 Each server needs to share some common configuration in order to be functioning
 properly. This includes set-up of some shared accounts, perhaps some hardening
 etc.
 .. note::
    Should you ever need to limit what hosts can connect to a server
    for some kind of maintenance or upgrade purposes, the ``common``
    role comes with ``maintenance`` and ``maintenance_allowed_sources``
    parameters. See :ref:`rolereference` for more information.
 Let's take care of this common configuration right away:
 . Create playbook for the communications server:
    :file:`~/mysite/playbooks/communications.yml`
    ::
       ---
       - hosts: communications
         remote_user: ansible
         become: yes
         roles:
           - common
 . Create playbook for the web server:
    :file:`~/mysite/playbooks/web.yml`
    ::
       ---
       - hosts: web
         remote_user: ansible
         become: yes
         roles:
           - common
 . Create playbook for the backup server:
    :file:`~/mysite/playbooks/backup.yml`
    ::
       ---
       - hosts: backup
         remote_user: ansible
         become: yes
         roles:
           - common
 . Create the global site playbook:
    :file:`~/mysite/playbooks/site.yml`
    ::
       ---
       - import_playbook: preseed.yml
       - import_playbook: communications.yml
       - import_playbook: web.yml
       - import_playbook: backup.yml
 . Time to create configuration for the role. Since this role is supposed to
    set-up a common base, we'll set-up the variables file that applies to all
    roles:
    :file:`~/mysite/group_vars/all.yml`
    ::
       ---
       ansible_key: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
       os_users:
         - name: admin
           uid: 1000
           additional_groups:
             - sudo
           authorized_keys:
             - "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
           password: "{{ 'admin' | password_hash('sha512') }}"
       common_packages:
         - emacs-nox
       ca_certificates:
         truststore: "{{ lookup('file', '~/mysite/tls/truststore.pem') }}"
    .. note::
       The ``common`` role comes with ability to set-up time
       synchronisation using NTP. This is not done by default. For
       details see the role parameter ``ntp_pools``.
    .. note::
       The ``ca_certificates`` parameter lets us deploy custom CA
       certificates on servers. The name we pick (in this case
       ``truststore``) can be set to anything. In this particular case,
       we want to deploy our own CA certificate for use as truststore,
       since this is what the services will use to validate server
       certificates when connecting to each-other.
 . That's all for configuration, time to apply the changes::
      workon mysite && ansible-playbook playbooks/site.yml
 . After this you should be able to *ssh* from Ansible server onto the
    managed servers as user ``admin`` using the *SSH* private key of
    the ``ansible`` user on controller machine. The ``admin`` user's
    password has also been set to ``admin``, and the user will be
    member of ``sudo`` group.
    .. note::
       Remote logins over SSH using password authentication are
       explicitly disabled as part of common set-up/hardening.
 Introducing LDAP
 ----------------
 Since some of the services actually depend on LDAP, we'll go ahead and set that
 one up first. This includes both the LDAP *server* and *client* configuration.
 . Update the playbook for communications server to include the LDAP client and
    server roles (``ldap_client`` and ``ldap_server``, respectively).
    :file:`~/mysite/playbooks/communications.yml`
    ::
       ---
       - hosts: communications
         remote_user: ansible
         become: yes
         roles:
           - common
           - ldap_client
           - ldap_server
 . Update the playbook for web server to include the LDAP client role
    (``ldap_client``). You never know when it might come in handy :)
    :file:`~/mysite/playbooks/web.yml`
    ::
       ---
       - hosts: web
         remote_user: ansible
         become: yes
         roles:
           - common
           - ldap_client
 . Time to configure the roles. For start, let us configure the LDAP
    server role. Keep in mind that there is a lot of default variables
    set-up by the role itself, making our config rather short. The
    ``ldap_server_domain`` parameter will be used to form the base DN
    of the LDAP directory (resulting in ``dc=example,dc=com``).
    :file:`~/mysite/group_vars/communications.yml`
    ::
       ---
       ldap_admin_password: admin
       ldap_server_domain: example.com
       ldap_server_organization: "Example Inc."
       ldap_server_tls_certificate: "{{ lookup('file', '~/mysite/tls/comms.example.com_ldap.pem') }}"
       ldap_server_tls_key: "{{ lookup('file', '~/mysite/tls/comms.example.com_ldap.key') }}"
 . Phew. That was... Well, actually, easy :) Technically, only the
    LDAP admin password, domain, and TLS certificate/key *must* be set,
    but it is nice to have organisation explicitly specified as well
    (instead of using whatever Debian picks as default). Let us add the
    LDAP client configuration next. We will start off with global LDAP
    client configuration. In case of the LDAP client role, we have got
    to be a bit more explicit.
    :file:`~/mysite/group_vars/all.yml`
    ::
       # Observe how we set the base DN. By default the ldap_server role
       # (defined up there) will use server's domain to form the base for LDAP.
       ldap_client_config:
         - comment: Set the base DN
           option: BASE
           value: dc=example,dc=com
         - comment: Set the default URI
           option: URI
           value: ldap://comms.example.com/
         - comment: Set the LDAP TLS truststore
           option: TLS_CACERT
           value: /etc/ssl/certs/truststore.pem
         - comment: Enforce TLS
           option: TLS_REQCERT
           value: demand
 . Ok, so this looks nice and dandy... But, let's have a bit better
    configuration on the communications server itself. Namely, on that
    one we should be able to connect to the LDAP server via unix socket
    instead of TCP.
    :file:`~/mysite/group_vars/communications.yml`
    ::
       ldap_client_config:
         - comment: Set the base DN
           option: BASE
           value: dc=example,dc=com
         - comment: Set the default URI
           option: URI
           value: ldapi:///
         - comment: Set the default bind DN, useful for administration.
           option: BINDDN
           value: cn=admin,dc=example,dc=com
         - comment: Set the LDAP TLS truststore
           option: TLS_CACERT
           value: /etc/ssl/certs/truststore.pem
         - comment: Enforce TLS
           option: TLS_REQCERT
           value: demand
 . Ok, time to re-run the playbooks again... Wait a minute, something
    is missing here... Ah, right, we have to generate the TLS private
    key and issue the X.509 certificate.
 . Create template for the ``certtool`` so it would know what
       extensions and content to have in the certificate:
       :file:`~/mysite/tls/comms.example.com_ldap.cfg`
       ::
          organization = "Example Inc."
          country = SE
          cn = "Exampe Inc. LDAP Server"
          expiration_days = 365
          dns_name = "comms.example.com"
          tls_www_server
          signing_key
          encryption_key
 . Almost there... Now let us generate the key and issue the certificate::
         certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/comms.example.com_ldap.key
         certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/comms.example.com_ldap.cfg --load-privkey ~/mysite/tls/comms.example.com_ldap.key --outfile ~/mysite/tls/comms.example.com_ldap.pem
 . And now, for the finishing touch, just run the playbooks again::
      workon mysite && ansible-playbook playbooks/site.yml
 Adding mail server
 ------------------
 The next thing in line is to implement the mail server capability. *Majic
 Ansible Roles* come with two distinct mail server-related roles. One for
 setting-up a mail server host (with authenticated IMAP, SMTP, mail storage etc),
 and one for setting-up a local SMTP mail forwarder (for having the rest of your
 servers relay their mails to the mail server host).
 .. note::
    Should you ever need to deploy the forwarder role on a laptop or
    machine behind NAT, make sure to look at the
    ``smtp_from_relay_allowed`` parameter. In case you need to connect
    to the SMTP relay via non-standard port (for example to work-around
    ISP blocks), have a look at the ``smtp_relay_host_port`` parameter.
 The mail server role looks-up available mail domains, users, and aliases in the
 LDAP directory. This has already been set-up on the server
 ``comms.example.com``, but some changes will be required.
 . Update the playbook for communications server to include the mail server
 role.
     :file:`~/mysite/playbooks/communications.yml`
     ::
       ---
       - hosts: communications
         remote_user: ansible
         become: yes
         roles:
           - common
           - ldap_client
           - ldap_server
           - mail_server
 . Let's configure the role next.
    :file:`~/mysite/group_vars/communications.yml`
    ::
       # Set the LDAP URL to connect through. Keep in mind TLS is required.
       mail_ldap_url: ldap://comms.example.com/
       # Here we need to point to the base DN of LDAP server. A bunch of entries
       # will need to exist under it for service to function correctly, though.
       mail_ldap_base_dn: dc=example,dc=com
       # Separate LDAP entries are used for Postfix/Dovecot
       # authentication. Therefore we have two passwords here.
       mail_ldap_postfix_password: postfix
       mail_ldap_dovecot_password: dovecot
       # Setting uid/gid is optional, but you might have a policy on how to
       # assign UIDs and GIDs, so it is convenient to be able to change this.
       mail_user_uid: 5000
       mail_user_gid: 5000
       # Set private keys and certificates to use for the IMAP service.
       imap_tls_certificate: "{{ lookup('file', '~/mysite/tls/comms.example.com_imap.pem') }}"
       imap_tls_key: "{{ lookup('file', '~/mysite/tls/comms.example.com_imap.key') }}"
       # Set private keys and certificates to use for the SMTP service.
       smtp_tls_certificate: "{{ lookup('file', '~/mysite/tls/comms.example.com_smtp.pem') }}"
       smtp_tls_key: "{{ lookup('file', '~/mysite/tls/comms.example.com_smtp.key') }}"
       # Set the X.509 certificate truststore to use for validating the
       # LDAP server certificate.
       mail_ldap_tls_truststore: "{{ lookup('file', '~/mysite/tls/truststore.pem') }}"
 . There are two distinct mail services that need to access the LDAP directory -
    *Postfix* (serving as an SMTP server), and *Dovecot* (serving as an IMAP
    server). These two need their own dedicated LDAP entries on the LDAP server in
    order to log-in. Luckily, it is easy to create such entries through the options
    provided by the LDAP server role. In addition to this, the Postfix and Dovecot
    services will check if users are members of ``mail`` group in LDAP directory
    before accepting them as valid mail users. Once again, the LDAP server role
    comes with a simple option for creating groups.
    :file:`~/mysite/group_vars/communications.yml`
    ::
       # Don't forget, the passwords here must match with passwords specified
       # under options mail_ldap_postfix_password/mail_ldap_dovecot_password.
       ldap_server_consumers:
         - name: postfix
           password: postfix
         - name: dovecot
           password: dovecot
       ldap_server_groups:
         - name: mail
 . Ok, so now our SMTP and IMAP service can log-in into the LDAP server to
    look-up the mail server information. We have also defined the mail group for
    limitting which users get mail service. However, we don't have any
    user/domain information yet. So let's change that, using the ``ldap_entries``
    option from LDAP server role.
    .. warning::
       Long-term, you probably want to manage these entries manually or through
       other means than the ``ldap_entries`` option. The reason for this is
       because this type of data in LDAP directory can be considered more of an
       operational/application data than configuration data that frequently
       changes (especially the user passwords/info). Backups of LDAP directory on
       regular basis are important. We will get to that at a later point.
    :file:`~/mysite/group_vars/communications.yml`
    ::
       ldap_entries:
         # Create first a couple of user entries. Don't forget to set the
         # "mail" attribute for them.
         - dn: uid=johndoe,ou=people,dc=example,dc=com
           attributes:
             objectClass:
               - inetOrgPerson
             uid: johndoe
             cn: John Doe
             sn: Doe
             userPassword: johndoe
             mail: john.doe@example.com
         - dn: uid=janedoe,ou=people,dc=example,dc=com
           attributes:
             objectClass:
               - inetOrgPerson
             uid: janedoe
             cn: Jane Doe
             sn: Doe
             userPassword: janedoe
             mail: jane.doe@example.com
         # Let's register our domain in LDAP directory.
         - dn: dc=example.com,ou=domains,ou=mail,ou=services,dc=example,dc=com
           attributes:
             objectClass: dNSDomain
             dc: "example.com"
           # Finally, for the lolz, let's also add the standard postmaster alias
           # for our domain. This one will also receive any undeliverable bounced
           # mails.
         - dn: cn=postmaster@example.com,ou=aliases,ou=mail,ou=services,dc=example,dc=com
           attributes:
             objectClass: nisMailAlias
             cn: postmaster@example.com
             rfc822MailMember: john.doe@example.com
 . Once again, before we apply the configuration, we must make sure the
    necessary TLS private keys and certificates are available. In this particular
    case, we need to set-up separate key/certificate pair for both the SMTP and
    IMAP service:
 . Create new templates for ``certtool``:

requirements.in

➞

Show inline comments

 # Ansible and role runtime.
 ansible~=10.3
 netaddr
 passlib
 python-ldap
 # Development and testing.
 ansible-lint
 defusedxml
 flake8
 gimmecert
 molecule[testinfra]~=24.8.0
 molecule-plugins[vagrant]~=23.5.0
 paramiko
 # Documentation.
 sphinx
 sphinx-rtd-theme
 # Python virtualenv management.
 pip
 pip-tools
 setuptools

0 comments (0 inline, 0 general)