About Majic Ansible Roles

=========================

Majic Ansible Roles is a collection of Ansible roles that are used on regular

basis for deployment and maintenance of Majic infrastructure.

The roles are kept as a separate project in hope of making them potentially

useful to wider audience, and for reference purposes.

Roles cover different aspects of infrastructure, such as mail servers, web

servers, web applications etc. The roles are mainly well-suited for smaller

installations.

Roles are written for use with *Debian GNU/Linux*. For more details on

supported releases, see :ref:`rolereference`.

At the moment, the roles have been written for and tested against **Ansible

2.9.x**.

The roles also utilise the ``ipv4/ipv6`` lookup plugins which require

Why were these roles created?

-----------------------------

For a long time I have had a couple of Internet-facing servers where I hosted

all the IT infrastructure I needed for my day-to-day life.

This started off with some basic services, like mail and XMPP server, and in

time extended to include a web server, code repository etc.

As the number of services I used grew, I found it more difficult to track

updates and upgrades, let alone test them in reliable way. The biggest problem

in particular was lack of time to properly document all the different things

I've set-up.

Being familiar with some Puppet-based deployments, I've started looking into the

possibility of using a configuration management system. Ansible emerged as

something that I thought would be easy to use, due to its agent-less nature.

Once I passed some basic tutorials and got to know the system a bit, I decided

to start my journey on implementing the different roles, in the way I want them,

that would let me easily set-up my servers (and reinstall them, amongst other

things).

The roles you see within this repository are the fruit of this labour. I hope

you find them useful.

Features

--------

*Majic Ansible Roles* have the following features:

.. warning::

   Of course, you may want to take some statements with a pinch of salt, and

   possibly attribute them to either delusions of grandeur, or bragging :)

* Emphasis on small, self-hosted deployments.

* Modular role design where possible and where necessary.

* A number of roles covering common set-up of servers, databases, web server,

  XMPP server, mail server, and LDAP server.

* Streamlined integration with LDAP server for most of the services.

* Well-documented, with role reference documentation, examples, and test/sample

  site.

* Balanced implementation allowing both configurability and ease of deployment.

* Free Software, released under liberal BSD license.

Available roles:

* ``bootstrap`` (for setting-up servers for Ansible use)

* ``common`` (for setting-up basic security, accounts, and configuration on

  servers)

* ``database`` (for creating databases to be used for applications)

* ``database_server`` (for deploying a database server, MariaDB)

* ``ldap_client`` (for setting-up LDAP client configuration)

* ``ldap_server`` (for deploying an LDAP server, OpenLDAP)

* ``mail_forwarder`` (for setting-up forwarding of local mails to smart host,

  Postfix)

* ``mail_server`` (for deploying Postfix, Dovecot, ClamAV)

* ``php_website`` (for deploying PHP websites)

* ``preseed`` (for preparing Debian preseed files)

* ``prosody`` (for deploying XMPP server, Prosody)

* ``web_server`` (for deploying web server, Nginx)

* ``wsgi_website`` (for deploying WSGI/Python applications)

Support

-------

In case of problems with the roles or provided code, please do not hestitate to

contact the author at **majic-ansible-roles (at) majic.rs**. The project can be

found at:

* https://code.majic.rs/majic-ansible-roles

* https://projects.majic.rs/majic-ansible-roles

License

-------

Majic Ansible Roles is released under terms of *BSD (3-Clause) License*::

  Copyright (c) 2018, Branko Majic

  All rights reserved.

  Redistribution and use in source and binary forms, with or without modification,

  are permitted provided that the following conditions are met:

    Redistributions of source code must retain the above copyright notice, this

    list of conditions and the following disclaimer.

    Redistributions in binary form must reproduce the above copyright notice, this

    list of conditions and the following disclaimer in the documentation and/or

    other materials provided with the distribution.

Majic Ansible Roles documentation

=================================

Majic Ansible Roles is a collection of Ansible roles that are used on regular

basis for deployment and maintenance of Majic infrastructure.

The roles are kept as a separate project in hope of making them potentially

useful to wider audience, and for reference purposes.

Roles cover different aspects of infrastructure, such as mail servers, web

servers, web applications etc. The roles are mainly well-suited for smaller

installations.

Roles are written for use with *Debian GNU/Linux*. For more details on

supported releases, see :ref:`rolereference`.

At the moment, the roles have been written for and tested against **Ansible

2.9.x**.

The roles also utilise the ``ipv4/ipv6`` lookup plugins which require

Contents

========

.. toctree::

   :maxdepth: 2

   about

   usage

   rolereference

   development

   releaseprocedures

   releasenotes

Indices and tables

==================

* :ref:`genindex`

* :ref:`modindex`

* :ref:`search`

Release notes

=============

x.y.z

-----

**Breaking changes:**

* All roles

  * Dropped support for Debian 11 (Bullseye).

**New features/improvements**

* ``backup_client`` role

  * Switched to using Paramiko + SFTP backend (instead of pexpect +

    SFTP), which should improve the backup performance.

**Bug fixes:**

* ``common`` role

  * Fixed permission errors with Python cache directories in the pip

    requirements upgrade checks virtual environment that can happen if

    the initial virtual environment set-up fails.

8.0.0

-----

Dropped support for Python 2.7 and Debian 10 Buster. Added support for

Debian 12 Bookworm. Numerous minor improvements and fixes.

**Breaking changes:**

* All roles

  * Dropped support for Debian 10 (Buster).

  * Added support for Debian 12 (Bookworm).

  * ``netaddr`` Python package is now required for using the roles.

  * ``dnspython`` Python package is no longer required for using the

    roles.

* ``backup_client`` role

  * Previously the backup would run even if pre-backup scripts would

    fail. This is no longer the case, and all pre-backup scripts must

    exit with non-zero exit code in order for backup process to

    kick-in.

  * Old backups are now automatically purged after successful

    backup. This could lead to longer runtimes for entire backup

    process, as well as higher CPU usage.

* ``common`` role

  * Dropped support for Python 2.7 pip requirements upgrade

    checks. Only Python 3 is supported now.

    Requirements (input) files for Python 3 are now put under the

    ``/etc/pip_check_requirements_upgrades`` directory instead of

    ``/etc/pip_check_requirements_upgrades-py3``.

    The ``pip_check_requirements_py3`` /

    ``pip_check_requirements_py3_in`` role parameters have been

    renamed to ``pip_check_requirements`` /

    ``pip_check_requirements_in``.

  * Parameter ``maintenance_allowed_hosts`` has been dropped and

    replaced with parameter ``maintenance_allowed_sources``. The new

    parameter expects a list of IPv4 and IPv6 addresses (or

    subnets). Resolvable names can no longer be specified.

  * NTP server configuration is now based on use of pools instead of

    servers. Parameter ``ntp_servers`` has been deprecated and

    replaced with parameter ``ntp_pools``.

* ``ldap_server`` role

  * Starting with Debian 12 Bookworm, the role no longer deploys

    *rsyslog* and *logrotate* configuration for writing and rotating

    the LDAP servers logs under ``/var/log/slapd.log``. Primary

    reason is that Debian 12 Bookworm no longer installs *rsyslog* by

    default, and it is considered to be deprecated at this point. The

    LDAP server logs can be read via ``journalctl -u slapd`` when

    necessary.

* ``mail_forwarder`` role

  * Firewall rules for incoming connections from the SMTP relay server

    are now based on relay's IPv4 and IPv6 addresses as resolved on

    managed machine during deployment time.

    In case the SMTP relay server's IP addresses change, the role

    needs to get reapplied against managed machines for those changes

    to take place.

    This change in behaviour was introduced to avoid firewall-related

    errors due to inability to resolve names via DNS servers during

    boot time.

* ``mail_server`` role

  * Parameter ``mail_server_tls_protocols`` has been dropped and

    replaced with parameter ``mail_server_minimum_tls_protocol``. Full

    list of TLS protocols can no longer be specified, only the minimum

    one.

  able to communicate over network with each-other, the Ansible

  servers, and with Internet. Debian Bookworm will be installed on top

  of these servers as part of the usage instructions.

* Debian Bookworm network installation CD.

* All servers should be on the same network.

* IP addresses for all servers should be known.

* Netmask for all servers should be known.

* Gateway for all servers should be known.

In case of the servers listed above, it might be safest to have them

as virtual machines - this is cheapest thing to do, and simplest (who

wants to deal with pesky hardware anyway?).

Usage instructions assume the following:

* Domain used for all servers is ``example.com``. If you wish to use a different

  domain, adjust the instructions accordingly.

* Server hostnames are ``ansible``, ``comms``, ``www``, and ``bak`` (for Ansible

  server, communications server, web server, and backup server, respectively).

Installing the OS on Ansible server

-----------------------------------

Start-off by installing the operating system on the Ansible server:

1. Fire-up the ``ansible`` server, and boot from the network installation CD.

2. Select the **Install** option.

3. Pick **English** as language.

4. Pick the country you are living in (or whatever else you want).

5. Pick the **en_US.UTF-8** locale.

6. Pick the **American English** keymap.

7. Configure the network if necessary.

8. Set the hostname to ``ansible``.

9. Set the domain to ``example.com``.

10. Set the root password.

11. Create a new user. For simplicity, call the user **Ansible user**, with

    username **ansible**.

12. Set-up partitioning in any way you want. You can go for **Guided - use

    entire disk** if you want to keep it simple and are just testing things.

13. Wait until the base system has been installed.

14. Pick whatever Debian archive mirror is closest to you.

15. If you have an HTTP proxy, provide its URL.

16. Pick if you want to participate in package survey or not.

17. Make sure that at least the **standard system utilities** and **SSH server**

    options are selected on task selection screen.

18. Wait for packages to be installed.

19. Install the GRUB boot loader on MBR.

20. Finalise the server install, and remove the installation media from server.

Installing required packages

----------------------------

With the operating system installed, it is necessary to install a couple of

packages, and to prepare the environment a bit on the Ansible server:

1. Install the necessary system packages (using the ``root`` account)::

     apt-get install -y virtualenv virtualenvwrapper git python3-pip python3-dev libffi-dev libssl-dev

2. Set-up loading of ``virtualenvwrapper`` via Bash completions (using the ``root`` account)::

     ln -s /usr/share/bash-completion/completions/virtualenvwrapper /etc/bash_completion.d/virtualenvwrapper

3. Set-up the virtual environment (using the ``ansible`` account):

   .. warning::

      If you are already logged-in as user ``ansible`` in the server, you will

      need to log-out and log-in again in order to be able to use

      ``virtualenvwrapper`` commands!

   ::

     mkdir ~/mysite/

     mkvirtualenv -a ~/mysite/ mysite

     pip install -U pip setuptools

.. warning::

   The ``netaddr`` package is needed for ``ipv4/ipv6`` lookup plugins

Cloning the *Majic Ansible Roles*

---------------------------------

With most of the software pieces in place, the only missing thing is the Majic

Ansible Roles:

1. Clone the git repository::

     git clone https://code.majic.rs/majic-ansible-roles ~/majic-ansible-roles

2. Checkout the correct version of the roles::

     cd ~/majic-ansible-roles/

     git checkout -b 8.0-dev 8.0-dev

Preparing the basic site configuration

--------------------------------------

Phew... Now that was a bit tedious and boring... But at least you are now ready

to set-up your own site :)

First of all, let's set-up some basic directory structure and configuration:

1. Create Ansible configuration file.

   .. warning::

      Since Ansible 2.x has introduced much stricter controls over security of

      deployed Python scripts, it is recommended (as in this example) to use the

      ``pipelining`` option (which should also improve performance). This is in

      particular necessary in cases where the SSH user connecting to remote

      machine is *not* ``root``, but there are tasks that use ``become`` with

      non-root ``become_user`` (which is the case in Majic Ansible Roles). See

      `official documentation

      <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_

      and other alternatives to this.

   :file:`~/mysite/ansible.cfg`

   ::

     [defaults]

     roles_path=/home/ansible/majic-ansible-roles/roles:/home/ansible/mysite/roles

     force_handlers = True

     inventory = /home/ansible/mysite/hosts

     interpreter_python = /usr/bin/python3

     [ssh_connection]

     pipelining = True

2. Create directory where retry files will be stored at (so they woudln't

   pollute your home directory)::

     mkdir ~/mysite/retry

3. Create the inventory file.

   :file:`~/mysite/hosts`

   ::

     [preseed]

     localhost ansible_connection=local

     [communications]

     comms.example.com

     [web]

     www.example.com

     [backup]

     bak.example.com

4. Create a number of directories for storing playbooks, group

   variables, SSH keys, X.509 artefacts (for TLS), and GnuPG keyring

   (we'll get to this later)::

     mkdir ~/mysite/playbooks/

     mkdir ~/mysite/group_vars/

     mkdir ~/mysite/ssh/

     mkdir ~/mysite/tls/

     mkdir ~/mysite/gnupg/

5. Create SSH private/public key pair that will be used by Ansible for

   connecting to destination servers, as well as for some roles::

     ssh-keygen -f ~/.ssh/id_rsa -N ''

Protecting communications using TLS

-----------------------------------

In order to protect the communications between users and servers, as

# Ansible and role runtime.

ansible~=10.3

netaddr

python-ldap

# Development and testing.

ansible-lint

defusedxml

flake8

gimmecert

molecule[testinfra]~=24.8.0

molecule-plugins[vagrant]~=23.5.0

paramiko

# Documentation.

sphinx

sphinx-rtd-theme

# Python virtualenv management.

pip

pip-tools

setuptools