About Majic Ansible Roles

=========================

Majic Ansible Roles is a collection of Ansible roles that are used on regular

basis for deployment and maintenance of Majic infrastructure.

The roles are kept as a separate project in hope of making them potentially

useful to wider audience, and for reference purposes.

Roles cover different aspects of infrastructure, such as mail servers, web

servers, web applications etc. The roles are mainly well-suited for smaller

installations.

Roles are written for use with *Debian GNU/Linux*. For more details on

supported releases, see :ref:`rolereference`.

At the moment, the roles have been written for and tested against **Ansible

2.9.x**.

The roles also utilise the ``ipv4/ipv6`` lookup plugins which require

Why were these roles created?

-----------------------------

For a long time I have had a couple of Internet-facing servers where I hosted

all the IT infrastructure I needed for my day-to-day life.

This started off with some basic services, like mail and XMPP server, and in

time extended to include a web server, code repository etc.

As the number of services I used grew, I found it more difficult to track

updates and upgrades, let alone test them in reliable way. The biggest problem

in particular was lack of time to properly document all the different things

I've set-up.

Being familiar with some Puppet-based deployments, I've started looking into the

possibility of using a configuration management system. Ansible emerged as

something that I thought would be easy to use, due to its agent-less nature.

Once I passed some basic tutorials and got to know the system a bit, I decided

to start my journey on implementing the different roles, in the way I want them,

that would let me easily set-up my servers (and reinstall them, amongst other

things).

The roles you see within this repository are the fruit of this labour. I hope

you find them useful.

Features

--------

*Majic Ansible Roles* have the following features:

.. warning::

   Of course, you may want to take some statements with a pinch of salt, and

   possibly attribute them to either delusions of grandeur, or bragging :)

* Emphasis on small, self-hosted deployments.

* Modular role design where possible and where necessary.

* A number of roles covering common set-up of servers, databases, web server,

  XMPP server, mail server, and LDAP server.

* Streamlined integration with LDAP server for most of the services.

* Well-documented, with role reference documentation, examples, and test/sample

  site.

* Balanced implementation allowing both configurability and ease of deployment.

* Free Software, released under liberal BSD license.

Majic Ansible Roles documentation

=================================

Majic Ansible Roles is a collection of Ansible roles that are used on regular

basis for deployment and maintenance of Majic infrastructure.

The roles are kept as a separate project in hope of making them potentially

useful to wider audience, and for reference purposes.

Roles cover different aspects of infrastructure, such as mail servers, web

servers, web applications etc. The roles are mainly well-suited for smaller

installations.

Roles are written for use with *Debian GNU/Linux*. For more details on

supported releases, see :ref:`rolereference`.

At the moment, the roles have been written for and tested against **Ansible

2.9.x**.

The roles also utilise the ``ipv4/ipv6`` lookup plugins which require

Contents

========

.. toctree::

   :maxdepth: 2

   about

   usage

   rolereference

   development

   releaseprocedures

   releasenotes

Indices and tables

==================

* :ref:`genindex`

* :ref:`modindex`

* :ref:`search`

Release notes

=============

x.y.z

-----

**Breaking changes:**

* All roles

  * Dropped support for Debian 11 (Bullseye).

**New features/improvements**

* ``backup_client`` role

  * Switched to using Paramiko + SFTP backend (instead of pexpect +

    SFTP), which should improve the backup performance.

**Bug fixes:**

* ``common`` role

  * Fixed permission errors with Python cache directories in the pip

    requirements upgrade checks virtual environment that can happen if

    the initial virtual environment set-up fails.

8.0.0

-----

Dropped support for Python 2.7 and Debian 10 Buster. Added support for

Debian 12 Bookworm. Numerous minor improvements and fixes.

**Breaking changes:**

* All roles

  * Dropped support for Debian 10 (Buster).

  * Added support for Debian 12 (Bookworm).

  * ``netaddr`` Python package is now required for using the roles.

  * ``dnspython`` Python package is no longer required for using the

    roles.

* ``backup_client`` role

  * Previously the backup would run even if pre-backup scripts would

    fail. This is no longer the case, and all pre-backup scripts must

    exit with non-zero exit code in order for backup process to

    kick-in.

  * Old backups are now automatically purged after successful

    backup. This could lead to longer runtimes for entire backup

    process, as well as higher CPU usage.

* ``common`` role

  * Dropped support for Python 2.7 pip requirements upgrade

    checks. Only Python 3 is supported now.

12. Set-up partitioning in any way you want. You can go for **Guided - use

    entire disk** if you want to keep it simple and are just testing things.

13. Wait until the base system has been installed.

14. Pick whatever Debian archive mirror is closest to you.

15. If you have an HTTP proxy, provide its URL.

16. Pick if you want to participate in package survey or not.

17. Make sure that at least the **standard system utilities** and **SSH server**

    options are selected on task selection screen.

18. Wait for packages to be installed.

19. Install the GRUB boot loader on MBR.

20. Finalise the server install, and remove the installation media from server.

Installing required packages

----------------------------

With the operating system installed, it is necessary to install a couple of

packages, and to prepare the environment a bit on the Ansible server:

1. Install the necessary system packages (using the ``root`` account)::

     apt-get install -y virtualenv virtualenvwrapper git python3-pip python3-dev libffi-dev libssl-dev

2. Set-up loading of ``virtualenvwrapper`` via Bash completions (using the ``root`` account)::

     ln -s /usr/share/bash-completion/completions/virtualenvwrapper /etc/bash_completion.d/virtualenvwrapper

3. Set-up the virtual environment (using the ``ansible`` account):

   .. warning::

      If you are already logged-in as user ``ansible`` in the server, you will

      need to log-out and log-in again in order to be able to use

      ``virtualenvwrapper`` commands!

   ::

     mkdir ~/mysite/

     mkvirtualenv -a ~/mysite/ mysite

     pip install -U pip setuptools

.. warning::

   The ``netaddr`` package is needed for ``ipv4/ipv6`` lookup plugins

Cloning the *Majic Ansible Roles*

---------------------------------

With most of the software pieces in place, the only missing thing is the Majic

Ansible Roles:

1. Clone the git repository::

     git clone https://code.majic.rs/majic-ansible-roles ~/majic-ansible-roles

2. Checkout the correct version of the roles::

     cd ~/majic-ansible-roles/

     git checkout -b 8.0-dev 8.0-dev

Preparing the basic site configuration

--------------------------------------

Phew... Now that was a bit tedious and boring... But at least you are now ready

to set-up your own site :)

First of all, let's set-up some basic directory structure and configuration:

1. Create Ansible configuration file.

   .. warning::

      Since Ansible 2.x has introduced much stricter controls over security of

      deployed Python scripts, it is recommended (as in this example) to use the

      ``pipelining`` option (which should also improve performance). This is in

      particular necessary in cases where the SSH user connecting to remote

      machine is *not* ``root``, but there are tasks that use ``become`` with

      non-root ``become_user`` (which is the case in Majic Ansible Roles). See

      `official documentation

      <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_

      and other alternatives to this.

   :file:`~/mysite/ansible.cfg`

   ::

     [defaults]

     roles_path=/home/ansible/majic-ansible-roles/roles:/home/ansible/mysite/roles

     force_handlers = True

     inventory = /home/ansible/mysite/hosts

# Ansible and role runtime.

ansible~=10.3

netaddr

python-ldap

# Development and testing.

ansible-lint

defusedxml

flake8

gimmecert

molecule[testinfra]~=24.8.0

molecule-plugins[vagrant]~=23.5.0

paramiko

# Documentation.

sphinx

sphinx-rtd-theme

# Python virtualenv management.

pip

pip-tools

setuptools