11. Create a new user. For simplicity, call the user **Ansible user**, with

    username **ansible**.

12. Set-up partitioning in any way you want. You can go for **Guided - use

    entire disk** if you want to keep it simple and are just testing things.

13. Wait until the base system has been installed.

14. Pick whatever Debian archive mirror is closest to you.

15. If you have an HTTP proxy, provide its URL.

16. Pick if you want to participate in package survey or not.

17. Make sure that at least the **standard system utilities** and **SSH server**

    options are selected on task selection screen.

18. Wait for packages to be installed.

19. Install the GRUB boot loader on MBR.

20. Finalise the server install, and remove the installation media from server.

Installing required packages

----------------------------

With the operating system installed, it is necessary to install a couple of

packages, and to prepare the environment a bit on the Ansible server:

1. Install the necessary system packages (using the ``root`` account)::

     apt-get install -y virtualenv virtualenvwrapper git python3-pip python3-dev libffi-dev libssl-dev

2. Set-up loading of ``virtualenvwrapper`` via Bash completions (using the ``root`` account)::

     ln -s /usr/share/bash-completion/completions/virtualenvwrapper /etc/bash_completion.d/virtualenvwrapper

3. Set-up the virtual environment (using the ``ansible`` account):

   .. warning::

      If you are already logged-in as user ``ansible`` in the server, you will

      need to log-out and log-in again in order to be able to use

      ``virtualenvwrapper`` commands!

   ::

     mkdir ~/mysite/

     pip install -U pip setuptools

.. warning::

   The ``netaddr`` package is needed for ``ipv4/ipv6`` lookup plugins

   which is used internally by some of the roles.

Cloning the *Majic Ansible Roles*

---------------------------------

With most of the software pieces in place, the only missing thing is the Majic

Ansible Roles:

1. Clone the git repository::

     git clone https://code.majic.rs/majic-ansible-roles ~/majic-ansible-roles

2. Checkout the correct version of the roles::

     cd ~/majic-ansible-roles/

     git checkout -b 8.0-dev 8.0-dev

Preparing the basic site configuration

--------------------------------------

Phew... Now that was a bit tedious and boring... But at least you are now ready

to set-up your own site :)

First of all, let's set-up some basic directory structure and configuration:

1. Create Ansible configuration file.

   .. warning::

      Since Ansible 2.x has introduced much stricter controls over security of

      deployed Python scripts, it is recommended (as in this example) to use the

      ``pipelining`` option (which should also improve performance). This is in

      particular necessary in cases where the SSH user connecting to remote

      machine is *not* ``root``, but there are tasks that use ``become`` with

      non-root ``become_user`` (which is the case in Majic Ansible Roles). See

      `official documentation

      <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_

      and other alternatives to this.

   :file:`~/mysite/ansible.cfg`

   ::

     [defaults]

     roles_path=/home/ansible/majic-ansible-roles/roles:/home/ansible/mysite/roles

     force_handlers = True

     inventory = /home/ansible/mysite/hosts

     [ssh_connection]

     pipelining = True

2. Create directory where retry files will be stored at (so they woudln't

   pollute your home directory)::

     mkdir ~/mysite/retry

3. Create the inventory file.

   :file:`~/mysite/hosts`

   ::

     [preseed]

     localhost ansible_connection=local

     [communications]

     comms.example.com

     [web]

     www.example.com

     [backup]

     bak.example.com

4. Create a number of directories for storing playbooks, group

   variables, SSH keys, X.509 artefacts (for TLS), and GnuPG keyring

   (we'll get to this later)::

     mkdir ~/mysite/playbooks/

     mkdir ~/mysite/group_vars/

     mkdir ~/mysite/ssh/

     mkdir ~/mysite/tls/

     mkdir ~/mysite/gnupg/

5. Create SSH private/public key pair that will be used by Ansible for

   connecting to destination servers, as well as for some roles::

     ssh-keygen -f ~/.ssh/id_rsa -N ''

Protecting communications using TLS

-----------------------------------

In order to protect the communications between users and servers, as

well as between servers themselves, it is important to set-up and

        # application.

        - role: database

          # This is both the database name, _and_ name of the database user

          # that will be granted full privileges on the database.

          db_name: nextcloud

          # Password for user used for accessing the database. Take note

          # that the user can only login from localhost.

          db_password: nextcloud

3. Now for my favourite part again - creating private keys and

   certificates!  Why?  Because the ``php_website`` role requires a

   private key/certificate pair to be deployed. So... Moving on:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/nextcloud.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

         cn = "Example Inc. Cloud Service"

         expiration_days = 365

         dns_name = "nextcloud.example.com"

         tls_www_server

         signing_key

         encryption_key

   2. Create the keys and certificates for the application::

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/nextcloud.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/nextcloud.example.com_https.cfg --load-privkey ~/mysite/tls/nextcloud.example.com_https.key --outfile ~/mysite/tls/nextcloud.example.com_https.pem

4. Time to get our hands a bit more dirty... Up until now we didn't

   have to write custom tasks, but that ends now.

   :file:`~/mysite/roles/nextcloud/tasks/main.yml`

   ::

      ---

      # Deployment

      # ==========

      - name: Download the application archive

          url: "https://download.nextcloud.com/server/releases/nextcloud-28.0.3.tar.bz2"

          dest: "/var/www/nextcloud.example.com/nextcloud-28.0.3.tar.gz"

          sha256sum: "9ed413c0de16f5b033ceeffcca99c0d61fc698dbeb8db851ac9adf9eef951906"

        become: yes

        become_user: admin-nextcloud_example_com

      - name: Unpack the application archive

          src: "/var/www/nextcloud.example.com/nextcloud-28.0.3.tar.gz"

          dest: "/var/www/nextcloud.example.com/"

          copy: no

          creates: "/var/www/nextcloud.example.com/nextcloud"

        become: yes

        become_user: admin-nextcloud_example_com

      # Majic Ansible Roles currently only support utf8 encoding.

      - name: Disable opportunistic use of utf8mb4 on fresh installs

          dest: "/var/www/nextcloud.example.com/nextcloud/lib/private/Setup/MySQL.php"

          line: "{{ '\t\t\t' }}$this->config->setValue('mysql.utf8mb4', true);"

          state: absent

      - name: Allow application user to install and update applications

          path: "/var/www/nextcloud.example.com/nextcloud/apps"

          mode: g+w

      - name: Allow CLI tool to be run by the user and group

          path: "/var/www/nextcloud.example.com/nextcloud/occ"

          mode: u+x,g+x

      - name: Create directory for storing data

          path: "/var/www/nextcloud.example.com/data"

          state: directory

          mode: 02770

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

      - name: Create directory for storing configuration files

          path: "/var/www/nextcloud.example.com/nextcloud/config"

          state: directory

          mode: 02750

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

      - name: Create an empty log file if it does not exist

          content: ""

          dest: "/var/www/nextcloud.example.com/data/nextcloud.log"

          force: no

      - name: Set-up log file permissions

          path: "/var/www/nextcloud.example.com/data/nextcloud.log"

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

          mode: 0660

      - name: Symlink the default path used by the web server for finding application files

          src: "/var/www/nextcloud.example.com/nextcloud"

          dest: "/var/www/nextcloud.example.com/htdocs"

          state: link

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

        notify:

          - Restart PHP-FPM

      # Installation

      # ============

      - name: Get application installation status

        become: yes

        become_user: "admin-nextcloud_example_com"

        register: nextcloud_status

        changed_when: False

        failed_when: False

      - name: Check if application is installed

          nextcloud_installed: "{{ 'Nextcloud is not installed' not in nextcloud_status.stderr }}"

      - name: Deploy installation script

          src: "install_nextcloud.py"

          dest: "/var/www/nextcloud.example.com/install_nextcloud.py"

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

          mode: 0700

        when: "not nextcloud_installed"

      - name: Install application

        become: yes

        become_user: "admin-nextcloud_example_com"

        when: "not nextcloud_installed"

      - name: Remove installation script

          path: "/var/www/nextcloud.example.com/install_nextcloud.py"

          state: absent

      - name: Fix data file permissions for application user/group

          path: "/var/www/nextcloud.example.com/data"

          mode: g+w

          recurse: yes

          follow: no

      - name: Deploy local configuration overrides

          src: "local.config.php"

          dest: "/var/www/nextcloud.example.com/nextcloud/config/local.config.php"

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

          mode: 0640

5. Set-up files that are deployed by the role.

   :file:`~/mysite/roles/nextcloud/files/local.config.php`

   ::

      <?php

      $CONFIG = array (

        'config_is_read_only' => true,

        'instanceid' => 'suqw2cvca8sp',

        'trusted_domains' =>

          array (

            0 => 'nextcloud.example.com',

          ),

      );

   :file:`~/mysite/roles/nextcloud/files/install_nextcloud.py`

   ::

      #!/usr/bin/env python3

      import pexpect

      # Spawn the process.

      install_process = pexpect.spawnu('/var/www/nextcloud.example.com/nextcloud/occ',

                                       args = [ 'maintenance:install',

                                                '--database', 'mysql',

                                                '--database-name', 'nextcloud',

                                                '--database-user', 'nextcloud',

                                                '--database-host', 'localhost',

                                                '--database-port', '3306',

                                                '--admin-user', 'admin',

                                                '--data-dir', '/var/www/nextcloud.example.com/data'])

      # If we get EOF, we probably already installed application, and ran

      # into error at the end since no patterns matched.

      try:

          # Provide database password.

          install_process.expect(u'What is the password to access the database with user.*\?', timeout=10)

          install_process.sendline(u'nextcloud')

          # Provide administrator password.

          install_process.expect(u'What is the password you like to use for the admin account.*\?', timeout=10)

            - wiki~=0.10.0

            - mysqlclient

          # This is the name of the WSGI application to

          # serve. wiki_example_com.wsgi will be the Python "module" that is

          # accesed, while application is the object instantiated within it (the

          # application itself). The module is referenced relative to the code

          # directory (in our case /var/www/wiki.example.com/code/).

          wsgi_application: wiki_example_com.wsgi:application

          # Specify explicitly requirements for installing Gunicorn.

          wsgi_requirements:

            - gunicorn==21.2.0

            - packaging==23.2

          wsgi_requirements_in:

            - gunicorn

        - role: database

          db_name: wiki

          db_password: wiki

3. Let's create a dedicated private key/certificate pair for the wiki website:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/wiki.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

         cn = "Exampe Inc. Wiki"

         expiration_days = 365

         dns_name = "wiki.example.com"

         tls_www_server

         signing_key

         encryption_key

   2. Create the keys and certificates for the application::

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/wiki.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/wiki.example.com_https.cfg --load-privkey ~/mysite/tls/wiki.example.com_https.key --outfile ~/mysite/tls/wiki.example.com_https.pem

4. At this point we have exhausted what we can do with the built-in roles. Time

   to add some custom tasks.

   :file:`~/mysite/roles/wiki/tasks/main.yml`

   ::

      ---

      - name: Create Django project directory

          dest: "/var/www/wiki.example.com/code"

          state: directory

          owner: admin-wiki_example_com

          group: web-wiki_example_com

          mode: 02750

      - name: Start Django project for the Wiki website

        args:

          chdir: "/var/www/wiki.example.com"

          creates: "/var/www/wiki.example.com/code/wiki_example_com"

        become: yes

        become_user: admin-wiki_example_com

      - name: Deploy settings for wiki website

          src: "{{ item }}"

          dest: "/var/www/wiki.example.com/code/wiki_example_com/{{ item }}"

          mode: 0640

          owner: admin-wiki_example_com

          group: web-wiki_example_com

        with_items:

          - settings.py

          - urls.py

        notify:

          - Restart wiki

      - name: Deploy project database and deploy static files

          command: "{{ item }}"

          app_path: "/var/www/wiki.example.com/code/"

          virtualenv: "/var/www/wiki.example.com/virtualenv/"

        become: yes

        become_user: admin-wiki_example_com

        with_items:

          - migrate

          - collectstatic

      - name: Deploy the superuser creation script

          src: "create_superuser.py"

          dest: "/var/www/wiki.example.com/code/create_superuser.py"

          owner: admin-wiki_example_com

          group: web-wiki_example_com

          mode: 0750

      - name: Create initial superuser

        args:

          chdir: "/var/www/wiki.example.com/code/"

        become: yes

        become_user: admin-wiki_example_com

        register: wiki_superuser

        changed_when: "wiki_superuser.stdout ==  'Created superuser.'"

   :file:`~/mysite/roles/wiki/handlers/main.yml`

   ::

      ---

      - name: Restart wiki

          name: wiki.example.com

          state: restarted

5. There is a couple of files that we are deploying through the above

   tasks. Let's create them as well.

   :file:`~/mysite/roles/wiki/files/settings.py`

   ::

      """

      Django settings for wiki_example_com project.

      Generated by 'django-admin startproject' using Django 4.2.11.

      For more information on this file, see

      https://docs.djangoproject.com/en/4.2/topics/settings/

      For the full list of settings and their values, see

      https://docs.djangoproject.com/en/4.2/ref/settings/

      """

      from pathlib import Path

      from django.urls import reverse_lazy

      # Build paths inside the project like this: BASE_DIR / 'subdir'.

      BASE_DIR = Path(__file__).resolve().parent.parent

      # Quick-start development settings - unsuitable for production

      # See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/

      # SECURITY WARNING: keep the secret key used in production secret!

      SECRET_KEY = 'django-insecure-*!yz4t12j5&x%-p%dd$uw!$-7(8vm)r%(87iz65-7t_7uh8j)0'

      # SECURITY WARNING: don't run with debug turned on in production!

      DEBUG = False

      ALLOWED_HOSTS = ["wiki.example.com", "localhost"]

      # Application definition

      INSTALLED_APPS = [

          'django.contrib.admin',

          'django.contrib.auth',

          'django.contrib.contenttypes',

          'django.contrib.sessions',

          'django.contrib.messages',

provided that a special variable (``run_handlers``) is passed in to playbook run. To

make the run shorter, the handlers in such a run are also tagged with

``handlers``. This doubling of environment variable + tagging stems from current

limitations of Ansible (it is not possible to specify that certain task should

be run only if a tag is specified, therefore an additional variable has to be

used).

Handlers alone can be invoked specifically with command similar to::

  ansible-playbook -t handlers -e run_handlers=true playbooks/site.yml

The ``run_handlers`` variable is treated as boolean, and by default it

is not set.

Checking for available package upgrades

---------------------------------------

One of the more annoying chores when you maintain your own infrastructure is

making sure everything is up-to-date. And this has to be done - both in order to

ensure for problem-free experience for users (yourself included), and for making

sure there are no security vulnerabilities that could be exploited by a (random)

adversary.

*Majic Ansible Roles* try to keep you covered on this front as well. As part of

regular deployment, the ``common`` role will deploy and configure ``apticron`` -

a nifty little script that runs on hourly basis and checks if any of your

system-provided packages are outdated.

If ``apticron`` detects an outdated package, it will output this information to

standard output, which will result in the cron daemon sending out an e-mail to

the local root account. These mails can be further directed towards other mail

accounts via aliases (easily achieveable if you use either the

``mail_forwarder`` or ``mail_server`` roles).

No packages will be upgraded automatically - ensuring you can make sure upgrades

work correctly and do not cause major outage without anyone being present to

fix them.

Another useful package you may want to look into is ``needrestart`` - which runs

as a hook during the upgrade process to detect any processes that seem to be

running with outdated libraries, allowing you to restart them as well. This

package is *not* installed by the ``common`` role out-of-the-box, but you can

easily do so by updating the ``common_packages`` setting.

In addition to system packages, the ``common`` role makes it easy to check if

any of the pip requirements files are outdated as well. It should be noted,

though, that this check does *not* verify the Python virtual environments

This is primarily useful when you use `pip-tools

<https://github.com/jazzband/pip-tools>`_ for maintaining the

requirements files. In fact, I would encourage you to utilise

``pip-tools`` for both this purpose and for keeping the virtual

environment in sync and up-to-date.

Roles that want to take advantage of this would:

- Create a sub-directory under

  ``/etc/pip_check_requirements_upgrades/``.

- Deploy ``.in`` and ``.txt`` files within the sub-directory (see ``pip-tools``

  docs for explanation of how the ``.in`` files work).

- Ensure the created sub-directory and files have ownership set to

  ``root:pipreqcheck``.

.. note::

   If you are using the ``wsgi_website`` role as dependency, simply set-up the

   ``wsgi_requirements`` parameter, and then deploy the ``.in`` and ``.txt``

   file into directory ``/etc/pip_check_requirements_upgrades/FQDN`` (this

   directory is automatically created when ``wsgi_requirements`` is specified).

Where to go next?

-----------------

Well, those were some rather lengthy usage instructions, but hopefuly they are

useful. Things you might want to check-out next:

* :ref:`rolereference`

* :ref:`testsite`

* Finally, if it tickles your interest, have a look at role implementations

  themselves.