.. _usage:

Usage

=====

Majic Ansible Roles are targeted at sysadmins who wish to deploy services for

their own, small-scale use. This chapter gives a simple tutorial-like set of

instructions for using all of the roles available.

.. contents:: :local:

Overview

--------

There is a number of different roles that can prove useful for setting-up a

small infrastructure of your own.

Some roles are suited for one-off operations during installation, like the

``preseed`` and ``bootstrap``, while some are better suited for periodic runs

for maintaining the users and integrity of the system.

By the end of following the instructions, you will have the following:

* Ansible server, used as controller for configuring and managing the

  remaining servers.

* Communications server, providing the LDAP, mail, and XMPP services.

* Web server, providing the web services.

* Backup server, used for storing all of the backups.

Pre-requisites

--------------

For the set-up outlined in this usage guide you'll need the following:

* One server where Ansible will be installed at. Debian Stretch will

  be installed on top of this server. The server will be set-up

  manually (this is currently out of scope for the *Majic Ansible

  Roles* automated set-up).

* Three servers where the services will be set-up. All servers must be able to

  communicate over network with each-other, the Ansible servers, and with

  Internet. Debian Stretch will be installed on top of this server as part of the

  usage instructions.

* Debian Stretch network install CD.

* All servers should be on the same network.

* IP addresses for all servers should be known.

* Netmask for all servers should be known.

* Gateway for all servers should be known.

In case of the servers listed above, it might be safest to have them as VMs -

this is cheapest thing to do, and simplest (who wants to deal with pesky hardware anyway?).

Usage instructions assume the following:

* Domain used for all servers is ``example.com``. If you wish to use a different

  domain, adjust the instructions accordingly.

* Server hostnames are ``ansible``, ``comms``, ``www``, and ``bak`` (for Ansible

  server, communications server, web server, and backup server, respectively).

Installing the OS on Ansible server

-----------------------------------

Start-off by installing the operating system on the Ansible server:

1. Fire-up the ``ansible`` server, and boot from the network installation CD.

2. Select the **Install** option.

3. Pick **English** as language.

4. Pick the country you are living in (or whatever else you want).

5. Pick the **en_US.UTF-8** locale.

6. Pick the **American English** keymap.

7. Configure the network if necessary.

8. Set the hostname to ``ansible``.

9. Set the domain to ``example.com``.

10. Set the root password.

11. Create a new user. For simplicity, call the user **Ansible user**, with

    username **ansible**.

12. Set-up partitioning in any way you want. You can go for **Guided - use

    entire disk** if you want to keep it simple and are just testing things.

13. Wait until the base system has been installed.

14. Pick whatever Debian archive mirror is closest to you.

15. If you have an HTTP proxy, provide its URL.

16. Pick if you want to participate in package survey or not.

17. Make sure that at least the **standard system utilities** and **SSH server**

    options are selected on task selection screen.

18. Wait for packages to be installed.

19. Install the GRUB boot loader on MBR.

20. Finalise the server install, and remove the installation media from server.

Installing required packages

----------------------------

With the operating system installed, it is necessary to install a couple of

packages, and to prepare the environment a bit on the Ansible server:

1. Install the necessary system packages (using the ``root`` account)::

2. Set-up the virtual environment (using the ``ansible`` account):

   .. warning::

      If you are already logged-in as user ``ansible`` in the server, you will

      need to log-out and log-in again in order to be able to use

      ``virtualenvwrapper`` commands!

   ::

     mkdir ~/mysite/

     pip install -U pip setuptools

     pip install 'ansible~=2.9.0' dnspython

.. warning::

   The ``dnspython`` package is important since it is used internally via

   ``dig`` lookup plugin.

Cloning the *Majic Ansible Roles*

---------------------------------

With most of the software pieces in place, the only missing thing is the Majic

Ansible Roles:

1. Clone the git repository::

     git clone https://code.majic.rs/majic-ansible-roles ~/majic-ansible-roles

2. Checkout the correct version of the roles::

     cd ~/majic-ansible-roles/

     git checkout -b 4.0-dev 4.0-dev

Preparing the basic site configuration

--------------------------------------

Phew... Now that was a bit tedious and boring... But at least you are now ready

to set-up your own site :)

First of all, let's set-up some basic directory structure and configuration:

1. Create Ansible configuration file.

   .. warning::

      Since Ansible 2.x has introduced much stricter controls over security of

      deployed Python scripts, it is recommended (as in this example) to use the

      ``pipelining`` option (which should also improve performance). This is in

      particular necessary in cases where the SSH user connecting to remote

      machine is *not* ``root``, but there are tasks that use ``become`` with

      non-root ``become_user`` (which is the case in Majic Ansible Roles). See

      `official documentation

      <http://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_

      and other alternatives to this.

   :file:`~/mysite/ansible.cfg`

   ::

     [defaults]

     roles_path=/home/ansible/majic-ansible-roles/roles:/home/ansible/mysite/roles

     force_handlers = True

     inventory = /home/ansible/mysite/hosts

     [ssh_connection]

     pipelining = True

2. Create directory where retry files will be stored at (so they woudln't

   pollute your home directory)::

     mkdir ~/mysite/retry

3. Create the inventory file.

   :file:`~/mysite/hosts`

   ::

     [preseed]

     localhost ansible_connection=local

     [communications]

     comms.example.com

     [web]

     www.example.com

     [backup]

     bak.example.com

4. Create a number of directories for storing playbooks, group variables, SSH

   keys, and GnuPG keyring (we'll get to this later)::

     mkdir ~/mysite/playbooks/

     mkdir ~/mysite/group_vars/

     mkdir ~/mysite/ssh/

     mkdir ~/mysite/gnupg/

5. Before moving ahead, we should also create SSH private/public key pair that

   will be used by Ansible for connecting to destination servers, as well as

   for some roles::

     ssh-keygen -f ~/.ssh/id_rsa -N ''

Preseed files

-------------

The ``preseed`` role is useful for generating Debian preseed files. Preseed

files can be used for automating the Debian installation process.

      # Set your default (initial) root password.

      preseed_root_password: changeit

      # Use manual network configuration (no DHCP).

      preseed_network_auto: no

      # Set the gateway for all servers.

      preseed_gateway: 10.32.64.1

      # Set the netmask for all servers.

      preseed_netmask: 255.255.255.0

      # Set the DNS for all servers.

      preseed_dns: 10.32.64.1

      # Set the domain for all servers.

      preseed_domain: example.com

      # Set the server-specific options.

      preseed_server_overrides:

        comms.example.com:

          hostname: comms

          ip: 10.32.64.19

        www.example.com:

          hostname: www

          ip: 10.32.64.20

        bak.example.com:

          hostname: bak

          ip: 10.32.64.23

5. Now re-run the preseed playbook::

     workon mysite && ansible-playbook playbooks/preseed.yml

6. The preseed files should have been updated now, and you should have the new

   customised configuration files in the ``preseed_files`` directory. You can

   now use these to install the servers.

Installing the servers with preseed files

-----------------------------------------

You have your preseed files now, so you can go ahead and install the

servers ``comms.example.com``, ``www.example.com``, and

``bak.example.com`` using them with network install CD. Have a look at

`Debian instructions

<https://www.debian.org/releases/stretch/amd64/apbs02.html.en>`_ for

more details.

If you need to, you can easily serve the preseed files from the Ansible server

with Python's built-in HTTP server::

  cd ~/mysite/preseed_files/

Then you can point installer to the preseed file selecting the

``Advanced options -> Automated install`` (don't press ``ENTER`` yet),

then pressing ``TAB``, and appending the following at the end (just

fill-in the correct hostname - ``comms``, ``www``, or ``bak``)::

  url=http://ansible.example.com:8000/HOSTNAME.example.com.cfg

Bootstrapping servers for Ansible set-up

----------------------------------------

In order to effectively use Ansible, a small initial bootstrap always has to be

done for managed servers. This mainly involves set-up of Ansible users on the

destination machine, and distributing the SSH public keys for authorisation.

When you use the preseed configuration files to deploy a server, you get the

benefit of having the authorized_keys set-up for the root operating system user,

making it easier to bootstrap the machines subsequently via Ansible.

Let's bootstrap our machines now:

1. For start, create a dedicated playbook for the bootstrap process.

   :file:`~/mysite/playbooks/bootstrap.yml`

   ::

      ---

      - hosts:

          - communications

          - web

          - backup

        remote_user: root

        roles:

          - bootstrap

2. The ``bootstrap`` role has only one parameter - an SSH key which should be

   deployed for the Ansible user on managed server (in the ``authorized_keys``

   file). This defaults to content of local file ``~/.ssh/id_rsa.pub``, so no

   need to make any changes so far.

3. SSH into all machines at least once from the Ansible server in order to

   store the SSH fingerprints into known hosts file::

     ssh root@comms.example.com date && \

     ssh root@www.example.com date && \

      - hosts: backup

        remote_user: ansible

        become: yes

        roles:

          - common

          - mail_forwarder

2. The next thing is to set-up the configuration for the new role. We can define

   this globally for all servers

   :file:`~/mysite/group_vars/all.yml`

   ::

      # First, let's make sure any mails directed to localhost root account get

      # forwarded to one of our mail users as well.

      local_mail_aliases:

        root: root john.doe@example.com

      # Now signal the local SMTP to relay any non-local mails via our

      # communications server. Don't forget to specify your own IP address (or

      # FQDN) here. Without this option, the SMTP would send out the mails

      # directly.

      smtp_relay_host: comms.example.com

3. Although we have told our web and backup servers to use the communications

   server as relay for non-local mail, the communications server is not aware of

   this. This would result in the communications server refusing all relay

   attempts (if not, it would be an open relay, which is bad).

   So, let's fix this a bit - we have a configuration option for the mail server

   for exactly this purpose.

   :file:`~/mysite/group_vars/communications.yml`

   ::

      # We want to allow relaying of mails from our web and backup servers

      # here.Beware the IP spoofing, though! Don't forget to change the bellow

      # IP for your server ;)

      smtp_allow_relay_from:

        - 10.32.64.20

        - 10.32.64.23

4. Let's apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

5. After this you may want to test out sending mail via web or backup server's

   local SMTP to the root user (to see if the aliasing works), and to some

   something similar to the following on your web server::

     swaks --to root@localhost --server localhost

     swaks --to YOUR_MAIL --server localhost

   If all went well, you should be able to see a new mail in John Doe's mailbox,

   as well as your own mailbox.

Adding XMPP server

------------------

Now that the users can communicate via mail server, we might as well add support

for some instant messaging. For this purpose, we will use the ``xmpp_server``

role.

1. Update the playbook for communications server to include the XMPP server

   role.

   :file:`~/mysite/playbooks/communications.yml`

   ::

      ---

      - hosts: communications

        remote_user: ansible

        become: yes

        roles:

          - common

          - ldap_client

          - ldap_server

          - mail_server

          - xmpp_server

2. Configure the role.

   :file:`~/mysite/group_vars/communications.yml`

   ::

      # Set one of the users to also be an XMPP administrator.

      xmpp_administrators:

        - john.doe@example.com

      # Unfortunately, XMPP can't look-up domains via LDAP, so we need to be

      # explicit here.

      xmpp_domains:

        - example.com

          url: "https://github.com/thebuggenie/thebuggenie/archive/v{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          sha256sum: "{{ tbg_archive_checksum }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Download Composer

        get_url:

          url: "https://getcomposer.org/download/1.7.2/composer.phar"

          dest: "/usr/local/bin/composer"

          sha256sum: "ec3428d049ae8877f7d102c2ee050dbd51a160fc2dde323f3e126a3b3846750e"

          owner: root

          group: root

          mode: 0755

      - name: Unpack TBG

        unarchive:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/"

          copy: no

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Create TBG cache directory

        file:

          path: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/cache"

          state: directory

          mode: 02770

        become: yes

        become_user: admin-tbg_example_com

      - name: Set-up the necessary write permissions for TBG directories

        file:

          path: "{{ item }}"

          mode: g+w

        with_items:

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/

      - name: Create symbolic link to TBG application

        file:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public"

          path: "/var/www/tbg.example.com/htdocs"

          state: link

          owner: "admin-tbg_example_com"

          group: "web-tbg_example_com"

        become: yes

        become_user: admin-tbg_example_com

      - name: Install TBG dependencies

        composer:

          command: install

          working_dir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Deploy database configuration file for TBG

        copy:

          src: "b2db.yml"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"

          mode: 0640

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Install pexpect package

        apt:

          name: python-pexpect

          state: present

      - name: Deploy expect script for installing TBG

        copy:

          src: "tbg_expect_install"

          dest: "/var/www/tbg.example.com/tbg_expect_install"

          mode: 0750

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Run TBG installer via expect script

        command: /var/www/tbg.example.com/tbg_expect_install

        args:

          chdir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed"

        become: yes

        become_user: admin-tbg_example_com

5. Set-up the files that are deployed by our role.

   :file:`~/mysite/roles/tbg/files/b2db.yml`

   ::

      b2db:

          username: "tbg"

          password: "tbg"

  have a look at ``environment_indicator`` role parameter. It lets you

  insert small strip with environment information at bottom of each

  HTML page served by the web server.

* Static content is served directly by *Nginx*.

* Each web application gets distinct sub-directory under ``/var/www``, named

  after the FQDN. All sub-directories created under there are created with

  ``2750`` permissions, with ownership set to admin user, and group set to the

  application's group. In other words, all directories will have ``SGID`` bit

  set, allowing you to create files/directories that will have their group

  automatically set to the group of the parent directory.

* Each WSGI website gets a dedicated virtual environment, stored in the

  sub-directory ``virtualenv`` of the website directory, for example

  ``/var/www/wiki.example.com/virtualenv``.

* Static files are served from sub-directory ``htdocs`` in the website

  directory, for example ``/var/www/wiki.example.com/htdocs/``.

* The base directory where your website/application code should be at is

  expected to be in sub-directory ``code`` in the website directory, for example

  ``/var/www/wiki.example.com/code/``.

* Combination of admin user membership in application group, ``SGID``

  permission, and the way ownership of sub-directories is set-up usually means

  that the administrator will be capable of managing application files, and

  application can be granted write permissions to a *minimum* of necessary

  files.

  .. warning::

     Just keep in mind that some file-management commands, like ``mv``, do *not*

     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you

     deploy new files to the directory instead (don't simply move them from your

     home directory).

1. Set-up the necessary directories first::

     mkdir -p ~/mysite/roles/wiki/{tasks,meta,files,handlers}/

2. Set-up some role dependencies, reusing the common role infrastructure.

   :file:`~/mysite/roles/wiki/meta/main.yml`

   ::

      ---

      dependencies:

        - role: wsgi_website

          fqdn: wiki.example.com

          # In many cases you need to have some development packages available

          # in order to build Python packages installed via pip

          packages:

            - build-essential

            - libjpeg-dev

          # Here we specify that anything accessing our website with "/static/"

          # URL should be treated as request to a static file, to be served

          # directly by Nginx instead of the WSGI server.

          static_locations:

            - /static/

          # Again, not mandatory, but it is good to have some sort of policy

          # for assigning UIDs.

          uid: 2001

          admin_uid: 3001

          # These are additional packages that should be installed in the

          # virtual environment.

          virtualenv_packages:

          # This is the name of the WSGI application to

          # serve. wiki_example_com.wsgi will be the Python "module" that is

          # accesed, while application is the object instantiated within it (the

          # application itself). The module is referenced relative to the code

          # directory (in our case /var/www/wiki.example.com/code/).

          wsgi_application: wiki_example_com.wsgi:application

        - role: database

          db_name: wiki

          db_password: wiki

3. Let's create a dedicated private key/certificate pair for the wiki website:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/wiki.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

         cn = "Exampe Inc. Wiki"

         expiration_days = 365

         dns_name = "wiki.example.com"

         tls_www_server

         signing_key

         encryption_key

   2. Create the keys and certificates for the application::

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/wiki.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/wiki.example.com_https.cfg --load-privkey ~/mysite/tls/wiki.example.com_https.key --outfile ~/mysite/tls/wiki.example.com_https.pem

4. At this point we have exhausted what we can do with the built-in roles. Time

   to add some custom tasks.

   :file:`~/mysite/roles/wiki/tasks/main.yml`

   ::

      ---

      - name: Create Django project directory

        file:

          dest: "/var/www/wiki.example.com/code"

          state: directory

          owner: admin-wiki_example_com

          group: web-wiki_example_com

          mode: 02750

      - name: Start Django project for the Wiki website

        command: "/var/www/wiki.example.com/virtualenv/bin/exec django-admin.py startproject wiki_example_com /var/www/wiki.example.com/code"

        args:

          chdir: "/var/www/wiki.example.com"

          creates: "/var/www/wiki.example.com/code/wiki_example_com"

        become: yes

        become_user: admin-wiki_example_com

      - name: Deploy settings for wiki website

        copy:

          src: "{{ item }}"

          dest: "/var/www/wiki.example.com/code/wiki_example_com/{{ item }}"

          mode: 0640

          owner: admin

          group: web-wiki_example_com

        with_items:

          - settings.py

          - urls.py

        notify:

          - Restart wiki

      - name: Deploy project database and deploy static files

        django_manage:

          command: "{{ item }}"

          app_path: "/var/www/wiki.example.com/code/"

          virtualenv: "/var/www/wiki.example.com/virtualenv/"

        become: yes

        become_user: admin-wiki_example_com

        with_items:

          - migrate

          - collectstatic

      - name: Deploy the superadmin creation script

        copy:

          src: "create_superadmin.py"

          dest: "/var/www/wiki.example.com/code/create_superadmin.py"

          owner: admin-wiki_example_com

          group: web-wiki_example_com

          mode: 0750

      - name: Create initial superuser

        command: "/var/www/wiki.example.com/virtualenv/bin/exec ./create_superadmin.py"

        args:

          chdir: "/var/www/wiki.example.com/code/"

        become: yes

        become_user: admin-wiki_example_com

        register: wiki_superuser

        changed_when: "wiki_superuser.stdout ==  'Created superuser.'"

   :file:`~/mysite/roles/wiki/handlers/main.yml`

   ::

      ---

      - name: Restart wiki

        service:

          name: wiki.example.com

          state: restarted

5. There is a couple of files that we are deploying through the above

   tasks. Let's create them as well.

   :file:`~/mysite/roles/wiki/files/settings.py`

   ::

      """

      Django settings for wiki_example_com project.

      For more information on this file, see

      For the full list of settings and their values, see

      """

      import os

      # Quick-start development settings - unsuitable for production

      # SECURITY WARNING: keep the secret key used in production secret!

      # SECURITY WARNING: don't run with debug turned on in production!

      DEBUG = False

      ALLOWED_HOSTS = ["wiki.example.com", "localhost"]

      # Application definition

          'django.contrib.admin',

          'django.contrib.auth',

          'django.contrib.contenttypes',

          'django.contrib.sessions',

          'django.contrib.messages',

          'django.contrib.staticfiles',

          'mptt',

          'sekizai',

          'sorl.thumbnail',

          'django.contrib.sessions.middleware.SessionMiddleware',

          'django.middleware.common.CommonMiddleware',

          'django.middleware.csrf.CsrfViewMiddleware',

          'django.contrib.auth.middleware.AuthenticationMiddleware',

          'django.contrib.messages.middleware.MessageMiddleware',

          'django.middleware.clickjacking.XFrameOptionsMiddleware',

      ROOT_URLCONF = 'wiki_example_com.urls'

      WSGI_APPLICATION = 'wiki_example_com.wsgi.application'

      # Database

      DATABASES = {

          'default': {

              'ENGINE': 'django.db.backends.mysql',

              'NAME': 'wiki',

              'USER': 'wiki',

              'PASSWORD': 'wiki',

              'HOST': '127.0.0.1',

              'PORT': '3306',

          }

      }