can be configured for static file serving.

* Within the website directory, systemd service will expect to find the website

  code within the ``code`` sub-directory (this can be symlink too).

* nginx communicates with WSGI server over a dedicated Unix socket for each

  website.

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **common**

* **web_server**

Parameters

~~~~~~~~~~

**additional_nginx_config** (list, optional, ``[]``)

  List providing additional Nginx configuration options to include. This can be

  useful for specifying things like error pages. Options are applied inside of a

  **server** context of Nginx configuration file.

  Each item is a dictionary with the following options describing the extra

  configuration option:

  **comment** (string, mandatory)

    Comment describing the configuration option.

  **value** (string, mandatory)

    Configuration option.

**admin_uid** (integer, optional, ``whatever OS picks``)

  UID of the dedicated website administrator user. The user will be member of

  website group.

**enforce_https** (boolean, optional, ``True``)

  Specify if HTTPS should be enforced for the website or not. If enforced,

  clients connecting via plaintext will be redirected to HTTPS, and clients will

  be served with ``Strict-Transport-Security`` header with value of

  ``max-age=31536000; includeSubDomains``.

**environment_indicator** (dictionary, optional, ``null``)

  Specify configuration for including environment indicator on all HTML

  pages. Indicator is a simple strip at bottom of a page with custom background

  colour, text colour, and text.

  Specifying environment indicator is useful for avoiding mistakes when testing

  by having better visibility what environment you are in

  (production/staging/test).

  The following keys need to be specified:

  **background_colour** (string, mandatory)

    Background colour to use for the strip at bottom. This should be value

    compatible with CSS ``background-color`` attribute.

  **text_colour** (string, mandatory

    Text colour to use for the strip at bottom. This should be value compatible

    with CSS ``color`` attribute.

  **text** (string, mandatory)

    Text to show in show in the strip at bottom.

**environment_variables** (dict, optional, ``{}``)

  Specify additional environment variables that should be set for running the

  service. Environment variables will be set in both the systemd service and for

  the application's administrator user (when logged in as one).

**fqdn** (string, mandatory)

  Fully-qualified domain name where the website is reachable. This value is used

  for calculating the user/group name for dedicated website user, as well as

  home directory of the website user (where data/code should be stored at).

**futures_version** (string, optional, ``3.0.5``)

  Version of ``futures`` package to deploy in virtual environment. Required by

  Gunicorn when using Python 2.7. Default version is tested with the test site.

**gunicorn_version** (string, optional, ``19.6.0``)

  Version of Gunicorn to deploy in virtual environment for running the WSGI

  application. Default version is tested with the test site.

**https_tls_certificate** (string, optional, ``{{ lookup('file', tls_certificate_dir + '/' + fqdn + '_https.pem') }}``)

  X.509 certificate used for TLS for HTTPS service. The file will be stored in

  directory ``/etc/ssl/certs/`` under name ``{{ fqdn }}_https.pem``.

**https_tls_key** (string, optional, ``{{ lookup('file', tls_private_key_dir + '/' + fqdn + '_https.key') }}``)

  Private key used for TLS for HTTPS service. The file will be stored in

  directory ``/etc/ssl/private/`` under name ``{{ fqdn }}_https.key``.

**packages** (list, optional, ``[]``)

  A list of additional packages to install for this particular WSGI

  website. This is usually going to be development libraries for building Python

  packages.

**rewrites** (list, optional, ``[]``)

  A list of rewrite rules that are applied to incoming requests. Each element of

  the list should be a string value compatible with the format of ``nginx``

  option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well

  as trailing semi-colon (``;``).

**static_locations** (list, optional, ``[]``)

  List of locations that should be treated as static-only, and not processed by

  the WSGI application at all. This is normally used for designating serving of

  static/media files by Nginx (for example, in case of Django projects for

  ``/static/`` and ``/media/``).

**uid** (integer, optional, ``whatever OS picks``)

  UID/GID (they are set-up to be the same) of the dedicated website

  user/group.

**use_paste** (boolean, optional, ``False``)

  Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a

  filename of a Python Paste ``ini`` file instead of WSGI application.

**virtuaelnv_packages** (list, optional, ``[]``)

  A list of additional packages to install for this particular WSGI appliction

  in its virtual environment using ``pip``.

**website_mail_recipients** (string, optional, ``root``)

  Space-separated list of e-mails or local users to which the mails, sent to

  either the website admin or website user, should be forwarded to. Forwarding

  is configured via ``~/.forward`` configuration file.

**wsgi_application** (string, mandatory)

  WSGI application that should be started by Gunicorn. The format should be

  conformant to what the ``gunicorn`` command-line tool accepts. If the

  ``use_paste`` option is enabled, the value should be equal to filename of the

  Python Paste ini file, located in the ``code`` sub-directory. It should be

  noted that in either case the value should be specsified relative to the

  ``code`` sub-directory. I.e. don't use full paths.

Examples

~~~~~~~~

Here is an example configuration for setting-up a (base) WSGI website (for

running a bare Django project):

.. code-block:: yaml

    - role: wsgi_website

      fqdn: django.example.com

      static_locations:

        - /static

        - /media

      uid: 2004

      virtualenv_packages:

        - django

      wsgi_application: django_example_com.wsgi:application

      environment_variables:

        DJANGO_SETTINGS_MODULE: "django_example_com.settings.production"

      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.key') }}"

      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.pem') }}"

      futures_version: 3.0.5

      gunicorn_version: 19.6.0

      additional_nginx_config:

        - comment: Use custom page for forbidden files.

          value: error_page 403 /static/403.html;

        - comment: Use custom page for non-existing locations/files.

          value: error_page 404 /static/404.html;

      website_mail_recipients: "root john.doe@example.com"

      environment_indicator:

        background_colour: "green"

        text_colour: "black"

        text: "TEST ENVIRONMENT"

Database Server

---------------

The ``database_server`` role can be used for setting-up a MariaDB database

server on destination machine.

The role implements the following:

* Installs MariaDB server and client.

* Configures MariaDB server and client to use *UTF-8* encoding by default.

* Sets password for the database root user.

* Deploys MariaDB client configuration in location ``/root/.my.cnf`` that

  contains username and password for the root database user.

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **common**

Parameters

~~~~~~~~~~

**db_root_password** (string, mandatory)

  Password for the *root* database user.

Examples

~~~~~~~~

Here is an example configuration for setting-up the database server:

.. code-block:: yaml

   ---

   db_root_password: root

Database

--------

The ``database`` role can be used for creating a MariaDB database and

accompanying user on destination machine.

The role implements the following:

* Creates MariaDB database.

* Creates a dedicated user capable of performing any operation on the created

  database. Username is set to be same as the name of database.

* Sets-up pre-backup task that creates database dump in location

  ``/srv/backup/mariadb/{{ db_name }}.sql``.

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **database_server**

* **backup_client**

Backups

~~~~~~~

If the backup for this role has been enabled, the following paths are backed-up:

**/srv/backup/maraidb/{{ db_name }}.sql**

  Dump of the database. Database dump is created every day at 01:45 in the

  morning.

Parameters

~~~~~~~~~~

**db_name** (string, mandatory)

  Name of the database that should be created.

**db_password** (string, mandatory)

  Password for the database user.

Examples

~~~~~~~~

Here is an example configuration for creating a single database (for some

website):

.. code-block:: yaml

      - name: Deploy expect script for installing TBG

        copy: src="tbg_expect_install" dest="/var/www/tbg.example.com/tbg_expect_install" mode=750

        become: yes

        become_user: admin-tbg_example_com

      - name: Run TBG installer via expect script

        command: /var/www/tbg.example.com/tbg_expect_install

                 chdir=/var/www/tbg.example.com/thebuggenie-{{ tbg_version }} 

                 creates=/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed

        become: yes

        become_user: admin-tbg_example_com

5. Set-up the files that are deployed by our role.

   :file:`~/mysite/roles/tbg/files/b2db.yml`

   ::

      b2db:

          username: "tbg"

          password: "tbg"

          dsn: "mysql:host=localhost;dbname=tbg"

          tableprefix: ''

          cacheclass: '\thebuggenie\core\framework\Cache'

   :file:`~/mysite/roles/tbg/files/tbg_expect_install`

   ::

      #!/usr/bin/expect

      spawn ./tbg_cli install --accept_license=yes --url_subdir=/ --use_existing_db_info=yes --enable_all_modules=yes --setup_htaccess=yes

      expect "Press ENTER to continue with the installation: "

      send "\r"

      expect "Press ENTER to continue: "

      send "\r"

      interact

6. And... Let's add the new role to our web server.

   :file:`~/mysite/playbooks/web.yml`

   ::

      ---

      - hosts: web

        remote_user: ansible

        become: yes

        roles:

          - common

          - ldap_client

          - mail_forwarder

          - web_server

          - database_server

          - tbg

7. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

8. At this point *The Bug Genie* has been installed, and you should be able to

   open the URL https://tbg.example.com/ (if you open http://tbg.example.com/ ,

   you will be redirected to the HTTPS URL) and log-in into *The Bug Genie*

   with username ``administrator`` and password ``admin``.

Deploying a WSGI application (Django Wiki)

------------------------------------------

Next thing up will be to deploy a WSGI Python application.

Similar to the PHP application deployment, we will use a couple of roles to make

it easier to deploy it in a standardised manner, and we will not have any kind

of parameters for configuring the role to keep things simple.

Most of the notes on how a ``php_website`` role is deployed also stand for the

``wsgi_website`` role, but we will reiterate and clarify them a bit just to be

on the safe side:

* The role is designed to execute every application via dedicated user and

  group. The user/group name is automatically derived from the FQDN of website,

  for example ``web-wiki_example_com``.

* While running the application, application user's umask is set to ``0007``

  (letting the administrator user be able to manage any files created while the

  application is running).

* An administrative user is created as well, and this user should be used when

  running maintenance and installation commands. Similar to application user,

  the name is also derived from the FQDN of website, for example

  ``admin-wiki_example_com``. Administrative user does not have a dedicated

  group, and instead belongs to same group as the application user. As

  convenience, whenever you switch to this user the Python virtual environment

  will be automatically activated for you.

* WSGI applications are executed via *Gunicorn*. The WSGI server listens on a

  Unix socket, making the socket accessible by *Nginx*.

* If you ever need to set some environment variables, this can easily be done

  via the ``environment_variables`` role parameter. This particular example does

  not set any, though.

* Mails deliverd to local admin/application users are forwarded to ``root``

  account instead (this can be configured via ``website_mail_recipients`` role

  parameter.

* If you ever find yourself mixing-up test and production websites, have a look

  at ``environment_indicator`` role parameter. It lets you insert small strip at

  bottom of each HTML page automatically.

* Static content is served directly by *Nginx*.

* Each web application gets distinct sub-directory under ``/var/www``, named

  after the FQDN. All sub-directories created under there are created with

  ``2750`` permissions, with ownership set to admin user, and group set to the

  application's group. In other words, all directories will have ``SGID`` bit

  set-up, allowing you to create files/directories that will have their group

  automatically set to the group of the parent directory.

* Each WSGI website gets a dedicated virtual environment, stored in the

  sub-directory ``virtualenv`` of the website directory, for example

  ``/var/www/wiki.example.com/virtualenv``.

* Static files are served from sub-directory ``htdocs`` in the website

  directory, for example ``/var/www/wiki.example.com/htdocs/``.

* The base directory where your website/application code should be at is

  expected to be in sub-directory ``code`` in the website directory, for example

  ``/var/www/wiki.example.com/code/``.

* Combination of admin user membership in application group, ``SGID``

  permission, and the way ownership of sub-directories is set-up usually means

  that the administrator will be capable of managing application files, and

  application can be granted write permissions to a *minimum* of necessary

  files.

  .. warning::

     Just keep in mind that some file-management commands, like ``mv``, do *not*

     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you

     deploy new files to the directory instead (don't simply move them from your

     home).

1. Set-up the necessary directories first::

     mkdir -p ~/mysite/roles/wiki/{tasks,meta,files}/

2. Set-up some role dependencies, reusing the common role infrastructure.

   :file:`~/mysite/roles/wiki/meta/main.yml`

   ::

      ---

      dependencies:

         - role: wsgi_website

           fqdn: wiki.example.com

           # In many cases you need to have some development packages available

           # in order to build Python packages installed via pip

           packages:

              - build-essential

              - python-dev

              - libjpeg-dev

              - libzip-dev

              - libtiff-dev

              - libfreetype6-dev

              - liblcms2-dev

              - libwebp-dev

              - libopenjpeg-dev

              - libmariadb-client-lgpl-dev

              - libmariadb-client-lgpl-dev-compat

           # Here we specify that anything accessing our website with "/static/"

           # URL should be treated as request to a static file, to be served

           # directly by Nginx instead of the WSGI server.

           static_locations:

              - /static/

           # Again, not mandatory, but it is good to have some sort of policy

           # for assigning UIDs.

           uid: 2001

           admin_uid: 3001

           # These are additional packages that should be installed in the

           # virtual environment.

           virtualenv_packages:

             - pillow

             - django==1.8.13

             - wiki

             - MySQL-python

           # This is the name of the WSGI application to

           # serve. wiki_example_com.wsgi will be the Python "module" that is

           # accesed, while application is the object instantiated within it (the

           # application itself). The module is referenced relative to the code

           # directory (in our case /var/www/wiki.example.com/code/).

           wsgi_application: wiki_example_com.wsgi:application

         - role: database

           db_name: wiki

           db_password: wiki

3. Let's create a dedicated private key/certificate pair for the wiki website:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/wiki.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

---

additional_nginx_config: {}

enforce_https: True

packages: []

rewrites: []

static_locations: []

use_paste: False

virtualenv_packages: []

environment_variables: {}

admin: "web-{{ fqdn | replace('.', '_') }}"

https_tls_certificate: "{{ lookup('file', tls_certificate_dir + '/' + fqdn + '_https.pem') }}"

https_tls_key: "{{ lookup('file', tls_private_key_dir + '/' + fqdn + '_https.key') }}"

gunicorn_version: "19.6.0"

futures_version: "3.0.5"

website_mail_recipients: "root"

{% if enforce_https -%}

server {

    # HTTP (plaintext) configuration.

    listen 80;

    server_name {{ fqdn }};

    # Redirect plaintext connections to HTTPS

    return 301 https://$host$request_uri;

}

{% endif -%}

server {

    # Base settings.

    root {{ home }}/htdocs/;

    server_name {{ fqdn }};

{% if not enforce_https %}

    # HTTP (plaintext) configuration.

    listen 80;

{% endif %}

    # HTTPS (TLS) configuration.

    listen 443 ssl;

    listen [::]:443 ssl;

    ssl_certificate_key /etc/ssl/private/{{ fqdn }}_https.key;

    ssl_certificate /etc/ssl/certs/{{ fqdn }}_https.pem;

{% if default_enforce_https -%}

    # Set-up HSTS header for preventing downgrades for users that visited the

    # site via HTTPS at least once.

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";

{% endif -%}

    {% for config in additional_nginx_config -%}

    # {{ config.comment }}

    {{ config.value }}

    {% endfor -%}

    {% if rewrites -%}

    # Site rewrites.

    {% for rewrite in rewrites -%}

    rewrite {{ rewrite }};

    {% endfor -%}

    {% endif %}

    {% if static_locations -%}

    # Static locations

    {% for location in static_locations -%}

    location {{ location }} {

        try_files $uri $uri/ =404;

    }

    {% endfor -%}

    {% endif %}

    # Pass remaining requests to the WSGI server.

    location / {

        try_files $uri @proxy_to_app;

    }

    location @proxy_to_app {

        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Host $http_host;

        proxy_redirect off;

        proxy_pass http://unix:/run/wsgi/{{ fqdn }}.sock;

    }

    {% if environment_indicator -%}

    # Show environment indicator on HTML pages.

    sub_filter_types text/html;

    sub_filter_once on;

    sub_filter "</body>" "<div id='website-environment' style='background-color: {{ environment_indicator.background_colour }}; width: 100%; text-align: center; position: fixed; bottom: 5px; color: {{ environment_indicator.text_colour }}; font-weight: bold;'>{{ environment_indicator.text }}</div></body>";

    {% endif -%}

    access_log /var/log/nginx/{{ fqdn }}-access.log;

    error_log /var/log/nginx/{{ fqdn }}-error.log;

}

---

local_mail_aliases:

  root: "root john.doe@{{ testsite_domain }}"

smtp_relay_host: mail.{{ testsite_domain }}

smtp_relay_truststore: "{{ lookup('file', inventory_dir + '/tls/ca.pem') }}"

default_https_tls_key: "{{ lookup('file', inventory_dir + '/tls/web.' + testsite_domain + '_https.key') }}"

default_https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/web.' + testsite_domain + '_https.pem') }}"

web_default_title: "Welcome to Example Inc."

web_default_message: "You are attempting to access the web server using a wrong name or an IP address. Please check your URL."

db_root_password: "root"

website_mail_recipients: "john.doe@example.com"

environment_indicator:

  background_colour: "purple"

  text_colour: "white"

#!/usr/bin/env python

import os

def application(environ, start_response):

    status = '200 OK'

    template = """<!DOCTYPE html>

<html lang="en">

  <head>

    <meta charset="utf-8">

    <title>{title}</title>

  </head>

  <body>

    <h1>Hello, world!</h1>

    <p>I am website {title}</p>

  </body>

</html>

"""

    response_headers = [('Content-type', 'text/html'),

                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)

    return [output]