as the basis for writing website-specific roles. Therefore the role is written

in quite generic way, allowing the integrator to write his/her own logic for

deploying the necessary Python applications/packages, while still reusing a

common base and reducing the workload.

The role implements the following:

* Creates a dedicated user/group for running the WSGI application.

* Creates a dedicated administrator user for maintaining the website.

* Creates a base directory where the website-specific code and data should be

  stored at.

* Adds nginx to website's group, so nginx could read the necessary files.

* Adds website administrator to website's group, so administrator could manage

  the code and data.

* Installs additional packages required for running the role (as configured).

* Sets-up a dedicated Python virtual environment for website.

* Install ``futures`` package in Python virtual environment (required for

  Gunicorn in combination withg Python 2.7).

* Install Gunicorn in Python virtual environment.

* Installs additional packages required for running the role in Python virtual

  environment (as configured).

* Configures systemd to run the website code (using Gunicorn)

* Deploys the HTTPS TLS private key and certificate (for website vhost).

* Configures nginx to serve the website (static files served directly, requests

  passed on to Gunicorn).

The role is implemented with the following layout/logic in mind:

* Website users are named after the ``FQDN`` (fully qualified domain name) of

  website, in format of ``web-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal to

  ``FQDN`` where dots have been replaced by underscores (for example,

  ``web-wiki_example_com``).

* Website users are set-up via GECOS field to have their umask set to ``0007``

  (in combination with ``pam_umask``).

* Administrator users are named after the ``FQDN`` (fully qualified domain name)

  of website, in format of ``admin-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal

  to ``FQDN`` where dots have been replaced by underscores (for example,

  ``admin-cloud_example_com``).

* All websites reside within a dedicated sub-directory in ``/var/www``. The

  sub-directory name is equal to the ``FQDN`` used for accessing the

  website. Owner of the directory is set to be the application administrator,

  while group is set to be the website group. Additionally, ``SGID`` bit is set

  on the directory. This allows admin, with correct umask, to create necessary

  files and directories that should be readable (and eventually writeable) by

  the website user (running the WSGI application) without having to become root.

* All files placed in the website directory should be either created there

  directly, or copied to the directory in order to make sure the ``SGID`` gets

  honored. **Do not move the files, the permissions will not be set correctly.**

* Within the website directory, Python virtual environment can be found within

  the ``virtualenv`` sub-directory. Switching to administrator user via login

  shell will automatically activate the virtual environment.

* Within the website directory, nginx will expect to find the static files

  within the ``htdocs`` sub-directory (this can be symlink too). Locations/aliases

  can be configured for static file serving.

* Within the website directory, systemd service will expect to find the website

  code within the ``code`` sub-directory (this can be symlink too).

* nginx communicates with WSGI server over a dedicated Unix socket for each

  website.

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **common**

* **web_server**

Parameters

~~~~~~~~~~

**additional_nginx_config** (list, optional, ``[]``)

  List providing additional Nginx configuration options to include. This can be

  useful for specifying things like error pages. Options are applied inside of a

  **server** context of Nginx configuration file.

  Each item is a dictionary with the following options describing the extra

  configuration option:

  **comment** (string, mandatory)

    Comment describing the configuration option.

  **value** (string, mandatory)

    Configuration option.

**admin_uid** (integer, optional, ``whatever OS picks``)

  UID of the dedicated website administrator user. The user will be member of

  website group.

**enforce_https** (boolean, optional, ``True``)

  Specify if HTTPS should be enforced for the website or not. If enforced,

  clients connecting via plaintext will be redirected to HTTPS, and clients will

  be served with ``Strict-Transport-Security`` header with value of

  ``max-age=31536000; includeSubDomains``.

**fqdn** (string, mandatory)

  Fully-qualified domain name where the website is reachable. This value is used

  for calculating the user/group name for dedicated website user, as well as

  home directory of the website user (where data/code should be stored at).

**futures_version** (string, optional, ``3.0.5``)

  Version of ``futures`` package to deploy in virtual environment. Required by

  Gunicorn when using Python 2.7. Default version is tested with the test site.

**gunicorn_version** (string, optional, ``19.6.0``)

  Version of Gunicorn to deploy in virtual environment for running the WSGI

  application. Default version is tested with the test site.

**https_tls_certificate** (string, optional, ``{{ lookup('file', tls_certificate_dir + '/' + fqdn + '_https.pem') }}``)

  X.509 certificate used for TLS for HTTPS service. The file will be stored in

  directory ``/etc/ssl/certs/`` under name ``{{ fqdn }}_https.pem``.

**https_tls_key** (string, optional, ``{{ lookup('file', tls_private_key_dir + '/' + fqdn + '_https.key') }}``)

  Private key used for TLS for HTTPS service. The file will be stored in

  directory ``/etc/ssl/private/`` under name ``{{ fqdn }}_https.key``.

**packages** (list, optional, ``[]``)

  A list of additional packages to install for this particular WSGI

  website. This is usually going to be development libraries for building Python

  packages.

**rewrites** (list, optional, ``[]``)

  A list of rewrite rules that are applied to incoming requests. Each element of

  the list should be a string value compatible with the format of ``nginx``

  option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well

  as trailing semi-colon (``;``).

**static_locations** (list, optional, ``[]``)

  List of locations that should be treated as static-only, and not processed by

  the WSGI application at all. This is normally used for designating serving of

  static/media files by Nginx (for example, in case of Django projects for

  ``/static/`` and ``/media/``).

**uid** (integer, optional, ``whatever OS picks``)

  UID/GID (they are set-up to be the same) of the dedicated website

  user/group.

**use_paste** (boolean, optional, ``False``)

  Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a

  filename of a Python Paste ``ini`` file instead of WSGI application.

**virtuaelnv_packages** (list, optional, ``[]``)

  A list of additional packages to install for this particular WSGI appliction

  in its virtual environment using ``pip``.

**wsgi_application** (string, mandatory)

  WSGI application that should be started by Gunicorn. The format should be

  conformant to what the ``gunicorn`` command-line tool accepts. If the

  ``use_paste`` option is enabled, the value should be equal to filename of the

  Python Paste ini file, located in the ``code`` sub-directory. It should be

  noted that in either case the value should be specsified relative to the

  ``code`` sub-directory. I.e. don't use full paths.

Examples

~~~~~~~~

Here is an example configuration for setting-up a (base) WSGI website (for

running a bare Django project):

.. code-block:: yaml

    - role: wsgi_website

      fqdn: django.example.com

      static_locations:

        - /static

        - /media

      uid: 2004

      virtualenv_packages:

        - django

      wsgi_application: django_example_com.wsgi:application

      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.key') }}"

      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.pem') }}"

      futures_version: 3.0.5

      gunicorn_version: 19.6.0

      additional_nginx_config:

        - comment: Use custom page for forbidden files.

          value: error_page 403 /static/403.html;

        - comment: Use custom page for non-existing locations/files.

          value: error_page 404 /static/404.html;

Database Server

---------------

The ``database_server`` role can be used for setting-up a MariaDB database

server on destination machine.

The role implements the following:

* Installs MariaDB server and client.

* Configures MariaDB server and client to use *UTF-8* encoding by default.

* Sets password for the database root user.

* Deploys MariaDB client configuration in location ``/root/.my.cnf`` that

  contains username and password for the root database user.

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **common**

Parameters

~~~~~~~~~~

**db_root_password** (string, mandatory)

  Password for the *root* database user.

Examples

~~~~~~~~

Here is an example configuration for setting-up the database server:

.. code-block:: yaml

   ---

   db_root_password: root

Database

--------

The ``database`` role can be used for creating a MariaDB database and

accompanying user on destination machine.

The role implements the following:

* Creates MariaDB database.

* Creates a dedicated user capable of performing any operation on the created

  database. Username is set to be same as the name of database.

* Sets-up pre-backup task that creates database dump in location

  ``/srv/backup/mariadb/{{ db_name }}.sql``.

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **database_server**

* **backup_client**

Backups

~~~~~~~

If the backup for this role has been enabled, the following paths are backed-up:

**/srv/backup/maraidb/{{ db_name }}.sql**

  Dump of the database. Database dump is created every day at 01:45 in the

  morning.

Parameters

~~~~~~~~~~

**db_name** (string, mandatory)

  Name of the database that should be created.

**db_password** (string, mandatory)

  Password for the database user.

      - name: Deploy database configuration file for TBG

        copy: src="b2db.yml" dest="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"

              mode=640 owner=admin-tbg_example_com group=web-tbg_example_com

      - name: Deploy expect script for installing TBG

        copy: src="tbg_expect_install" dest="/var/www/tbg.example.com/tbg_expect_install" mode=750

        become: yes

        become_user: admin-tbg_example_com

      - name: Run TBG installer via expect script

        command: /var/www/tbg.example.com/tbg_expect_install

                 chdir=/var/www/tbg.example.com/thebuggenie-{{ tbg_version }} 

                 creates=/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed

        become: yes

        become_user: admin-tbg_example_com

5. Set-up the files that are deployed by our role.

   :file:`~/mysite/roles/tbg/files/b2db.yml`

   ::

      b2db:

          username: "tbg"

          password: "tbg"

          dsn: "mysql:host=localhost;dbname=tbg"

          tableprefix: ''

          cacheclass: '\thebuggenie\core\framework\Cache'

   :file:`~/mysite/roles/tbg/files/tbg_expect_install`

   ::

      #!/usr/bin/expect

      spawn ./tbg_cli install --accept_license=yes --url_subdir=/ --use_existing_db_info=yes --enable_all_modules=yes --setup_htaccess=yes

      expect "Press ENTER to continue with the installation: "

      send "\r"

      expect "Press ENTER to continue: "

      send "\r"

      interact

6. And... Let's add the new role to our web server.

   :file:`~/mysite/playbooks/web.yml`

   ::

      ---

      - hosts: web

        remote_user: ansible

        become: yes

        roles:

          - common

          - ldap_client

          - mail_forwarder

          - web_server

          - database_server

          - tbg

7. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

8. At this point *The Bug Genie* has been installed, and you should be able to

   open the URL https://tbg.example.com/ (if you open http://tbg.example.com/ ,

   you will be redirected to the HTTPS URL) and log-in into *The Bug Genie*

   with username ``administrator`` and password ``admin``.

Deploying a WSGI application (Django Wiki)

------------------------------------------

Next thing up will be to deploy a WSGI Python application.

Similar to the PHP application deployment, we will use a couple of roles to make

it easier to deploy it in a standardised manner, and we will not have any kind

of parameters for configuring the role to keep things simple.

Most of the notes on how a ``php_website`` role is deployed also stand for the

``wsgi_website`` role, but we will reiterate and clarify them a bit just to be

on the safe side:

* The role is designed to execute every application via dedicated user and

  group. The user/group name is automatically derived from the FQDN of website,

  for example ``web-wiki_example_com``.

* While running the application, application user's umask is set to ``0007``

  (letting the administrator user be able to manage any files created while the

  application is running).

* An administrative user is created as well, and this user should be used when

  running maintenance and installation commands. Similar to application user,

  the name is also derived from the FQDN of website, for example

  ``admin-wiki_example_com``. Administrative user does not have a dedicated

  group, and instead belongs to same group as the application user. As

  convenience, whenever you switch to this user the Python virtual environment

  will be automatically activated for you.

* WSGI applications are executed via *Gunicorn*. The WSGI server listens on a

  Unix socket, making the socket accessible by *Nginx*.

* Static content is served directly by *Nginx*.

* Each web application gets distinct sub-directory under ``/var/www``, named

  after the FQDN. All sub-directories created under there are created with

  ``2750`` permissions, with ownership set to admin user, and group set to the

  application's group. In other words, all directories will have ``SGID`` bit

  set-up, allowing you to create files/directories that will have their group

  automatically set to the group of the parent directory.

* Each WSGI website gets a dedicated virtual environment, stored in the

  sub-directory ``virtualenv`` of the website directory, for example

  ``/var/www/wiki.example.com/virtualenv``.

* Static files are served from sub-directory ``htdocs`` in the website

  directory, for example ``/var/www/wiki.example.com/htdocs/``.

* The base directory where your website/application code should be at is

  expected to be in sub-directory ``code`` in the website directory, for example

  ``/var/www/wiki.example.com/code/``.

* Combination of admin user membership in application group, ``SGID``

  permission, and the way ownership of sub-directories is set-up usually means

  that the administrator will be capable of managing application files, and

  application can be granted write permissions to a *minimum* of necessary

  files.

  .. warning::

     Just keep in mind that some file-management commands, like ``mv``, do *not*

     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you

     deploy new files to the directory instead (don't simply move them from your

     home).

1. Set-up the necessary directories first::

     mkdir -p ~/mysite/roles/wiki/{tasks,meta,files}/

2. Set-up some role dependencies, reusing the common role infrastructure.

   :file:`~/mysite/roles/wiki/meta/main.yml`

   ::

      ---

      dependencies:

         - role: wsgi_website

           fqdn: wiki.example.com

           # In many cases you need to have some development packages available

           # in order to build Python packages installed via pip

           packages:

              - build-essential

              - python-dev

              - libjpeg-dev

              - libzip-dev

              - libtiff-dev

              - libfreetype6-dev

              - liblcms2-dev

              - libwebp-dev

              - libopenjpeg-dev

              - libmariadb-client-lgpl-dev

              - libmariadb-client-lgpl-dev-compat

           # Here we specify that anything accessing our website with "/static/"

           # URL should be treated as request to a static file, to be served

           # directly by Nginx instead of the WSGI server.

           static_locations:

              - /static/

           # Again, not mandatory, but it is good to have some sort of policy

           # for assigning UIDs.

           uid: 2001

           admin_uid: 3001

           # These are additional packages that should be installed in the

           # virtual environment.

           virtualenv_packages:

             - pillow

             - django==1.8.13

             - wiki

             - MySQL-python

           # This is the name of the WSGI application to

           # serve. wiki_example_com.wsgi will be the Python "module" that is

           # accesed, while application is the object instantiated within it (the

           # application itself). The module is referenced relative to the code

           # directory (in our case /var/www/wiki.example.com/code/).

           wsgi_application: wiki_example_com.wsgi:application

         - role: database

           db_name: wiki

           db_password: wiki

3. Let's create a dedicated private key/certificate pair for the wiki website:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/wiki.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

         cn = "Exampe Inc. Wiki"

         expiration_days = 365

         dns_name = "wiki.example.com"

         tls_www_server

         signing_key

         encryption_key

---

additional_nginx_config: {}

enforce_https: True

packages: []

rewrites: []

static_locations: []

use_paste: False

virtualenv_packages: []

admin: "web-{{ fqdn | replace('.', '_') }}"

https_tls_certificate: "{{ lookup('file', tls_certificate_dir + '/' + fqdn + '_https.pem') }}"

https_tls_key: "{{ lookup('file', tls_private_key_dir + '/' + fqdn + '_https.key') }}"

gunicorn_version: "19.6.0"

futures_version: "3.0.5"

---

- set_fact:

    admin: "admin-{{ fqdn | replace('.', '_') }}"

    user: "web-{{ fqdn | replace('.', '_') }}"

    home: "/var/www/{{ fqdn }}"

- name: Create WSGI website group

  group: name="{{ user }}" gid="{{ uid | default(omit) }}" state=present

- name: Create WSGI website admin user

  user: name="{{ admin }}" uid="{{ admin_uid | default(omit) }}" group="{{ user }}"

        shell=/bin/bash createhome=yes home="{{ home }}" state=present

- name: Set-up directory for storing user profile configuration files

  file: path="{{ home }}/.profile.d" state=directory

        owner="{{ admin }}" group="{{ user }}" mode=750

- name: Deploy profile configuration file for auto-activating the virtual environment

  copy: src="profile_virtualenv.sh" dest="{{ home }}/.profile.d/virtualenv.sh"

        owner="root" group="{{ user }}" mode="640"

- name: Create WSGI website user

  user: name="{{ user }}" uid="{{ uid | default(omit) }}" group="{{ user }}" comment="umask=0007"

        system=yes createhome=no state=present

- name: Add nginx user to website group

  user: name="www-data" groups="{{ user }}" append="yes"

  notify:

    - Restart nginx

- name: Install extra packages for website

  apt: name="{{ item }}" state=present

  with_items: "{{ packages }}"

  notify:

    - "Restart website {{ fqdn }}"

- name: Set-up MariaDB mysql_config symbolic link for compatibility (workaround for Debian bug 766996)

  file: src="/usr/bin/mariadb_config" dest="/usr/bin/mysql_config" state=link

  when: "'libmariadb-client-lgpl-dev-compat' in packages"

- name: Create directory for storing the Python virtual environment

  file: path="{{ home }}/virtualenv" state=directory

        owner="{{ admin }}" group="{{ user }}" mode="2750"

- name: Create Python virtual environment

  become_user: "{{ admin }}"

  command: /usr/bin/virtualenv --prompt "({{ fqdn }})" "{{ home }}/virtualenv" creates="{{ home }}/virtualenv/bin/activate"

- name: Configure project directory for the Python virtual environment

  template: src="venv_project.j2" dest="{{ home }}/virtualenv/.project"

            owner="{{ admin }}" group="{{ user }}" mode="640"

- name: Deploy virtualenv wrapper

  template: src="venv_exec.j2" dest="{{ home }}/virtualenv/bin/exec"

            owner="{{ admin }}" group="{{ user }}" mode="750"

- name: Install futures package for use with Gunicorn thread workers

  become_user: "{{ admin }}"

  pip: name=futures version="{{ futures_version }}" state=present virtualenv="{{ home }}/virtualenv"

  notify:

    - "Restart website {{ fqdn }}"

- name: Install Gunicorn in Python virtual environment

  become_user: "{{ admin }}"

  pip: name=gunicorn version="{{ gunicorn_version }}" state=present virtualenv="{{ home }}/virtualenv"

  notify:

    - "Restart website {{ fqdn }}"

- name: Install additional packages in Python virtual environment

  become_user: "{{ admin }}"

  pip: name="{{ item }}" state=present virtualenv="{{ home }}/virtualenv"

  with_items: "{{ virtualenv_packages }}"

  notify:

    - "Restart website {{ fqdn }}"

- name: Deploy systemd socket configuration for website

  template: src="systemd_wsgi_website.socket.j2" dest="/etc/systemd/system/{{ fqdn }}.socket"

            owner=root group=root mode=644

  notify:

    - Reload systemd

    - "Restart website {{ fqdn }}"

- name: Deploy systemd service configuration for website

  template: src="systemd_wsgi_website.service.j2" dest="/etc/systemd/system/{{ fqdn }}.service"

            owner=root group=root mode=644

  notify:

    - Reload systemd

    - "Restart website {{ fqdn }}"

- name: Enable the website service

  service: name="{{ fqdn }}" enabled=yes state=started

- name: Create directory where static files can be served from

  file: path="{{ home }}/htdocs/" state=directory

        owner="{{ admin }}" group="{{ user }}" mode="2750"

- name: Deploy nginx TLS private key for website

  copy: dest="/etc/ssl/private/{{ fqdn }}_https.key" content="{{ https_tls_key }}"

        mode=640 owner=root group=root

  notify:

    - Restart nginx

- name: Deploy nginx TLS certificate for website

  copy: dest="/etc/ssl/certs/{{ fqdn }}_https.pem" content="{{ https_tls_certificate }}"

        mode=644 owner=root group=root

  notify:

    - Restart nginx

- name: Deploy nginx configuration file for website

  template: src="nginx_site.j2" dest="/etc/nginx/sites-available/{{ fqdn }}"

            owner=root group=root mode=640 validate="/usr/local/bin/nginx_verify_site.sh -n '{{ fqdn }}' %s"

  notify:

    - Restart nginx

- name: Enable nginx website

  file: src="/etc/nginx/sites-available/{{ fqdn }}" dest="/etc/nginx/sites-enabled/{{ fqdn }}"

        state=link

[Unit]

Description=Website {{ fqdn }}

Requires={{ fqdn }}.socket

After=network.target

[Service]

User={{ user }}

Group={{ user }}

WorkingDirectory={{ home }}/code

ExecStart={{ home }}/virtualenv/bin/gunicorn --bind unix:/run/wsgi/{{ fqdn }}.sock {% if use_paste %}--paste {{home}}/code/{{ wsgi_application }}{% else %}{{ wsgi_application }}{% endif %}

ExecReload=/bin/kill -s HUP $MAINPID

ExecStop=/bin/kill -s TERM $MAINPID

PrivateTmp=true

UMask=0007

[Install]

#!/usr/bin/env python

def application(environ, start_response):

    status = '200 OK'

    response_headers = [('Content-type', 'text/plain'),

                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)

    return [output]

---

dependencies:

  - role: wsgi_website

    fqdn: wsgi.{{ testsite_domain }}

    admin_uid: 3001

    uid: 2001

    wsgi_application: wsgi:application

    static_locations:

      - /static/

    https_tls_key: "{{ lookup('file', inventory_dir + '/tls/wsgi.' + testsite_domain + '_https.key') }}"

    https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/wsgi.' + testsite_domain + '_https.pem') }}"

  - role: database

    db_name: wsgi_{{ testsite_domain_underscores }}

    db_password: wsgi_{{ testsite_domain_underscores }}