From 981584549895a4e032b205a10fd05af2d93a9f4d 2015-05-10 00:48:54
From: Branko Majic <branko@majic.rs>
Date: 2015-05-10 00:48:54
Subject: [PATCH] MAR-5: Adding the initial implementation of the WSGI website role.

---

diff --git a/docs/rolereference.rst b/docs/rolereference.rst
index 3ec43a8204b4a5a82d57cdba71bc445d8a26fbd5..86f3b8ee7e72204c9efe9043b4a26e741f34e513 100644
--- a/docs/rolereference.rst
+++ b/docs/rolereference.rst
@@ -896,7 +896,7 @@ Here is an example configuration for setting-up web server:
 
 
 PHP Website
---------
+-----------
 
 The ``php_website`` role can be used for setting-up a website powered by PHP on
 destination machine.
@@ -1013,3 +1013,133 @@ Here is an example configuration for setting-up a (base) PHP website (for runnin
         - php5-json
         - php5-mysql
         - php5-curl
+
+
+WSGI Website
+------------
+
+The ``wsgi_website`` role can be used for setting-up a website powered by Python
+on destination machine. The website needs to use the WSGI specification for
+making the Python web application(s) available.
+
+This role is normally not supposed to be used directly, but should instead serve
+as the basis for writing website-specific roles. Therefore the role is written
+in quite generic way, allowing the integrator to write his/her own logic for
+deploying the necessary Python applications/packages, while still reusing a
+common base and reducing the workload.
+
+The role implements the following:
+
+* Creates a dedicated user/group for running the WSGI application.
+* Creates a base directory where the website-specific code and data should be
+  stored at.
+* Adds nginx to website's group, so nginx could read the necessary files.
+* Adds website administrator to website's group, so administrator could manage
+  the code and data.
+* Installs additional packages required for running the role (as configured).
+* Sets-up a dedicated Python virtual environment for website.
+* Install Gunicorn in Python virtual environment.
+* Installs additional packages required for running the role in Python virtual
+  environment (as configured).
+* Configures systemd to run the website code (using Gunicorn)
+* Configures nginx to serve the website (static files served directly, requests
+  passed on to Gunicorn).
+
+The role is implemented with the following layout/logic in mind:
+
+* Website users are named after the ``FQDN`` (fully qualified domain name) of
+  website, in format of ``web-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal to
+  ``FQDN`` where dots have been replaced by underscores (for example,
+  ``web-wiki_example_com``).
+* All websites reside within a dedicated sub-directory in ``/var/www``. The
+  sub-directory name is equal to the ``FQDN`` used for accessing the
+  website. Owner of the directory is set to be the application administrator,
+  while group is set to be the website group. Additionally, ``SGID`` bit is set
+  on the directory. This allows admin, with correct umask, to create necessary
+  files and directories that should be readable (and eventually writeable) by
+  the website user (running the WSGI application) without having to become root.
+* All files placed in the website directory should be either created there
+  directly, or copied to the directory in order to make sure the ``SGID`` gets
+  honored. **Do not move the files, the permissions will not be set correctly.**
+* Within the website directory, Python virtual environment can be found within
+  the ``virtualenv`` sub-directory. The virtual environment is also symlinked to
+  website admin's ``~/.virtualenvs/`` directory for easier access (and
+  auto-completion with virtualenvwrapper).
+* Within the website directory, nginx will expect to find the static files
+  within the ``htdocs`` sub-directory (this can be symlink too). Locations/aliases
+  can be configured for static file serving.
+* Within the website directory, systemd service will expect to find the website
+  code within the ``code`` sub-directory (this can be symlink too).
+* nginx communicates with WSGI server over a dedicated Unix socket for each
+  website.
+
+
+Parameters
+~~~~~~~~~~
+
+**admin** (string, mandatory)
+  Name of the operating system user in charge of maintaining the website. This
+  user is capable of making modifications to website configuration anda data
+  stored within the website directory.
+
+**fqdn** (string, mandatory)
+  Fully-qualified domain name where the website is reachable. This value is used
+  for calculating the user/group name for dedicated website user, as well as
+  home directory of the website user (where data/code should be stored at).
+
+**packages** (list, optional)
+  A list of additional packages to install for this particular WSGI
+  website. This is usually going to be development libraries for building Python
+  packages.
+
+**rewrites** (list, optional)
+  A list of rewrite rules that are applied to incoming requests. Each element of
+  the list should be a string value compatible with the format of ``nginx``
+  option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well
+  as trailing semi-colon (``;``).
+
+**static_locations** (list, optional)
+  List of locations that should be treated as static-only, and not processed by
+  the WSGI application at all. This is normally used for designating serving of
+  static/media files by Nginx (for example, in case of Django projects for
+  ``/static/`` and ``/media/``).
+
+**uid** (integer, mandatory)
+  UID/GID (they are set-up to be the same) of the dedicated website
+  user/group.
+
+**use_paste** (boolean, optional)
+  Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a
+  filename of a Python Paste ``ini`` file instead of WSGI application.
+
+**virtuaelnv_packages** (list, optional)
+  A list of additional packages to install for this particular PHP
+  appliction. This is usually going to be different PHP extensions.
+
+**wsgi_application** (string, mandatory)
+  WSGI application that should be started by Gunicorn. The format should be
+  conformant to what the ``gunicorn`` command-line tool accepts. If the
+  ``use_paste`` option is enabled, the value should be equal to filename of the
+  Python Paste ini file, located in the ``code`` sub-directory.
+
+
+Examples
+~~~~~~~~
+
+Here is an example configuration for setting-up a (base) WSGI website (for
+running a bare Django project):
+
+.. code-block:: yaml
+
+    ---
+
+    - role: wsgi_website
+      admin: admin
+      fqdn: django.example.com
+      static_locations:
+        - /static
+        - /media
+      uid: 2004
+      virtualenv_packages:
+        - django
+      wsgi_application: django_example_com.wsgi:application
diff --git a/roles/wsgi_website/defaults/main.yml b/roles/wsgi_website/defaults/main.yml
new file mode 100644
index 0000000000000000000000000000000000000000..ae0b1396890577f0f970773945ba9d5ae0ea24e0
--- /dev/null
+++ b/roles/wsgi_website/defaults/main.yml
@@ -0,0 +1,7 @@
+---
+
+packages: []
+rewrites: []
+static_locations: []
+use_paste: False
+virtualenv_packages: []
\ No newline at end of file
diff --git a/roles/wsgi_website/handlers/main.yml b/roles/wsgi_website/handlers/main.yml
new file mode 100644
index 0000000000000000000000000000000000000000..205f7aeda9b6b99b9f7a6d1d6e2765cce994fabc
--- /dev/null
+++ b/roles/wsgi_website/handlers/main.yml
@@ -0,0 +1,4 @@
+---
+
+- name: Restart website
+  service: name="{{ fqdn }}" state=restarted
diff --git a/roles/wsgi_website/tasks/main.yml b/roles/wsgi_website/tasks/main.yml
new file mode 100644
index 0000000000000000000000000000000000000000..f2f8f2440d6e074bd9ec266b1de91325988ae03e
--- /dev/null
+++ b/roles/wsgi_website/tasks/main.yml
@@ -0,0 +1,94 @@
+---
+
+- set_fact:
+    user: "web-{{ fqdn | replace('.', '_') }}"
+    home: "/var/www/{{ fqdn }}"
+
+- name: Create WSGI website group
+  group: name="{{ user }}" gid="{{ uid }}" state=present
+
+- name: Create home directory for the user (avoid populating with skeleton)
+  file: path="{{ home }}" state=directory
+        owner="{{ admin }}" group="{{ user }}" mode=2750
+
+- name: Create WSGI website user
+  user: name="{{ user }}" uid="{{ uid }}" group="{{ user }}"
+        system=yes createhome=no state=present
+
+- name: Add nginx user to website group
+  user: name="www-data" groups="{{ user }}" append="yes"
+  notify:
+    - Restart nginx
+
+- name: Add admin to website group
+  user: name="{{ admin }}" groups="{{ user }}" append="yes"
+
+- name: Create directory for storing socket file
+  file: path="/var/run/wsgi/{{ fqdn }}" state="directory"
+        owner="{{ user }}" group="www-data" mode="750"
+
+- name: Install extra packages for website
+  apt: name="{{ item }}" state=present
+  with_items: packages
+
+- name: Create directory for storing the Python virtual environment
+  file: path="{{ home }}/virtualenv" state=directory
+        owner="{{ admin }}" group="{{ user }}" mode="2750"
+
+- name: Create Python virtual environment
+  sudo_user: "{{ admin }}"
+  command: /usr/bin/virtualenv "{{ home }}/virtualenv" creates="{{ home }}/virtualenv/bin/activate"
+
+- name: Create directory where virtualenvs will be symlinked to
+  sudo_user: "{{ admin }}"
+  file: path="~/.virtualenvs" state=directory mode=750
+
+- name: Create convenience symlink for Python virtual environment wrapper utility
+  sudo_user: "{{ admin }}"
+  file: src="{{ home }}/virtualenv" dest="~/.virtualenvs/{{ fqdn }}" state=link
+
+- name: Deploy virtualenv wrapper
+  template: src="venv_exec.j2" dest="{{ home }}/virtualenv/bin/exec"
+            owner="{{ admin }}" group="{{ user }}" mode="750"
+
+- name: Install Gunicorn in Python virtual environment
+  sudo_user: "{{ admin }}"
+  pip: name=gunicorn state=present virtualenv="{{ home }}/virtualenv"
+
+- name: Install additional packages in Python virtual environment
+  sudo_user: "{{ admin }}"
+  pip: name="{{ item }}" state=present virtualenv="{{ home }}/virtualenv"
+  with_items: virtualenv_packages
+
+- name: Deploy systemd socket configuration for website
+  template: src="systemd_wsgi_website.socket.j2" dest="/etc/systemd/system/{{ fqdn }}.socket"
+            owner=root group=root mode=644
+  notify:
+    - Reload systemd
+    - Restart website
+
+- name: Deploy systemd service configuration for website
+  template: src="systemd_wsgi_website.service.j2" dest="/etc/systemd/system/{{ fqdn }}.service"
+            owner=root group=root mode=644
+  notify:
+    - Reload systemd
+    - Restart website
+
+- name: Enable the website service
+  service: name="{{ fqdn }}" enabled=yes state=started
+
+- name: Create directory where static files can be served from
+  file: path="{{ home }}/htdocs/" state=directory
+        owner="{{ admin }}" group="{{ user }}" mode="2750"
+
+- name: Deploy nginx configuration file for website
+  template: src="nginx_site.j2" dest="/etc/nginx/sites-available/{{ fqdn }}"
+            owner=root group=root mode=640 validate="/usr/local/bin/nginx_verify_site.sh -n '{{ fqdn }}' %s"
+  notify:
+    - Restart nginx
+
+- name: Enable nginx website
+  file: src="/etc/nginx/sites-available/{{ fqdn }}" dest="/etc/nginx/sites-enabled/{{ fqdn }}"
+        state=link
+  notify:
+    - Restart nginx
diff --git a/roles/wsgi_website/templates/nginx_site.j2 b/roles/wsgi_website/templates/nginx_site.j2
new file mode 100644
index 0000000000000000000000000000000000000000..dbc13bc3e954b107d751bf3f6bc1d248a1123480
--- /dev/null
+++ b/roles/wsgi_website/templates/nginx_site.j2
@@ -0,0 +1,35 @@
+server {
+    listen 80;
+
+    root {{ home }}/htdocs/;
+
+    server_name {{ fqdn }};
+
+    # Site rewrites.
+    {% for rewrite in rewrites -%}
+    rewrite {{ rewrite }};
+    {% endfor %}
+
+    # Static locations
+    {% for location in static_locations -%}
+    location {{ location }} {
+        try_files $uri $uri/ =404;
+    }
+    {% endfor %}
+
+    # Pass remaining requests to the WSGI server.
+    location / {
+        try_files $uri @proxy_to_app;
+    }
+
+    location @proxy_to_app {
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header Host $http_host;
+        proxy_redirect off;
+
+        proxy_pass http://unix:/run/wsgi/{{ fqdn }}.sock;
+    }
+
+    access_log /var/log/nginx/{{ fqdn }}-access.log;
+    error_log /var/log/nginx/{{ fqdn }}-error.log;
+}
diff --git a/roles/wsgi_website/templates/supervisor_site.conf.j2 b/roles/wsgi_website/templates/supervisor_site.conf.j2
new file mode 100644
index 0000000000000000000000000000000000000000..9988fd63275b75c7f02c16cc1f4bc64388b7e1a5
--- /dev/null
+++ b/roles/wsgi_website/templates/supervisor_site.conf.j2
@@ -0,0 +1,10 @@
+[program:{{ fqdn }}]
+command={{ home }}/virtualenv/bin/gunicorn --bind unix:/var/run/wsgi/{{ fqdn }}/socket {% if use_paste %}--paste{% endif %} {{ wsgi_application }}
+directory={{ home }}/code
+user={{ user }}
+group={{ user }}
+autostart=true
+autorestart=true
+umask=0007
+redirect_stderr=true
+environment=HOME="{{ home }}"
diff --git a/roles/wsgi_website/templates/systemd_wsgi_website.service.j2 b/roles/wsgi_website/templates/systemd_wsgi_website.service.j2
new file mode 100644
index 0000000000000000000000000000000000000000..7d22006393d5ee7e40c9c0f87cf1958264042469
--- /dev/null
+++ b/roles/wsgi_website/templates/systemd_wsgi_website.service.j2
@@ -0,0 +1,17 @@
+[Unit]
+Description=Website {{ fqdn }}
+Requires={{ fqdn }}.socket
+After=network.target
+
+[Service]
+User={{ user }}
+Group={{ user }}
+WorkingDirectory={{ home }}/code
+ExecStart={{ home }}/virtualenv/bin/gunicorn --bind unix:/run/wsgi/{{ fqdn }}.sock {% if use_paste %}--paste {{home}}/code/{{ wsgi_application }}{% else %}{{ wsgi_application }}{% endif %}
+
+ExecReload=/bin/kill -s HUP $MAINPID
+ExecStop=/bin/kill -s TERM $MAINPID
+PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target
\ No newline at end of file
diff --git a/roles/wsgi_website/templates/systemd_wsgi_website.socket.j2 b/roles/wsgi_website/templates/systemd_wsgi_website.socket.j2
new file mode 100644
index 0000000000000000000000000000000000000000..57e48ccfe4681ede2a13bcdca067768e527a7c0a
--- /dev/null
+++ b/roles/wsgi_website/templates/systemd_wsgi_website.socket.j2
@@ -0,0 +1,11 @@
+[Unit]
+Description=Socket for website {{ fqdn }}
+
+[Socket]
+ListenStream=/run/wsgi/{{ fqdn }}.sock
+SocketUser=www-data
+SocketGroup=www-data
+SocketMode=0660
+
+[Install]
+WantedBy=sockets.target
diff --git a/roles/wsgi_website/templates/venv_exec.j2 b/roles/wsgi_website/templates/venv_exec.j2
new file mode 100644
index 0000000000000000000000000000000000000000..3ae1e271735ce413d708f21e69070ab416b70fd7
--- /dev/null
+++ b/roles/wsgi_website/templates/venv_exec.j2
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+source {{ home }}/virtualenv/bin/activate
+$@