From 981584549895a4e032b205a10fd05af2d93a9f4d 2015-05-10 00:48:54 From: Branko Majic Date: 2015-05-10 00:48:54 Subject: [PATCH] MAR-5: Adding the initial implementation of the WSGI website role. --- diff --git a/docs/rolereference.rst b/docs/rolereference.rst index 3ec43a8204b4a5a82d57cdba71bc445d8a26fbd5..86f3b8ee7e72204c9efe9043b4a26e741f34e513 100644 --- a/docs/rolereference.rst +++ b/docs/rolereference.rst @@ -896,7 +896,7 @@ Here is an example configuration for setting-up web server: PHP Website --------- +----------- The ``php_website`` role can be used for setting-up a website powered by PHP on destination machine. @@ -1013,3 +1013,133 @@ Here is an example configuration for setting-up a (base) PHP website (for runnin - php5-json - php5-mysql - php5-curl + + +WSGI Website +------------ + +The ``wsgi_website`` role can be used for setting-up a website powered by Python +on destination machine. The website needs to use the WSGI specification for +making the Python web application(s) available. + +This role is normally not supposed to be used directly, but should instead serve +as the basis for writing website-specific roles. Therefore the role is written +in quite generic way, allowing the integrator to write his/her own logic for +deploying the necessary Python applications/packages, while still reusing a +common base and reducing the workload. + +The role implements the following: + +* Creates a dedicated user/group for running the WSGI application. +* Creates a base directory where the website-specific code and data should be + stored at. +* Adds nginx to website's group, so nginx could read the necessary files. +* Adds website administrator to website's group, so administrator could manage + the code and data. +* Installs additional packages required for running the role (as configured). +* Sets-up a dedicated Python virtual environment for website. +* Install Gunicorn in Python virtual environment. +* Installs additional packages required for running the role in Python virtual + environment (as configured). +* Configures systemd to run the website code (using Gunicorn) +* Configures nginx to serve the website (static files served directly, requests + passed on to Gunicorn). + +The role is implemented with the following layout/logic in mind: + +* Website users are named after the ``FQDN`` (fully qualified domain name) of + website, in format of ``web-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal to + ``FQDN`` where dots have been replaced by underscores (for example, + ``web-wiki_example_com``). +* All websites reside within a dedicated sub-directory in ``/var/www``. The + sub-directory name is equal to the ``FQDN`` used for accessing the + website. Owner of the directory is set to be the application administrator, + while group is set to be the website group. Additionally, ``SGID`` bit is set + on the directory. This allows admin, with correct umask, to create necessary + files and directories that should be readable (and eventually writeable) by + the website user (running the WSGI application) without having to become root. +* All files placed in the website directory should be either created there + directly, or copied to the directory in order to make sure the ``SGID`` gets + honored. **Do not move the files, the permissions will not be set correctly.** +* Within the website directory, Python virtual environment can be found within + the ``virtualenv`` sub-directory. The virtual environment is also symlinked to + website admin's ``~/.virtualenvs/`` directory for easier access (and + auto-completion with virtualenvwrapper). +* Within the website directory, nginx will expect to find the static files + within the ``htdocs`` sub-directory (this can be symlink too). Locations/aliases + can be configured for static file serving. +* Within the website directory, systemd service will expect to find the website + code within the ``code`` sub-directory (this can be symlink too). +* nginx communicates with WSGI server over a dedicated Unix socket for each + website. + + +Parameters +~~~~~~~~~~ + +**admin** (string, mandatory) + Name of the operating system user in charge of maintaining the website. This + user is capable of making modifications to website configuration anda data + stored within the website directory. + +**fqdn** (string, mandatory) + Fully-qualified domain name where the website is reachable. This value is used + for calculating the user/group name for dedicated website user, as well as + home directory of the website user (where data/code should be stored at). + +**packages** (list, optional) + A list of additional packages to install for this particular WSGI + website. This is usually going to be development libraries for building Python + packages. + +**rewrites** (list, optional) + A list of rewrite rules that are applied to incoming requests. Each element of + the list should be a string value compatible with the format of ``nginx`` + option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well + as trailing semi-colon (``;``). + +**static_locations** (list, optional) + List of locations that should be treated as static-only, and not processed by + the WSGI application at all. This is normally used for designating serving of + static/media files by Nginx (for example, in case of Django projects for + ``/static/`` and ``/media/``). + +**uid** (integer, mandatory) + UID/GID (they are set-up to be the same) of the dedicated website + user/group. + +**use_paste** (boolean, optional) + Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a + filename of a Python Paste ``ini`` file instead of WSGI application. + +**virtuaelnv_packages** (list, optional) + A list of additional packages to install for this particular PHP + appliction. This is usually going to be different PHP extensions. + +**wsgi_application** (string, mandatory) + WSGI application that should be started by Gunicorn. The format should be + conformant to what the ``gunicorn`` command-line tool accepts. If the + ``use_paste`` option is enabled, the value should be equal to filename of the + Python Paste ini file, located in the ``code`` sub-directory. + + +Examples +~~~~~~~~ + +Here is an example configuration for setting-up a (base) WSGI website (for +running a bare Django project): + +.. code-block:: yaml + + --- + + - role: wsgi_website + admin: admin + fqdn: django.example.com + static_locations: + - /static + - /media + uid: 2004 + virtualenv_packages: + - django + wsgi_application: django_example_com.wsgi:application diff --git a/roles/wsgi_website/defaults/main.yml b/roles/wsgi_website/defaults/main.yml new file mode 100644 index 0000000000000000000000000000000000000000..ae0b1396890577f0f970773945ba9d5ae0ea24e0 --- /dev/null +++ b/roles/wsgi_website/defaults/main.yml @@ -0,0 +1,7 @@ +--- + +packages: [] +rewrites: [] +static_locations: [] +use_paste: False +virtualenv_packages: [] \ No newline at end of file diff --git a/roles/wsgi_website/handlers/main.yml b/roles/wsgi_website/handlers/main.yml new file mode 100644 index 0000000000000000000000000000000000000000..205f7aeda9b6b99b9f7a6d1d6e2765cce994fabc --- /dev/null +++ b/roles/wsgi_website/handlers/main.yml @@ -0,0 +1,4 @@ +--- + +- name: Restart website + service: name="{{ fqdn }}" state=restarted diff --git a/roles/wsgi_website/tasks/main.yml b/roles/wsgi_website/tasks/main.yml new file mode 100644 index 0000000000000000000000000000000000000000..f2f8f2440d6e074bd9ec266b1de91325988ae03e --- /dev/null +++ b/roles/wsgi_website/tasks/main.yml @@ -0,0 +1,94 @@ +--- + +- set_fact: + user: "web-{{ fqdn | replace('.', '_') }}" + home: "/var/www/{{ fqdn }}" + +- name: Create WSGI website group + group: name="{{ user }}" gid="{{ uid }}" state=present + +- name: Create home directory for the user (avoid populating with skeleton) + file: path="{{ home }}" state=directory + owner="{{ admin }}" group="{{ user }}" mode=2750 + +- name: Create WSGI website user + user: name="{{ user }}" uid="{{ uid }}" group="{{ user }}" + system=yes createhome=no state=present + +- name: Add nginx user to website group + user: name="www-data" groups="{{ user }}" append="yes" + notify: + - Restart nginx + +- name: Add admin to website group + user: name="{{ admin }}" groups="{{ user }}" append="yes" + +- name: Create directory for storing socket file + file: path="/var/run/wsgi/{{ fqdn }}" state="directory" + owner="{{ user }}" group="www-data" mode="750" + +- name: Install extra packages for website + apt: name="{{ item }}" state=present + with_items: packages + +- name: Create directory for storing the Python virtual environment + file: path="{{ home }}/virtualenv" state=directory + owner="{{ admin }}" group="{{ user }}" mode="2750" + +- name: Create Python virtual environment + sudo_user: "{{ admin }}" + command: /usr/bin/virtualenv "{{ home }}/virtualenv" creates="{{ home }}/virtualenv/bin/activate" + +- name: Create directory where virtualenvs will be symlinked to + sudo_user: "{{ admin }}" + file: path="~/.virtualenvs" state=directory mode=750 + +- name: Create convenience symlink for Python virtual environment wrapper utility + sudo_user: "{{ admin }}" + file: src="{{ home }}/virtualenv" dest="~/.virtualenvs/{{ fqdn }}" state=link + +- name: Deploy virtualenv wrapper + template: src="venv_exec.j2" dest="{{ home }}/virtualenv/bin/exec" + owner="{{ admin }}" group="{{ user }}" mode="750" + +- name: Install Gunicorn in Python virtual environment + sudo_user: "{{ admin }}" + pip: name=gunicorn state=present virtualenv="{{ home }}/virtualenv" + +- name: Install additional packages in Python virtual environment + sudo_user: "{{ admin }}" + pip: name="{{ item }}" state=present virtualenv="{{ home }}/virtualenv" + with_items: virtualenv_packages + +- name: Deploy systemd socket configuration for website + template: src="systemd_wsgi_website.socket.j2" dest="/etc/systemd/system/{{ fqdn }}.socket" + owner=root group=root mode=644 + notify: + - Reload systemd + - Restart website + +- name: Deploy systemd service configuration for website + template: src="systemd_wsgi_website.service.j2" dest="/etc/systemd/system/{{ fqdn }}.service" + owner=root group=root mode=644 + notify: + - Reload systemd + - Restart website + +- name: Enable the website service + service: name="{{ fqdn }}" enabled=yes state=started + +- name: Create directory where static files can be served from + file: path="{{ home }}/htdocs/" state=directory + owner="{{ admin }}" group="{{ user }}" mode="2750" + +- name: Deploy nginx configuration file for website + template: src="nginx_site.j2" dest="/etc/nginx/sites-available/{{ fqdn }}" + owner=root group=root mode=640 validate="/usr/local/bin/nginx_verify_site.sh -n '{{ fqdn }}' %s" + notify: + - Restart nginx + +- name: Enable nginx website + file: src="/etc/nginx/sites-available/{{ fqdn }}" dest="/etc/nginx/sites-enabled/{{ fqdn }}" + state=link + notify: + - Restart nginx diff --git a/roles/wsgi_website/templates/nginx_site.j2 b/roles/wsgi_website/templates/nginx_site.j2 new file mode 100644 index 0000000000000000000000000000000000000000..dbc13bc3e954b107d751bf3f6bc1d248a1123480 --- /dev/null +++ b/roles/wsgi_website/templates/nginx_site.j2 @@ -0,0 +1,35 @@ +server { + listen 80; + + root {{ home }}/htdocs/; + + server_name {{ fqdn }}; + + # Site rewrites. + {% for rewrite in rewrites -%} + rewrite {{ rewrite }}; + {% endfor %} + + # Static locations + {% for location in static_locations -%} + location {{ location }} { + try_files $uri $uri/ =404; + } + {% endfor %} + + # Pass remaining requests to the WSGI server. + location / { + try_files $uri @proxy_to_app; + } + + location @proxy_to_app { + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header Host $http_host; + proxy_redirect off; + + proxy_pass http://unix:/run/wsgi/{{ fqdn }}.sock; + } + + access_log /var/log/nginx/{{ fqdn }}-access.log; + error_log /var/log/nginx/{{ fqdn }}-error.log; +} diff --git a/roles/wsgi_website/templates/supervisor_site.conf.j2 b/roles/wsgi_website/templates/supervisor_site.conf.j2 new file mode 100644 index 0000000000000000000000000000000000000000..9988fd63275b75c7f02c16cc1f4bc64388b7e1a5 --- /dev/null +++ b/roles/wsgi_website/templates/supervisor_site.conf.j2 @@ -0,0 +1,10 @@ +[program:{{ fqdn }}] +command={{ home }}/virtualenv/bin/gunicorn --bind unix:/var/run/wsgi/{{ fqdn }}/socket {% if use_paste %}--paste{% endif %} {{ wsgi_application }} +directory={{ home }}/code +user={{ user }} +group={{ user }} +autostart=true +autorestart=true +umask=0007 +redirect_stderr=true +environment=HOME="{{ home }}" diff --git a/roles/wsgi_website/templates/systemd_wsgi_website.service.j2 b/roles/wsgi_website/templates/systemd_wsgi_website.service.j2 new file mode 100644 index 0000000000000000000000000000000000000000..7d22006393d5ee7e40c9c0f87cf1958264042469 --- /dev/null +++ b/roles/wsgi_website/templates/systemd_wsgi_website.service.j2 @@ -0,0 +1,17 @@ +[Unit] +Description=Website {{ fqdn }} +Requires={{ fqdn }}.socket +After=network.target + +[Service] +User={{ user }} +Group={{ user }} +WorkingDirectory={{ home }}/code +ExecStart={{ home }}/virtualenv/bin/gunicorn --bind unix:/run/wsgi/{{ fqdn }}.sock {% if use_paste %}--paste {{home}}/code/{{ wsgi_application }}{% else %}{{ wsgi_application }}{% endif %} + +ExecReload=/bin/kill -s HUP $MAINPID +ExecStop=/bin/kill -s TERM $MAINPID +PrivateTmp=true + +[Install] +WantedBy=multi-user.target \ No newline at end of file diff --git a/roles/wsgi_website/templates/systemd_wsgi_website.socket.j2 b/roles/wsgi_website/templates/systemd_wsgi_website.socket.j2 new file mode 100644 index 0000000000000000000000000000000000000000..57e48ccfe4681ede2a13bcdca067768e527a7c0a --- /dev/null +++ b/roles/wsgi_website/templates/systemd_wsgi_website.socket.j2 @@ -0,0 +1,11 @@ +[Unit] +Description=Socket for website {{ fqdn }} + +[Socket] +ListenStream=/run/wsgi/{{ fqdn }}.sock +SocketUser=www-data +SocketGroup=www-data +SocketMode=0660 + +[Install] +WantedBy=sockets.target diff --git a/roles/wsgi_website/templates/venv_exec.j2 b/roles/wsgi_website/templates/venv_exec.j2 new file mode 100644 index 0000000000000000000000000000000000000000..3ae1e271735ce413d708f21e69070ab416b70fd7 --- /dev/null +++ b/roles/wsgi_website/templates/venv_exec.j2 @@ -0,0 +1,4 @@ +#!/bin/bash + +source {{ home }}/virtualenv/bin/activate +$@