Changeset - 67dd87d59abb
[Not reviewed]
0 6 0
Branko Majic (branko) - 9 years ago 2016-11-28 22:39:12
branko@majic.rs
MAR-83: Added support to wsgi_website role for specifying headers that should be passed on to Gunicorn by Nginx. Updated hello.wsgi app to demonstrate the feature.
6 files changed with 24 insertions and 3 deletions:
0 comments (0 inline, 0 general)
docs/rolereference.rst
Show inline comments
 
@@ -1381,455 +1381,464 @@ running *ownCloud* and *The Bug Genie* applications):
 
        - ^/apps/contacts/carddav\.php /remote.php/carddav/
 
        - ^/remote/(.*) /remote.php
 
      deny_files_regex:
 
        - ^(\.|autotest|occ|issue|indie|db_|console|build/|tests/|config/|lib/|3rdparty/|templates/).*
 
      packages:
 
        # For ownCloud
 
        - php5-gd
 
        - php5-json
 
        - php5-mysql
 
        - php5-curl
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/cloud.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/cloud.example.com_https.pem') }}"
 
      additional_nginx_config:
 
        - comment: Use custom page for forbidden files.
 
          value: error_page 403 /core/templates/403.php;
 
        - comment: Use custom page for non-existing locations/files.
 
          value: error_page 404 /core/templates/404.php;
 
      additional_fpm_config:
 
        "env[PATH]": "\"/usr/local/bin:/usr/bin:/bin\""
 
      website_mail_recipients: "root john.doe@example.com"
 
      environment_indicator:
 
        background_colour: "green"
 
        text_colour: "black"
 
        text: "TEST ENVIRONMENT"
 
    - role: php_website
 
      deny_files_regex:
 
        - ^\..*
 
      php_rewrite_urls:
 
        - ^(.*) /index.php?url=$1
 
      fqdn: tbg.example.com
 
      uid: 2007
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/tbg.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/tbg.example.com_https.pem') }}"
 

	
 

	
 
WSGI Website
 
------------
 

	
 
The ``wsgi_website`` role can be used for setting-up a website powered by Python
 
on destination machine. The website needs to use the WSGI specification for
 
making the Python web application(s) available.
 

	
 
This role is normally not supposed to be used directly, but should instead serve
 
as the basis for writing website-specific roles. Therefore the role is written
 
in quite generic way, allowing the integrator to write his/her own logic for
 
deploying the necessary Python applications/packages, while still reusing a
 
common base and reducing the workload.
 

	
 
The role implements the following:
 

	
 
* Creates a dedicated user/group for running the WSGI application.
 
* Creates a dedicated administrator user for maintaining the website.
 
* Creates a base directory where the website-specific code and data should be
 
  stored at.
 
* Adds nginx to website's group, so nginx could read the necessary files.
 
* Adds website administrator to website's group, so administrator could manage
 
  the code and data.
 
* Installs additional packages required for running the role (as configured).
 
* Sets-up a dedicated Python virtual environment for website.
 
* Install ``futures`` package in Python virtual environment (required for
 
  Gunicorn in combination withg Python 2.7).
 
* Install Gunicorn in Python virtual environment.
 
* Installs additional packages required for running the role in Python virtual
 
  environment (as configured).
 
* Configures systemd to run the website code (using Gunicorn)
 
* Deploys the HTTPS TLS private key and certificate (for website vhost).
 
* Configures nginx to serve the website (static files served directly, requests
 
  passed on to Gunicorn).
 

	
 
The role is implemented with the following layout/logic in mind:
 

	
 
* Website users are named after the ``FQDN`` (fully qualified domain name) of
 
  website, in format of ``web-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal to
 
  ``FQDN`` where dots have been replaced by underscores (for example,
 
  ``web-wiki_example_com``).
 
* Website users are set-up via GECOS field to have their umask set to ``0007``
 
  (in combination with ``pam_umask``).
 
* Administrator users are named after the ``FQDN`` (fully qualified domain name)
 
  of website, in format of ``admin-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal
 
  to ``FQDN`` where dots have been replaced by underscores (for example,
 
  ``admin-cloud_example_com``).
 
* All websites reside within a dedicated sub-directory in ``/var/www``. The
 
  sub-directory name is equal to the ``FQDN`` used for accessing the
 
  website. Owner of the directory is set to be the application administrator,
 
  while group is set to be the website group. Additionally, ``SGID`` bit is set
 
  on the directory. This allows admin, with correct umask, to create necessary
 
  files and directories that should be readable (and eventually writeable) by
 
  the website user (running the WSGI application) without having to become root.
 
* All files placed in the website directory should be either created there
 
  directly, or copied to the directory in order to make sure the ``SGID`` gets
 
  honored. **Do not move the files, the permissions will not be set correctly.**
 
* Within the website directory, Python virtual environment can be found within
 
  the ``virtualenv`` sub-directory. Switching to administrator user via login
 
  shell will automatically activate the virtual environment.
 
* Within the website directory, nginx will expect to find the static files
 
  within the ``htdocs`` sub-directory (this can be symlink too). Locations/aliases
 
  can be configured for static file serving.
 
* Within the website directory, systemd service will expect to find the website
 
  code within the ``code`` sub-directory (this can be symlink too).
 
* nginx communicates with WSGI server over a dedicated Unix socket for each
 
  website.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 
* **web_server**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**additional_nginx_config** (list, optional, ``[]``)
 
  List providing additional Nginx configuration options to include. This can be
 
  useful for specifying things like error pages. Options are applied inside of a
 
  **server** context of Nginx configuration file.
 

	
 
  Each item is a dictionary with the following options describing the extra
 
  configuration option:
 

	
 
  **comment** (string, mandatory)
 
    Comment describing the configuration option.
 

	
 
  **value** (string, mandatory)
 
    Configuration option.
 

	
 
**admin_uid** (integer, optional, ``whatever OS picks``)
 
  UID of the dedicated website administrator user. The user will be member of
 
  website group.
 

	
 
**enforce_https** (boolean, optional, ``True``)
 
  Specify if HTTPS should be enforced for the website or not. If enforced,
 
  clients connecting via plaintext will be redirected to HTTPS, and clients will
 
  be served with ``Strict-Transport-Security`` header with value of
 
  ``max-age=31536000; includeSubDomains``.
 

	
 
**environment_indicator** (dictionary, optional, ``null``)
 
  Specify configuration for including environment indicator on all HTML
 
  pages. Indicator is a simple strip at bottom of a page with custom background
 
  colour, text colour, and text.
 

	
 
  Specifying environment indicator is useful for avoiding mistakes when testing
 
  by having better visibility what environment you are in
 
  (production/staging/test).
 

	
 
  The following keys need to be specified:
 

	
 
  **background_colour** (string, mandatory)
 
    Background colour to use for the strip at bottom. This should be value
 
    compatible with CSS ``background-color`` attribute.
 

	
 
  **text_colour** (string, mandatory
 
    Text colour to use for the strip at bottom. This should be value compatible
 
    with CSS ``color`` attribute.
 

	
 
  **text** (string, mandatory)
 
    Text to show in show in the strip at bottom.
 

	
 
**environment_variables** (dict, optional, ``{}``)
 
  Specify additional environment variables that should be set for running the
 
  service. Environment variables will be set in both the systemd service and for
 
  the application's administrator user (when logged in as one).
 

	
 
**fqdn** (string, mandatory)
 
  Fully-qualified domain name where the website is reachable. This value is used
 
  for calculating the user/group name for dedicated website user, as well as
 
  home directory of the website user (where data/code should be stored at).
 

	
 
**futures_version** (string, optional, ``3.0.5``)
 
  Version of ``futures`` package to deploy in virtual environment. Required by
 
  Gunicorn when using Python 2.7. Default version is tested with the test site.
 

	
 
**gunicorn_version** (string, optional, ``19.6.0``)
 
  Version of Gunicorn to deploy in virtual environment for running the WSGI
 
  application. Default version is tested with the test site.
 

	
 
**https_tls_certificate** (string, optional, ``{{ lookup('file', tls_certificate_dir + '/' + fqdn + '_https.pem') }}``)
 
  X.509 certificate used for TLS for HTTPS service. The file will be stored in
 
  directory ``/etc/ssl/certs/`` under name ``{{ fqdn }}_https.pem``.
 

	
 
**https_tls_key** (string, optional, ``{{ lookup('file', tls_private_key_dir + '/' + fqdn + '_https.key') }}``)
 
  Private key used for TLS for HTTPS service. The file will be stored in
 
  directory ``/etc/ssl/private/`` under name ``{{ fqdn }}_https.key``.
 

	
 
**packages** (list, optional, ``[]``)
 
  A list of additional packages to install for this particular WSGI
 
  website. This is usually going to be development libraries for building Python
 
  packages.
 

	
 
**proxy_headers** (dictionary, optional, ``{}``)
 
  Additional headers to set when proxying request to Gunicorn. Keys are header
 
  names, values are header values. Both should be compatible with Nginx
 
  ``proxy_set_header``. If you need to provide an empty value, use quotes (don't
 
  forget to surround them by another set of quotes for YAML syntax, for example
 
  ``"\"\""`` or ``'""'``).
 

	
 
**rewrites** (list, optional, ``[]``)
 
  A list of rewrite rules that are applied to incoming requests. Each element of
 
  the list should be a string value compatible with the format of ``nginx``
 
  option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well
 
  as trailing semi-colon (``;``).
 

	
 
**static_locations** (list, optional, ``[]``)
 
  List of locations that should be treated as static-only, and not processed by
 
  the WSGI application at all. This is normally used for designating serving of
 
  static/media files by Nginx (for example, in case of Django projects for
 
  ``/static/`` and ``/media/``).
 

	
 
**uid** (integer, optional, ``whatever OS picks``)
 
  UID/GID (they are set-up to be the same) of the dedicated website
 
  user/group.
 

	
 
**use_paste** (boolean, optional, ``False``)
 
  Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a
 
  filename of a Python Paste ``ini`` file instead of WSGI application.
 

	
 
**virtuaelnv_packages** (list, optional, ``[]``)
 
  A list of additional packages to install for this particular WSGI appliction
 
  in its virtual environment using ``pip``.
 

	
 
**website_mail_recipients** (string, optional, ``root``)
 
  Space-separated list of e-mails or local users to which the mails, sent to
 
  either the website admin or website user, should be forwarded to. Forwarding
 
  is configured via ``~/.forward`` configuration file.
 

	
 
**wsgi_application** (string, mandatory)
 
  WSGI application that should be started by Gunicorn. The format should be
 
  conformant to what the ``gunicorn`` command-line tool accepts. If the
 
  ``use_paste`` option is enabled, the value should be equal to filename of the
 
  Python Paste ini file, located in the ``code`` sub-directory. It should be
 
  noted that in either case the value should be specsified relative to the
 
  ``code`` sub-directory. I.e. don't use full paths.
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up a (base) WSGI website (for
 
running a bare Django project):
 

	
 
.. code-block:: yaml
 

	
 
    - role: wsgi_website
 
      fqdn: django.example.com
 
      static_locations:
 
        - /static
 
        - /media
 
      uid: 2004
 
      virtualenv_packages:
 
        - django
 
      wsgi_application: django_example_com.wsgi:application
 
      environment_variables:
 
        DJANGO_SETTINGS_MODULE: "django_example_com.settings.production"
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.pem') }}"
 
      futures_version: 3.0.5
 
      gunicorn_version: 19.6.0
 
      additional_nginx_config:
 
        - comment: Use custom page for forbidden files.
 
          value: error_page 403 /static/403.html;
 
        - comment: Use custom page for non-existing locations/files.
 
          value: error_page 404 /static/404.html;
 
      website_mail_recipients: "root john.doe@example.com"
 
      environment_indicator:
 
        background_colour: "green"
 
        text_colour: "black"
 
        text: "TEST ENVIRONMENT"
 
      proxy_headers:
 
        Accept-Encoding: '""'
 

	
 

	
 
Database Server
 
---------------
 

	
 
The ``database_server`` role can be used for setting-up a MariaDB database
 
server on destination machine.
 

	
 
The role implements the following:
 

	
 
* Installs MariaDB server and client.
 
* Configures MariaDB server and client to use *UTF-8* encoding by default.
 
* Sets password for the database root user.
 
* Deploys MariaDB client configuration in location ``/root/.my.cnf`` that
 
  contains username and password for the root database user.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**db_root_password** (string, mandatory)
 
  Password for the *root* database user.
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up the database server:
 

	
 
.. code-block:: yaml
 

	
 
   ---
 

	
 
   db_root_password: root
 

	
 

	
 
Database
 
--------
 

	
 
The ``database`` role can be used for creating a MariaDB database and
 
accompanying user on destination machine.
 

	
 
The role implements the following:
 

	
 
* Creates MariaDB database.
 
* Creates a dedicated user capable of performing any operation on the created
 
  database. Username is set to be same as the name of database.
 
* Sets-up pre-backup task that creates database dump in location
 
  ``/srv/backup/mariadb/{{ db_name }}.sql``.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **database_server**
 
* **backup_client**
 

	
 

	
 
Backups
 
~~~~~~~
 

	
 
If the backup for this role has been enabled, the following paths are backed-up:
 

	
 
**/srv/backup/maraidb/{{ db_name }}.sql**
 
  Dump of the database. Database dump is created every day at 01:45 in the
 
  morning.
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**db_name** (string, mandatory)
 
  Name of the database that should be created.
 

	
 
**db_password** (string, mandatory)
 
  Password for the database user.
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for creating a single database (for some
 
website):
 

	
 
.. code-block:: yaml
 

	
 
  - role: database
 
    db_name: phpinfo_example_com
 
    db_password: phpinfo_example_com
 

	
 

	
 
Backup Server
 
-------------
 

	
 
The ``backup_server`` role can be used for setting-up a server to act as backup
 
storage for the backup clients. Storage is made available to the clients
 
exclusively via SFTP on a dedicated port and dedicated OpenSSH server
 
instance. This instance is specifically configured and tailored for this
 
purpose.
 

	
 
The role is primarily aimed for use with `Duplicity
 
<http://duplicity.nongnu.org/>`_, but should be also usable for generic SFTP
 
uploads.
 

	
 
The role implements the following:
 

	
 
* Installs backup software (Duplicity, Duply).
 
* Creates a dedicated directory structure for backups with the following structure:
 

	
 
  * ``/srv/backups/`` - main directory under which all the backups reside.
 
  * ``/srv/backups/SERVER_NAME/`` - home directory for the backup user, name
 
    after the server. Backup users are confined to their respective home
 
    directory via chroot. Backup users can't write to their own home directory,
 
    though.
 
  * ``/srv/backups/SERVER_NAME/duplicity/`` - directory where the Duplicity
 
    backups are stored at. This directory is writable by the respective backup
 
    user.
 
  * ``SERVER_NAME/.ssh/`` - directory where authorized keys are stored. Backup
 
    user is not allowed to make modifications to this directory and files
 
    contained within (i.e. backup users can't add more keys to the
 
    ``authorized_keys`` file).
 
* Creates dedicated operating system users for backup clients. These users will
 
  be made members of the ``backup`` group as well (as an additional group).
 
* Sets-up ``authorized_keys`` for the backup clients.
 
* Makes sure the backup users can't log-in via regular OpenSSH server instance.
 
* Sets-up dedicated OpenSSH server instances to be used exclusively by backup
 
  clients. The instance listens on TCP port ``2222``.
 
* Updates firewall to allow incoming TCP connections to port
 
  ``2222``. Connections are allowed only from the configured IP addresses
 
  associated with backup clients.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**backup_clients** (list, optional)
 
  List of backup clients that are connecting to the backup server. This is
 
  usually done on a per-server basis. Each item in the list is a dictionary
 
  describing the backup client. The following keys are available:
 

	
 
  **server** (string, mandatory)
 
    Name of the server that is backed up. It is highly recommended to use
 
    server's FQDN for this purpose. The dedicated operating system user created
 
    will have the name of format ``bak-ESCAPED_SERVER_NAME``, where
 
    ``ESCAPED_SERVER_NAME`` is calculated by taking the passed-in server name
 
    and replacing all dots (``.``) with undescores (``_``). For example,
 
    ``web.example.com`` will be turned into ``bak-web_example_com``.
 

	
 
  **uid** (integer, optional, ``whatever OS picks``)
 
    Uid for the operating system user. User's default group will have a GID
 
    identical to the user's UID if specified. Otherwise user's default group
 
    will have OS-determined GID.
 

	
 
  **ip** (IPv4 address, mandatory)
 
    IPv4 address from which the backup client server is connecting to the backup
 
    server. Used for introducing stricter firewall rules.
 

	
 
  **public_key** (string, mandatory)
 
    SSH public key used by backup client to connect to the backup server.
 

	
 
**backup_host_ssh_private_keys** (dictionary, mandatory)
 
  Defines host keys used for the dedicated OpenSSH server instance for
 
  backup. Key values that must be provided are: **dsa**, **rsa**, **ed25519**,
 
  and **ecdsa**, with values for each one of them corresponding to a private key
 
  generated using the appropriate algorithm. Keys for this purpose can be easily
 
  created via commands::
 

	
 
    ssh-keygen -f backup_server_dsa_key -N '' -t dsa
 
    ssh-keygen -f backup_server_rsa_key -N '' -t rsa
 
    ssh-keygen -f backup_server_ed25519_key -N '' -t ed25519
 
    ssh-keygen -f backup_server_ecdsa_key -N '' -t ecdsa
 

	
 

	
 
Examples
docs/usage.rst
Show inline comments
 
@@ -1296,384 +1296,386 @@ Before we start, here is a couple of useful pointers regarding the
 
           # uid/gid usage, this is useful. Take note that below value is used
 
           # for both the dedicated uid and gid for application user.
 
           uid: 2000
 
           admin_uid: 3000
 
         # And this role sets up a new dedicated database for our web
 
         # application.
 
         - role: database
 
           # This is both the database name, _and_ name of the database user
 
           # that will be granted full privileges on the database.
 
           db_name: tbg
 
           # This will be the password of our user 'tbg' for accessing the
 
           # database. Take note the user can only login from localhost.
 
           db_password: tbg
 

	
 
3. Now for my favourite part again - creating private keys and certificates!
 
   Why?  Because the ``php_website`` role requires a private key/certificate
 
   pair to be deployed. So... Moving on:
 

	
 
   1. Create new template for ``certtool``:
 

	
 
      :file:`~/mysite/tls/tbg.example.com_https.cfg`
 
      ::
 

	
 
         organization = "Example Inc."
 
         country = SE
 
         cn = "Exampe Inc. Issue Tracker"
 
         expiration_days = 365
 
         dns_name = "tbg.example.com"
 
         tls_www_server
 
         signing_key
 
         encryption_key
 

	
 
   2. Create the keys and certificates for the application::
 

	
 
        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/tbg.example.com_https.key
 
        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/tbg.example.com_https.cfg --load-privkey ~/mysite/tls/tbg.example.com_https.key --outfile ~/mysite/tls/tbg.example.com_https.pem
 

	
 
4. Time to get our hands a bit more dirty... Up until now we didn't have to write
 
   custom tasks, but at this point we need to.
 

	
 
   :file:`~/mysite/roles/tbg/tasks/main.yml`
 
   ::
 

	
 
      ---
 

	
 
      - name: Define TBG version
 
        set_fact: tbg_version=4.1.0
 

	
 
      - name: Download the TBG archive
 
        get_url: url=https://github.com/thebuggenie/thebuggenie/archive/v{{ tbg_version }}.tar.gz
 
                 dest="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"
 
                 sha256sum=0fd0a680ba281adc97d5d2c720e63b995225c99716a36eca6a198b8a5ebf8057
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Download Composer
 
        get_url: url=https://getcomposer.org/download/1.0.0-alpha10/composer.phar
 
                 dest="/usr/local/bin/composer"
 
                 sha256sum=9f2c7d0364bc743bcde9cfe1fe84749e5ac38c46d47cf42966ce499135fd4628
 
                 owner=root group=root mode=755
 

	
 
      - name: Unpack TBG
 
        unarchive: src="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"
 
                   dest="/var/www/tbg.example.com/" copy=no
 
                   creates="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Create TBG cache directory
 
        file: path="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/cache" state=directory mode=2770
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Set-up the necessary write permissions for TBG directories
 
        file: path="{{ item }}" mode=g+w
 
        with_items:
 
           - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/
 
           - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public/
 
           - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/
 

	
 
      - name: Create symbolic link to TBG application
 
        file: src="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public"
 
              path="/var/www/tbg.example.com/htdocs"
 
              state=link
 
              owner="admin-tbg_example_com" group="web-tbg_example_com" mode=2750
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Install TBG dependencies
 
        composer: command=install working_dir="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Deploy database configuration file for TBG
 
        copy: src="b2db.yml" dest="/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"
 
              mode=640 owner=admin-tbg_example_com group=web-tbg_example_com
 

	
 
      - name: Deploy expect script for installing TBG
 
        copy: src="tbg_expect_install" dest="/var/www/tbg.example.com/tbg_expect_install" mode=750
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Run TBG installer via expect script
 
        command: /var/www/tbg.example.com/tbg_expect_install
 
                 chdir=/var/www/tbg.example.com/thebuggenie-{{ tbg_version }} 
 
                 creates=/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
5. Set-up the files that are deployed by our role.
 

	
 
   :file:`~/mysite/roles/tbg/files/b2db.yml`
 
   ::
 

	
 
      b2db:
 
          username: "tbg"
 
          password: "tbg"
 
          dsn: "mysql:host=localhost;dbname=tbg"
 
          tableprefix: ''
 
          cacheclass: '\thebuggenie\core\framework\Cache'
 

	
 
   :file:`~/mysite/roles/tbg/files/tbg_expect_install`
 
   ::
 

	
 
      #!/usr/bin/expect
 

	
 
      spawn ./tbg_cli install --accept_license=yes --url_subdir=/ --use_existing_db_info=yes --enable_all_modules=yes --setup_htaccess=yes
 

	
 
      expect "Press ENTER to continue with the installation: "
 
      send "\r"
 
      expect "Press ENTER to continue: "
 
      send "\r"
 
      interact
 

	
 
6. And... Let's add the new role to our web server.
 

	
 
   :file:`~/mysite/playbooks/web.yml`
 
   ::
 

	
 
      ---
 
      - hosts: web
 
        remote_user: ansible
 
        become: yes
 
        roles:
 
          - common
 
          - ldap_client
 
          - mail_forwarder
 
          - web_server
 
          - database_server
 
          - tbg
 

	
 
7. Apply the changes::
 

	
 
     workon mysite && ansible-playbook playbooks/site.yml
 

	
 
8. At this point *The Bug Genie* has been installed, and you should be able to
 
   open the URL https://tbg.example.com/ (if you open http://tbg.example.com/ ,
 
   you will be redirected to the HTTPS URL) and log-in into *The Bug Genie*
 
   with username ``administrator`` and password ``admin``.
 

	
 

	
 
Deploying a WSGI application (Django Wiki)
 
------------------------------------------
 

	
 
Next thing up will be to deploy a WSGI Python application.
 

	
 
Similar to the PHP application deployment, we will use a couple of roles to make
 
it easier to deploy it in a standardised manner, and we will not have any kind
 
of parameters for configuring the role to keep things simple.
 

	
 
Most of the notes on how a ``php_website`` role is deployed also stand for the
 
``wsgi_website`` role, but we will reiterate and clarify them a bit just to be
 
on the safe side:
 

	
 
* The role is designed to execute every application via dedicated user and
 
  group. The user/group name is automatically derived from the FQDN of website,
 
  for example ``web-wiki_example_com``.
 
* While running the application, application user's umask is set to ``0007``
 
  (letting the administrator user be able to manage any files created while the
 
  application is running).
 
* An administrative user is created as well, and this user should be used when
 
  running maintenance and installation commands. Similar to application user,
 
  the name is also derived from the FQDN of website, for example
 
  ``admin-wiki_example_com``. Administrative user does not have a dedicated
 
  group, and instead belongs to same group as the application user. As
 
  convenience, whenever you switch to this user the Python virtual environment
 
  will be automatically activated for you.
 
* WSGI applications are executed via *Gunicorn*. The WSGI server listens on a
 
  Unix socket, making the socket accessible by *Nginx*.
 
* If you ever need to set some environment variables, this can easily be done
 
  via the ``environment_variables`` role parameter. This particular example does
 
  not set any, though.
 
* You can also specify headers to be passed on via Nginx ``proxy_set_header``
 
  directive to Gunicorn running the application.
 
* Mails deliverd to local admin/application users are forwarded to ``root``
 
  account instead (this can be configured via ``website_mail_recipients`` role
 
  parameter.
 
* If you ever find yourself mixing-up test and production websites, have a look
 
  at ``environment_indicator`` role parameter. It lets you insert small strip at
 
  bottom of each HTML page automatically.
 
* Static content is served directly by *Nginx*.
 
* Each web application gets distinct sub-directory under ``/var/www``, named
 
  after the FQDN. All sub-directories created under there are created with
 
  ``2750`` permissions, with ownership set to admin user, and group set to the
 
  application's group. In other words, all directories will have ``SGID`` bit
 
  set-up, allowing you to create files/directories that will have their group
 
  automatically set to the group of the parent directory.
 
* Each WSGI website gets a dedicated virtual environment, stored in the
 
  sub-directory ``virtualenv`` of the website directory, for example
 
  ``/var/www/wiki.example.com/virtualenv``.
 
* Static files are served from sub-directory ``htdocs`` in the website
 
  directory, for example ``/var/www/wiki.example.com/htdocs/``.
 
* The base directory where your website/application code should be at is
 
  expected to be in sub-directory ``code`` in the website directory, for example
 
  ``/var/www/wiki.example.com/code/``.
 
* Combination of admin user membership in application group, ``SGID``
 
  permission, and the way ownership of sub-directories is set-up usually means
 
  that the administrator will be capable of managing application files, and
 
  application can be granted write permissions to a *minimum* of necessary
 
  files.
 

	
 
  .. warning::
 
     Just keep in mind that some file-management commands, like ``mv``, do *not*
 
     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you
 
     deploy new files to the directory instead (don't simply move them from your
 
     home).
 

	
 
1. Set-up the necessary directories first::
 

	
 
     mkdir -p ~/mysite/roles/wiki/{tasks,meta,files}/
 

	
 
2. Set-up some role dependencies, reusing the common role infrastructure.
 

	
 
   :file:`~/mysite/roles/wiki/meta/main.yml`
 
   ::
 

	
 
      ---
 

	
 
      dependencies:
 
         - role: wsgi_website
 
           fqdn: wiki.example.com
 
           # In many cases you need to have some development packages available
 
           # in order to build Python packages installed via pip
 
           packages:
 
              - build-essential
 
              - python-dev
 
              - libjpeg-dev
 
              - libzip-dev
 
              - libtiff-dev
 
              - libfreetype6-dev
 
              - liblcms2-dev
 
              - libwebp-dev
 
              - libopenjpeg-dev
 
              - libmariadb-client-lgpl-dev
 
              - libmariadb-client-lgpl-dev-compat
 
           # Here we specify that anything accessing our website with "/static/"
 
           # URL should be treated as request to a static file, to be served
 
           # directly by Nginx instead of the WSGI server.
 
           static_locations:
 
              - /static/
 
           # Again, not mandatory, but it is good to have some sort of policy
 
           # for assigning UIDs.
 
           uid: 2001
 
           admin_uid: 3001
 
           # These are additional packages that should be installed in the
 
           # virtual environment.
 
           virtualenv_packages:
 
             - pillow
 
             - django==1.8.13
 
             - wiki
 
             - MySQL-python
 
           # This is the name of the WSGI application to
 
           # serve. wiki_example_com.wsgi will be the Python "module" that is
 
           # accesed, while application is the object instantiated within it (the
 
           # application itself). The module is referenced relative to the code
 
           # directory (in our case /var/www/wiki.example.com/code/).
 
           wsgi_application: wiki_example_com.wsgi:application
 
         - role: database
 
           db_name: wiki
 
           db_password: wiki
 

	
 
3. Let's create a dedicated private key/certificate pair for the wiki website:
 

	
 
   1. Create new template for ``certtool``:
 

	
 
      :file:`~/mysite/tls/wiki.example.com_https.cfg`
 
      ::
 

	
 
         organization = "Example Inc."
 
         country = SE
 
         cn = "Exampe Inc. Wiki"
 
         expiration_days = 365
 
         dns_name = "wiki.example.com"
 
         tls_www_server
 
         signing_key
 
         encryption_key
 

	
 
   2. Create the keys and certificates for the application::
 

	
 
        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/wiki.example.com_https.key
 
        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/wiki.example.com_https.cfg --load-privkey ~/mysite/tls/wiki.example.com_https.key --outfile ~/mysite/tls/wiki.example.com_https.pem
 

	
 
4. At this point we have exhausted what we can do with the built-in roles. Time
 
   to add some custom tasks.
 

	
 
   :file:`~/mysite/roles/wiki/tasks/main.yml`
 
   ::
 

	
 
      ---
 

	
 
      - name: Create Django project directory
 
        file: dest="/var/www/wiki.example.com/code" state=directory
 
              owner=admin-wiki_example_com group=web-wiki_example_com
 
              mode=2750
 

	
 
      - name: Start Django project for the Wiki website
 
        command: /var/www/wiki.example.com/virtualenv/bin/exec django-admin.py startproject wiki_example_com /var/www/wiki.example.com/code
 
                 chdir=/var/www/wiki.example.com
 
                 creates=/var/www/wiki.example.com/code/wiki_example_com
 
        become: yes
 
        become_user: admin-wiki_example_com
 

	
 
      - name: Deploy settings for wiki website
 
        copy: src="{{ item }}" dest="/var/www/wiki.example.com/code/wiki_example_com/{{ item }}"
 
              mode=640 owner=admin group=web-wiki_example_com
 
        with_items:
 
           - settings.py
 
           - urls.py
 
        notify:
 
           - Restart website wiki.example.com
 

	
 
      - name: Deploy project database and deploy static files
 
        django_manage: command="{{ item }}"
 
                       app_path="/var/www/wiki.example.com/code/"
 
                       virtualenv="/var/www/wiki.example.com/virtualenv/"
 
        become: yes
 
        become_user: admin-wiki_example_com
 
        with_items:
 
           - syncdb
 
           - migrate
 
           - collectstatic
 

	
 
      - name: Deploy the superadmin creation script
 
        copy: src="create_superadmin.py" dest="/var/www/wiki.example.com/code/create_superadmin.py"
 
              owner=admin-wiki_example_com group=web-wiki_example_com mode=750
 

	
 
      - name: Create initial superuser
 
        command: /var/www/wiki.example.com/virtualenv/bin/exec ./create_superadmin.py
 
                 chdir=/var/www/wiki.example.com/code/
 
        become: yes
 
        become_user: admin-wiki_example_com
 
        register: wiki_superuser
 
        changed_when: wiki_superuser.stdout == "Created superuser."
 

	
 
5. There is a couple of files that we are deploying through the above
 
   tasks. Let's create them as well.
 

	
 
   :file:`~/mysite/roles/wiki/files/settings.py`
 
   ::
 

	
 
      """
 
      Django settings for wiki_example_com project.
 

	
 
      For more information on this file, see
 
      https://docs.djangoproject.com/en/1.6/topics/settings/
 

	
 
      For the full list of settings and their values, see
 
      https://docs.djangoproject.com/en/1.6/ref/settings/
 
      """
 

	
 
      # Build paths inside the project like this: os.path.join(BASE_DIR, ...)
 
      import os
 
      BASE_DIR = os.path.dirname(os.path.dirname(__file__))
 

	
 

	
 
      # Quick-start development settings - unsuitable for production
 
      # See https://docs.djangoproject.com/en/1.6/howto/deployment/checklist/
 

	
 
      # SECURITY WARNING: keep the secret key used in production secret!
 
      SECRET_KEY = 'l^q+t$7h$ebls)v34+w9m9v4$n+^(9guxqntu&#cc4m&lfd-6_'
 

	
 
      # SECURITY WARNING: don't run with debug turned on in production!
 
      DEBUG = False
 

	
 
      TEMPLATE_DEBUG = False
 

	
roles/wsgi_website/defaults/main.yml
Show inline comments
 
---
 

	
 
additional_nginx_config: {}
 
enforce_https: True
 
packages: []
 
rewrites: []
 
static_locations: []
 
use_paste: False
 
virtualenv_packages: []
 
environment_variables: {}
 
admin: "web-{{ fqdn | replace('.', '_') }}"
 
https_tls_certificate: "{{ lookup('file', tls_certificate_dir + '/' + fqdn + '_https.pem') }}"
 
https_tls_key: "{{ lookup('file', tls_private_key_dir + '/' + fqdn + '_https.key') }}"
 
gunicorn_version: "19.6.0"
 
futures_version: "3.0.5"
 
website_mail_recipients: "root"
 
environment_indicator: null
 
\ No newline at end of file
 
environment_indicator: null
 
proxy_headers: {}
roles/wsgi_website/templates/nginx_site.j2
Show inline comments
 
{% if enforce_https -%}
 
server {
 
    # HTTP (plaintext) configuration.
 
    listen 80;
 
    server_name {{ fqdn }};
 

	
 
    # Redirect plaintext connections to HTTPS
 
    return 301 https://$host$request_uri;
 
}
 

	
 
{% endif -%}
 
server {
 
    # Base settings.
 
    root {{ home }}/htdocs/;
 
    server_name {{ fqdn }};
 
{% if not enforce_https %}
 

	
 
    # HTTP (plaintext) configuration.
 
    listen 80;
 

	
 
{% endif %}
 
    # HTTPS (TLS) configuration.
 
    listen 443 ssl;
 
    listen [::]:443 ssl;
 
    ssl_certificate_key /etc/ssl/private/{{ fqdn }}_https.key;
 
    ssl_certificate /etc/ssl/certs/{{ fqdn }}_https.pem;
 

	
 
{% if default_enforce_https -%}
 
    # Set-up HSTS header for preventing downgrades for users that visited the
 
    # site via HTTPS at least once.
 
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
 
{% endif -%}
 

	
 
    {% for config in additional_nginx_config -%}
 
    # {{ config.comment }}
 
    {{ config.value }}
 
    {% endfor -%}
 

	
 
    {% if rewrites -%}
 
    # Site rewrites.
 
    {% for rewrite in rewrites -%}
 
    rewrite {{ rewrite }};
 
    {% endfor -%}
 
    {% endif %}
 

	
 
    {% if static_locations -%}
 
    # Static locations
 
    {% for location in static_locations -%}
 
    location {{ location }} {
 
        try_files $uri $uri/ =404;
 
    }
 
    {% endfor -%}
 
    {% endif %}
 

	
 
    # Pass remaining requests to the WSGI server.
 
    location / {
 
        try_files $uri @proxy_to_app;
 
    }
 

	
 
    location @proxy_to_app {
 
        proxy_set_header X-Forwarded-Proto $scheme;
 
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
        proxy_set_header Host $http_host;
 
        proxy_redirect off;
 

	
 
    {% for header, value in proxy_headers.iteritems() -%}
 
    proxy_set_header {{ header }} {{ value }};
 
    {% endfor -%}
 

	
 
        proxy_pass http://unix:/run/wsgi/{{ fqdn }}.sock;
 
    }
 

	
 
    {% if environment_indicator -%}
 
    # Show environment indicator on HTML pages.
 
    sub_filter_types text/html;
 
    sub_filter_once on;
 
    sub_filter "</body>" "<div id='website-environment' style='background-color: {{ environment_indicator.background_colour }}; width: 100%; text-align: center; position: fixed; bottom: 5px; color: {{ environment_indicator.text_colour }}; font-weight: bold;'>{{ environment_indicator.text }}</div></body>";
 
    {% endif -%}
 

	
 
    access_log /var/log/nginx/{{ fqdn }}-access.log;
 
    error_log /var/log/nginx/{{ fqdn }}-error.log;
 
}
testsite/group_vars/web.yml
Show inline comments
 
---
 

	
 
local_mail_aliases:
 
  root: "root john.doe@{{ testsite_domain }}"
 

	
 
smtp_relay_host: mail.{{ testsite_domain }}
 

	
 
smtp_relay_truststore: "{{ lookup('file', inventory_dir + '/tls/ca.pem') }}"
 

	
 
default_https_tls_key: "{{ lookup('file', inventory_dir + '/tls/web.' + testsite_domain + '_https.key') }}"
 
default_https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/web.' + testsite_domain + '_https.pem') }}"
 

	
 
web_default_title: "Welcome to Example Inc."
 
web_default_message: "You are attempting to access the web server using a wrong name or an IP address. Please check your URL."
 

	
 
db_root_password: "root"
 

	
 
website_mail_recipients: "john.doe@example.com"
 

	
 
environment_indicator:
 
  background_colour: "purple"
 
  text_colour: "white"
 
  text: "Majic Ansible Roles Test Site"
 
\ No newline at end of file
 
  text: "Majic Ansible Roles Test Site"
 

	
 
proxy_headers:
 
  Accept-Encoding: '"gzip"'
testsite/playbooks/roles/wsgihello/files/hello.wsgi
Show inline comments
 
#!/usr/bin/env python
 

	
 
import os
 

	
 
def application(environ, start_response):
 
    status = '200 OK'
 

	
 
    template = """<!DOCTYPE html>
 
<html lang="en">
 
  <head>
 
    <meta charset="utf-8">
 
    <title>{title}</title>
 
  </head>
 
  <body>
 
    <h1>Hello, world!</h1>
 
    <p>I am website {title}</p>
 
    <p>Accept-Encoding header was set to {acceptencoding}</p>
 
  </body>
 
</html>
 
"""
 
    output = template.format(title=os.environ.get("WEBSITE_NAME", "that nobody set a name for :("))
 
    output = template.format(title=os.environ.get("WEBSITE_NAME", "that nobody set a name for :("),
 
                             acceptencoding=environ.get("HTTP_ACCEPT_ENCODING"))
 

	
 
    response_headers = [('Content-type', 'text/html'),
 
                        ('Content-Length', str(len(output)))]
 
    start_response(status, response_headers)
 

	
 
    return [output]
0 comments (0 inline, 0 general)