Of course, no database has been created for either of the web applications,

   but we will get to that one later (there is a dedicated ``database`` role

   which can be combined with web app roles for this purpose).

Deploying a PHP web application (The Bug Genie)

-----------------------------------------------

We have some basic infrastructure up and running on our web server, so

now we can move on to setting-up a PHP web application on it. As

mentioned before, we will take *The Bug Genie* as an example.

For this we will create a local role in our site to take care of it. This role

will in turn utilise two roles coming from *Majic Ansible Roles* that will make

our life (a little) easier.

To make the example a bit simpler, no parameters will be introduced

for this role (not even the password for database, we'll hard-code

everything).

Before we start, here is a couple of useful pointers regarding the

``php_website`` role we'll be using for the PHP part:

* The role is designed to execute every application via dedicated user and

  group. The user/group name is automatically derived from the FQDN of website,

  for example ``web-tbg_example_com``.

* While running the application, application user's umask is set to ``0007``

  (letting the administrator user be able to manage any files created while the

  application is running).

* An administrative user is created as well, and this user should be used when

  running maintenance and installation commands. Similar to application user,

  the name is also derived from the FQDN of website, for example

  ``admin-tbg_example_com``. Administrative user does not have a dedicated

  group, and instead belongs to same group as the application user.

* PHP applications are executed via FastCGI, using *PHP-FPM*.

* If you ever need to set some additional PHP FPM settings, this can easily be

  done via the ``additional_fpm_config`` role parameter. This particular example

  does not set any, though.

* Mails delivered to local admin/application users are forwarded to ``root``

  account instead (this can be configured via ``website_mail_recipients`` role

  parameter.

* If you ever find yourself mixing-up test and production websites,

  have a look at ``environment_indicator`` role parameter. It lets you

  insert small strip with environment information at bottom of each

  HTML page served by the web server.

* Static content (non-PHP) is served directly by *Nginx*.

* Each web application gets distinct sub-directory under ``/var/www``, named

  after the FQDN. All sub-directories created under there are created with

  ``02750`` permissions, with ownership set to admin user, and group set to the

  application's group. In other words, all directories will have ``SGID`` bit

  set, allowing you to create files/directories that will have their group

  automatically set to the group of the parent directory.

* Files are served (both by *Nginx* and *PHP-FPM*) from sub-directory called

  ``htdocs`` (located in website directory). For example

  ``/var/www/tbg.example.com/htdocs/``. Normally, this can be a symlink to some

  other sub-directory within the website directory (useful for having multiple

  versions for easier downgrades etc).

* Combination of admin user membership in application group, ``SGID``

  permission, and the way ownership of sub-directories is set-up usually means

  that the administrator will be capable of managing application files, and

  application can be granted write permissions to a *minimum* of necessary

  files.

  .. warning::

     Just keep in mind that some file-management commands, like ``mv``, do *not*

     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you

     deploy new files to the directory instead (don't simply move them from your

     home directory).

1. Start-off with creating the necessary directories for the new role::

     mkdir -p ~/mysite/roles/tbg/{tasks,meta,files}/

2. Let's set-up role dependencies, reusing some common roles to make our life

   easier.

   :file:`~/mysite/roles/tbg/meta/main.yml`

   ::

      ---

      dependencies:

        # Ok, so this role helps us set-up Nginx virtual host for serving our

        # app.

        - role: php_website

          # Our virtual host will for PHP website will respond to this name.

          fqdn: tbg.example.com

          # TLS key and certificate to use for the virtual host.

          https_tls_certificate: "{{ lookup('file', '~/mysite/tls/tbg.example.com_https.pem') }}"

          https_tls_key: "{{ lookup('file', '~/mysite/tls/tbg.example.com_https.key') }}"

          # Some additional packages are required in order to deploy and use TBG.

          packages:

            - php-gd

            - php-curl

            - php-mbstring

            - php-xml

            - git

            - php-mysql

            - php-apcu

            - php-zip

          # Set-up URL rewriting. This is based on public/.htaccess file from

          # TBG.

          php_rewrite_urls:

            - ^(.*)$ /index.php?url=$1

          # We don't necessarily need this, but in case you have a policy on

          # uid/gid usage, this is useful. Take note that below value is used

          # for both the dedicated uid and gid for application user.

          uid: 2000

          admin_uid: 3000

        # And this role sets up a new dedicated database for our web

        # application.

        - role: database

          # This is both the database name, _and_ name of the database user

          # that will be granted full privileges on the database.

          db_name: tbg

          # This will be the password of our user 'tbg' for accessing the

          # database. Take note the user can only login from localhost.

          db_password: tbg

3. Now for my favourite part again - creating private keys and certificates!

   Why?  Because the ``php_website`` role requires a private key/certificate

   pair to be deployed. So... Moving on:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/tbg.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

         cn = "Exampe Inc. Issue Tracker"

         expiration_days = 365

         dns_name = "tbg.example.com"

         tls_www_server

         signing_key

         encryption_key

   2. Create the keys and certificates for the application::

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/tbg.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/tbg.example.com_https.cfg --load-privkey ~/mysite/tls/tbg.example.com_https.key --outfile ~/mysite/tls/tbg.example.com_https.pem

4. Time to get our hands a bit more dirty... Up until now we didn't have to write

   custom tasks, but at this point we need to.

   :file:`~/mysite/roles/tbg/tasks/main.yml`

   ::

      ---

      - name: Define TBG version

        set_fact:

          tbg_version: "4.3.1"

          tbg_archive_checksum: "45de72b1ef82142ad46686577d593375ba370156df4367d17386b4e26a37f342"

      - name: Download the TBG archive

        get_url:

          url: "https://github.com/thebuggenie/thebuggenie/archive/v{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          sha256sum: "{{ tbg_archive_checksum }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Download Composer

        get_url:

          url: "https://getcomposer.org/download/1.10.19/composer.phar"

          dest: "/usr/local/bin/composer"

          sha256sum: "688bf8f868643b420dded326614fcdf969572ac8ad7fbbb92c28a631157d39e8"

          owner: root

          group: root

          mode: 0755

      - name: Unpack TBG

        unarchive:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/"

          copy: no

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Allow use of lib-pcre version 10 (since PHP is built against it in Debian Buster)

        lineinfile:

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/{{ item }}"

          state: present

          regexp: '.*"lib-pcre".*'

          line: '        "lib-pcre": ">=8.0",'

        with_items:

          - "composer.json"

          - "composer.lock"

      - name: Create TBG cache directory

        file:

          path: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/cache"

          state: directory

          mode: 02770

        become: yes

        become_user: admin-tbg_example_com

      - name: Set-up the necessary write permissions for TBG directories

        file:

          path: "{{ item }}"

          mode: g+w

        with_items:

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/

      - name: Create symbolic link to TBG application

        file:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public"

          path: "/var/www/tbg.example.com/htdocs"

          state: link

          owner: "admin-tbg_example_com"

          group: "web-tbg_example_com"

        become: yes

        become_user: admin-tbg_example_com

      - name: Install TBG dependencies

        composer:

          command: install

          working_dir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Deploy database configuration file for TBG

        copy:

          src: "b2db.yml"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"

          mode: 0640

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Install pexpect package

        apt:

          name: python3-pexpect

          state: present

      - name: Deploy expect script for installing TBG

        copy:

          src: "tbg_expect_install"

          dest: "/var/www/tbg.example.com/tbg_expect_install"

          mode: 0750

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Run TBG installer via expect script

        command: /var/www/tbg.example.com/tbg_expect_install

        args:

          chdir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed"

        become: yes

        become_user: admin-tbg_example_com

5. Set-up the files that are deployed by our role.

   :file:`~/mysite/roles/tbg/files/b2db.yml`

   ::

      b2db:

          username: "tbg"

          password: "tbg"

          dsn: "mysql:host=localhost;dbname=tbg"

          tableprefix: ''

          cacheclass: '\thebuggenie\core\framework\Cache'

   :file:`~/mysite/roles/tbg/files/tbg_expect_install`

   ::

      #!/usr/bin/env python3

      import pexpect

      # Spawn the process.

      install_process = pexpect.spawnu('./tbg_cli', args = ["install",

                                                                        "--accept_license=yes",

                                                                        "--url_subdir=/",

                                                                        "--use_existing_db_info=yes",

                                                                        "--enable_all_modules=no",

                                                                        "--setup_htaccess=yes"])

      # If we get EOF, we probably already installed application, and ran

      # into error at the end since no patterns matched.

      try:

          # First confirmation.

          install_process.expect(u'Press ENTER to continue with the installation: ', timeout=5)

          install_process.sendline(u'')

          # Second confirmation.

          install_process.expect(u'Press ENTER to continue: ', timeout=5)

          install_process.sendline(u'')

          # Wait for application to finish.

          install_process.expect(pexpect.EOF, timeout=60)

      except pexpect.EOF as e:

          pass

      # Close application.

      install_process.close()

      # Print text output.

      print(install_process.before)

      # Return same exit code like child process.

      exit(install_process.exitstatus)

6. And... Let's add the new role to our web server.

   :file:`~/mysite/playbooks/web.yml`

   ::

      ---

      - hosts: web

        remote_user: ansible

        become: yes

        roles:

          - common

          - ldap_client

          - mail_forwarder

          - web_server

          - database_server

          - tbg

7. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

8. At this point *The Bug Genie* has been installed, and you should be able to

   open the URL https://tbg.example.com/ and log-in into *The Bug Genie*

   with username ``administrator`` and password ``admin``.

Deploying a WSGI application (Django Wiki)

------------------------------------------

Next thing up will be to deploy a WSGI Python application.

Similar to the PHP application deployment, we will use a couple of roles to make

it easier to deploy it in a standardised manner, and we will not have any kind

of parameters for configuring the role to keep things simple.

Most of the notes on how a ``php_website`` role is deployed also stand for the

``wsgi_website`` role, but we will reiterate and clarify them a bit just to be

on the safe side:

* The role is designed to execute every application via dedicated user and

  group. The user/group name is automatically derived from the FQDN of website,

  for example ``web-wiki_example_com``.

* While running the application, application user's umask is set to ``0007``

  (letting the administrator user be able to manage any files created while the

  application is running).

* An administrative user is created as well, and this user should be used when

  running maintenance and installation commands. Similar to application user,

  the name is also derived from the FQDN of website, for example

  ``admin-wiki_example_com``. Administrative user does not have a dedicated

  group, and instead belongs to same group as the application user. As

  convenience, whenever you switch to this user the Python virtual environment

  will be automatically activated for you.

* WSGI applications are executed via *Gunicorn*. The WSGI server listens on a

  Unix socket, making the socket accessible by *Nginx*.

* If you ever need to set some environment variables, this can easily be done

  via the ``environment_variables`` role parameter. This particular example does

  not set any, though.

* You can also specify headers to be passed on via Nginx ``proxy_set_header``

  directive to Gunicorn running the application.

* Mails deliverd to local admin/application users are forwarded to ``root``

  account instead (this can be configured via ``website_mail_recipients`` role

  parameter.

* If you ever find yourself mixing-up test and production websites,

  have a look at ``environment_indicator`` role parameter. It lets you

  insert small strip with environment information at bottom of each

  HTML page served by the web server.

* Static content is served directly by *Nginx*.

* Each web application gets distinct sub-directory under ``/var/www``, named

  after the FQDN. All sub-directories created under there are created with

  ``2750`` permissions, with ownership set to admin user, and group set to the

  application's group. In other words, all directories will have ``SGID`` bit

  set, allowing you to create files/directories that will have their group

  automatically set to the group of the parent directory.

* Each WSGI website gets a dedicated virtual environment, stored in the

  sub-directory ``virtualenv`` of the website directory, for example

      backup_host_ssh_private_keys:

        rsa: "{{ lookup('file', inventory_dir + '/ssh/bak_rsa_key') }}"

        ed25519: "{{ lookup('file', inventory_dir + '/ssh/bak_ed25519_key') }}"

        ecdsa: "{{ lookup('file', inventory_dir + '/ssh/bak_ecdsa_key') }}"

3. Since we have decided to specify the keys above through file lookup, the

   above-listed keys now need to be generated::

     ssh-keygen -f ~/mysite/ssh/bak_rsa_key -N '' -t rsa

     ssh-keygen -f ~/mysite/ssh/bak_ed25519_key -N '' -t ed25519

     ssh-keygen -f ~/mysite/ssh/bak_ecdsa_key -N '' -t ecdsa

Adding backup clients

---------------------

Well, that was all nice and dandy, but it does look like something is

missing... Ah, we haven't really configured any backup clients, right?

Surprisingly enough, specifying backup clients is optional, but that won't get

you far.

Luckily for you, all relevant *Majic Ansible Roles* are *backup-aware*. In other

words, all the roles have been implemented with support for doing back-ups - it

is just that by default this functionality is disabled (since you might be

relying on some other schema to back things up - LVM snapshots or what-not).

All that is needed is to enable the backups for each role, and provide some

extra variables required by the ``backup_client`` role.

For this a set of GnuPG private keys are necessary. These need to be provided as

ASCII-armoured GnuPG-exported files. For simplicity sake, this example documents

use of GnuPG keyring in conjunction with Ansible's ``pipe`` lookup.

So, back to the business:

1. Update the backup server configuration - each client needs to be explicitly

   registered:

   :file:`~/mysite/group_vars/backup.yml`

   ::

      backup_clients:

        - server: comms.example.com

          public_key: "{{ lookup('file', inventory_dir + '/ssh/comms.example.com.pub') }}"

          ip: 10.32.64.19

        - server: www.example.com

          public_key: "{{ lookup('file', inventory_dir + '/ssh/www.example.com.pub') }}"

          ip: 10.32.64.20

        # Ah, this is a bit interesting - we can back-up the backup server

        # itself! Don't worry, though, this is mainly so the logs and home

        # directories are preserved, so it won't take too much space ;)

        - server: bak.example.com

          public_key: "{{ lookup('file', inventory_dir + '/ssh/bak.example.com.pub') }}"

          ip: 127.0.0.1

2. And now to configure backup clients for all servers:

   .. warning::

      By default Ansible's file lookup plugin will strip newlines and

      spaces from the end of the file. This is a problem when

      deploying the RSA ssh keys, since if there is no newline after

      the ``-----END OPENSSH PRIVATE KEY-----`` delimeter, ssh client

      will report error about the format of the key file being

      invalid. Therefore the example below explicitly disables

      stripping newline from the end of the file.

   :file:`~/mysite/group_vars/all.yml`

   ::

      enable_backup: yes

      backup_encryption_key: "{{ lookup('pipe', 'gpg --homedir ~/mysite/gnupg/ --armour --export-secret-keys ' + ansible_fqdn ) }}"

      backup_server: bak.example.com

      backup_server_host_ssh_public_keys:

        - "{{ lookup('file', inventory_dir + '/ssh/bak_rsa_key.pub') }}"

        - "{{ lookup('file', inventory_dir + '/ssh/bak_ed25519_key.pub') }}"

        - "{{ lookup('file', inventory_dir + '/ssh/bak_ecdsa_key.pub') }}"

      backup_ssh_key: "{{ lookup('file', inventory_dir + '/ssh/' + ansible_fqdn, rstrip=False) }}"

3. So, looking at the configuration up there, there is a couple of file lookups

   for getting the variable values, as well as one pipe lookup for fetching the

   encryption keys. For start, let's create the SSH private keys used for client

   log-ins to backup server::

     ssh-keygen -f ~/mysite/ssh/comms.example.com -N ''

     ssh-keygen -f ~/mysite/ssh/www.example.com -N ''

     ssh-keygen -f ~/mysite/ssh/bak.example.com -N ''

4. Next off, a GnuPG keyring needs to be populated with private keys that will

   be used for backup encryption and signing. In total, we need three keys, one

   for each server. The keys should not be encrypted, and they should be named

   after the respective server's FQDN. For simplicity sake, here is a nice

   copy-pastable batch version for doing so:

   .. note:: Key generation requires a lot of entropy. If you are running this

             command on a VM, you may want to ``apt-get install haveged`` to

             speed this up. Please do read up on haveged if deploying to a real

             system, though! Don't trust it blindly!

   ::

     chmod 700 ~/mysite/gnupg

     pkill gpg-agent

     gpg --homedir ~/mysite/gnupg --batch --generate-key << EOF

     Key-Type:RSA

     Key-Length:1024

     Name-Real:comms.example.com

     Expire-Date:0

     %no-protection

     %commit

     Key-Type:RSA

     Key-Length:1024

     Name-Real:www.example.com

     Expire-Date:0

     %no-protection

     %commit

     Key-Type:RSA

     Key-Length:1024

     Name-Real:bak.example.com

     Expire-Date:0

     %no-protection

     %commit

     EOF

5. And... Apply::

     workon mysite && ansible-playbook playbooks/site.yml

6. At this point the backup roles have been set-up everywhere, and the backups

   will be running every day at 02:00 in the morning. Of course, you may want to

   test out a backup yourself immediatelly by running the following command on

   servers::

     duply main backup

   .. note:: For more information on available commands and how to work with

             backup tool, please have look at `Official Duply Pages

             <http://duply.net/>`_.

Adding backup support to custom roles

-------------------------------------

As mentioned before, all of the supplied roles coming with *Majic Ansible Roles*

include backup support. What gets backed-up depends on the role implementation

(see role reference for details). What about backup support for custom roles?

This is something that has to be done by hand. However, it is quite simple to do

so.

Backup integration will be demonstrated with the previously implemented

``tbg`` role.

*The Bug Genie* stores most of its data in database, but thanks to the

``database`` role its backup is already handled for us. As a side-note, just

before every backup run the database is dumped and stored in location

``/srv/backup/tbg.sql``. That file is subsequently backed-up via *Duply* run.

What is not backed-up for us, though, are the files uploaded to *The Bug

Genie*. So let's fix that one.

1. Add the ``backup`` role to list of dependencies. Take note that while the

   ``backup_client`` role deals with basic set-up of backup client and its

   configuration, the ``backup`` role is used to define what should be

   backed-up. It is important to define unique filename for the backup patterns

   file. Take into account that you can use pretty much any globbing pattern

   supported by Duplicity.

   .. warning::

      Make sure the addition is properly aligned in the yaml file to previous

      role dependency definitions.

   :file:`~/mysite/roles/tbg/meta/main.yml`

   .. Small workaround for Sphinx not preserving leading spaces in

      case all lines have the same amount of leading spaces.

   .. code-block:: none

      :name: sphinx_workaround

        - role: backup

          when: enable_backup

          backup_patterns_filename: "tbg"

          backup_patterns:

            - "/var/www/tbg.example.com/files"

2. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

   ::

     duply main backup

4. Verify that the files have been backed-up:

   ::

      duply main list

.. note:: If you wanted to run a script prior to backup run, you would simply

          deploy a shell script with desired content to

          ``/etc/duply/main/pre.d/``. Just make sure the permissions for it are

          ok (it has to be executable by the root user).

Dealing with failures

---------------------

While the roles have been designed to be fairly robust, it should be taken into

account that certain handlers are used to bring the system into consistent

state. These handlers are mostly the ones dealing with service restarts, but

there are also a couple of handlers that take care of transforming certain data

into the required formats, import of files etc.

This means that failure to successfully execute such handlers could result in

inconsistent state on the server. Think of service configuration files being

updated, yet the service itself is not restarted and therefore continues to run

with the old configuration.

Handler execution failure can depend on a couple of things, including the loss

of SSH connectivity to managed machine, or some kind of unusual time-out during

handler execution.

To help handle this situation, Majic Ansible Roles all come with a special way

to invoke the handlers explicitly. Each role will include handlers as tasks,

provided that a special variable (``run_handlers``) is passed in to playbook run. To

make the run shorter, the handlers in such a run are also tagged with

``handlers``. This doubling of environment variable + tagging stems from current

limitations of Ansible (it is not possible to specify that certain task should

be run only if a tag is specified, therefore an additional variable has to be

used).

Handlers alone can be invoked specifically with command similar to::

  ansible-playbook -t handlers -e run_handlers=true playbooks/site.yml

The ``run_handlers`` variable is treated as boolean, and by default it

is not set.

Checking for available package upgrades

---------------------------------------

One of the more annoying chores when you maintain your own infrastructure is

making sure everything is up-to-date. And this has to be done - both in order to

ensure for problem-free experience for users (yourself included), and for making

sure there are no security vulnerabilities that could be exploited by a (random)

adversary.

*Majic Ansible Roles* try to keep you covered on this front as well. As part of

regular deployment, the ``common`` role will deploy and configure ``apticron`` -

a nifty little script that runs on hourly basis and checks if any of your

system-provided packages are outdated.

If ``apticron`` detects an outdated package, it will output this information to

standard output, which will result in the cron daemon sending out an e-mail to

the local root account. These mails can be further directed towards other mail

accounts via aliases (easily achieveable if you use either the

``mail_forwarder`` or ``mail_server`` roles).

No packages will be upgraded automatically - ensuring you can make sure upgrades

work correctly and do not cause major outage without anyone being present to

fix them.

Another useful package you may want to look into is ``needrestart`` - which runs

as a hook during the upgrade process to detect any processes that seem to be

running with outdated libraries, allowing you to restart them as well. This

package is *not* installed by the ``common`` role out-of-the-box, but you can

easily do so by updating the ``common_packages`` setting.

In addition to system packages, the ``common`` role makes it easy to check if

any of the pip requirements files are outdated as well. It should be noted,

though, that this check does *not* verify the Python virtual environments

themselves.

This is primarily useful when you use `pip-tools

<https://github.com/jazzband/pip-tools>`_ for maintaining the

requirements files. In fact, I would encourage you to utilise

``pip-tools`` for both this purpose and for keeping the virtual

environment in sync and up-to-date.

Roles that want to take advantage of this would:

- Create a sub-directory under

  ``/etc/pip_check_requirements_upgrades/`` (for Python 2

  applications) or ``/etc/pip_check_requirements_upgrades-py3/`` (for

  Python 3 applications).

- Deploy ``.in`` and ``.txt`` files within the sub-directory (see ``pip-tools``

  docs for explanation of how the ``.in`` files work).

- Ensure the created sub-directory and files have ownership set to

  ``root:pipreqcheck``.

.. note::

   If you are using the ``wsgi_website`` role as dependency, simply set-up the

   ``wsgi_requirements`` parameter, and then deploy the ``.in`` and ``.txt``

   file into directory ``/etc/pip_check_requirements_upgrades/FQDN`` (this

   directory is automatically created when ``wsgi_requirements`` is specified).

Where to go next?

-----------------

Well, those were some rather lengthy usage instructions, but hopefuly they are

useful. Things you might want to check-out next:

* :ref:`rolereference`

* :ref:`testsite`

* Finally, if it tickles your interest, have a look at role implementations

  themselves.