- php-xml

            - git

            - php-mysql

            - php-apcu

            - php-zip

          # Set-up URL rewriting. This is based on public/.htaccess file from

          # TBG.

          php_rewrite_urls:

            - ^(.*)$ /index.php?url=$1

          # We don't necessarily need this, but in case you have a policy on

          # uid/gid usage, this is useful. Take note that below value is used

          # for both the dedicated uid and gid for application user.

          uid: 2000

          admin_uid: 3000

        # And this role sets up a new dedicated database for our web

        # application.

        - role: database

          # This is both the database name, _and_ name of the database user

          # that will be granted full privileges on the database.

          db_name: tbg

          # This will be the password of our user 'tbg' for accessing the

          # database. Take note the user can only login from localhost.

          db_password: tbg

3. Now for my favourite part again - creating private keys and certificates!

   Why?  Because the ``php_website`` role requires a private key/certificate

   pair to be deployed. So... Moving on:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/tbg.example.com_https.cfg`

      ::

         organization = "Example Inc."

         country = SE

         cn = "Exampe Inc. Issue Tracker"

         expiration_days = 365

         dns_name = "tbg.example.com"

         tls_www_server

         signing_key

         encryption_key

   2. Create the keys and certificates for the application::

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/tbg.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/tbg.example.com_https.cfg --load-privkey ~/mysite/tls/tbg.example.com_https.key --outfile ~/mysite/tls/tbg.example.com_https.pem

4. Time to get our hands a bit more dirty... Up until now we didn't have to write

   custom tasks, but at this point we need to.

   :file:`~/mysite/roles/tbg/tasks/main.yml`

   ::

      ---

      - name: Define TBG version

        set_fact:

          tbg_version: "4.3.1"

          tbg_archive_checksum: "45de72b1ef82142ad46686577d593375ba370156df4367d17386b4e26a37f342"

      - name: Download the TBG archive

        get_url:

          url: "https://github.com/thebuggenie/thebuggenie/archive/v{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          sha256sum: "{{ tbg_archive_checksum }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Download Composer

        get_url:

          url: "https://getcomposer.org/download/1.10.19/composer.phar"

          dest: "/usr/local/bin/composer"

          sha256sum: "688bf8f868643b420dded326614fcdf969572ac8ad7fbbb92c28a631157d39e8"

          owner: root

          group: root

          mode: 0755

      - name: Unpack TBG

        unarchive:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/"

          copy: no

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Allow use of lib-pcre version 10 (since PHP is built against it in Debian Buster)

        lineinfile:

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/{{ item }}"

          state: present

          regexp: '.*"lib-pcre".*'

          line: '        "lib-pcre": ">=8.0",'

        with_items:

          - "composer.json"

          - "composer.lock"

      - name: Create TBG cache directory

        file:

          path: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/cache"

          state: directory

          mode: 02770

        become: yes

        become_user: admin-tbg_example_com

      - name: Set-up the necessary write permissions for TBG directories

        file:

          path: "{{ item }}"

          mode: g+w

        with_items:

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/

      - name: Create symbolic link to TBG application

        file:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public"

          path: "/var/www/tbg.example.com/htdocs"

          state: link

          owner: "admin-tbg_example_com"

          group: "web-tbg_example_com"

        become: yes

        become_user: admin-tbg_example_com

      - name: Install TBG dependencies

        composer:

          command: install

          working_dir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Deploy database configuration file for TBG

        copy:

          src: "b2db.yml"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"

          mode: 0640

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Install pexpect package

        apt:

          name: python3-pexpect

          state: present

      - name: Deploy expect script for installing TBG

        copy:

          src: "tbg_expect_install"

          dest: "/var/www/tbg.example.com/tbg_expect_install"

          mode: 0750

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Run TBG installer via expect script

        command: /var/www/tbg.example.com/tbg_expect_install

        args:

          chdir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed"

        become: yes

        become_user: admin-tbg_example_com

5. Set-up the files that are deployed by our role.

   :file:`~/mysite/roles/tbg/files/b2db.yml`

   ::

      b2db:

          username: "tbg"

          password: "tbg"

          dsn: "mysql:host=localhost;dbname=tbg"

          tableprefix: ''

          cacheclass: '\thebuggenie\core\framework\Cache'

   :file:`~/mysite/roles/tbg/files/tbg_expect_install`

   ::

      #!/usr/bin/env python3

      import pexpect

      # Spawn the process.

      install_process = pexpect.spawnu('./tbg_cli', args = ["install",

                                                                        "--accept_license=yes",

                                                                        "--url_subdir=/",

                                                                        "--use_existing_db_info=yes",

                                                                        "--enable_all_modules=no",

                                                                        "--setup_htaccess=yes"])

      # If we get EOF, we probably already installed application, and ran

      # into error at the end since no patterns matched.

      try:

          # First confirmation.

          install_process.expect(u'Press ENTER to continue with the installation: ', timeout=5)

             speed this up. Please do read up on haveged if deploying to a real

             system, though! Don't trust it blindly!

   ::

     chmod 700 ~/mysite/gnupg

     pkill gpg-agent

     gpg --homedir ~/mysite/gnupg --batch --generate-key << EOF

     Key-Type:RSA

     Key-Length:1024

     Name-Real:comms.example.com

     Expire-Date:0

     %no-protection

     %commit

     Key-Type:RSA

     Key-Length:1024

     Name-Real:www.example.com

     Expire-Date:0

     %no-protection

     %commit

     Key-Type:RSA

     Key-Length:1024

     Name-Real:bak.example.com

     Expire-Date:0

     %no-protection

     %commit

     EOF

5. And... Apply::

     workon mysite && ansible-playbook playbooks/site.yml

6. At this point the backup roles have been set-up everywhere, and the backups

   will be running every day at 02:00 in the morning. Of course, you may want to

   test out a backup yourself immediatelly by running the following command on

   servers::

     duply main backup

   .. note:: For more information on available commands and how to work with

             backup tool, please have look at `Official Duply Pages

             <http://duply.net/>`_.

Adding backup support to custom roles

-------------------------------------

As mentioned before, all of the supplied roles coming with *Majic Ansible Roles*

include backup support. What gets backed-up depends on the role implementation

(see role reference for details). What about backup support for custom roles?

This is something that has to be done by hand. However, it is quite simple to do

so.

Backup integration will be demonstrated with the previously implemented

``tbg`` role.

*The Bug Genie* stores most of its data in database, but thanks to the

``database`` role its backup is already handled for us. As a side-note, just

before every backup run the database is dumped and stored in location

``/srv/backup/tbg.sql``. That file is subsequently backed-up via *Duply* run.

What is not backed-up for us, though, are the files uploaded to *The Bug

Genie*. So let's fix that one.

1. Add the ``backup`` role to list of dependencies. Take note that while the

   ``backup_client`` role deals with basic set-up of backup client and its

   configuration, the ``backup`` role is used to define what should be

   backed-up. It is important to define unique filename for the backup patterns

   file. Take into account that you can use pretty much any globbing pattern

   supported by Duplicity.

   .. warning::

      Make sure the addition is properly aligned in the yaml file to previous

      role dependency definitions.

   :file:`~/mysite/roles/tbg/meta/main.yml`

   .. Small workaround for Sphinx not preserving leading spaces in

      case all lines have the same amount of leading spaces.

   .. code-block:: none

      :name: sphinx_workaround

        - role: backup

          when: enable_backup

          backup_patterns_filename: "tbg"

          backup_patterns:

            - "/var/www/tbg.example.com/files"

2. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

   ::

     duply main backup

4. Verify that the files have been backed-up:

   ::

      duply main list

.. note:: If you wanted to run a script prior to backup run, you would simply

          deploy a shell script with desired content to

          ``/etc/duply/main/pre.d/``. Just make sure the permissions for it are

          ok (it has to be executable by the root user).

Dealing with failures

---------------------

While the roles have been designed to be fairly robust, it should be taken into

account that certain handlers are used to bring the system into consistent

state. These handlers are mostly the ones dealing with service restarts, but

there are also a couple of handlers that take care of transforming certain data

into the required formats, import of files etc.

This means that failure to successfully execute such handlers could result in

inconsistent state on the server. Think of service configuration files being

updated, yet the service itself is not restarted and therefore continues to run

with the old configuration.

Handler execution failure can depend on a couple of things, including the loss

of SSH connectivity to managed machine, or some kind of unusual time-out during

handler execution.

To help handle this situation, Majic Ansible Roles all come with a special way

to invoke the handlers explicitly. Each role will include handlers as tasks,

provided that a special variable (``run_handlers``) is passed in to playbook run. To

make the run shorter, the handlers in such a run are also tagged with

``handlers``. This doubling of environment variable + tagging stems from current

limitations of Ansible (it is not possible to specify that certain task should

be run only if a tag is specified, therefore an additional variable has to be

used).

Handlers alone can be invoked specifically with command similar to::

  ansible-playbook -t handlers -e run_handlers=true playbooks/site.yml

The ``run_handlers`` variable is treated as boolean, and by default it

is not set.

Checking for available package upgrades

---------------------------------------

One of the more annoying chores when you maintain your own infrastructure is

making sure everything is up-to-date. And this has to be done - both in order to

ensure for problem-free experience for users (yourself included), and for making

sure there are no security vulnerabilities that could be exploited by a (random)

adversary.

*Majic Ansible Roles* try to keep you covered on this front as well. As part of

regular deployment, the ``common`` role will deploy and configure ``apticron`` -

a nifty little script that runs on hourly basis and checks if any of your

system-provided packages are outdated.

If ``apticron`` detects an outdated package, it will output this information to

standard output, which will result in the cron daemon sending out an e-mail to

the local root account. These mails can be further directed towards other mail

accounts via aliases (easily achieveable if you use either the

``mail_forwarder`` or ``mail_server`` roles).

No packages will be upgraded automatically - ensuring you can make sure upgrades

work correctly and do not cause major outage without anyone being present to

fix them.

Another useful package you may want to look into is ``needrestart`` - which runs

as a hook during the upgrade process to detect any processes that seem to be

running with outdated libraries, allowing you to restart them as well. This

package is *not* installed by the ``common`` role out-of-the-box, but you can

easily do so by updating the ``common_packages`` setting.

In addition to system packages, the ``common`` role makes it easy to check if

any of the pip requirements files are outdated as well. It should be noted,

though, that this check does *not* verify the Python virtual environments

themselves.

This is primarily useful when you use `pip-tools

<https://github.com/jazzband/pip-tools>`_ for maintaining the

requirements files. In fact, I would encourage you to utilise

``pip-tools`` for both this purpose and for keeping the virtual

environment in sync and up-to-date.

Roles that want to take advantage of this would:

- Create a sub-directory under