majic-ansible-roles Changeset

@@ -1230,10 +1230,11 @@ Let us first define what we want to deploy on the web server. Here is the plan:

   setting-up a couple of web applications. For the purpose of the usage

   instructions, we will deploy the following two:

   1. `The Bug Genie <http://thebuggenie.org/>`_ - an issue tracker. To keep

      things simple, we will not integrate it with our LDAP server (although

      this is supported and possible). Being written in PHP, this will

      demonstrate the role for PHP web application deployment.

   1. `Nextcloud <https://nextcloud.com/>`_ - extendable solution for

      file sharing, calendars etc. To keep things simple, we will not

      integrate it with our LDAP server (although this is supported

      and possible). Being written in PHP, this application will be

      used to demonstrate the role for PHP web application deployment.

   2. `Django Wiki <https://github.com/django-wiki/django-wiki>`_ - a wiki

      application written in Django. This will serve as a demo of how the WSGI

@@ -1380,285 +1381,322 @@ server.

   which can be combined with web app roles for this purpose).

Deploying a PHP web application (The Bug Genie)

Deploying a PHP web application (Nextcloud)

-----------------------------------------------

We have some basic infrastructure up and running on our web server, so

now we can move on to setting-up a PHP web application on it. As

mentioned before, we will take *The Bug Genie* as an example.

mentioned before, we will roll-out *Nextcloud*.

For this we will create a local role in our site to take care of it. This role

will in turn utilise two roles coming from *Majic Ansible Roles* that will make

our life (a little) easier.

For this we will create a local role in our site to take care of

it. This role will in turn utilise two roles coming from *Majic

Ansible Roles* that will make our life (a little) easier.

To make the example a bit simpler, no parameters will be introduced

for this role (not even the password for database, we'll hard-code

everything).

for this role (not even the password for database) - we'll hard-code

everything.

Before we start, here is a couple of useful pointers regarding the

``php_website`` role we'll be using for the PHP part:

* The role is designed to execute every application via dedicated user and

  group. The user/group name is automatically derived from the FQDN of website,

  for example ``web-tbg_example_com``.

* While running the application, application user's umask is set to ``0007``

  (letting the administrator user be able to manage any files created while the

  application is running).

* An administrative user is created as well, and this user should be used when

  running maintenance and installation commands. Similar to application user,

  the name is also derived from the FQDN of website, for example

  ``admin-tbg_example_com``. Administrative user does not have a dedicated

  group, and instead belongs to same group as the application user.

``php_website`` role:

* The role is designed to execute every application via dedicated user

  and group. The user/group name is automatically derived from the

  FQDN of website, for example ``web-nextcloud_example_com``.

* While running the application, application user's umask is set to

  ``0007`` (letting the administrator user be able to manage any files

  created while the application is running).

* An administrative user is created as well, and this user should be

  used when running maintenance and installation commands. Similar to

  application user, the name is also derived from the FQDN of website,

  for example ``admin-nextcloud_example_com``. Administrative user

  does not have a dedicated group, and instead belongs to same group

  as the application user.

* PHP applications are executed via FastCGI, using *PHP-FPM*.

* If you ever need to set some additional PHP FPM settings, this can easily be

  done via the ``additional_fpm_config`` role parameter. This particular example

  does not set any, though.

* Mails delivered to local admin/application users are forwarded to ``root``

  account instead (this can be configured via ``website_mail_recipients`` role

* If you ever need to set some additional PHP FPM settings, this can

  easily be done via the ``additional_fpm_config`` role

  parameter. This particular example does not set any, though.

* Mails delivered to local admin/application users are forwarded to

  ``root`` account (configurable via ``website_mail_recipients`` role

  parameter.

* If you ever find yourself mixing-up test and production websites,

  have a look at ``environment_indicator`` role parameter. It lets you

  insert small strip with environment information at bottom of each

  HTML page served by the web server.

* Static content (non-PHP) is served directly by *Nginx*.

* Each web application gets distinct sub-directory under ``/var/www``, named

  after the FQDN. All sub-directories created under there are created with

  ``02750`` permissions, with ownership set to admin user, and group set to the

  application's group. In other words, all directories will have ``SGID`` bit

  set, allowing you to create files/directories that will have their group

  automatically set to the group of the parent directory.

* Files are served (both by *Nginx* and *PHP-FPM*) from sub-directory called

  ``htdocs`` (located in website directory). For example

  ``/var/www/tbg.example.com/htdocs/``. Normally, this can be a symlink to some

  other sub-directory within the website directory (useful for having multiple

  versions for easier downgrades etc).

* Each web application gets distinct sub-directory under ``/var/www``,

  named after the FQDN. All sub-directories created under there are

  created with ``02750`` permissions, with ownership set to admin

  user, and group set to the application's group. In other words, all

  directories will have ``SGID`` bit set, allowing you to create

  files/directories that will have their group automatically set to

  the group of the parent directory.

* Files are served (both by *Nginx* and *PHP-FPM*) from sub-directory

  called ``htdocs`` (located in website directory). For example

  ``/var/www/nextcloud.example.com/htdocs/``. Normally, this can be a

  symlink to some other sub-directory within the website directory

  (useful for having multiple versions for easier downgrades etc).

* Combination of admin user membership in application group, ``SGID``

  permission, and the way ownership of sub-directories is set-up usually means

  that the administrator will be capable of managing application files, and

  application can be granted write permissions to a *minimum* of necessary

  files.

  permission, and the way ownership of sub-directories is set-up

  usually means that the administrator will be capable of managing

  application files, and application can be granted write permissions

  to a *minimum* of necessary files.

  .. warning::

     Just keep in mind that some file-management commands, like ``mv``, do *not*

     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you

     deploy new files to the directory instead (don't simply move them from your

     home directory).

     Just keep in mind that some file-management commands, like

     ``mv``, do *not* respect the ``SGID`` bit. In fact, I would

     recommend using ``cp`` when you deploy new files to the directory

     instead (don't simply move them from your home directory).

1. Start-off with creating the necessary directories for the new role::

     mkdir -p ~/mysite/roles/tbg/{tasks,meta,files}/

     mkdir -p ~/mysite/roles/nextcloud/{tasks,meta,files}/

2. Let's set-up role dependencies, reusing some common roles to make our life

   easier.

2. Let's set-up role dependencies, reusing some common roles to make

   our life easier.

   :file:`~/mysite/roles/tbg/meta/main.yml`

   :file:`~/mysite/roles/nextcloud/meta/main.yml`

::

---

      dependencies:

        # Ok, so this role helps us set-up Nginx virtual host for serving our

        # app.

        # Role helps us set-up Nginx virtual host for serving our app.

        - role: php_website

          # Our virtual host will for PHP website will respond to this name.

          fqdn: tbg.example.com

          # Name that will be bound to specific virtual host definition.

          fqdn: nextcloud.example.com

          # TLS key and certificate to use for the virtual host.

          https_tls_certificate: "{{ lookup('file', '~/mysite/tls/tbg.example.com_https.pem') }}"

          https_tls_key: "{{ lookup('file', '~/mysite/tls/tbg.example.com_https.key') }}"

          # Some additional packages are required in order to deploy and use TBG.

          https_tls_certificate: "{{ lookup('file', '~/mysite/tls/nextcloud.example.com_https.pem') }}"

          https_tls_key: "{{ lookup('file', '~/mysite/tls/nextcloud.example.com_https.key') }}"

          # Additional packages required for deploying and running Nextcloud.

          packages:

            - php-gd

            - php-json

            - php-mysql

            - php-curl

            - php-intl

            - php-mbstring

            - php-imagick

            - php-ldap

            - php-xml

            - git

            - php-mysql

            - php-apcu

            - php-zip

          # Set-up URL rewriting. This is based on public/.htaccess file from

          # TBG.

          php_rewrite_urls:

            - ^(.*)$ /index.php?url=$1

          # We don't necessarily need this, but in case you have a policy on

          # uid/gid usage, this is useful. Take note that below value is used

          # for both the dedicated uid and gid for application user.

            - php-gmp

            - python3-pexpect

            - php-apcu

            - php-bcmath

          # Set-up URL rewrites for well-known URIs (see https://en.wikipedia.org/wiki/Well-known_URIs).

          rewrites:

            - '^/\.well-known/carddav /remote.php/dav/ permanent'

            - '^/\.well-known/caldav /remote.php/dav/ permanent'

            - '^/remote/(.*) /remote.php last'

          # Prevent specific files from ever being served by the web server (for security reasons etc).

          deny_files_regex:

            - '^/(build|tests|config|lib|3rdparty|templates|data)/'

            - '^/(?:\.|autotest|occ|issue|indie|db_|console)'

          # Custom regex defining what files shouled be processed via PHP

          # interpreter.

          php_file_regex: \.php(?:$|/)

          # Not necessarily needed, but in case you have a policy on uid/gid

          # usage, this is useful. Take note that the uid value is also used

          # for the application group (gid == uid).

          uid: 2000

          admin_uid: 3000

        # And this role sets up a new dedicated database for our web

        # Role that sets up a new dedicated database for our web

        # application.

        - role: database

          # This is both the database name, _and_ name of the database user

          # that will be granted full privileges on the database.

          db_name: tbg

          # This will be the password of our user 'tbg' for accessing the

          # database. Take note the user can only login from localhost.

          db_password: tbg

          db_name: nextcloud

          # Password for user used for accessing the database. Take note

          # that the user can only login from localhost.

          db_password: nextcloud

3. Now for my favourite part again - creating private keys and certificates!

   Why?  Because the ``php_website`` role requires a private key/certificate

   pair to be deployed. So... Moving on:

3. Now for my favourite part again - creating private keys and

   certificates!  Why?  Because the ``php_website`` role requires a

   private key/certificate pair to be deployed. So... Moving on:

   1. Create new template for ``certtool``:

      :file:`~/mysite/tls/tbg.example.com_https.cfg`

      :file:`~/mysite/tls/nextcloud.example.com_https.cfg`

::

         organization = "Example Inc."

         country = SE

         cn = "Exampe Inc. Issue Tracker"

         cn = "Example Inc. Cloud Service"

         expiration_days = 365

         dns_name = "tbg.example.com"

         dns_name = "nextcloud.example.com"

         tls_www_server

         signing_key

         encryption_key

   2. Create the keys and certificates for the application::

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/tbg.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/tbg.example.com_https.cfg --load-privkey ~/mysite/tls/tbg.example.com_https.key --outfile ~/mysite/tls/tbg.example.com_https.pem

        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/nextcloud.example.com_https.key

        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/nextcloud.example.com_https.cfg --load-privkey ~/mysite/tls/nextcloud.example.com_https.key --outfile ~/mysite/tls/nextcloud.example.com_https.pem

4. Time to get our hands a bit more dirty... Up until now we didn't have to write

   custom tasks, but at this point we need to.

4. Time to get our hands a bit more dirty... Up until now we didn't

   have to write custom tasks, but that ends now.

   :file:`~/mysite/roles/tbg/tasks/main.yml`

   :file:`~/mysite/roles/nextcloud/tasks/main.yml`

::

---

      - name: Define TBG version

        set_fact:

          tbg_version: "4.3.1"

          tbg_archive_checksum: "45de72b1ef82142ad46686577d593375ba370156df4367d17386b4e26a37f342"

      # Deployment

      # ==========

      - name: Download the TBG archive

      - name: Download the application archive

        get_url:

          url: "https://github.com/thebuggenie/thebuggenie/archive/v{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          sha256sum: "{{ tbg_archive_checksum }}"

          url: "https://download.nextcloud.com/server/releases/nextcloud-20.0.14.tar.bz2"

          dest: "/var/www/nextcloud.example.com/nextcloud-20.0.14.tar.gz"

          sha256sum: "09652c459c99e0b77dbd91603151715c6cd09e4f4abbdecb64f495581af09367"

        become: yes

        become_user: admin-tbg_example_com

        become_user: admin-nextcloud_example_com

      - name: Download Composer

        get_url:

          url: "https://getcomposer.org/download/1.10.19/composer.phar"

          dest: "/usr/local/bin/composer"

          sha256sum: "688bf8f868643b420dded326614fcdf969572ac8ad7fbbb92c28a631157d39e8"

          owner: root

          group: root

          mode: 0755

      - name: Unpack TBG

      - name: Unpack the application archive

        unarchive:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"

          dest: "/var/www/tbg.example.com/"

          src: "/var/www/nextcloud.example.com/nextcloud-20.0.14.tar.gz"

          dest: "/var/www/nextcloud.example.com/"

          copy: no

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

          creates: "/var/www/nextcloud.example.com/nextcloud"

        become: yes

        become_user: admin-tbg_example_com

        become_user: admin-nextcloud_example_com

      - name: Allow use of lib-pcre version 10 (since PHP is built against it in Debian Buster)

      # Majic Ansible Roles currently only support utf8 encoding.

      - name: Disable opportunistic use of utf8mb4 on fresh installs

        lineinfile:

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/{{ item }}"

          state: present

          regexp: '.*"lib-pcre".*'

          line: '        "lib-pcre": ">=8.0",'

        with_items:

          - "composer.json"

          - "composer.lock"

          dest: "/var/www/nextcloud.example.com/nextcloud/lib/private/Setup/MySQL.php"

          line: "{{ '\t\t\t' }}$this->config->setValue('mysql.utf8mb4', true);"

          state: absent

      - name: Create directory for storing uploaded files

      - name: Allow application user to install and update applications

        file:

          path: "/var/www/tbg.example.com/files"

          state: directory

          mode: 02770

        become: yes

        become_user: admin-tbg_example_com

          path: "/var/www/nextcloud.example.com/nextcloud/apps"

          mode: g+w

      - name: Create symlink towards directory where uploaded files are stored

      - name: Allow CLI tool to be run by the user and group

        file:

          src: "/var/www/tbg.example.com/files"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/files"

          state: link

        become: yes

        become_user: admin-tbg_example_com

          path: "/var/www/nextcloud.example.com/nextcloud/occ"

          mode: u+x,g+x

      - name: Create TBG cache directory

      - name: Create directory for storing data

        file:

          path: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/cache"

          path: "/var/www/nextcloud.example.com/data"

          state: directory

          mode: 02770

        become: yes

        become_user: admin-tbg_example_com

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

      - name: Set-up the necessary write permissions for TBG directories

      - name: Create directory for storing configuration files

        file:

          path: "{{ item }}"

          mode: g+w

        with_items:

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public/

          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/

          path: "/var/www/nextcloud.example.com/nextcloud/config"

          state: directory

          mode: 02750

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

      - name: Create symbolic link to TBG application

      - name: Create an empty log file if it does not exist

        copy:

          content: ""

          dest: "/var/www/nextcloud.example.com/data/nextcloud.log"

          force: no

      - name: Set-up log file permissions

        file:

          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public"

          path: "/var/www/tbg.example.com/htdocs"

          path: "/var/www/nextcloud.example.com/data/nextcloud.log"

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

          mode: 0660

      - name: Symlink the default path used by the web server for finding application files

        file:

          src: "/var/www/nextcloud.example.com/nextcloud"

          dest: "/var/www/nextcloud.example.com/htdocs"

          state: link

          owner: "admin-tbg_example_com"

          group: "web-tbg_example_com"

        become: yes

        become_user: admin-tbg_example_com

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

        notify:

          - Restart PHP-FPM

      - name: Install TBG dependencies

        composer:

          command: install

          working_dir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

        become: yes

        become_user: admin-tbg_example_com

      - name: Deploy database configuration file for TBG

        copy:

          src: "b2db.yml"

          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"

          mode: 0640

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      # Installation

      # ============

      - name: Get application installation status

        command: "/var/www/nextcloud.example.com/nextcloud/occ status"

        become: yes

        become_user: "admin-nextcloud_example_com"

        register: nextcloud_status

        changed_when: False

        failed_when: False

      - name: Install pexpect package

        apt:

          name: python3-pexpect

          state: present

      - name: Check if application is installed

        set_fact:

          nextcloud_installed: "{{ 'Nextcloud is not installed' not in nextcloud_status.stdout }}"

      - name: Deploy expect script for installing TBG

      - name: Deploy installation script

        copy:

          src: "tbg_expect_install"

          dest: "/var/www/tbg.example.com/tbg_expect_install"

          mode: 0750

          owner: admin-tbg_example_com

          group: web-tbg_example_com

      - name: Run TBG installer via expect script

        command: /var/www/tbg.example.com/tbg_expect_install

        args:

          chdir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"

          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed"

          src: "install_nextcloud.py"

          dest: "/var/www/nextcloud.example.com/install_nextcloud.py"

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

          mode: 0700

        when: "not nextcloud_installed"

      - name: Install application

        command: "/var/www/nextcloud.example.com/install_nextcloud.py"

        become: yes

        become_user: admin-tbg_example_com

        become_user: "admin-nextcloud_example_com"

        when: "not nextcloud_installed"

      - name: Remove installation script

        file:

          path: "/var/www/nextcloud.example.com/install_nextcloud.py"

          state: absent

5. Set-up the files that are deployed by our role.

      - name: Fix data file permissions for application user/group

        file:

          path: "/var/www/nextcloud.example.com/data"

          mode: g+w

          recurse: yes

          follow: no

   :file:`~/mysite/roles/tbg/files/b2db.yml`

::

      - name: Deploy local configuration overrides

        copy:

          src: "local.config.php"

          dest: "/var/www/nextcloud.example.com/nextcloud/config/local.config.php"

          owner: "admin-nextcloud_example_com"

          group: "web-nextcloud_example_com"

          mode: 0640

5. Set-up files that are deployed by the role.

      b2db:

          username: "tbg"

          password: "tbg"

          dsn: "mysql:host=localhost;dbname=tbg"

          tableprefix: ''

          cacheclass: '\thebuggenie\core\framework\Cache'

   :file:`~/mysite/roles/nextcloud/files/local.config.php`

::

   :file:`~/mysite/roles/tbg/files/tbg_expect_install`

      <?php

      $CONFIG = array (

        'config_is_read_only' => true,

        'instanceid' => 'suqw2cvca8sp',

        'trusted_domains' =>

          array (

            0 => 'nextcloud.example.com',

),

);

   :file:`~/mysite/roles/nextcloud/files/install_nextcloud.py`

::

      #!/usr/bin/env python3

@@ -1666,35 +1704,48 @@ Before we start, here is a couple of useful pointers regarding the

      import pexpect

      # Spawn the process.

      install_process = pexpect.spawnu('./tbg_cli', args = ["install",

                                                                        "--accept_license=yes",

                                                                        "--url_subdir=/",

                                                                        "--use_existing_db_info=yes",

                                                                        "--enable_all_modules=no",

                                                                        "--setup_htaccess=yes"])

      install_process = pexpect.spawnu('/var/www/nextcloud.example.com/nextcloud/occ',

                                       args = [ 'maintenance:install',

                                                '--database', 'mysql',

                                                '--database-name', 'nextcloud',

                                                '--database-user', 'nextcloud',

                                                '--database-host', 'localhost',

                                                '--database-port', '3306',

                                                '--admin-user', 'admin',

                                                '--data-dir', '/var/www/nextcloud.example.com/data'])

      # If we get EOF, we probably already installed application, and ran

      # into error at the end since no patterns matched.

      try:

          # First confirmation.

          install_process.expect(u'Press ENTER to continue with the installation: ', timeout=5)

          install_process.sendline(u'')

          # Second confirmation.

          install_process.expect(u'Press ENTER to continue: ', timeout=5)

          install_process.sendline(u'')

          # Provide database password.

          install_process.expect(u'What is the password to access the database with user.*\?', timeout=10)

          install_process.sendline(u'nextcloud')

          # Provide administrator password.

          install_process.expect(u'What is the password you like to use for the admin account.*\?', timeout=10)

          install_process.sendline(u'admin')

          # Wait for application to finish.

          install_process.expect(pexpect.EOF, timeout=60)

          install_process.expect(pexpect.EOF, timeout=120)

      except pexpect.EOF as e:

          pass

      # Close application.

      # Print command output. Has to be done prior to final wait for

      # pexpect.EOF.

      print(install_process.before.encode('utf-8'))

      # Close application. Additional wait for pexpect.EOF prevents the

      # process from getting killed prematurely in case it exits

      # immediatelly (due to wrong command line arguments etc). Some

      # background information can be found at (although it is a very old

      # post):

      # https://www.heikkitoivonen.net/blog/2009/01/28/pexpect-and-inconsistent-exit-status/

      install_process.expect(pexpect.EOF)

      install_process.close()

      # Print text output.

      print(install_process.before)

      # Return same exit code like child process.

      exit(install_process.exitstatus)

@@ -1714,15 +1765,15 @@ Before we start, here is a couple of useful pointers regarding the

          - mail_forwarder

          - web_server

          - database_server

          - tbg

          - nextcloud

7. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

8. At this point *The Bug Genie* has been installed, and you should be able to

   open the URL https://tbg.example.com/ and log-in into *The Bug Genie*

   with username ``administrator`` and password ``admin``.

8. At this point *Nextcloud* has been installed, and you should be

   able to open the URL https://nextcloud.example.com/ and log-in into

   *Nextcloud* with username ``admin`` and password ``admin``.

Deploying a WSGI application (Django Wiki)

@@ -2144,7 +2195,7 @@ on the safe side:

          - mail_forwarder

          - web_server

          - database_server

          - tbg

          - nextcloud

          - wiki

7. Apply the changes:

@@ -2373,36 +2424,39 @@ So, back to the business:

Adding backup support to custom roles

-------------------------------------

As mentioned before, all of the supplied roles coming with *Majic Ansible Roles*

include backup support. What gets backed-up depends on the role implementation

(see role reference for details). What about backup support for custom roles?

This is something that has to be done by hand. However, it is quite simple to do

so.

Backup integration will be demonstrated with the previously implemented

``tbg`` role.

*The Bug Genie* stores most of its data in database, but thanks to the

``database`` role its backup is already handled for us. As a side-note, just

before every backup run the database is dumped and stored in location

``/srv/backup/tbg.sql``. That file is subsequently backed-up via *Duply* run.

What is not backed-up for us, though, are the files uploaded to *The Bug

Genie*. So let's fix that one.

1. Add the ``backup`` role to list of dependencies. Take note that while the

   ``backup_client`` role deals with basic set-up of backup client and its

   configuration, the ``backup`` role is used to define what should be

   backed-up. It is important to define unique filename for the backup patterns

   file. Take into account that you can use pretty much any globbing pattern

   supported by Duplicity.

As mentioned before, all of the supplied roles coming with *Majic

Ansible Roles* include backup support. What gets backed-up depends on

the role implementation (see role reference for details). What about

backup support for custom roles?  This is something that has to be

done by hand. However, it is quite simple to do so.

Backup integration will be demonstrated with the previously

implemented ``nextcloud`` role.

*Nextcloud* stores most of its data on the filesystem, but also in the

database. Thanks to the ``database`` role, though, database backup is

already handled for us. As a side-note, just before every backup run

the database is dumped and stored in location

``/srv/backup/nextcloud.sql``. That file is subsequently backed-up via

*Duply* run.

What is not backed-up for us, though, are the uploaded files

themselves, as well as configuration (which is generated during

application installation). So let's fix that one.

1. Add the ``backup`` role to list of dependencies. Take note that

   while the ``backup_client`` role deals with basic set-up of backup

   client and its configuration, the ``backup`` role is used to define

   what should be backed-up. It is important to define unique filename

   for the backup patterns file. Take into account that you can use

   pretty much any globbing pattern supported by Duplicity.

   .. warning::

      Make sure the addition is properly aligned in the yaml file to previous

      role dependency definitions.

      Make sure the addition is properly aligned in the yaml file to

      previous role dependency definitions.

   :file:`~/mysite/roles/tbg/meta/main.yml`

   :file:`~/mysite/roles/nextcloud/meta/main.yml`

   .. Small workaround for Sphinx not preserving leading spaces in

      case all lines have the same amount of leading spaces.

@@ -2412,22 +2466,18 @@ Genie*. So let's fix that one.

        - role: backup

          when: enable_backup

          backup_patterns_filename: "tbg"

          backup_patterns_filename: "nextcloud"

          backup_patterns:

            - "/var/www/tbg.example.com/files"

            - "/var/www/nextcloud.example.com/data"

            - "/var/www/nextcloud.example.com/nextcloud/config"

2. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

3. Now rerun the backup on server ``www.example.com`` (as root). If

   you haven't uploaded any files, you may want to do so before

   testing to make sure something is backed-up. This will require

   enabling file uploads in `The Bug Genie settings

   <https://tbg.example.com/configure/files>`_, creating a test

   project, and then adding a new project release (via project's

   release center). While creating a new project release, it is

   possible to upload a release file.

   you haven't uploaded any files to Nextcloud, you may want to do so

   before testing to make sure something is backed-up.

::

@@ -2439,10 +2489,11 @@ Genie*. So let's fix that one.

      duply main list

.. note:: If you wanted to run a script prior to backup run, you would simply

          deploy a shell script with desired content to

          ``/etc/duply/main/pre.d/``. Just make sure the permissions for it are

          ok (it has to be executable by the root user).

.. note::

   If you wanted to run a script prior to backup run, you would simply

   deploy a shell script with desired content to

   ``/etc/duply/main/pre.d/``. Just make sure the permissions for it

   are ok (it has to be executable by the root user).

Dealing with failures