Changeset - dd1b35170e07
[Not reviewed]
0 1 0
Branko Majic (branko) - 10 months ago 2023-12-20 22:59:24
branko@majic.rs
MAR-189: Replace The Bug Genie deployment as part of usage instructions with Nextcloud:

- Nextcloud is still being properly maintained, and should be way less
hacky to roll-out as opposed to TBG.
1 file changed with 298 insertions and 247 deletions:
0 comments (0 inline, 0 general)
docs/usage.rst
Show inline comments
 
@@ -1227,16 +1227,17 @@ Let us first define what we want to deploy on the web server. Here is the plan:
 
   try to take that one out of the way.
 

	
 
3. With this basic deployment for a web server in place, we can move on to
 
   setting-up a couple of web applications. For the purpose of the usage
 
   instructions, we will deploy the following two:
 

	
 
   1. `The Bug Genie <http://thebuggenie.org/>`_ - an issue tracker. To keep
 
      things simple, we will not integrate it with our LDAP server (although
 
      this is supported and possible). Being written in PHP, this will
 
      demonstrate the role for PHP web application deployment.
 
   1. `Nextcloud <https://nextcloud.com/>`_ - extendable solution for
 
      file sharing, calendars etc. To keep things simple, we will not
 
      integrate it with our LDAP server (although this is supported
 
      and possible). Being written in PHP, this application will be
 
      used to demonstrate the role for PHP web application deployment.
 

	
 
   2. `Django Wiki <https://github.com/django-wiki/django-wiki>`_ - a wiki
 
      application written in Django. This will serve as a demo of how the WSGI
 
      role works.
 

	
 
It should be noted that the web application deployment roles are a bit more
 
@@ -1377,327 +1378,377 @@ server.
 

	
 
   Of course, no database has been created for either of the web applications,
 
   but we will get to that one later (there is a dedicated ``database`` role
 
   which can be combined with web app roles for this purpose).
 

	
 

	
 
Deploying a PHP web application (The Bug Genie)
 
Deploying a PHP web application (Nextcloud)
 
-----------------------------------------------
 

	
 
We have some basic infrastructure up and running on our web server, so
 
now we can move on to setting-up a PHP web application on it. As
 
mentioned before, we will take *The Bug Genie* as an example.
 
mentioned before, we will roll-out *Nextcloud*.
 

	
 
For this we will create a local role in our site to take care of it. This role
 
will in turn utilise two roles coming from *Majic Ansible Roles* that will make
 
our life (a little) easier.
 
For this we will create a local role in our site to take care of
 
it. This role will in turn utilise two roles coming from *Majic
 
Ansible Roles* that will make our life (a little) easier.
 

	
 
To make the example a bit simpler, no parameters will be introduced
 
for this role (not even the password for database, we'll hard-code
 
everything).
 
for this role (not even the password for database) - we'll hard-code
 
everything.
 

	
 
Before we start, here is a couple of useful pointers regarding the
 
``php_website`` role we'll be using for the PHP part:
 

	
 
* The role is designed to execute every application via dedicated user and
 
  group. The user/group name is automatically derived from the FQDN of website,
 
  for example ``web-tbg_example_com``.
 
* While running the application, application user's umask is set to ``0007``
 
  (letting the administrator user be able to manage any files created while the
 
  application is running).
 
* An administrative user is created as well, and this user should be used when
 
  running maintenance and installation commands. Similar to application user,
 
  the name is also derived from the FQDN of website, for example
 
  ``admin-tbg_example_com``. Administrative user does not have a dedicated
 
  group, and instead belongs to same group as the application user.
 
``php_website`` role:
 

	
 
* The role is designed to execute every application via dedicated user
 
  and group. The user/group name is automatically derived from the
 
  FQDN of website, for example ``web-nextcloud_example_com``.
 
* While running the application, application user's umask is set to
 
  ``0007`` (letting the administrator user be able to manage any files
 
  created while the application is running).
 
* An administrative user is created as well, and this user should be
 
  used when running maintenance and installation commands. Similar to
 
  application user, the name is also derived from the FQDN of website,
 
  for example ``admin-nextcloud_example_com``. Administrative user
 
  does not have a dedicated group, and instead belongs to same group
 
  as the application user.
 
* PHP applications are executed via FastCGI, using *PHP-FPM*.
 
* If you ever need to set some additional PHP FPM settings, this can easily be
 
  done via the ``additional_fpm_config`` role parameter. This particular example
 
  does not set any, though.
 
* Mails delivered to local admin/application users are forwarded to ``root``
 
  account instead (this can be configured via ``website_mail_recipients`` role
 
* If you ever need to set some additional PHP FPM settings, this can
 
  easily be done via the ``additional_fpm_config`` role
 
  parameter. This particular example does not set any, though.
 
* Mails delivered to local admin/application users are forwarded to
 
  ``root`` account (configurable via ``website_mail_recipients`` role
 
  parameter.
 
* If you ever find yourself mixing-up test and production websites,
 
  have a look at ``environment_indicator`` role parameter. It lets you
 
  insert small strip with environment information at bottom of each
 
  HTML page served by the web server.
 
* Static content (non-PHP) is served directly by *Nginx*.
 
* Each web application gets distinct sub-directory under ``/var/www``, named
 
  after the FQDN. All sub-directories created under there are created with
 
  ``02750`` permissions, with ownership set to admin user, and group set to the
 
  application's group. In other words, all directories will have ``SGID`` bit
 
  set, allowing you to create files/directories that will have their group
 
  automatically set to the group of the parent directory.
 
* Files are served (both by *Nginx* and *PHP-FPM*) from sub-directory called
 
  ``htdocs`` (located in website directory). For example
 
  ``/var/www/tbg.example.com/htdocs/``. Normally, this can be a symlink to some
 
  other sub-directory within the website directory (useful for having multiple
 
  versions for easier downgrades etc).
 
* Each web application gets distinct sub-directory under ``/var/www``,
 
  named after the FQDN. All sub-directories created under there are
 
  created with ``02750`` permissions, with ownership set to admin
 
  user, and group set to the application's group. In other words, all
 
  directories will have ``SGID`` bit set, allowing you to create
 
  files/directories that will have their group automatically set to
 
  the group of the parent directory.
 
* Files are served (both by *Nginx* and *PHP-FPM*) from sub-directory
 
  called ``htdocs`` (located in website directory). For example
 
  ``/var/www/nextcloud.example.com/htdocs/``. Normally, this can be a
 
  symlink to some other sub-directory within the website directory
 
  (useful for having multiple versions for easier downgrades etc).
 
* Combination of admin user membership in application group, ``SGID``
 
  permission, and the way ownership of sub-directories is set-up usually means
 
  that the administrator will be capable of managing application files, and
 
  application can be granted write permissions to a *minimum* of necessary
 
  files.
 
  permission, and the way ownership of sub-directories is set-up
 
  usually means that the administrator will be capable of managing
 
  application files, and application can be granted write permissions
 
  to a *minimum* of necessary files.
 

	
 
  .. warning::
 
     Just keep in mind that some file-management commands, like ``mv``, do *not*
 
     respect the ``SGID`` bit. In fact, I would recommend using ``cp`` when you
 
     deploy new files to the directory instead (don't simply move them from your
 
     home directory).
 
     Just keep in mind that some file-management commands, like
 
     ``mv``, do *not* respect the ``SGID`` bit. In fact, I would
 
     recommend using ``cp`` when you deploy new files to the directory
 
     instead (don't simply move them from your home directory).
 

	
 
1. Start-off with creating the necessary directories for the new role::
 

	
 
     mkdir -p ~/mysite/roles/tbg/{tasks,meta,files}/
 
     mkdir -p ~/mysite/roles/nextcloud/{tasks,meta,files}/
 

	
 
2. Let's set-up role dependencies, reusing some common roles to make our life
 
   easier.
 
2. Let's set-up role dependencies, reusing some common roles to make
 
   our life easier.
 

	
 
   :file:`~/mysite/roles/tbg/meta/main.yml`
 
   :file:`~/mysite/roles/nextcloud/meta/main.yml`
 
   ::
 

	
 
      ---
 

	
 
      dependencies:
 
        # Ok, so this role helps us set-up Nginx virtual host for serving our
 
        # app.
 

	
 
        # Role helps us set-up Nginx virtual host for serving our app.
 
        - role: php_website
 
          # Our virtual host will for PHP website will respond to this name.
 
          fqdn: tbg.example.com
 

	
 
          # Name that will be bound to specific virtual host definition.
 
          fqdn: nextcloud.example.com
 

	
 
          # TLS key and certificate to use for the virtual host.
 
          https_tls_certificate: "{{ lookup('file', '~/mysite/tls/tbg.example.com_https.pem') }}"
 
          https_tls_key: "{{ lookup('file', '~/mysite/tls/tbg.example.com_https.key') }}"
 
          # Some additional packages are required in order to deploy and use TBG.
 
          https_tls_certificate: "{{ lookup('file', '~/mysite/tls/nextcloud.example.com_https.pem') }}"
 
          https_tls_key: "{{ lookup('file', '~/mysite/tls/nextcloud.example.com_https.key') }}"
 

	
 
          # Additional packages required for deploying and running Nextcloud.
 
          packages:
 
            - php-gd
 
            - php-json
 
            - php-mysql
 
            - php-curl
 
            - php-intl
 
            - php-mbstring
 
            - php-imagick
 
            - php-ldap
 
            - php-xml
 
            - git
 
            - php-mysql
 
            - php-apcu
 
            - php-zip
 
          # Set-up URL rewriting. This is based on public/.htaccess file from
 
          # TBG.
 
          php_rewrite_urls:
 
            - ^(.*)$ /index.php?url=$1
 
          # We don't necessarily need this, but in case you have a policy on
 
          # uid/gid usage, this is useful. Take note that below value is used
 
          # for both the dedicated uid and gid for application user.
 
            - php-gmp
 
            - python3-pexpect
 
            - php-apcu
 
            - php-bcmath
 

	
 
          # Set-up URL rewrites for well-known URIs (see https://en.wikipedia.org/wiki/Well-known_URIs).
 
          rewrites:
 
            - '^/\.well-known/carddav /remote.php/dav/ permanent'
 
            - '^/\.well-known/caldav /remote.php/dav/ permanent'
 
            - '^/remote/(.*) /remote.php last'
 

	
 
          # Prevent specific files from ever being served by the web server (for security reasons etc).
 
          deny_files_regex:
 
            - '^/(build|tests|config|lib|3rdparty|templates|data)/'
 
            - '^/(?:\.|autotest|occ|issue|indie|db_|console)'
 

	
 
          # Custom regex defining what files shouled be processed via PHP
 
          # interpreter.
 
          php_file_regex: \.php(?:$|/)
 

	
 
          # Not necessarily needed, but in case you have a policy on uid/gid
 
          # usage, this is useful. Take note that the uid value is also used
 
          # for the application group (gid == uid).
 
          uid: 2000
 
          admin_uid: 3000
 
        # And this role sets up a new dedicated database for our web
 

	
 
        # Role that sets up a new dedicated database for our web
 
        # application.
 
        - role: database
 

	
 
          # This is both the database name, _and_ name of the database user
 
          # that will be granted full privileges on the database.
 
          db_name: tbg
 
          # This will be the password of our user 'tbg' for accessing the
 
          # database. Take note the user can only login from localhost.
 
          db_password: tbg
 
          db_name: nextcloud
 

	
 
          # Password for user used for accessing the database. Take note
 
          # that the user can only login from localhost.
 
          db_password: nextcloud
 

	
 

	
 
3. Now for my favourite part again - creating private keys and certificates!
 
   Why?  Because the ``php_website`` role requires a private key/certificate
 
   pair to be deployed. So... Moving on:
 

	
 
3. Now for my favourite part again - creating private keys and
 
   certificates!  Why?  Because the ``php_website`` role requires a
 
   private key/certificate pair to be deployed. So... Moving on:
 

	
 
   1. Create new template for ``certtool``:
 

	
 
      :file:`~/mysite/tls/tbg.example.com_https.cfg`
 
      :file:`~/mysite/tls/nextcloud.example.com_https.cfg`
 
      ::
 

	
 
         organization = "Example Inc."
 
         country = SE
 
         cn = "Exampe Inc. Issue Tracker"
 
         cn = "Example Inc. Cloud Service"
 
         expiration_days = 365
 
         dns_name = "tbg.example.com"
 
         dns_name = "nextcloud.example.com"
 
         tls_www_server
 
         signing_key
 
         encryption_key
 

	
 
   2. Create the keys and certificates for the application::
 

	
 
        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/tbg.example.com_https.key
 
        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/tbg.example.com_https.cfg --load-privkey ~/mysite/tls/tbg.example.com_https.key --outfile ~/mysite/tls/tbg.example.com_https.pem
 
        certtool --sec-param normal --generate-privkey --outfile ~/mysite/tls/nextcloud.example.com_https.key
 
        certtool --generate-certificate --load-ca-privkey ~/mysite/tls/ca.key --load-ca-certificate ~/mysite/tls/ca.pem --template ~/mysite/tls/nextcloud.example.com_https.cfg --load-privkey ~/mysite/tls/nextcloud.example.com_https.key --outfile ~/mysite/tls/nextcloud.example.com_https.pem
 

	
 
4. Time to get our hands a bit more dirty... Up until now we didn't have to write
 
   custom tasks, but at this point we need to.
 
4. Time to get our hands a bit more dirty... Up until now we didn't
 
   have to write custom tasks, but that ends now.
 

	
 
   :file:`~/mysite/roles/tbg/tasks/main.yml`
 
   :file:`~/mysite/roles/nextcloud/tasks/main.yml`
 
   ::
 

	
 
      ---
 

	
 
      - name: Define TBG version
 
        set_fact:
 
          tbg_version: "4.3.1"
 
          tbg_archive_checksum: "45de72b1ef82142ad46686577d593375ba370156df4367d17386b4e26a37f342"
 
      # Deployment
 
      # ==========
 

	
 
      - name: Download the TBG archive
 
      - name: Download the application archive
 
        get_url:
 
          url: "https://github.com/thebuggenie/thebuggenie/archive/v{{ tbg_version }}.tar.gz"
 
          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"
 
          sha256sum: "{{ tbg_archive_checksum }}"
 
          url: "https://download.nextcloud.com/server/releases/nextcloud-20.0.14.tar.bz2"
 
          dest: "/var/www/nextcloud.example.com/nextcloud-20.0.14.tar.gz"
 
          sha256sum: "09652c459c99e0b77dbd91603151715c6cd09e4f4abbdecb64f495581af09367"
 
        become: yes
 
        become_user: admin-tbg_example_com
 
        become_user: admin-nextcloud_example_com
 

	
 
      - name: Download Composer
 
        get_url:
 
          url: "https://getcomposer.org/download/1.10.19/composer.phar"
 
          dest: "/usr/local/bin/composer"
 
          sha256sum: "688bf8f868643b420dded326614fcdf969572ac8ad7fbbb92c28a631157d39e8"
 
          owner: root
 
          group: root
 
          mode: 0755
 

	
 
      - name: Unpack TBG
 
      - name: Unpack the application archive
 
        unarchive:
 
          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}.tar.gz"
 
          dest: "/var/www/tbg.example.com/"
 
          src: "/var/www/nextcloud.example.com/nextcloud-20.0.14.tar.gz"
 
          dest: "/var/www/nextcloud.example.com/"
 
          copy: no
 
          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"
 
          creates: "/var/www/nextcloud.example.com/nextcloud"
 
        become: yes
 
        become_user: admin-tbg_example_com
 
        become_user: admin-nextcloud_example_com
 

	
 
      - name: Allow use of lib-pcre version 10 (since PHP is built against it in Debian Buster)
 
      # Majic Ansible Roles currently only support utf8 encoding.
 
      - name: Disable opportunistic use of utf8mb4 on fresh installs
 
        lineinfile:
 
          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/{{ item }}"
 
          state: present
 
          regexp: '.*"lib-pcre".*'
 
          line: '        "lib-pcre": ">=8.0",'
 
        with_items:
 
          - "composer.json"
 
          - "composer.lock"
 
          dest: "/var/www/nextcloud.example.com/nextcloud/lib/private/Setup/MySQL.php"
 
          line: "{{ '\t\t\t' }}$this->config->setValue('mysql.utf8mb4', true);"
 
          state: absent
 

	
 
      - name: Create directory for storing uploaded files
 
      - name: Allow application user to install and update applications
 
        file:
 
          path: "/var/www/tbg.example.com/files"
 
          state: directory
 
          mode: 02770
 
        become: yes
 
        become_user: admin-tbg_example_com
 
          path: "/var/www/nextcloud.example.com/nextcloud/apps"
 
          mode: g+w
 

	
 
      - name: Create symlink towards directory where uploaded files are stored
 
      - name: Allow CLI tool to be run by the user and group
 
        file:
 
          src: "/var/www/tbg.example.com/files"
 
          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/files"
 
          state: link
 
        become: yes
 
        become_user: admin-tbg_example_com
 
          path: "/var/www/nextcloud.example.com/nextcloud/occ"
 
          mode: u+x,g+x
 

	
 
      - name: Create TBG cache directory
 
      - name: Create directory for storing data
 
        file:
 
          path: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/cache"
 
          path: "/var/www/nextcloud.example.com/data"
 
          state: directory
 
          mode: 02770
 
        become: yes
 
        become_user: admin-tbg_example_com
 
          owner: "admin-nextcloud_example_com"
 
          group: "web-nextcloud_example_com"
 

	
 
      - name: Set-up the necessary write permissions for TBG directories
 
      - name: Create directory for storing configuration files
 
        file:
 
          path: "{{ item }}"
 
          mode: g+w
 
        with_items:
 
          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/
 
          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public/
 
          - /var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/
 
          path: "/var/www/nextcloud.example.com/nextcloud/config"
 
          state: directory
 
          mode: 02750
 
          owner: "admin-nextcloud_example_com"
 
          group: "web-nextcloud_example_com"
 

	
 
      - name: Create symbolic link to TBG application
 
      - name: Create an empty log file if it does not exist
 
        copy:
 
          content: ""
 
          dest: "/var/www/nextcloud.example.com/data/nextcloud.log"
 
          force: no
 

	
 
      - name: Set-up log file permissions
 
        file:
 
          src: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/public"
 
          path: "/var/www/tbg.example.com/htdocs"
 
          path: "/var/www/nextcloud.example.com/data/nextcloud.log"
 
          owner: "admin-nextcloud_example_com"
 
          group: "web-nextcloud_example_com"
 
          mode: 0660
 

	
 
      - name: Symlink the default path used by the web server for finding application files
 
        file:
 
          src: "/var/www/nextcloud.example.com/nextcloud"
 
          dest: "/var/www/nextcloud.example.com/htdocs"
 
          state: link
 
          owner: "admin-tbg_example_com"
 
          group: "web-tbg_example_com"
 
        become: yes
 
        become_user: admin-tbg_example_com
 
          owner: "admin-nextcloud_example_com"
 
          group: "web-nextcloud_example_com"
 
        notify:
 
          - Restart PHP-FPM
 

	
 
      - name: Install TBG dependencies
 
        composer:
 
          command: install
 
          working_dir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"
 
        become: yes
 
        become_user: admin-tbg_example_com
 

	
 
      - name: Deploy database configuration file for TBG
 
        copy:
 
          src: "b2db.yml"
 
          dest: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/core/config/b2db.yml"
 
          mode: 0640
 
          owner: admin-tbg_example_com
 
          group: web-tbg_example_com
 
      # Installation
 
      # ============
 

	
 
      - name: Get application installation status
 
        command: "/var/www/nextcloud.example.com/nextcloud/occ status"
 
        become: yes
 
        become_user: "admin-nextcloud_example_com"
 
        register: nextcloud_status
 
        changed_when: False
 
        failed_when: False
 

	
 
      - name: Install pexpect package
 
        apt:
 
          name: python3-pexpect
 
          state: present
 
      - name: Check if application is installed
 
        set_fact:
 
          nextcloud_installed: "{{ 'Nextcloud is not installed' not in nextcloud_status.stdout }}"
 

	
 
      - name: Deploy expect script for installing TBG
 
      - name: Deploy installation script
 
        copy:
 
          src: "tbg_expect_install"
 
          dest: "/var/www/tbg.example.com/tbg_expect_install"
 
          mode: 0750
 
          owner: admin-tbg_example_com
 
          group: web-tbg_example_com
 

	
 
      - name: Run TBG installer via expect script
 
        command: /var/www/tbg.example.com/tbg_expect_install
 
        args:
 
          chdir: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}"
 
          creates: "/var/www/tbg.example.com/thebuggenie-{{ tbg_version }}/installed"
 
          src: "install_nextcloud.py"
 
          dest: "/var/www/nextcloud.example.com/install_nextcloud.py"
 
          owner: "admin-nextcloud_example_com"
 
          group: "web-nextcloud_example_com"
 
          mode: 0700
 
        when: "not nextcloud_installed"
 

	
 
      - name: Install application
 
        command: "/var/www/nextcloud.example.com/install_nextcloud.py"
 
        become: yes
 
        become_user: admin-tbg_example_com
 
        become_user: "admin-nextcloud_example_com"
 
        when: "not nextcloud_installed"
 

	
 
      - name: Remove installation script
 
        file:
 
          path: "/var/www/nextcloud.example.com/install_nextcloud.py"
 
          state: absent
 

	
 
5. Set-up the files that are deployed by our role.
 
      - name: Fix data file permissions for application user/group
 
        file:
 
          path: "/var/www/nextcloud.example.com/data"
 
          mode: g+w
 
          recurse: yes
 
          follow: no
 

	
 
   :file:`~/mysite/roles/tbg/files/b2db.yml`
 
   ::
 
      - name: Deploy local configuration overrides
 
        copy:
 
          src: "local.config.php"
 
          dest: "/var/www/nextcloud.example.com/nextcloud/config/local.config.php"
 
          owner: "admin-nextcloud_example_com"
 
          group: "web-nextcloud_example_com"
 
          mode: 0640
 

	
 
5. Set-up files that are deployed by the role.
 

	
 
      b2db:
 
          username: "tbg"
 
          password: "tbg"
 
          dsn: "mysql:host=localhost;dbname=tbg"
 
          tableprefix: ''
 
          cacheclass: '\thebuggenie\core\framework\Cache'
 
   :file:`~/mysite/roles/nextcloud/files/local.config.php`
 
   ::
 

	
 
   :file:`~/mysite/roles/tbg/files/tbg_expect_install`
 
      <?php
 
      $CONFIG = array (
 
        'config_is_read_only' => true,
 
        'instanceid' => 'suqw2cvca8sp',
 
        'trusted_domains' =>
 
          array (
 
            0 => 'nextcloud.example.com',
 
          ),
 
      );
 

	
 
   :file:`~/mysite/roles/nextcloud/files/install_nextcloud.py`
 
   ::
 

	
 
      #!/usr/bin/env python3
 

	
 
      import pexpect
 

	
 
      # Spawn the process.
 
      install_process = pexpect.spawnu('./tbg_cli', args = ["install",
 
                                                                        "--accept_license=yes",
 
                                                                        "--url_subdir=/",
 
                                                                        "--use_existing_db_info=yes",
 
                                                                        "--enable_all_modules=no",
 
                                                                        "--setup_htaccess=yes"])
 
      install_process = pexpect.spawnu('/var/www/nextcloud.example.com/nextcloud/occ',
 
                                       args = [ 'maintenance:install',
 
                                                '--database', 'mysql',
 
                                                '--database-name', 'nextcloud',
 
                                                '--database-user', 'nextcloud',
 
                                                '--database-host', 'localhost',
 
                                                '--database-port', '3306',
 
                                                '--admin-user', 'admin',
 
                                                '--data-dir', '/var/www/nextcloud.example.com/data'])
 

	
 
      # If we get EOF, we probably already installed application, and ran
 
      # into error at the end since no patterns matched.
 
      try:
 
          # First confirmation.
 
          install_process.expect(u'Press ENTER to continue with the installation: ', timeout=5)
 
          install_process.sendline(u'')
 
          # Second confirmation.
 
          install_process.expect(u'Press ENTER to continue: ', timeout=5)
 
          install_process.sendline(u'')
 
          # Provide database password.
 
          install_process.expect(u'What is the password to access the database with user.*\?', timeout=10)
 
          install_process.sendline(u'nextcloud')
 

	
 
          # Provide administrator password.
 
          install_process.expect(u'What is the password you like to use for the admin account.*\?', timeout=10)
 
          install_process.sendline(u'admin')
 

	
 
          # Wait for application to finish.
 
          install_process.expect(pexpect.EOF, timeout=60)
 
          install_process.expect(pexpect.EOF, timeout=120)
 

	
 
      except pexpect.EOF as e:
 
          pass
 

	
 
      # Close application.
 
      # Print command output. Has to be done prior to final wait for
 
      # pexpect.EOF.
 
      print(install_process.before.encode('utf-8'))
 

	
 
      # Close application. Additional wait for pexpect.EOF prevents the
 
      # process from getting killed prematurely in case it exits
 
      # immediatelly (due to wrong command line arguments etc). Some
 
      # background information can be found at (although it is a very old
 
      # post):
 
      #
 
      # https://www.heikkitoivonen.net/blog/2009/01/28/pexpect-and-inconsistent-exit-status/
 
      #
 
      install_process.expect(pexpect.EOF)
 
      install_process.close()
 

	
 
      # Print text output.
 
      print(install_process.before)
 

	
 
      # Return same exit code like child process.
 
      exit(install_process.exitstatus)
 

	
 
6. And... Let's add the new role to our web server.
 

	
 
   :file:`~/mysite/playbooks/web.yml`
 
@@ -1711,21 +1762,21 @@ Before we start, here is a couple of useful pointers regarding the
 
        roles:
 
          - common
 
          - ldap_client
 
          - mail_forwarder
 
          - web_server
 
          - database_server
 
          - tbg
 
          - nextcloud
 

	
 
7. Apply the changes::
 

	
 
     workon mysite && ansible-playbook playbooks/site.yml
 

	
 
8. At this point *The Bug Genie* has been installed, and you should be able to
 
   open the URL https://tbg.example.com/ and log-in into *The Bug Genie*
 
   with username ``administrator`` and password ``admin``.
 
8. At this point *Nextcloud* has been installed, and you should be
 
   able to open the URL https://nextcloud.example.com/ and log-in into
 
   *Nextcloud* with username ``admin`` and password ``admin``.
 

	
 

	
 
Deploying a WSGI application (Django Wiki)
 
------------------------------------------
 

	
 
Next thing up will be to deploy a WSGI Python application.
 
@@ -2141,13 +2192,13 @@ on the safe side:
 
        roles:
 
          - common
 
          - ldap_client
 
          - mail_forwarder
 
          - web_server
 
          - database_server
 
          - tbg
 
          - nextcloud
 
          - wiki
 

	
 
7. Apply the changes:
 

	
 
   ::
 

	
 
@@ -2370,82 +2421,82 @@ So, back to the business:
 
             <http://duply.net/>`_.
 

	
 

	
 
Adding backup support to custom roles
 
-------------------------------------
 

	
 
As mentioned before, all of the supplied roles coming with *Majic Ansible Roles*
 
include backup support. What gets backed-up depends on the role implementation
 
(see role reference for details). What about backup support for custom roles?
 
This is something that has to be done by hand. However, it is quite simple to do
 
so.
 

	
 
Backup integration will be demonstrated with the previously implemented
 
``tbg`` role.
 

	
 
*The Bug Genie* stores most of its data in database, but thanks to the
 
``database`` role its backup is already handled for us. As a side-note, just
 
before every backup run the database is dumped and stored in location
 
``/srv/backup/tbg.sql``. That file is subsequently backed-up via *Duply* run.
 

	
 
What is not backed-up for us, though, are the files uploaded to *The Bug
 
Genie*. So let's fix that one.
 

	
 
1. Add the ``backup`` role to list of dependencies. Take note that while the
 
   ``backup_client`` role deals with basic set-up of backup client and its
 
   configuration, the ``backup`` role is used to define what should be
 
   backed-up. It is important to define unique filename for the backup patterns
 
   file. Take into account that you can use pretty much any globbing pattern
 
   supported by Duplicity.
 
As mentioned before, all of the supplied roles coming with *Majic
 
Ansible Roles* include backup support. What gets backed-up depends on
 
the role implementation (see role reference for details). What about
 
backup support for custom roles?  This is something that has to be
 
done by hand. However, it is quite simple to do so.
 

	
 
Backup integration will be demonstrated with the previously
 
implemented ``nextcloud`` role.
 

	
 
*Nextcloud* stores most of its data on the filesystem, but also in the
 
database. Thanks to the ``database`` role, though, database backup is
 
already handled for us. As a side-note, just before every backup run
 
the database is dumped and stored in location
 
``/srv/backup/nextcloud.sql``. That file is subsequently backed-up via
 
*Duply* run.
 

	
 
What is not backed-up for us, though, are the uploaded files
 
themselves, as well as configuration (which is generated during
 
application installation). So let's fix that one.
 

	
 
1. Add the ``backup`` role to list of dependencies. Take note that
 
   while the ``backup_client`` role deals with basic set-up of backup
 
   client and its configuration, the ``backup`` role is used to define
 
   what should be backed-up. It is important to define unique filename
 
   for the backup patterns file. Take into account that you can use
 
   pretty much any globbing pattern supported by Duplicity.
 

	
 
   .. warning::
 

	
 
      Make sure the addition is properly aligned in the yaml file to previous
 
      role dependency definitions.
 
      Make sure the addition is properly aligned in the yaml file to
 
      previous role dependency definitions.
 

	
 
   :file:`~/mysite/roles/tbg/meta/main.yml`
 
   :file:`~/mysite/roles/nextcloud/meta/main.yml`
 

	
 
   .. Small workaround for Sphinx not preserving leading spaces in
 
      case all lines have the same amount of leading spaces.
 

	
 
   .. code-block:: none
 
      :name: sphinx_workaround
 

	
 
        - role: backup
 
          when: enable_backup
 
          backup_patterns_filename: "tbg"
 
          backup_patterns_filename: "nextcloud"
 
          backup_patterns:
 
            - "/var/www/tbg.example.com/files"
 
            - "/var/www/nextcloud.example.com/data"
 
            - "/var/www/nextcloud.example.com/nextcloud/config"
 

	
 
2. Apply the changes::
 

	
 
     workon mysite && ansible-playbook playbooks/site.yml
 

	
 
3. Now rerun the backup on server ``www.example.com`` (as root). If
 
   you haven't uploaded any files, you may want to do so before
 
   testing to make sure something is backed-up. This will require
 
   enabling file uploads in `The Bug Genie settings
 
   <https://tbg.example.com/configure/files>`_, creating a test
 
   project, and then adding a new project release (via project's
 
   release center). While creating a new project release, it is
 
   possible to upload a release file.
 
   you haven't uploaded any files to Nextcloud, you may want to do so
 
   before testing to make sure something is backed-up.
 

	
 
   ::
 

	
 
     duply main backup
 

	
 
4. Verify that the files have been backed-up:
 

	
 
   ::
 

	
 
      duply main list
 

	
 
.. note:: If you wanted to run a script prior to backup run, you would simply
 
          deploy a shell script with desired content to
 
          ``/etc/duply/main/pre.d/``. Just make sure the permissions for it are
 
          ok (it has to be executable by the root user).
 
.. note::
 
   If you wanted to run a script prior to backup run, you would simply
 
   deploy a shell script with desired content to
 
   ``/etc/duply/main/pre.d/``. Just make sure the permissions for it
 
   are ok (it has to be executable by the root user).
 

	
 

	
 
Dealing with failures
 
---------------------
 

	
 
While the roles have been designed to be fairly robust, it should be taken into
0 comments (0 inline, 0 general)