Changeset - 62adfbb316a3
[Not reviewed]
0 3 0
Branko Majic (branko) - 5 years ago 2021-01-13 22:15:13
branko@majic.rs
MAR-151: Added support for Debian 10 Buster to database_server role:

- Updated role reference documentaiton.
- Updated role meta information.
- Updated tests.
- Deploy MariaDB client login configuration prior to setting-up
users/passwords for deprecated feature testing to avoid errors when
password gets changed in the middle of a task loop.
3 files changed with 23 insertions and 8 deletions:
0 comments (0 inline, 0 general)
docs/rolereference.rst
Show inline comments
 
@@ -1661,768 +1661,769 @@ Distribution compatibility
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up two (base) PHP websites (for
 
running *ownCloud* and *The Bug Genie* applications):
 

	
 
.. code-block:: yaml
 

	
 
    - role: php_website
 
      fqdn: cloud.example.com
 
      uid: 2001
 
      php_file_regex: \.php($|/)
 
      rewrites:
 
        - ^/\.well-known/host-meta /public.php?service=host-meta
 
        - ^/\.well-known/host-meta\.json /public.php?service=host-meta-json
 
        - ^/\.well-known/carddav /remote.php/carddav/ redirect
 
        - ^/\.well-known/caldav /remote.php/caldav/ redirect
 
        - ^/apps/calendar/caldav\.php /remote.php/caldav/
 
        - ^/apps/contacts/carddav\.php /remote.php/carddav/
 
        - ^/remote/(.*) /remote.php
 
      deny_files_regex:
 
        - ^(\.|autotest|occ|issue|indie|db_|console|build/|tests/|config/|lib/|3rdparty/|templates/).*
 
      packages:
 
        # For ownCloud
 
        - php5-gd
 
        - php5-json
 
        - php5-mysql
 
        - php5-curl
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/cloud.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/cloud.example.com_https.pem') }}"
 
      additional_nginx_config:
 
        - comment: Use custom page for forbidden files.
 
          value: error_page 403 /core/templates/403.php;
 
        - comment: Use custom page for non-existing locations/files.
 
          value: error_page 404 /core/templates/404.php;
 
      additional_fpm_config:
 
        "env[PATH]": "\"/usr/local/bin:/usr/bin:/bin\""
 
      website_mail_recipients: "root john.doe@example.com"
 
      environment_indicator:
 
        background_colour: "green"
 
        text_colour: "black"
 
        text: "TEST ENVIRONMENT"
 
    - role: php_website
 
      deny_files_regex:
 
        - ^\..*
 
      php_rewrite_urls:
 
        - ^(.*) /index.php?url=$1
 
      fqdn: tbg.example.com
 
      uid: 2007
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/tbg.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/tbg.example.com_https.pem') }}"
 

	
 

	
 
WSGI Website
 
------------
 

	
 
The ``wsgi_website`` role can be used for setting-up a website powered by Python
 
on destination machine. The website needs to use the WSGI specification for
 
making the Python web application(s) available.
 

	
 
This role is normally not supposed to be used directly, but should instead serve
 
as the basis for writing website-specific roles. Therefore the role is written
 
in quite generic way, allowing the integrator to write his/her own logic for
 
deploying the necessary Python applications/packages, while still reusing a
 
common base and reducing the workload.
 

	
 
The role implements the following:
 

	
 
* Creates a dedicated user/group for running the WSGI application.
 
* Creates a dedicated administrator user for maintaining the website.
 
* Creates a base directory where the website-specific code and data should be
 
  stored at.
 
* Adds nginx to website's group, so nginx could read the necessary files.
 
* Adds website administrator to website's group, so administrator could manage
 
  the code and data.
 
* Installs additional packages required for running the role (as configured).
 
* Sets-up a dedicated Python virtual environment for website. Python
 
  version can be specified (default is Python 2).
 
* Install ``futures`` package in Python virtual environment (required for
 
  Gunicorn in combination with Python 2.7).
 
* Install Gunicorn in Python virtual environment.
 
* Installs additional packages required for running the role in Python virtual
 
  environment (as configured).
 
* Configures systemd to run the website code (using Gunicorn)
 
* Deploys the HTTPS TLS private key and certificate (for website vhost).
 
* Configures nginx to serve the website (static files served directly, requests
 
  passed on to Gunicorn).
 

	
 
The role is implemented with the following layout/logic in mind:
 

	
 
* No plaintext HTTP is allowed, HTTPS is mandatory. Clients connecting
 
  via plaintext HTTP are redirected to HTTPS.
 
* Clients are served with ``Strict-Transport-Security`` header with
 
  value of ``max-age=31536000; includeSubDomains``. This forces
 
  compliant clients to always connect using HTTPS to the web server
 
  when accessing its domain, as well as any subdomains served
 
  by this web server or any other. The (client-side) cached header
 
  value expires after one year.
 
* Website users are named after the ``FQDN`` (fully qualified domain name) of
 
  website, in format of ``web-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal to
 
  ``FQDN`` where dots have been replaced by underscores (for example,
 
  ``web-wiki_example_com``).
 
* Website users are set-up via GECOS field to have their umask set to ``0007``
 
  (in combination with ``pam_umask``).
 
* Administrator users are named after the ``FQDN`` (fully qualified domain name)
 
  of website, in format of ``admin-ESCAPEDFQDN``, where ``ESCAPEDFQDN`` is equal
 
  to ``FQDN`` where dots have been replaced by underscores (for example,
 
  ``admin-cloud_example_com``).
 
* All websites reside within a dedicated sub-directory in ``/var/www``. The
 
  sub-directory name is equal to the ``FQDN`` used for accessing the
 
  website. Owner of the directory is set to be the application administrator,
 
  while group is set to be the website group. Additionally, ``SGID`` bit is set
 
  on the directory. This allows admin, with correct umask, to create necessary
 
  files and directories that should be readable (and eventually writeable) by
 
  the website user (running the WSGI application) without having to become root.
 
* All files placed in the website directory should be either created there
 
  directly, or copied to the directory in order to make sure the ``SGID`` gets
 
  honored. **Do not move the files, the permissions will not be set correctly.**
 
* Within the website directory, Python virtual environment can be found within
 
  the ``virtualenv`` sub-directory. Switching to administrator user via login
 
  shell will automatically activate the virtual environment.
 
* Within the website directory, nginx will expect to find the static files
 
  within the ``htdocs`` sub-directory (this can be symlink too). Locations/aliases
 
  can be configured for static file serving.
 
* Within the website directory, systemd service will expect to find the website
 
  code within the ``code`` sub-directory (this can be symlink too).
 
* nginx communicates with WSGI server over a dedicated Unix socket for each
 
  website.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 
* **web_server**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**additional_nginx_config** (list, optional, ``[]``)
 
  List providing additional Nginx configuration options to include. This can be
 
  useful for specifying things like error pages. Options are applied inside of a
 
  **server** context of Nginx configuration file.
 

	
 
  Each item is a dictionary with the following options describing the extra
 
  configuration option:
 

	
 
  **comment** (string, mandatory)
 
    Comment describing the configuration option.
 

	
 
  **value** (string, mandatory)
 
    Configuration option.
 

	
 
**admin_uid** (integer, optional, ``whatever OS picks``)
 
  UID of the dedicated website administrator user. The user will be member of
 
  website group.
 

	
 
**environment_indicator** (dictionary, optional, ``null``)
 
  Specify configuration for including environment indicator on all HTML
 
  pages. Indicator is a simple strip at bottom of a page with custom background
 
  colour, text colour, and text.
 

	
 
  Specifying environment indicator is useful for avoiding mistakes when testing
 
  by having better visibility what environment you are in
 
  (production/staging/test).
 

	
 
  The following keys need to be specified:
 

	
 
  **background_colour** (string, mandatory)
 
    Background colour to use for the strip at bottom. This should be value
 
    compatible with CSS ``background-color`` attribute.
 

	
 
  **text_colour** (string, mandatory
 
    Text colour to use for the strip at bottom. This should be value compatible
 
    with CSS ``color`` attribute.
 

	
 
  **text** (string, mandatory)
 
    Text to show in show in the strip at bottom.
 

	
 
**environment_variables** (dict, optional, ``{}``)
 
  Specify additional environment variables that should be set for running the
 
  service. Environment variables will be set in both the systemd service and for
 
  the application's administrator user (when logged in as one).
 

	
 
**fqdn** (string, mandatory)
 
  Fully-qualified domain name where the website is reachable. This value is used
 
  for calculating the user/group name for dedicated website user, as well as
 
  home directory of the website user (where data/code should be stored at).
 

	
 
**https_tls_certificate** (string, mandatory)
 
  X.509 certificate used for TLS for HTTPS service. The file will be stored in
 
  directory ``/etc/ssl/certs/`` under name ``{{ fqdn }}_https.pem``.
 

	
 
**https_tls_key** (string, mandatory)
 
  Private key used for TLS for HTTPS service. The file will be stored in
 
  directory ``/etc/ssl/private/`` under name ``{{ fqdn }}_https.key``.
 

	
 
**packages** (list, optional, ``[]``)
 
  A list of additional packages to install for this particular WSGI
 
  website. This is usually going to be development libraries for building Python
 
  packages.
 

	
 
**proxy_headers** (dictionary, optional, ``{}``)
 
  Additional headers to set when proxying request to Gunicorn. Keys are header
 
  names, values are header values. Both should be compatible with Nginx
 
  ``proxy_set_header``. If you need to provide an empty value, use quotes (don't
 
  forget to surround them by another set of quotes for YAML syntax, for example
 
  ``"\"\""`` or ``'""'``).
 

	
 
**python_version** (string, optional, ``2``)
 
  Python version to use when setting-up the Python virtual environment.
 

	
 
**rewrites** (list, optional, ``[]``)
 
  A list of rewrite rules that are applied to incoming requests. Each element of
 
  the list should be a string value compatible with the format of ``nginx``
 
  option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well
 
  as trailing semi-colon (``;``).
 

	
 
**static_locations** (list, optional, ``[]``)
 
  List of locations that should be treated as static-only, and not processed by
 
  the WSGI application at all. This is normally used for designating serving of
 
  static/media files by Nginx (for example, in case of Django projects for
 
  ``/static/`` and ``/media/``).
 

	
 
**uid** (integer, optional, ``whatever OS picks``)
 
  UID/GID (they are set-up to be the same) of the dedicated website
 
  user/group.
 

	
 
**use_paste** (boolean, optional, ``False``)
 
  Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a
 
  filename of a Python Paste ``ini`` file instead of WSGI application.
 

	
 
**virtuaelnv_packages** (list, optional, ``[]``)
 
  A list of additional packages to install for this particular WSGI appliction
 
  in its virtual environment using ``pip``.
 

	
 
**website_mail_recipients** (string, optional, ``root``)
 
  Space-separated list of e-mails or local users to which the mails, sent to
 
  either the website admin or website user, should be forwarded to. Forwarding
 
  is configured via ``~/.forward`` configuration file.
 

	
 
**wsgi_application** (string, mandatory)
 
  WSGI application that should be started by Gunicorn. The format should be
 
  conformant to what the ``gunicorn`` command-line tool accepts. If the
 
  ``use_paste`` option is enabled, the value should be equal to filename of the
 
  Python Paste ini file, located in the ``code`` sub-directory. It should be
 
  noted that in either case the value should be specsified relative to the
 
  ``code`` sub-directory. I.e. don't use full paths.
 

	
 
**wsgi_requirements** (list, optional, ``[ futures==3.3.0, gunicorn==19.10.0 ]``)
 
  Complete list of pip requirements used for deploying Gunicorn. If
 
  specified, this list will be used to create requirements file and
 
  install Gunicorn and its dependencies from that one. This allows to
 
  have pinned packages for both Gunicorn, futures, and their
 
  dependencies. The ``futures`` package is required by Gunicorn when
 
  using Python 2.7.
 

	
 
  It should be noted that this installation method is meant primarily in case of
 
  roles that want to take advantage of upgrade checks for pip requirements
 
  files, and that employ `pip-tools <https://github.com/jazzband/pip-tools>`_.
 

	
 
  In addition to change of installation method, when this parameter is specified
 
  the role will deploy necessary files for running the pip requirements upgrade
 
  check (see the ``common`` role for description). For this a directory is
 
  created under ``/etc/pip_check_requirements_upgrades/FQDN``. The same
 
  directory should be used by dependant roles to deploy their own ``.in`` and
 
  ``.txt`` files. Make sure the file ownership is set to ``root:pipreqcheck``.
 

	
 
  Should you need to utilise the requirements file in some manner (other than
 
  checking for its upgrades), it will be also stored (and made accessible to
 
  application user/admin)) in application's home directory under the name
 
  ``.wsgi_requirements.txt``.
 

	
 
  To create complete requirements list, it is recommended to use `pip-tools
 
  <https://github.com/jazzband/pip-tools>`_ (the ``pip-compile`` utility) with
 
  ``gunicorn`` and ``futures`` in the ``.in.`` file.
 

	
 
**wsgi_requirements_in** (list, optional, ``[ futures, gunicorn ]``)
 
  List of top level packages to use when performing the pip
 
  requirements upgrade checks for the Gunicorn requirements (listed
 
  via ``wsgi_requirements`` parameter). For Python 3-based websites,
 
  it should be sufficient to list only ``gunicorn`` (``futures`` is
 
  required for Python 2).
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up a (base) WSGI website (for
 
running a bare Django project):
 

	
 
.. code-block:: yaml
 

	
 
    # Sample for a Django installation.
 
    - role: wsgi_website
 
      fqdn: django.example.com
 
      static_locations:
 
        - /static
 
        - /media
 
      uid: 2004
 
      virtualenv_packages:
 
        - django
 
      wsgi_application: django_example_com.wsgi:application
 
      environment_variables:
 
        DJANGO_SETTINGS_MODULE: "django_example_com.settings.production"
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.pem') }}"
 
      additional_nginx_config:
 
        - comment: Use custom page for forbidden files.
 
          value: error_page 403 /static/403.html;
 
        - comment: Use custom page for non-existing locations/files.
 
          value: error_page 404 /static/404.html;
 
      website_mail_recipients: "root john.doe@example.com"
 
      environment_indicator:
 
        background_colour: "green"
 
        text_colour: "black"
 
        text: "TEST ENVIRONMENT"
 
      proxy_headers:
 
        Accept-Encoding: '""'
 

	
 
    # Use wsgi_requirements to deploy Gunicorn.
 
    - role: wsgi_website
 
      fqdn: wsgi.example.com
 
      wsgi_application: wsgi:main
 
      wsgi_requirements:
 
        - gunicorn==19.7.1
 
	- futures==3.1.1
 

	
 

	
 
Database Server
 
---------------
 

	
 
The ``database_server`` role can be used for setting-up a MariaDB database
 
server on destination machine.
 

	
 
The role implements the following:
 

	
 
* Installs MariaDB server and client.
 
* Configures MariaDB server and client to use *UTF-8* encoding by default.
 
* Sets-up the database root user for passwordless login via UNIX
 
  socket authentication.
 
* Drops the ``debian-sys-maint`` database user (which was used in
 
  Debian Jessie and earlier for maintenance tasks) if it is present,
 
  and updates the Debian system maintenance configuration file to use
 
  the root account over unix socket authentication.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
This role has no parameters.
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 
- Debian 10 (Buster)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
This role has no parameters which can be configured configure.
 

	
 

	
 
Database
 
--------
 

	
 
The ``database`` role can be used for creating a MariaDB database and
 
accompanying user on destination machine.
 

	
 
The role implements the following:
 

	
 
* Creates MariaDB database.
 
* Creates a dedicated user capable of performing any operation on the created
 
  database. Username is set to be same as the name of database.
 
* Sets-up pre-backup task that creates database dump in location
 
  ``/srv/backup/mariadb/{{ db_name }}.sql``.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **database_server**
 
* **backup_client**
 

	
 

	
 
Backups
 
~~~~~~~
 

	
 
If the backup for this role has been enabled, the following paths are backed-up:
 

	
 
**/srv/backup/maraidb/{{ db_name }}.sql**
 
  Dump of the database. Database dump is created every day at 01:45 in the
 
  morning.
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**db_name** (string, mandatory)
 
  Name of the database that should be created.
 

	
 
**db_password** (string, mandatory)
 
  Password for the database user.
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for creating a single database (for some
 
website):
 

	
 
.. code-block:: yaml
 

	
 
  - role: database
 
    db_name: phpinfo_example_com
 
    db_password: phpinfo_example_com
 

	
 

	
 
Backup Server
 
-------------
 

	
 
The ``backup_server`` role can be used for setting-up a server to act as backup
 
storage for the backup clients. Storage is made available to the clients
 
exclusively via SFTP on a dedicated port and dedicated OpenSSH server
 
instance. This instance is specifically configured and tailored for this
 
purpose.
 

	
 
The role is primarily aimed for use with `Duplicity
 
<http://duplicity.nongnu.org/>`_, but should be also usable for generic SFTP
 
uploads.
 

	
 
The role implements the following:
 

	
 
* Installs backup software (Duplicity, Duply).
 
* Creates a dedicated directory structure for backups with the following structure:
 

	
 
  * ``/srv/backups/`` - main directory under which all the backups reside.
 
  * ``/srv/backups/SERVER_NAME/`` - home directory for the backup user, name
 
    after the server. Backup users are confined to their respective home
 
    directory via chroot. Backup users can't write to their own home directory,
 
    though.
 
  * ``/srv/backups/SERVER_NAME/duplicity/`` - directory where the Duplicity
 
    backups are stored at. This directory is writable by the respective backup
 
    user.
 
  * ``SERVER_NAME/.ssh/`` - directory where authorized keys are stored. Backup
 
    user is not allowed to make modifications to this directory and files
 
    contained within (i.e. backup users can't add more keys to the
 
    ``authorized_keys`` file).
 
* Creates dedicated operating system users for backup clients. These users will
 
  be made members of the ``backup`` group as well (as an additional group).
 
* Sets-up ``authorized_keys`` for the backup clients.
 
* Makes sure the backup users can't log-in via regular OpenSSH server instance.
 
* Sets-up dedicated OpenSSH server instances to be used exclusively by backup
 
  clients. The instance listens on TCP port ``2222``.
 
* Updates firewall to allow incoming TCP connections to port
 
  ``2222``. Connections are allowed only from the configured IP addresses
 
  associated with backup clients.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**backup_clients** (list, optional)
 
  List of backup clients that are connecting to the backup server. This is
 
  usually done on a per-server basis. Each item in the list is a dictionary
 
  describing the backup client. The following keys are available:
 

	
 
  **server** (string, mandatory)
 
    Name of the server that is backed up. It is highly recommended to use
 
    server's FQDN for this purpose. The dedicated operating system user created
 
    will have the name of format ``bak-ESCAPED_SERVER_NAME``, where
 
    ``ESCAPED_SERVER_NAME`` is calculated by taking the passed-in server name
 
    and replacing all dots (``.``) with undescores (``_``). For example,
 
    ``web.example.com`` will be turned into ``bak-web_example_com``.
 

	
 
  **uid** (integer, optional, ``whatever OS picks``)
 
    Uid for the operating system user. User's default group will have a GID
 
    identical to the user's UID if specified. Otherwise user's default group
 
    will have OS-determined GID.
 

	
 
  **ip** (IPv4 address, mandatory)
 
    IPv4 address from which the backup client server is connecting to the backup
 
    server. Used for introducing stricter firewall rules.
 

	
 
  **public_key** (string, mandatory)
 
    SSH public key used by backup client to connect to the backup server.
 

	
 
**backup_host_ssh_private_keys** (dictionary, mandatory)
 
  Defines host keys used for the dedicated OpenSSH server instance for
 
  backup. Key values that must be provided are:
 

	
 
  - **rsa**
 
  - **ed25519**
 
  - **ecdsa**
 

	
 
  Values for each key should be the corresponding private key
 
  generated using the appropriate algorithm. Keys for this purpose can
 
  be easily created via commands::
 

	
 
    ssh-keygen -f backup_server_rsa_key -N '' -t rsa
 
    ssh-keygen -f backup_server_ed25519_key -N '' -t ed25519
 
    ssh-keygen -f backup_server_ecdsa_key -N '' -t ecdsa
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 
- Debian 10 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up the backup server role:
 

	
 
.. code-block:: yaml
 

	
 
  - role: backup_server
 
    backup_clients:
 
      - server: web.example.com
 
        uid: 3000
 
        public_key: "{{ lookup('file', inventory_dir + '/ssh/web.example.com.pub') }}"
 
        ip: 10.32.64.18
 
      - server: mail.example.com
 
        public_key: "{{ lookup('file', inventory_dir + '/ssh/mail.example.com.pub') }}"
 
        ip: 10.32.64.15
 
    backup_host_ssh_private_keys:
 
      rsa: "{{ lookup('file', inventory_dir + '/ssh/backup_server_rsa_key') }}"
 
      ed25519: "{{ lookup('file', inventory_dir + '/ssh/backup_server_ed25519_key') }}"
 
      ecdsa: "{{ lookup('file', inventory_dir + '/ssh/backup_server_ecdsa_key') }}"
 

	
 

	
 
Backup Client
 
-------------
 

	
 
The ``backup_client`` role can be used for setting-up the server as a backup
 
client so it can perform backups to the backup server.
 

	
 
Backup clients utilise duplicity (via the duply convenience wrapper) for
 
performing the backups to a backup server via *SFTP* protocol.
 

	
 
The role itself will take care of deploying the necessary software,
 
configuration files, and encryption/signing private key to the backup client in
 
order to be able to perform backup.
 

	
 
Files that should be backed-up are specified using the ``backup`` role.
 

	
 
The role implements the following:
 

	
 
* Installs backup software (Duplicity, Duply).
 
* Sets-up Duply configuration under directory ``/etc/duply/main/``.
 
* Deploys encryption/signing private key (usually host-specific), as well as
 
  additional encryption public keys to the server, and imports them into local
 
  GnuPG keyring used by backup software.
 
* Deploys private SSH key for logging-in into the backup server over SFTP.
 
* Deploys ``known_hosts`` file for SFTP fingerprint verification.
 
* Sets-up a handler that runs scripts/binaries before the actual backup
 
  run. This is helpful for producing database backups. Such scripts/binaries
 
  should be deployed to directory ``/etc/duply/main/pre.d/``, and marked
 
  executable by the root user.
 

	
 
Duply is configured as follows:
 

	
 
* GnuPG keyring is stored under ``/etc/duply/main/gnupg/``. The keyring should
 
  not be managed manually.
 
* SSH private key for logging-in into backup server is stored in location
 
  ``/etc/duply/main/ssh/identity``. Backup server SFTP fingerprint is stored in
 
  location ``/etc/duply/main/ssh/known_hosts``.
 
* Base directory for back-ups is root (``/``), but *all* files are excluded by
 
  default to prevent huge back-ups. Ansible roles that want to utilise the
 
  backup client role can specify which patterns should be included in the backup
 
  when including the ``backup`` role. Include pattern file is assembled and
 
  stored in location ``/etc/duply/main/include``.
 
* Backups are encrypted and signed with the specified encryption key.
 
* Maximum age for old backups is set to 6 months.
 
* Maximum age for full backups is set to 1 month.
 
* Volume size is set to 1GB.
 
* Pre-backup scripts are run via ``/etc/duply/main/pre`` handler that tries to
 
  execute scripts/binaries from directory ``/etc/duply/main/pre.d/``.
 

	
 
.. note::
 
   Since at time of this writing there are no lookup plugins for extracting key
 
   material/information from GnuPG keyring, you may want to resort to extraction
 
   of keys on the controller machine via lookups similar to::
 

	
 
     lookup('pipe', 'gpg2 --homedir /path/to/your/keyring --armor --export some_identifier')
 
     lookup('pipe', 'gpg2 --homedir /path/to/your/keyring --armor --export-secret-keys some_identifier')
 

	
 
   This may not be the most elegant solution, but for now it offers better
 
   flexibility (theoretically, you could store all those keys etc as plaintext
 
   files instead).
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**backup_additional_encryption_keys** (list, optional, ``[]``)
 
  List of additional public encryption keys used for backup operation. Each item
 
  in the list should be an ASCII armour-encoded public key exported from a GnuPG
 
  keyring. These additional public keys are useful in cases where the backups
 
  should be decryptable with some master key in addition to server-specific key.
 

	
 
**backup_client_username** (string, optional, ``bak-{{ ansible_fqdn | replace('.', '_') }}``)
 
  Username for connecting to the backup server via SFTP.
 

	
 
**backup_encryption_key** (string, mandatory)
 
  Private GnuPG key, encoded using ASCII armor, used for encryption and signing
 
  operations when running the backup on the client server. This *must* be a
 
  private key! This is normally host-specific encryption key that is distributed
 
  to destination server and that can be also used for the restore operations
 
  (for data decryption). The key must not be password-protected.
 

	
 
**backup_server** (string, mandatory)
 
  Backup server to connect to.
 

	
 
**backup_server_destination** (string, optional, ``/duplicity``)
 
  Target directory on the backup server where the backups are stored.
 

	
 
**backup_server_host_ssh_public_keys** (list, mandatory)
 
  SSH public keys presented by the server during client authentication. These
 
  public keys are used for populating the known hosts on the backup client side
 
  for host verification purposes.
 

	
 
**backup_server_port** (int, optional, ``2222``)
 
  Port on the backup server to connect to for accessing the SFTP service.
 

	
 
**backup_ssh_key** (string, mandatory)
 
  SSH private key for logging-in into the backup server.
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 
- Debian 10 (Buster)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up the role (take note that lookup
 
plugin is quite useful here for fetching key values from some local directory):
 

	
 
.. code-block:: yaml
 

	
 
  - role: backup_client
 
    backup_additional_encryption_keys: "-----BEGIN PGP PUBLIC KEY BLOCK-----\n...\n-----END PGP PUBLIC KEY BLOCK-----"
 
    backup_client_username: "user"
 
    backup_encryption_key: "-----BEGIN PGP PRIVATE KEY BLOCK-----\n...\n-----END PGP PRIVATE KEY BLOCK-----"
 
    backup_server: "backup.example.com"
 
    backup_server_destination: "//example/host"
 
    backup_server_host_ssh_public_keys:
 
      - "{{ lookup('file', inventory_dir + '/ssh/backup_server_ecdsa_key.pub') }}"
 
      - "{{ lookup('file', inventory_dir + '/ssh/backup_server_ed25519_key.pub') }}"
 
      - "{{ lookup('file', inventory_dir + '/ssh/backup_server_rsa_key.pub') }}"
 
    backup_server_port: 22
 
    backup_ssh_key: "{{ lookup('file', inventory_dir + '/ssh/web.example.com') }}"
 

	
 

	
 
Backup
 
------
 

	
 
The ``backup`` role can be used to specify what files should be backed-up to the
 
backup server.
 

	
 
The role provides a convenient way to deploy a file containing file and
 
directory patterns describing the file/directory paths that should be included
 
in the back-up.
 

	
 
The role implements the following:
 

	
 
* Installs a file with provided patterns in directory
 
  ``/etc/duply/main/patterns/``.
 
* Assembles/refresshes the main include pattern file at
 
  ``/etc/duply/main/include``.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **backup_client**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**backup_patterns_filename** (string, mandatory)
 
  Name of the backup patterns file. The file is stored in directory
 
  ``/etc/duply/main/patterns/``. This should be a unique filename amongst all
 
  roles. If role can be included multiple times, make sure the filename is
 
  always unique when depending on the backup role.
 

	
 
**backup_patterns** (list, optional, ``[]``)
 
  List of globbing patterns defining which files or directories should be
 
  backed-up.
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 
- Debian 10 (Buster)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up the role:
 

	
 
.. code-block:: yaml
 

	
 
  - role: backup
roles/database_server/molecule/default/molecule.yml
Show inline comments
 
---
 

	
 
dependency: {}
 

	
 
driver:
 
  name: vagrant
 
  provider:
 
    name: virtualbox
 

	
 
lint:
 
  name: yamllint
 
  options:
 
    config-file: ../../.yamllint.yml
 

	
 
platforms:
 

	
 
  - name: parameters-mandatory-stretch64
 
    groups:
 
      - parameters-mandatory
 
    box: debian/contrib-stretch64
 
    memory: 512
 
    cpus: 1
 

	
 
  - name: deprecated-stretch64
 
    groups:
 
      - deprecated
 
    box: debian/contrib-stretch64
 
    memory: 512
 
    cpus: 1
 

	
 
  - name: parameters-mandatory-buster64
 
    groups:
 
      - parameters-mandatory
 
    box: debian/contrib-buster64
 
    memory: 512
 
    cpus: 1
 

	
 
  - name: deprecated-buster64
 
    groups:
 
      - deprecated
 
    box: debian/contrib-buster64
 
    memory: 512
 
    cpus: 1
 

	
 
provisioner:
 
  name: ansible
 
  config_options:
 
    defaults:
 
      force_valid_group_names: "ignore"
 
      interpreter_python: "/usr/bin/python3"
 
    ssh_connection:
 
      pipelining: "True"
 
  lint:
 
    name: ansible-lint
 

	
 
scenario:
 
  name: default
 

	
 
verifier:
 
  name: testinfra
 
  lint:
 
    name: flake8
roles/database_server/molecule/default/prepare.yml
Show inline comments
 
---
 

	
 
- name: Prepare
 
  hosts: all
 
  gather_facts: false
 
  tasks:
 
    - name: Install python for Ansible
 
      raw: test -e /usr/bin/python3 || (apt -y update && apt install -y python3-minimal)
 
      become: true
 
      changed_when: false
 

	
 
- hosts: all
 
  become: true
 
  tasks:
 

	
 
    - name: Update all caches to avoid errors due to missing remote archives
 
      apt:
 
        update_cache: true
 
      changed_when: false
 

	
 
- hosts: deprecated
 
  become: true
 
  tasks:
 

	
 
    - name: Install MariaDB
 
      apt:
 
        name:
 
          - mariadb-client
 
          - mariadb-server
 
          - python3-mysqldb
 
        state: present
 

	
 
    - name: Enable and start MariaDB
 
      service:
 
        name: mysql
 
        state: started
 
        enabled: true
 

	
 
    - name: Deploy username and password for the root database user
 
      copy:
 
        src: "deprecated-root-my.cnf"
 
        dest: "/root/.my.cnf"
 
        owner: root
 
        group: root
 
        mode: 0400
 

	
 
    - name: Set password for the root database user (creating separate entry for different hosts)
 
      mysql_user:
 
        check_implicit_admin: true
 
        name: root
 
        host: "{{ item }}"
 
        password: "root_password"
 
      with_items:
 
        - "localhost"
 
        - "127.0.0.1"
 
        - "::1"
 
        - "{{ ansible_hostname }}"
 

	
 
    - name: Deploy username and password for the root database user
 
      copy:
 
        src: "deprecated-root-my.cnf"
 
        dest: "/root/.my.cnf"
 
        owner: root
 
        group: root
 
        mode: 0400
 

	
 
    - name: Disable use of unix socket login
 
      command: "mysql -B -e \"update mysql.user set plugin='' where user='root' and plugin='unix_socket'; flush privileges;\""
 

	
 
    - name: Create Debian system maintenance user
 
      mysql_user:
 
        name: debian-sys-maint
 
        password: debian-sys-maint-password
 

	
 
    - name: Deploy Debian system maintenance user login configuration
 
      copy:
 
        src: "deprecated-debian.cnf"
 
        dest: "/etc/mysql/debian.cnf"
 
        owner: root
 
        group: root
 
        mode: 0600
0 comments (0 inline, 0 general)