Changeset - 62adfbb316a3
[Not reviewed]
0 3 0
Branko Majic (branko) - 5 years ago 2021-01-13 22:15:13
branko@majic.rs
MAR-151: Added support for Debian 10 Buster to database_server role:

- Updated role reference documentaiton.
- Updated role meta information.
- Updated tests.
- Deploy MariaDB client login configuration prior to setting-up
users/passwords for deprecated feature testing to avoid errors when
password gets changed in the middle of a task loop.
3 files changed with 23 insertions and 8 deletions:
0 comments (0 inline, 0 general)
docs/rolereference.rst
Show inline comments
 
@@ -1853,384 +1853,385 @@ Parameters
 

	
 
**fqdn** (string, mandatory)
 
  Fully-qualified domain name where the website is reachable. This value is used
 
  for calculating the user/group name for dedicated website user, as well as
 
  home directory of the website user (where data/code should be stored at).
 

	
 
**https_tls_certificate** (string, mandatory)
 
  X.509 certificate used for TLS for HTTPS service. The file will be stored in
 
  directory ``/etc/ssl/certs/`` under name ``{{ fqdn }}_https.pem``.
 

	
 
**https_tls_key** (string, mandatory)
 
  Private key used for TLS for HTTPS service. The file will be stored in
 
  directory ``/etc/ssl/private/`` under name ``{{ fqdn }}_https.key``.
 

	
 
**packages** (list, optional, ``[]``)
 
  A list of additional packages to install for this particular WSGI
 
  website. This is usually going to be development libraries for building Python
 
  packages.
 

	
 
**proxy_headers** (dictionary, optional, ``{}``)
 
  Additional headers to set when proxying request to Gunicorn. Keys are header
 
  names, values are header values. Both should be compatible with Nginx
 
  ``proxy_set_header``. If you need to provide an empty value, use quotes (don't
 
  forget to surround them by another set of quotes for YAML syntax, for example
 
  ``"\"\""`` or ``'""'``).
 

	
 
**python_version** (string, optional, ``2``)
 
  Python version to use when setting-up the Python virtual environment.
 

	
 
**rewrites** (list, optional, ``[]``)
 
  A list of rewrite rules that are applied to incoming requests. Each element of
 
  the list should be a string value compatible with the format of ``nginx``
 
  option ``rewrite``. The keyword ``rewrite`` itself should be omitted, as well
 
  as trailing semi-colon (``;``).
 

	
 
**static_locations** (list, optional, ``[]``)
 
  List of locations that should be treated as static-only, and not processed by
 
  the WSGI application at all. This is normally used for designating serving of
 
  static/media files by Nginx (for example, in case of Django projects for
 
  ``/static/`` and ``/media/``).
 

	
 
**uid** (integer, optional, ``whatever OS picks``)
 
  UID/GID (they are set-up to be the same) of the dedicated website
 
  user/group.
 

	
 
**use_paste** (boolean, optional, ``False``)
 
  Tell Gunicorn to assume that the passed-in ``wsgi_application`` value is a
 
  filename of a Python Paste ``ini`` file instead of WSGI application.
 

	
 
**virtuaelnv_packages** (list, optional, ``[]``)
 
  A list of additional packages to install for this particular WSGI appliction
 
  in its virtual environment using ``pip``.
 

	
 
**website_mail_recipients** (string, optional, ``root``)
 
  Space-separated list of e-mails or local users to which the mails, sent to
 
  either the website admin or website user, should be forwarded to. Forwarding
 
  is configured via ``~/.forward`` configuration file.
 

	
 
**wsgi_application** (string, mandatory)
 
  WSGI application that should be started by Gunicorn. The format should be
 
  conformant to what the ``gunicorn`` command-line tool accepts. If the
 
  ``use_paste`` option is enabled, the value should be equal to filename of the
 
  Python Paste ini file, located in the ``code`` sub-directory. It should be
 
  noted that in either case the value should be specsified relative to the
 
  ``code`` sub-directory. I.e. don't use full paths.
 

	
 
**wsgi_requirements** (list, optional, ``[ futures==3.3.0, gunicorn==19.10.0 ]``)
 
  Complete list of pip requirements used for deploying Gunicorn. If
 
  specified, this list will be used to create requirements file and
 
  install Gunicorn and its dependencies from that one. This allows to
 
  have pinned packages for both Gunicorn, futures, and their
 
  dependencies. The ``futures`` package is required by Gunicorn when
 
  using Python 2.7.
 

	
 
  It should be noted that this installation method is meant primarily in case of
 
  roles that want to take advantage of upgrade checks for pip requirements
 
  files, and that employ `pip-tools <https://github.com/jazzband/pip-tools>`_.
 

	
 
  In addition to change of installation method, when this parameter is specified
 
  the role will deploy necessary files for running the pip requirements upgrade
 
  check (see the ``common`` role for description). For this a directory is
 
  created under ``/etc/pip_check_requirements_upgrades/FQDN``. The same
 
  directory should be used by dependant roles to deploy their own ``.in`` and
 
  ``.txt`` files. Make sure the file ownership is set to ``root:pipreqcheck``.
 

	
 
  Should you need to utilise the requirements file in some manner (other than
 
  checking for its upgrades), it will be also stored (and made accessible to
 
  application user/admin)) in application's home directory under the name
 
  ``.wsgi_requirements.txt``.
 

	
 
  To create complete requirements list, it is recommended to use `pip-tools
 
  <https://github.com/jazzband/pip-tools>`_ (the ``pip-compile`` utility) with
 
  ``gunicorn`` and ``futures`` in the ``.in.`` file.
 

	
 
**wsgi_requirements_in** (list, optional, ``[ futures, gunicorn ]``)
 
  List of top level packages to use when performing the pip
 
  requirements upgrade checks for the Gunicorn requirements (listed
 
  via ``wsgi_requirements`` parameter). For Python 3-based websites,
 
  it should be sufficient to list only ``gunicorn`` (``futures`` is
 
  required for Python 2).
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up a (base) WSGI website (for
 
running a bare Django project):
 

	
 
.. code-block:: yaml
 

	
 
    # Sample for a Django installation.
 
    - role: wsgi_website
 
      fqdn: django.example.com
 
      static_locations:
 
        - /static
 
        - /media
 
      uid: 2004
 
      virtualenv_packages:
 
        - django
 
      wsgi_application: django_example_com.wsgi:application
 
      environment_variables:
 
        DJANGO_SETTINGS_MODULE: "django_example_com.settings.production"
 
      https_tls_key: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.key') }}"
 
      https_tls_certificate: "{{ lookup('file', inventory_dir + '/tls/wsgi.example.com_https.pem') }}"
 
      additional_nginx_config:
 
        - comment: Use custom page for forbidden files.
 
          value: error_page 403 /static/403.html;
 
        - comment: Use custom page for non-existing locations/files.
 
          value: error_page 404 /static/404.html;
 
      website_mail_recipients: "root john.doe@example.com"
 
      environment_indicator:
 
        background_colour: "green"
 
        text_colour: "black"
 
        text: "TEST ENVIRONMENT"
 
      proxy_headers:
 
        Accept-Encoding: '""'
 

	
 
    # Use wsgi_requirements to deploy Gunicorn.
 
    - role: wsgi_website
 
      fqdn: wsgi.example.com
 
      wsgi_application: wsgi:main
 
      wsgi_requirements:
 
        - gunicorn==19.7.1
 
	- futures==3.1.1
 

	
 

	
 
Database Server
 
---------------
 

	
 
The ``database_server`` role can be used for setting-up a MariaDB database
 
server on destination machine.
 

	
 
The role implements the following:
 

	
 
* Installs MariaDB server and client.
 
* Configures MariaDB server and client to use *UTF-8* encoding by default.
 
* Sets-up the database root user for passwordless login via UNIX
 
  socket authentication.
 
* Drops the ``debian-sys-maint`` database user (which was used in
 
  Debian Jessie and earlier for maintenance tasks) if it is present,
 
  and updates the Debian system maintenance configuration file to use
 
  the root account over unix socket authentication.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
This role has no parameters.
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 
- Debian 10 (Buster)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
This role has no parameters which can be configured configure.
 

	
 

	
 
Database
 
--------
 

	
 
The ``database`` role can be used for creating a MariaDB database and
 
accompanying user on destination machine.
 

	
 
The role implements the following:
 

	
 
* Creates MariaDB database.
 
* Creates a dedicated user capable of performing any operation on the created
 
  database. Username is set to be same as the name of database.
 
* Sets-up pre-backup task that creates database dump in location
 
  ``/srv/backup/mariadb/{{ db_name }}.sql``.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **database_server**
 
* **backup_client**
 

	
 

	
 
Backups
 
~~~~~~~
 

	
 
If the backup for this role has been enabled, the following paths are backed-up:
 

	
 
**/srv/backup/maraidb/{{ db_name }}.sql**
 
  Dump of the database. Database dump is created every day at 01:45 in the
 
  morning.
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**db_name** (string, mandatory)
 
  Name of the database that should be created.
 

	
 
**db_password** (string, mandatory)
 
  Password for the database user.
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for creating a single database (for some
 
website):
 

	
 
.. code-block:: yaml
 

	
 
  - role: database
 
    db_name: phpinfo_example_com
 
    db_password: phpinfo_example_com
 

	
 

	
 
Backup Server
 
-------------
 

	
 
The ``backup_server`` role can be used for setting-up a server to act as backup
 
storage for the backup clients. Storage is made available to the clients
 
exclusively via SFTP on a dedicated port and dedicated OpenSSH server
 
instance. This instance is specifically configured and tailored for this
 
purpose.
 

	
 
The role is primarily aimed for use with `Duplicity
 
<http://duplicity.nongnu.org/>`_, but should be also usable for generic SFTP
 
uploads.
 

	
 
The role implements the following:
 

	
 
* Installs backup software (Duplicity, Duply).
 
* Creates a dedicated directory structure for backups with the following structure:
 

	
 
  * ``/srv/backups/`` - main directory under which all the backups reside.
 
  * ``/srv/backups/SERVER_NAME/`` - home directory for the backup user, name
 
    after the server. Backup users are confined to their respective home
 
    directory via chroot. Backup users can't write to their own home directory,
 
    though.
 
  * ``/srv/backups/SERVER_NAME/duplicity/`` - directory where the Duplicity
 
    backups are stored at. This directory is writable by the respective backup
 
    user.
 
  * ``SERVER_NAME/.ssh/`` - directory where authorized keys are stored. Backup
 
    user is not allowed to make modifications to this directory and files
 
    contained within (i.e. backup users can't add more keys to the
 
    ``authorized_keys`` file).
 
* Creates dedicated operating system users for backup clients. These users will
 
  be made members of the ``backup`` group as well (as an additional group).
 
* Sets-up ``authorized_keys`` for the backup clients.
 
* Makes sure the backup users can't log-in via regular OpenSSH server instance.
 
* Sets-up dedicated OpenSSH server instances to be used exclusively by backup
 
  clients. The instance listens on TCP port ``2222``.
 
* Updates firewall to allow incoming TCP connections to port
 
  ``2222``. Connections are allowed only from the configured IP addresses
 
  associated with backup clients.
 

	
 

	
 
Role dependencies
 
~~~~~~~~~~~~~~~~~
 

	
 
Depends on the following roles:
 

	
 
* **common**
 

	
 

	
 
Parameters
 
~~~~~~~~~~
 

	
 
**backup_clients** (list, optional)
 
  List of backup clients that are connecting to the backup server. This is
 
  usually done on a per-server basis. Each item in the list is a dictionary
 
  describing the backup client. The following keys are available:
 

	
 
  **server** (string, mandatory)
 
    Name of the server that is backed up. It is highly recommended to use
 
    server's FQDN for this purpose. The dedicated operating system user created
 
    will have the name of format ``bak-ESCAPED_SERVER_NAME``, where
 
    ``ESCAPED_SERVER_NAME`` is calculated by taking the passed-in server name
 
    and replacing all dots (``.``) with undescores (``_``). For example,
 
    ``web.example.com`` will be turned into ``bak-web_example_com``.
 

	
 
  **uid** (integer, optional, ``whatever OS picks``)
 
    Uid for the operating system user. User's default group will have a GID
 
    identical to the user's UID if specified. Otherwise user's default group
 
    will have OS-determined GID.
 

	
 
  **ip** (IPv4 address, mandatory)
 
    IPv4 address from which the backup client server is connecting to the backup
 
    server. Used for introducing stricter firewall rules.
 

	
 
  **public_key** (string, mandatory)
 
    SSH public key used by backup client to connect to the backup server.
 

	
 
**backup_host_ssh_private_keys** (dictionary, mandatory)
 
  Defines host keys used for the dedicated OpenSSH server instance for
 
  backup. Key values that must be provided are:
 

	
 
  - **rsa**
 
  - **ed25519**
 
  - **ecdsa**
 

	
 
  Values for each key should be the corresponding private key
 
  generated using the appropriate algorithm. Keys for this purpose can
 
  be easily created via commands::
 

	
 
    ssh-keygen -f backup_server_rsa_key -N '' -t rsa
 
    ssh-keygen -f backup_server_ed25519_key -N '' -t ed25519
 
    ssh-keygen -f backup_server_ecdsa_key -N '' -t ecdsa
 

	
 

	
 
Distribution compatibility
 
~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Role is compatible with the following distributions:
 

	
 
- Debian 9 (Stretch)
 
- Debian 10 (Stretch)
 

	
 

	
 
Examples
 
~~~~~~~~
 

	
 
Here is an example configuration for setting-up the backup server role:
 

	
 
.. code-block:: yaml
 

	
 
  - role: backup_server
 
    backup_clients:
 
      - server: web.example.com
 
        uid: 3000
 
        public_key: "{{ lookup('file', inventory_dir + '/ssh/web.example.com.pub') }}"
 
        ip: 10.32.64.18
 
      - server: mail.example.com
 
        public_key: "{{ lookup('file', inventory_dir + '/ssh/mail.example.com.pub') }}"
 
        ip: 10.32.64.15
roles/database_server/molecule/default/molecule.yml
Show inline comments
 
---
 

	
 
dependency: {}
 

	
 
driver:
 
  name: vagrant
 
  provider:
 
    name: virtualbox
 

	
 
lint:
 
  name: yamllint
 
  options:
 
    config-file: ../../.yamllint.yml
 

	
 
platforms:
 

	
 
  - name: parameters-mandatory-stretch64
 
    groups:
 
      - parameters-mandatory
 
    box: debian/contrib-stretch64
 
    memory: 512
 
    cpus: 1
 

	
 
  - name: deprecated-stretch64
 
    groups:
 
      - deprecated
 
    box: debian/contrib-stretch64
 
    memory: 512
 
    cpus: 1
 

	
 
  - name: parameters-mandatory-buster64
 
    groups:
 
      - parameters-mandatory
 
    box: debian/contrib-buster64
 
    memory: 512
 
    cpus: 1
 

	
 
  - name: deprecated-buster64
 
    groups:
 
      - deprecated
 
    box: debian/contrib-buster64
 
    memory: 512
 
    cpus: 1
 

	
 
provisioner:
 
  name: ansible
 
  config_options:
 
    defaults:
 
      force_valid_group_names: "ignore"
 
      interpreter_python: "/usr/bin/python3"
 
    ssh_connection:
 
      pipelining: "True"
 
  lint:
 
    name: ansible-lint
 

	
 
scenario:
 
  name: default
 

	
 
verifier:
 
  name: testinfra
 
  lint:
 
    name: flake8
roles/database_server/molecule/default/prepare.yml
Show inline comments
 
---
 

	
 
- name: Prepare
 
  hosts: all
 
  gather_facts: false
 
  tasks:
 
    - name: Install python for Ansible
 
      raw: test -e /usr/bin/python3 || (apt -y update && apt install -y python3-minimal)
 
      become: true
 
      changed_when: false
 

	
 
- hosts: all
 
  become: true
 
  tasks:
 

	
 
    - name: Update all caches to avoid errors due to missing remote archives
 
      apt:
 
        update_cache: true
 
      changed_when: false
 

	
 
- hosts: deprecated
 
  become: true
 
  tasks:
 

	
 
    - name: Install MariaDB
 
      apt:
 
        name:
 
          - mariadb-client
 
          - mariadb-server
 
          - python3-mysqldb
 
        state: present
 

	
 
    - name: Enable and start MariaDB
 
      service:
 
        name: mysql
 
        state: started
 
        enabled: true
 

	
 
    - name: Deploy username and password for the root database user
 
      copy:
 
        src: "deprecated-root-my.cnf"
 
        dest: "/root/.my.cnf"
 
        owner: root
 
        group: root
 
        mode: 0400
 

	
 
    - name: Set password for the root database user (creating separate entry for different hosts)
 
      mysql_user:
 
        check_implicit_admin: true
 
        name: root
 
        host: "{{ item }}"
 
        password: "root_password"
 
      with_items:
 
        - "localhost"
 
        - "127.0.0.1"
 
        - "::1"
 
        - "{{ ansible_hostname }}"
 

	
 
    - name: Deploy username and password for the root database user
 
      copy:
 
        src: "deprecated-root-my.cnf"
 
        dest: "/root/.my.cnf"
 
        owner: root
 
        group: root
 
        mode: 0400
 

	
 
    - name: Disable use of unix socket login
 
      command: "mysql -B -e \"update mysql.user set plugin='' where user='root' and plugin='unix_socket'; flush privileges;\""
 

	
 
    - name: Create Debian system maintenance user
 
      mysql_user:
 
        name: debian-sys-maint
 
        password: debian-sys-maint-password
 

	
 
    - name: Deploy Debian system maintenance user login configuration
 
      copy:
 
        src: "deprecated-debian.cnf"
 
        dest: "/etc/mysql/debian.cnf"
 
        owner: root
 
        group: root
 
        mode: 0600
0 comments (0 inline, 0 general)