Release notes

=============

x.y.z

-----

* ``common`` role

  * Fixed permission errors with Python cache directories in the pip

    requirements upgrade checks virtual environment that can happen if

    the initial virtual environment set-up fails.

8.0.0

-----

Dropped support for Python 2.7 and Debian 10 Buster. Added support for

Debian 12 Bookworm. Numerous minor improvements and fixes.

**Breaking changes:**

* All roles

  * Dropped support for Debian 10 (Buster).

  * Added support for Debian 12 (Bookworm).

  * ``netaddr`` Python package is now required for using the roles.

  * ``dnspython`` Python package is no longer required for using the

    roles.

* ``backup_client`` role

  * Previously the backup would run even if pre-backup scripts would

    fail. This is no longer the case, and all pre-backup scripts must

    exit with non-zero exit code in order for backup process to

    kick-in.

  * Old backups are now automatically purged after successful

    backup. This could lead to longer runtimes for entire backup

    process, as well as higher CPU usage.

* ``common`` role

  * Dropped support for Python 2.7 pip requirements upgrade

    checks. Only Python 3 is supported now.

    Requirements (input) files for Python 3 are now put under the

    ``/etc/pip_check_requirements_upgrades`` directory instead of

    ``/etc/pip_check_requirements_upgrades-py3``.

    The ``pip_check_requirements_py3`` /

    ``pip_check_requirements_py3_in`` role parameters have been

    renamed to ``pip_check_requirements`` /

    ``pip_check_requirements_in``.

  * Parameter ``maintenance_allowed_hosts`` has been dropped and

    replaced with parameter ``maintenance_allowed_sources``. The new

    parameter expects a list of IPv4 and IPv6 addresses (or

    subnets). Resolvable names can no longer be specified.

  * NTP server configuration is now based on use of pools instead of

    servers. Parameter ``ntp_servers`` has been deprecated and

    replaced with parameter ``ntp_pools``.

* ``ldap_server`` role

  * Starting with Debian 12 Bookworm, the role no longer deploys

    *rsyslog* and *logrotate* configuration for writing and rotating

    the LDAP servers logs under ``/var/log/slapd.log``. Primary

    reason is that Debian 12 Bookworm no longer installs *rsyslog* by

    default, and it is considered to be deprecated at this point. The

    LDAP server logs can be read via ``journalctl -u slapd`` when

    necessary.

* ``mail_forwarder`` role

  * Firewall rules for incoming connections from the SMTP relay server

    are now based on relay's IPv4 and IPv6 addresses as resolved on

    managed machine during deployment time.

    In case the SMTP relay server's IP addresses change, the role

    needs to get reapplied against managed machines for those changes

    to take place.

    This change in behaviour was introduced to avoid firewall-related

    errors due to inability to resolve names via DNS servers during

    boot time.

* ``mail_server`` role

  * Parameter ``mail_server_tls_protocols`` has been dropped and

    replaced with parameter ``mail_server_minimum_tls_protocol``. Full

    list of TLS protocols can no longer be specified, only the minimum

    one.

* ``wsgi_website`` role

  * Dropped support for Python 2.7. Only Python 3 is supported now.

    The ``python_version`` role parameter has been dropped. The

    ``python_interpreter`` parameter is still available, but it

    defaults to Python 3 binary.

    Python (input) requirements files are now placed under the

---

dependency: {}

driver:

  name: vagrant

  provider:

    name: virtualbox

lint:

  name: yamllint

  options:

    config-file: ../../.yamllint.yml

# NOTE: Hostnames are shortened because default values for backup

# usernames are calculated by appending hostname to the "bak-" string,

# which will easily exceed the maximum username length of 32. Yay

# stupid legacy design decisions!

platforms:

  - name: backup-server

    box: debian/bookworm64

    memory: 512

    cpus: 1

    provider_raw_config_args:

      - "customize ['modifyvm', :id, '--paravirtprovider', 'minimal']"

    interfaces:

      - auto_config: true

        ip: 192.168.56.10

        network_name: private_network

        type: static

  - name: param-mandatory-bookworm

    groups:

      - parameters-mandatory

    box: debian/bookworm64

    memory: 256

    cpus: 1

    provider_raw_config_args:

      - "customize ['modifyvm', :id, '--paravirtprovider', 'minimal']"

    interfaces:

      - auto_config: true

        ip: 192.168.56.20

        network_name: private_network

        type: static

  - name: param-optional-bookworm

    groups:

      - parameters-optional

    box: debian/bookworm64

    memory: 256

    cpus: 1

    provider_raw_config_args:

      - "customize ['modifyvm', :id, '--paravirtprovider', 'minimal']"

    interfaces:

      - auto_config: true

        ip: 192.168.56.21

        network_name: private_network

        type: static

provisioner:

  name: ansible

  config_options:

    defaults:

      force_valid_group_names: "ignore"

      interpreter_python: "/usr/bin/python3"

    ssh_connection:

      pipelining: "True"

  lint:

    name: ansible-lint

scenario:

  name: default

verifier:

  name: testinfra

  lint:

    name: flake8

    - name: Deploy SSH server keys

      copy:

        content: "{{ lookup('file', item.key) + '\n' }}"

        dest: "{{ item.value }}"

        owner: root

        group: root

        mode: 0600

      with_dict:

        tests/data/ssh/server_rsa: /etc/ssh/ssh_host_rsa_key

        tests/data/ssh/server_ed25519: /etc/ssh/ssh_host_ed25519_key

        tests/data/ssh/server_ecdsa: /etc/ssh/ssh_host_ecdsa_key

      notify:

        - Restart ssh

    - name: Drop the outdated public keys

      file:

        path: "{{ item }}"

        state: absent

      with_items:

        - /etc/ssh/ssh_host_rsa_key.pub

        - /etc/ssh/ssh_host_ed25519_key.pub

        - /etc/ssh/ssh_host_ecdsa_key.pub

    - name: Force the use of internal-sftp subsystem for SFTP

      lineinfile:

        path: /etc/ssh/sshd_config

        regexp: "^Subsystem.*sftp"

        line: "Subsystem sftp internal-sftp"

        state: present

    - name: Deploy custom SSH server configuration that chroots users

      copy:

        src: "tests/data/backup_server-sshd-chroot_backup_users.conf"

        dest: "/etc/ssh/sshd_config.d/chroot_backup_users.conf"

        owner: root

        group: root

        mode: 0600

      notify:

        - Restart ssh

    - name: Set-up backup group that will contain all backup users

      group:

        name: "backup-users"

    - name: Set-up backup user groups

      group:

        name: "{{ item.name }}"

      with_items: "{{ backup_users }}"

    - name: Set-up backup users

      user:

        name: "{{ item.name }}"

        group: "{{ item.name }}"

        groups:

          - "backup-users"

      with_items: "{{ backup_users }}"

    - name: Set-up authorised keys

      authorized_key:

        user: "{{ item.name }}"

        key: "{{ item.key }}"

      with_items: "{{ backup_users }}"

    - name: Set-up port forwarding

      command: "iptables -t nat -A PREROUTING -p tcp -m tcp --dport '{{ item }}' -j REDIRECT --to-ports 22"

      changed_when: false

      with_items:

        - 2222

        - 3333

    - name: Change ownership of home directories for SFTP chroot to work

      file:

        path: "/home/{{ item.name }}"

        state: directory

        owner: root

        group: root

        mode: 0755

      with_items: "{{ backup_users }}"

    - name: Set-up duplicity backup directories

      file:

        path: "~{{ item.name }}/duplicity"

        state: directory

        owner: root

        group: backup-users

        mode: 0770

      with_items: "{{ backup_users }}"

  handlers:

    - name: Restart ssh

      service:

        name: ssh

        state: restarted

  vars:

    backup_users:

      - name: bak-param-mandatory-bookworm

        key: "{{ lookup('file', 'tests/data/ssh/parameters-mandatory.pub') }}"

      - name: backupuser

        key: "{{ lookup('file', 'tests/data/ssh/parameters-optional.pub') }}"

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(

    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('parameters-*')

def test_installed_packages(host):

    """

    Tests if the necessary packages are installed.

    """

    assert host.package('duply').is_installed

    assert host.package('duplicity').is_installed

def test_duply_directories(host):

    """

    Tests if Duply directories have been set-up correctly.

    """

    with host.sudo():

        for directory_path in ["/etc/duply",

                               "/etc/duply/main",

                               "/etc/duply/main/patterns",

                               "/etc/duply/main/gnupg",

                               "/etc/duply/main/ssh",

                               "/var/cache/duply",

                               "/var/cache/duply/main"]:

            directory = host.file(directory_path)

            assert directory.is_directory

            assert directory.user == 'root'

            assert directory.group == 'root'

            assert directory.mode == 0o700

def test_gnupg_private_keys_file(host):

    """

    Tests if file containing GnuPG private keys has been created and has correct

    permissions.

    """

    with host.sudo():

        gnupg_private_keys = host.file('/etc/duply/main/private_keys.asc')

        assert gnupg_private_keys.is_file

        assert gnupg_private_keys.user == 'root'

        assert gnupg_private_keys.group == 'root'

        assert gnupg_private_keys.mode == 0o600

def test_gnupg_public_keys_file(host):

    """

    Tests if file containing additional GnuPG public keys used for encryption

    has been created and has correct permissions.

    """

    with host.sudo():

        gnupg_public_keys = host.file('/etc/duply/main/public_keys.asc')

        assert gnupg_public_keys.is_file

        assert gnupg_public_keys.user == 'root'

        assert gnupg_public_keys.group == 'root'

        assert gnupg_public_keys.mode == 0o600

def test_private_ssh_key_file(host):

    """

    Tests if the file containing client SSH key used for logging-in into the

    backup server has been deployed and has correct permissions.

    """

    with host.sudo():

        ssh_key = host.file('/etc/duply/main/ssh/identity')

        assert ssh_key.is_file

        assert ssh_key.user == 'root'

        assert ssh_key.group == 'root'

        assert ssh_key.mode == 0o600

def test_known_hosts(host):

    """

    Tests if the Duply known_hosts file has been deployed and has correct

    permissions.

    """

    with host.sudo():

        known_hosts = host.file('/etc/duply/main/ssh/known_hosts')

        assert known_hosts.is_file

        assert known_hosts.user == 'root'

        assert known_hosts.group == 'root'

        assert known_hosts.mode == 0o600

def test_duply_configuration(host):

    """

    Tests if Duply configuraiton file has been deployed and has correct file

    permissions.

    """

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(

    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('parameters-mandatory')

def test_gnupg_private_keys_file_content(host):

    """

    Tests if correct GnuPG private key used for encryption and signing has been

    deployed.

    """

    with host.sudo():

        gnupg_private_keys = host.file('/etc/duply/main/private_keys.asc')

        # The rstrip() is added because Ansible strips last newline

        # when using the file lookup plugin.

        assert gnupg_private_keys.content_string == open('tests/data/gnupg/parameters-mandatory.asc', 'r').read().rstrip()

def test_gnupg_public_keys_file_content(host):

    """

    Tests if no additional public GnuPG keys have been deployed (should be

    default without optional parameters).

    """

    with host.sudo():

        gnupg_public_keys = host.file('/etc/duply/main/public_keys.asc')

        assert gnupg_public_keys.content_string == ""

def test_backup_ssh_key_file_content(host):

    """

    Tests if correct key has been deployed for SSH client authentication.

    """

    with host.sudo():

        ssh_key = host.file('/etc/duply/main/ssh/identity')

        assert ssh_key.content_string == open('tests/data/ssh/parameters-mandatory', 'r').read()

def test_known_hosts_content(host):

    """

    Tests if known hosts file has been set-up with correct content.

    """

    with host.sudo():

        known_hosts = host.file('/etc/duply/main/ssh/known_hosts')

        assert known_hosts.content_string == open('tests/data/ssh/parameters-mandatory-known_hosts', 'r').read()

def test_duply_configuration_content(host):

    """

    Tests if duply configuration has been set-up correctly.

    """

    hostname = host.run('hostname').stdout.strip()

    with host.sudo():

        duply_configuration = host.file('/etc/duply/main/conf')

        assert "GPG_KEYS_ENC='59C26F031A129C54'" in duply_configuration.content_string

        assert "GPG_KEY_SIGN='59C26F031A129C54'" in duply_configuration.content_string

            "-oGlobalKnownHostsFile=/etc/duply/main/ssh/known_hosts -oIdentityFile=/etc/duply/main/ssh/identity'\"" in duply_configuration.content_string

def test_duply_gnupg_keyring_private_keys(host):

    """

    Tests if private key used for encryption/signing has been correctly

    imporeted into Duply GnuPG keyring.

    """

    with host.sudo():

        private_key_listing = host.run('gpg --homedir /etc/duply/main/gnupg --list-public-keys')

        assert private_key_listing.rc == 0

        assert '59C26F031A129C54' in private_key_listing.stdout

import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(

    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('parameters-optional')

def test_gnupg_private_keys_file_content(host):

    """

    Tests if correct GnuPG private key used for encryption and signing has been

    deployed.

    """

    with host.sudo():

        gnupg_private_keys = host.file('/etc/duply/main/private_keys.asc')

        assert gnupg_private_keys.content_string == open('tests/data/gnupg/parameters-optional.asc', 'r').read().strip()

def test_gnupg_public_keys_file_content(host):

    """

    Tests if correct additional public GnuPG keys have been deployed.

    """

    with host.sudo():

        gnupg_public_keys = host.file('/etc/duply/main/public_keys.asc')

        assert open('tests/data/gnupg/additional_encryption_key_1.asc', 'r').read().strip() in gnupg_public_keys.content_string

        assert open('tests/data/gnupg/additional_encryption_key_2.asc', 'r').read().strip() in gnupg_public_keys.content_string

        assert open('tests/data/gnupg/additional_encryption_key_3.asc', 'r').read().strip() in gnupg_public_keys.content_string

def test_backup_ssh_key_file_content(host):

    """

    Tests if correct key has been deployed for SSH client authentication.

    """

    with host.sudo():

        ssh_key = host.file('/etc/duply/main/ssh/identity')

        assert ssh_key.content_string == open('tests/data/ssh/parameters-optional', 'r').read().strip()

def test_known_hosts_content(host):

    """

    Tests if known hosts file has been set-up with correct content.

    """

    with host.sudo():

        known_hosts = host.file('/etc/duply/main/ssh/known_hosts')

        assert known_hosts.content_string == open('tests/data/ssh/parameters-optional-known_hosts', 'r').read()

def test_duply_configuration_content(host):

    """

    Tests if duply configuration has been set-up correctly.

    """

    hostname = host.run('hostname').stdout.strip()

    with host.sudo():

        duply_configuration = host.file('/etc/duply/main/conf')

        assert "GPG_KEYS_ENC='C4B2AE9F7A4F400A,3093C91BC3A9444B,86816FD928063B3F,8A14CD6C71223B72'" in duply_configuration.content_string

        assert "GPG_KEY_SIGN='C4B2AE9F7A4F400A'" in duply_configuration.content_string

            "-oGlobalKnownHostsFile=/etc/duply/main/ssh/known_hosts -oIdentityFile=/etc/duply/main/ssh/identity'\"" in duply_configuration.content_string

def test_duply_gnupg_keyring_private_keys(host):

    """

    Tests if private key used for encryption/signing has been correctly

    imporeted into Duply GnuPG keyring.

    """

    with host.sudo():

        private_key_listing = host.run('gpg --homedir /etc/duply/main/gnupg --list-public-keys')

        assert private_key_listing.rc == 0

        assert 'C4B2AE9F7A4F400A' in private_key_listing.stdout

def test_duply_gnupg_keyring_public_keys(host):

    """

    Tests if additional public keys used for encryption have been correctly

    imporeted into Duply GnuPG keyring.

    """

    with host.sudo():

        public_key_listing = host.run('gpg --homedir /etc/duply/main/gnupg --list-public-keys')

        keys = ['3093C91BC3A9444B', '86816FD928063B3F', '8A14CD6C71223B72']

        assert public_key_listing.rc == 0

        for key in keys:

            assert key in public_key_listing.stdout

---

- name: Install backup software

  apt:

    name:

      - duplicity

      - duply

    state: present

- name: Set-up Duply directories

  file:

    path: "{{ item }}"

    state: directory

    owner: root

    group: root

    mode: 0700

  with_items:

    - "/etc/duply"

    - "/etc/duply/main"

    - "/etc/duply/main/patterns"

    - "/etc/duply/main/gnupg"

    - "/etc/duply/main/ssh"

    - "/var/cache/duply"

    - "/var/cache/duply/main"

- name: Deploy GnuPG private keys

  copy:

    content: "{{ backup_encryption_key }}"

    dest: "/etc/duply/main/private_keys.asc"

    owner: root

    group: root

    mode: 0600

  notify:

    - Remove current keyring

    - Create keyring directory

    - Import private keys

    - Import public keys

- name: Deploy GnuPG public keys

  copy:

    content: "{{ backup_additional_encryption_keys | join('\n') }}"

    dest: "/etc/duply/main/public_keys.asc"

    owner: root

    group: root

    mode: 0600

  notify:

    - Remove current keyring

    - Create keyring directory

    - Import private keys

    - Import public keys

- name: Extract encryption key identifier (Duplicty requires key ID in hexadecimal format)

  shell: "set -o pipefail && gpg --no-tty --list-packets /etc/duply/main/private_keys.asc | grep keyid: |

    head -n1 | sed -e 's/.*: //'"

  args:

    executable: /bin/bash

  register: backup_encryption_key_id

  changed_when: false

  failed_when: not backup_encryption_key_id.stdout

- name: Extract additional encryption keys identifiers (Duplicty requires key ID in hexadecimal format)

  shell: "set -o pipefail &&  gpg --no-tty --list-packets /etc/duply/main/public_keys.asc | grep keyid: |

    sed -e 's/.*: //' | sort -u | tr '\n' ',' | sed -e 's/,$//'"

  args:

    executable: /bin/bash

  when: backup_additional_encryption_keys | length > 0

  register: backup_additional_encryption_keys_ids

  changed_when: false

  failed_when: not backup_additional_encryption_keys_ids.stdout

- name: Deploy private SSH key for logging-in into backup server

  copy:

    content: "{{ backup_ssh_key }}"

    dest: "/etc/duply/main/ssh/identity"

    owner: root

    group: root

    mode: 0600

  no_log: true

- name: Deploy custom known_hosts for backup purposes

  template:

    src: "known_hosts.j2"

    dest: "/etc/duply/main/ssh/known_hosts"

    owner: root

    group: root

    mode: 0600

- name: Deploy Duply configuration file

  template:

    src: "duply_main_conf.j2"

    dest: "/etc/duply/main/conf"

    owner: root

    group: root

    mode: 0600

- name: Deploy base exclude pattern (exclude all by default)

  copy:

    content: "- **"

# GnuPG keys that should be used for encryption. Normally the encryption key is

# not available locally.

GPG_KEYS_ENC='{{ backup_encryption_key_id.stdout }}{% if backup_additional_encryption_keys %},{{ backup_additional_encryption_keys_ids.stdout }}{% endif %}'

# GnuPG key used for signing.

GPG_KEY_SIGN='{{ backup_encryption_key_id.stdout }}'

# Trust all keys available in the GnuPG keyring.

GPG_OPTS="--homedir /etc/duply/main/gnupg/ --trust-model always"

# Destination where the backups are stored at.

# Base directory to backup (root). File selection is done via include/exclude

# patterns.

SOURCE='/'

# Maximum age for preserving old backups. Used when running the "purge"

# command.

MAX_AGE=6M

# Maximum age of the last full backup performed before a new full backup is

# taken.

MAX_FULLBKP_AGE=1M

DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE " 

# Duplicity volume size in megabytes.

VOLSIZE=1024

DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

# Output verbosity (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)

# Path to a directory used for restoring files from backups. The file is stored

# there temporarily.

TEMP_DIR="/tmp"

# Directory for storing (caching) unencrypted metadata. This metadata is used

# for producting incremental backups.

ARCH_DIR="/var/cache/duply/main/"

# Use the GnuPG agent for passwords prompts. Since we deploy the signing key

# without any encryption, this effectively means no prompts.

DUPL_PARAMS="$DUPL_PARAMS --use-agent"

# Rely only on global known_hosts file (which should only contain

# resolvable names), bypassing addition of IP addresses to root's

# known_hosts file. Log level is configured to reduce verbosity

# (mentions of IP address additions to user's known_hosts file). Use

# dedicated private key for performing logins towards the backup

# server.

# By default we exclude everything, and then include only specific patterns.

DUPL_PARAMS="$DUPL_PARAMS --include-filelist /etc/duply/main/include"