x.y.z

-----

Dropped support for Debian 10 (Buster).

**Breaking changes:**

* All roles

  * Dropped support for Debian 10 (Buster).

**Bug fixes:**

* ``common`` role

  * Fix deprecation warnings for Python requirements upgrade checks

    when using pip-tools 7.3.0. This would result in unnecessary

    notifications being sent out to server administrator.

7.1.0

-----

  pacakges and (new) TCP connections. The rate-limitting is based on the source

  IP address, using the ``iptables hashlimit`` module.

* Sets-up system for performing checks on certificates (currently only if they

  expire within less than 30 days). Roles that want their certificates checked

  should deploy a ``.conf`` to directory ``/etc/check_certificate/`` with paths

  to certificate files, one per line. Certificates are checked on

  daily basis, using crontab (resulting in failures being sent out to

  the ``root`` user).

* Deploys ``apticron`` package that performs checks for available package

  upgrades on daily basis. Mails are delivered to local ``root`` account, and

  can be redirected elsewhere via aliases. If using ``mail_forwarder`` or

  ``mail_server`` roles on the same server, aliases can be set-up through them.

* Optionally configures time synchronisation using NTP (if

  ``ntp_servers`` parameter is set).

Role dependencies

~~~~~~~~~~~~~~~~~

Depends on the following roles:

* **backup_client**

  machine using NTP. If no time synchronisation should be set-up, set

  to empty list. Default is not to configure time synchronisation.

  If setting this parameter, it is recommended to set the list of

  servers to list shipped by default Debian configuration::

    - "0.debian.pool.ntp.org"

    - "1.debian.pool.ntp.org"

    - "2.debian.pool.ntp.org"

    - "3.debian.pool.ntp.org"

**pip_check_requirements_in** (list, optional, ``[pip, pip-tools, setuptools, wheel]``)

  List of Python package requirements inputs to use for checking for

  package upgrades for the Python 3 virtual environment used to run

  List of Python package requirements to install in Python 3 virtual

  environment in order to be able to run the ``pip-tools``

  applications as part of pip requirements upgrade checks. This list

  needs to be updated from time to time as the new releases of

  Default value is:

  .. code-block:: yaml

    - build==1.0.3

    - click==8.1.7

    - importlib-metadata==6.7.0

    - packaging==23.2

    - pip-tools==6.14.0

    - pip==23.1.2

    - pyproject-hooks==1.0.0

work correctly and do not cause major outage without anyone being present to

fix them.

Another useful package you may want to look into is ``needrestart`` - which runs

as a hook during the upgrade process to detect any processes that seem to be

running with outdated libraries, allowing you to restart them as well. This

package is *not* installed by the ``common`` role out-of-the-box, but you can

easily do so by updating the ``common_packages`` setting.

In addition to system packages, the ``common`` role makes it easy to check if

any of the pip requirements files are outdated as well. It should be noted,

though, that this check does *not* verify the Python virtual environments

This is primarily useful when you use `pip-tools

<https://github.com/jazzband/pip-tools>`_ for maintaining the

requirements files. In fact, I would encourage you to utilise

``pip-tools`` for both this purpose and for keeping the virtual

environment in sync and up-to-date.

Roles that want to take advantage of this would:

- Create a sub-directory under

- Deploy ``.in`` and ``.txt`` files within the sub-directory (see ``pip-tools``

  docs for explanation of how the ``.in`` files work).

- Ensure the created sub-directory and files have ownership set to

  ``root:pipreqcheck``.

.. note::

   If you are using the ``wsgi_website`` role as dependency, simply set-up the

   ``wsgi_requirements`` parameter, and then deploy the ``.in`` and ``.txt``

   file into directory ``/etc/pip_check_requirements_upgrades/FQDN`` (this

   directory is automatically created when ``wsgi_requirements`` is specified).

incoming_connection_limit_burst: 9

prompt_colour: none

prompt_id: null

extra_backup_patterns:

  - "/root"

  - "/home"

pip_check_requirements_in:

  - pip

  - pip-tools

  - setuptools

  - wheel

pip_check_requirements:

  - build==1.0.3

  - click==8.1.7

  - importlib-metadata==6.7.0

  - packaging==23.2

  - pip-tools==6.14.0

  - pip==23.1.2

  - pyproject-hooks==1.0.0

  - setuptools==68.0.0

  - tomli==2.0.1

  - typing-extensions==4.7.1

  - wheel==0.41.3

  - zipp==3.15.0

prompt_colour: cyan

prompt_id: test

# Purposefully set this to 3 servers to make sure we are

# overriding the default configuration.

ntp_servers:

  - "0.debian.pool.ntp.org"

  - "1.debian.pool.ntp.org"

  - "2.debian.pool.ntp.org"

maintenance: true

maintenance_allowed_hosts:

  - client1

pip_check_requirements_in:

  - pip >= 0.3.1

  - pip-tools >= 0.3.2

  - setuptools >= 0.3.3

  - wheel >= 0.3.4

# From backup_client role meta dependency.

backup_encryption_key: "{{ lookup('file', 'tests/data/gnupg/backup_encryption_key') }}"

backup_server: backup-server

backup_server_host_ssh_public_keys:

  - bougs-backup-server-key-1

  - bougs-backup-server-key-2

backup_ssh_key: "bogus-backup-client-key"

    - name: Set-up directories for testing pip requirements upgrade checks script

      file:

        path: "{{ item }}"

        state: directory

        owner: root

        group: pipreqcheck

        mode: 0750

      with_items:

        - "/tmp/pip_check_requirements_upgrades"

        - "/tmp/pip_check_requirements_upgrades/with_updates"

        - "/tmp/pip_check_requirements_upgrades/without_updates"

    - name: Deploy files for testing pip requirements upgrade checks script

      copy:

        src: "{{ item }}"

        dest: "/tmp/{{ item }}"

        owner: root

        group: pipreqcheck

        mode: 0640

        directory_mode: 0750

      with_items:

        - "pip_check_requirements_upgrades/with_updates/requirements.in"

        - "pip_check_requirements_upgrades/with_updates/requirements.txt"

        - "pip_check_requirements_upgrades/without_updates/requirements.in"

        - "pip_check_requirements_upgrades/without_updates/requirements.txt"

    - name: Install web server for testing connectivity

      apt:

        name: nginx

        state: present

    - name: Deploy firewall configuration file for the web server

      copy:

        src: ferm_http.conf

        dest: /etc/ferm/conf.d/99-http.conf

        owner: root

        group: root

        - security

        - raw

    - name: Create some custom legacy ip6tables chains for testing their removal (max chain name length is 29)

      command: "ip6tables-legacy -t '{{ item }}' -N '{{ (ansible_date_time.iso8601_micro | to_uuid)[:28] }}'"

      with_items:

        - filter

        - nat

        - mangle

        - security

        - raw

- hosts: parameters-mandatory,parameters-optional

  become: true

  tasks:

    - name: Remove the ss utility (see https://github.com/philpep/testinfra/pull/320)

      file:

        path: "/bin/ss"

        state: absent

    Tests deployment of cron job for checking certificates.

    """

    check_certificate_crontab = host.file('/etc/cron.d/check_certificate')

    assert check_certificate_crontab.is_file

    assert check_certificate_crontab.user == 'root'

    assert check_certificate_crontab.group == 'root'

    assert check_certificate_crontab.mode == 0o644

    assert "0 0 * * * nobody /usr/local/bin/check_certificate.sh -q expiration" in check_certificate_crontab.content_string

    """

    Tests creation of Python virtual environment used for performing pip

    requirements upgrade checks.

    """

    with host.sudo():

        virtualenv_activate = host.file(virtualenv_activate_path)

        assert virtualenv_activate.is_file

        assert virtualenv_activate.user == 'pipreqcheck'

        assert virtualenv_activate.group == 'pipreqcheck'

        assert virtualenv_activate.mode == 0o644

    """

    Tests creation of directories used for storing configuration used by script

    that performs pip requirements upgrade checks.

    """

    with host.sudo():

        pipreqcheck_config_directory = host.file(config_dir)

        assert pipreqcheck_config_directory.is_directory

        assert pipreqcheck_config_directory.user == 'root'

        assert pipreqcheck_config_directory.group == 'pipreqcheck'

        assert pipreqcheck_config_directory.mode == 0o750

        pipreqcheck_config_directory_pipreqcheck = host.file(os.path.join(config_dir, 'pipreqcheck'))

        assert pipreqcheck_config_directory_pipreqcheck.is_directory

        assert pipreqcheck_config_directory_pipreqcheck.user == 'root'

        assert pipreqcheck_config_directory_pipreqcheck.group == 'pipreqcheck'

        assert pipreqcheck_config_directory_pipreqcheck.mode == 0o750

    """

    Tests deployment of requirements input and text file used for virtual

    environment utilised by script that perform pip requirements upgrade checks.

    """

    with host.sudo():

        requirements_in = host.file(requirements_in_path)

        assert requirements_in.is_file

        assert requirements_in.user == 'root'

        assert requirements_in.group == 'pipreqcheck'

        assert requirements_in.mode == 0o640

        requirements_txt = host.file(requirements_txt_path)

        requirements_txt.is_file

        assert requirements_txt.user == 'root'

        assert requirements_txt.group == 'pipreqcheck'

        assert requirements_txt.mode == 0o640

        "build==1.0.3",

        "click==8.1.7",

        "importlib-metadata==6.7.0",

        "packaging==23.2",

        "pip-tools==6.14.0",

        "pip==23.1.2",

        "pyproject_hooks==1.0.0",

        "setuptools==68.0.0",

        "tomli==2.0.1",

        "typing_extensions==4.7.1",

        "wheel==0.41.3",

        "zipp==3.15.0",

    packages = host.run("sudo -u pipreqcheck %s freeze --all", pip_path)

    # Normalise package names and order.

    expected_packages = sorted([p.lower() for p in expected_packages])

    actual_packages = sorted(packages.stdout.lower().strip().split("\n"))

    # This is a dummy distro-provided package ignored by the

    # pip-tools. pkg-resources for Python 2.7, pkg_resources for

    # Python 3.

    if "pkg-resources==0.0.0" in actual_packages:

        actual_packages.remove("pkg-resources==0.0.0")

    """

    Tests script used for performing pip requirements upgrade checks.

    """

    pipreqcheck_script = host.file('/usr/local/bin/pip_check_requirements_upgrades.sh')

    assert pipreqcheck_script.is_file

    assert pipreqcheck_script.user == 'root'

    assert pipreqcheck_script.group == 'root'

    assert pipreqcheck_script.mode == 0o755

    """

    Tests if crontab entry is set-up correctly for running the pip requirements

    upgrade checks.

    """

    crontab = host.file(crontab_path)

    assert crontab.is_file

    assert crontab.user == 'root'

    assert crontab.group == 'root'

    assert crontab.mode == 0o644

    assert "MAILTO=root" in crontab.content_string

    assert virtualenv_path in crontab.content_string.split(" ")

    """

    Tests if Python virtual environment for pipreqcheck has been

    set-up correctly.

    """

    with host.sudo('pipreqcheck'):

        major_version = host.run("%s -c %s", python_path, "import sys; print(sys.version_info.major)")

    assert major_version.rc == 0

    assert major_version.stdout.strip() == expected_major_version

    """

    Tests if the pip_check_requirements_upgrades.sh script properly

    reports available updates or not.

    """

    expected_line_count = 9

    expected_warning_message = "[WARN]  Upgrades available for: %s/with_updates/requirements.txt" % config_directory

    expected_package_diff = "@@ -1 +1 @@\n-urllib3==1.24.2\n+urllib3==1.24.3"

    with host.sudo("pipreqcheck"):

        # Clean-up the SSH warning from the beginning of stderr if

        # present.

        stderr = re.sub("^Warning: Permanently added.*?\r\n", "", report.stderr)

    assert stderr == ""

    assert len(report.stdout.split("\n")) == expected_line_count

    assert expected_warning_message in report.stdout

    assert expected_package_diff in report.stdout

@pytest.mark.parametrize('default_path', [

import os

import socket

import paramiko

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(

    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('parameters-mandatory')

def test_apt_proxy(host):

    """

    Tests if proxy configuration for apt is missing.

    """

    assert not host.file('/etc/apt/apt.conf.d/00proxy').exists

    assert not host.package('ntp').is_installed

    assert not host.package('ntpdate').is_installed

def test_ntp_listening_interfaces(host):

    """

    Tests if NTP server is not listening.

    """

    assert not host.socket('udp://:::123').is_listening

    """

    Tests content of requirements input file used for virtual

    environment utilised by script that performs pip requirements

    upgrade checks.

    """

    with host.sudo():

        deployed_requirements = host.file(requirements_path).content_string

        expected_requirements = sorted([line.lower() for line in expected_requirements])

        actual_requirements = sorted(deployed_requirements.lower().strip().split("\n"))

        assert actual_requirements == expected_requirements

    assert ntpq.rc == 0

    assert len(ntpq.stdout.strip().split("\n")) == expected_stdout_line_count

def test_ntp_listening_interfaces(host):

    """

    Tests if NTP server is listening on correct ports.

    """

    assert host.socket('udp://:::123').is_listening

    """

    Tests content of requirements input file used for virtual

    environment utilised by script that performs pip requirements

    upgrade checks.

    """

    with host.sudo():

        deployed_requirements = host.file(requirements_path).content_string

        expected_requirements = sorted([line.lower() for line in expected_requirements])

        actual_requirements = sorted(deployed_requirements.lower().strip().split("\n"))

        assert actual_requirements == expected_requirements

---

- name: Enable use of proxy for retrieving system packages via apt

  template:

    src: "apt_proxy.j2"

    dest: "/etc/apt/apt.conf.d/00proxy"

    owner: root

    group: root

    mode: 0644

  when: apt_proxy is defined

- name: Disable use of proxy for retrieving system packages via apt

  file:

    path: "/etc/apt/apt.conf.d/00proxy"

- name: Create user for running pip requirements checks

  user:

    name: "pipreqcheck"

    uid: "{{ pipreqcheck_uid | default(omit) }}"

    group: "pipreqcheck"

    home: "/var/lib/pipreqcheck"

    state: present

- name: Check for presence of wrong Python versions

  stat:

    path: "{{ item.offending_binary }}"

  with_items:

      virtualenv_directory: "/var/lib/pipreqcheck/virtualenv"

  register: "pipreqcheck_wrong_python_version_check"

- name: Drop Python virtual environment if wrong Python version has been detected within

  file:

    path: "{{ item.item.virtualenv_directory }}"

    state: absent

  when: "item.stat.exists"

  with_items: "{{ pipreqcheck_wrong_python_version_check.results }}"

- name: Create directory for Python virtual environment used for installing/running pip-tools

  file:

    path: "{{ item }}"

    state: directory

    owner: pipreqcheck

    group: pipreqcheck

    mode: 0750

  with_items:

    - "/var/lib/pipreqcheck/virtualenv"

- name: Create Python virtual environment used for installing/running pip-tools

  command: "/usr/bin/virtualenv --python '{{ item.python_path }}' --prompt '({{ item.name }})' '{{ item.virtualenv_path }}'"

  args:

    creates: "{{ item.creates }}"

  become: true

  become_user: "pipreqcheck"