From 5f9f8aef347353a2fcb77b0dcf6a4f8e43cf8ee4 2024-02-10 23:58:28 From: Branko Majic Date: 2024-02-10 23:58:28 Subject: [PATCH] MAR-154: Drop support for Python 2.7 from the common role: - This primarily concerns removal of pip requirements upgrade checks for Python 2.7. - The old Python 3-specific paths are getting deprecated, and the Python 2 paths will be used instead. - Set permissions on pipreqcheck directory explicitly (because of deprecation testing). --- diff --git a/docs/releasenotes.rst b/docs/releasenotes.rst index 74712fc7f03990d9ed3ce78e08ae5473e32de9ff..099233c19915a06a84fe015a2a40b63fe23ba917 100644 --- a/docs/releasenotes.rst +++ b/docs/releasenotes.rst @@ -13,6 +13,20 @@ Dropped support for Debian 10 (Buster). * Dropped support for Debian 10 (Buster). +* ``common`` role + + * Dropped support for Python 2.7 pip requirements upgrade + checks. Only Python 3 is supported now. + + Requirements (input) files for Python 3 are now put under the + ``/etc/pip_check_requirements_upgrades`` directory instead of + ``/etc/pip_check_requirements_upgrades-py3``. + + The ``pip_check_requirements_py3`` / + ``pip_check_requirements_py3_in`` role parameters have been + renamed to ``pip_check_requirements`` / + ``pip_check_requirements_in``. + **Bug fixes:** * ``common`` role diff --git a/docs/rolereference.rst b/docs/rolereference.rst index 692be27da5cd78a7e345b351989cb0b0c5c714ff..03e197ef3c2488b552fe9cef69655de766108db0 100644 --- a/docs/rolereference.rst +++ b/docs/rolereference.rst @@ -272,23 +272,21 @@ The role implements the following: upgrades on daily basis. Mails are delivered to local ``root`` account, and can be redirected elsewhere via aliases. If using ``mail_forwarder`` or ``mail_server`` roles on the same server, aliases can be set-up through them. -* Sets-up system for performing checks on pip requirements - files. Roles that want their requirements files checked should - create a sub-directory inside of - ``/etc/pip_check_requirements_upgrades`` (for Python 2 - applications), or ``/etc/pip_check_requirements_upgrades-py3`` (for - Python 3 applications), and place ``.txt`` and ``.in`` files inside - (with same base name). The ``.txt`` files should be standard - requirements files with fixed versions (the ones installed by the - role). The ``.in`` files should contain only the top-level packages - (no dependencies). Avoid hard-coding versions in the ``.in`` file - unless really needed. For packages where you want to stick to - stable/LTS version branch, you should be able to use ``~=`` operator - (for example ``django~=1.8.0``. Checks are implemented via - `pip-tools `_ and a custom - script that outputs diffs if upgrades are available. Script is run - via cronjob on daily basis, and any output will be delivered to - local ``root`` user. +* Sets-up system for performing checks on pip requirements files. Only + Python 3 is supported. Roles that want their requirements files + checked should create a sub-directory inside of + ``/etc/pip_check_requirements_upgrades``, and place ``.txt`` and + ``.in`` files inside (with same base name). The ``.txt`` files + should be standard requirements files with fixed versions (the ones + installed by the role). The ``.in`` files should contain only the + top-level packages (no dependencies). Avoid hard-coding versions in + the ``.in`` file unless really needed. For packages where you want + to stick to stable/LTS version branch, you should be able to use + ``~=`` operator (for example ``django~=1.8.0``. Checks are + implemented via `pip-tools `_ + and a custom script that outputs diffs if upgrades are + available. Script is run via cronjob on daily basis, and any output + will be delivered to local ``root`` user. * Optionally configures time synchronisation using NTP (if ``ntp_servers`` parameter is set). @@ -417,32 +415,16 @@ Parameters - "3.debian.pool.ntp.org" **pip_check_requirements_in** (list, optional, ``[pip, pip-tools, setuptools, wheel]``) - List of Python package requirements inputs to use for checking for - package upgrades for the Python 2 virtual environment used to run - the check itself. For Python 3, see the dedicated parameter - ``pip_check_requirements_py3`` below. - -**pip_check_requirements** (list, optional, ``[click==7.1.2, pip-tools==5.5.0, pip==20.3.4, setuptools==44.1.1, wheel==0.37.1]``) - List of Python package requirements to install in Python 2 virtual - environment in order to be able to run the ``pip-tools`` - applications as part of pip requirements upgrade checks. This list - needs to be updated from time to time as the new releases of - ``pip-tools`` and related packages are coming out. For Python 3, see - the dedicated parameter ``pip_check_requirements_py3`` below. - -**pip_check_requirements_py3_in** (list, optional, ``[pip, pip-tools, setuptools, wheel]``) List of Python package requirements inputs to use for checking for package upgrades for the Python 3 virtual environment used to run - the check itself. For Python 2, see the dedicated parameter - ``pip_check_requirements`` above. + the check itself. -**pip_check_requirements_py3** (list, optional, ``see below``) +**pip_check_requirements** (list, optional, ``see below``) List of Python package requirements to install in Python 3 virtual environment in order to be able to run the ``pip-tools`` applications as part of pip requirements upgrade checks. This list needs to be updated from time to time as the new releases of - ``pip-tools`` and related packages are coming out. For Python 2, see - the dedicated parameter ``pip_check_requirements`` above. + ``pip-tools`` and related packages are coming out. Default value is: diff --git a/docs/usage.rst b/docs/usage.rst index 339512e28627bc267a905d6a0ad26720fe78fe49..667db9dff3e009f7e78f9097f1f5dfb7a274d7be 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -2564,7 +2564,7 @@ easily do so by updating the ``common_packages`` setting. In addition to system packages, the ``common`` role makes it easy to check if any of the pip requirements files are outdated as well. It should be noted, though, that this check does *not* verify the Python virtual environments -themselves. +themselves. Only Python 3 is supported at this time. This is primarily useful when you use `pip-tools `_ for maintaining the @@ -2575,9 +2575,7 @@ environment in sync and up-to-date. Roles that want to take advantage of this would: - Create a sub-directory under - ``/etc/pip_check_requirements_upgrades/`` (for Python 2 - applications) or ``/etc/pip_check_requirements_upgrades-py3/`` (for - Python 3 applications). + ``/etc/pip_check_requirements_upgrades/``. - Deploy ``.in`` and ``.txt`` files within the sub-directory (see ``pip-tools`` docs for explanation of how the ``.in`` files work). - Ensure the created sub-directory and files have ownership set to diff --git a/roles/common/defaults/main.yml b/roles/common/defaults/main.yml index 3ce67b4abb9616d0543d5d343da839830039e0d6..b8a6d9f48f4def5114344c263534f7a1ba1851bf 100644 --- a/roles/common/defaults/main.yml +++ b/roles/common/defaults/main.yml @@ -18,17 +18,6 @@ pip_check_requirements_in: - setuptools - wheel pip_check_requirements: - - click==7.1.2 - - pip-tools==5.5.0 - - pip==20.3.4 - - setuptools==44.1.1 - - wheel==0.37.1 -pip_check_requirements_py3_in: - - pip - - pip-tools - - setuptools - - wheel -pip_check_requirements_py3: - build==1.0.3 - click==8.1.7 - importlib-metadata==6.7.0 diff --git a/roles/common/files/cron_check_pip_requirements-py3 b/roles/common/files/cron_check_pip_requirements-py3 deleted file mode 100644 index 3f9b727e72f1aaaaeded5f4996a25cc8894a9875..0000000000000000000000000000000000000000 --- a/roles/common/files/cron_check_pip_requirements-py3 +++ /dev/null @@ -1,2 +0,0 @@ -MAILTO=root -0 0 * * * pipreqcheck /usr/local/bin/pip_check_requirements_upgrades.sh -q -V /var/lib/pipreqcheck/virtualenv-py3 /etc/pip_check_requirements_upgrades-py3 diff --git a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/with_updates/requirements.in b/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/with_updates/requirements.in deleted file mode 100644 index 68f017c3bbb61b0cb4f1adcaacab91c89326e036..0000000000000000000000000000000000000000 --- a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/with_updates/requirements.in +++ /dev/null @@ -1 +0,0 @@ -urllib3~=1.24.0 diff --git a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/with_updates/requirements.txt b/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/with_updates/requirements.txt deleted file mode 100644 index 5c118370aa97ff0863958d2373515789b78d76d0..0000000000000000000000000000000000000000 --- a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/with_updates/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -# -# This file is autogenerated by pip-compile -# To update, run: -# -# pip-compile -# -urllib3==1.24.2 # via -r requirements.in diff --git a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/without_updates/requirements.in b/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/without_updates/requirements.in deleted file mode 100644 index 68f017c3bbb61b0cb4f1adcaacab91c89326e036..0000000000000000000000000000000000000000 --- a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/without_updates/requirements.in +++ /dev/null @@ -1 +0,0 @@ -urllib3~=1.24.0 diff --git a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/without_updates/requirements.txt b/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/without_updates/requirements.txt deleted file mode 100644 index 0b886c8ea31924572b67de67a0c5a102450a5e0f..0000000000000000000000000000000000000000 --- a/roles/common/molecule/default/files/pip_check_requirements_upgrades-py3/without_updates/requirements.txt +++ /dev/null @@ -1,7 +0,0 @@ -# -# This file is autogenerated by pip-compile -# To update, run: -# -# pip-compile -# -urllib3==1.24.3 # via -r requirements.in diff --git a/roles/common/molecule/default/group_vars/parameters-optional.yml b/roles/common/molecule/default/group_vars/parameters-optional.yml index 067b0ba528a1e897e79274c12beff5df569c3a19..5501c5be31e304c2cf40558e0837ea9bc6ceaddf 100644 --- a/roles/common/molecule/default/group_vars/parameters-optional.yml +++ b/roles/common/molecule/default/group_vars/parameters-optional.yml @@ -54,11 +54,6 @@ maintenance: true maintenance_allowed_hosts: - client1 pip_check_requirements_in: - - pip >= 0.2.1 - - pip-tools >= 0.2.2 - - setuptools >= 0.2.3 - - wheel >= 0.2.4 -pip_check_requirements_py3_in: - pip >= 0.3.1 - pip-tools >= 0.3.2 - setuptools >= 0.3.3 diff --git a/roles/common/molecule/default/playbook.yml b/roles/common/molecule/default/playbook.yml index e03c3eef640c6d0ce20aea824833b599b8cf74d3..f5717e7309012549c92b471c713dd321dd22ce0d 100644 --- a/roles/common/molecule/default/playbook.yml +++ b/roles/common/molecule/default/playbook.yml @@ -20,9 +20,6 @@ - "/tmp/pip_check_requirements_upgrades" - "/tmp/pip_check_requirements_upgrades/with_updates" - "/tmp/pip_check_requirements_upgrades/without_updates" - - "/tmp/pip_check_requirements_upgrades-py3" - - "/tmp/pip_check_requirements_upgrades-py3/with_updates" - - "/tmp/pip_check_requirements_upgrades-py3/without_updates" - name: Deploy files for testing pip requirements upgrade checks script copy: @@ -37,10 +34,6 @@ - "pip_check_requirements_upgrades/with_updates/requirements.txt" - "pip_check_requirements_upgrades/without_updates/requirements.in" - "pip_check_requirements_upgrades/without_updates/requirements.txt" - - "pip_check_requirements_upgrades-py3/with_updates/requirements.in" - - "pip_check_requirements_upgrades-py3/with_updates/requirements.txt" - - "pip_check_requirements_upgrades-py3/without_updates/requirements.in" - - "pip_check_requirements_upgrades-py3/without_updates/requirements.txt" - name: Install web server for testing connectivity apt: diff --git a/roles/common/molecule/default/prepare.yml b/roles/common/molecule/default/prepare.yml index 26e612e2d4cde04e31db5765783029c4310c79f7..6523320949d40cd847e72bab81c741529787028f 100644 --- a/roles/common/molecule/default/prepare.yml +++ b/roles/common/molecule/default/prepare.yml @@ -129,6 +129,30 @@ - security - raw + - name: Create deprecated directory for storing requirements files created using Python 3 (pip requirements upgrade checks) + file: + path: "/etc/pip_check_requirements_upgrades-py3" + state: directory + owner: root + group: root + mode: 0750 + + - name: Create deprecated directory for Python 3 virtual environment (pip requirements upgrade checks) + file: + path: "/var/lib/pipreqcheck/virtualenv-py3/" + state: directory + owner: root + group: root + mode: 0750 + + - name: Create deprecated cronjob file for Python 3 (pip requirements upgrade checks) + file: + path: "/etc/cron.d/check_pip_requirements-py3" + state: touch + owner: root + group: root + mode: 0644 + - hosts: parameters-mandatory,parameters-optional become: true tasks: diff --git a/roles/common/molecule/default/tests/test_default.py b/roles/common/molecule/default/tests/test_default.py index e9e79a64e361564a811284d68034b1ba29741959..fa177a842db5f8070c8286762feddb4ad69eb9a5 100644 --- a/roles/common/molecule/default/tests/test_default.py +++ b/roles/common/molecule/default/tests/test_default.py @@ -234,16 +234,14 @@ def test_check_certificate_crontab(host): assert "0 0 * * * nobody /usr/local/bin/check_certificate.sh -q expiration" in check_certificate_crontab.content_string -@pytest.mark.parametrize('virtualenv_activate_path', [ - '/var/lib/pipreqcheck/virtualenv/bin/activate', - '/var/lib/pipreqcheck/virtualenv-py3/bin/activate', -]) -def test_pipreqcheck_virtualenv(host, virtualenv_activate_path): +def test_pipreqcheck_virtualenv(host): """ Tests creation of Python virtual environment used for performing pip requirements upgrade checks. """ + virtualenv_activate_path = '/var/lib/pipreqcheck/virtualenv/bin/activate' + with host.sudo(): virtualenv_activate = host.file(virtualenv_activate_path) @@ -253,16 +251,14 @@ def test_pipreqcheck_virtualenv(host, virtualenv_activate_path): assert virtualenv_activate.mode == 0o644 -@pytest.mark.parametrize('config_dir', [ - '/etc/pip_check_requirements_upgrades', - '/etc/pip_check_requirements_upgrades-py3', -]) -def test_pipreqcheck_directories(host, config_dir): +def test_pipreqcheck_directories(host): """ Tests creation of directories used for storing configuration used by script that performs pip requirements upgrade checks. """ + config_dir = '/etc/pip_check_requirements_upgrades' + with host.sudo(): pipreqcheck_config_directory = host.file(config_dir) assert pipreqcheck_config_directory.is_directory @@ -277,18 +273,15 @@ def test_pipreqcheck_directories(host, config_dir): assert pipreqcheck_config_directory_pipreqcheck.mode == 0o750 -@pytest.mark.parametrize('requirements_in_path, requirements_txt_path', [ - ('/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in', - '/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt'), - ('/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.in', - '/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt'), -]) -def test_pipreqcheck_requirements(host, requirements_in_path, requirements_txt_path): +def test_pipreqcheck_requirements(host): """ Tests deployment of requirements input and text file used for virtual environment utilised by script that perform pip requirements upgrade checks. """ + requirements_in_path = '/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in' + requirements_txt_path = '/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt' + with host.sudo(): requirements_in = host.file(requirements_in_path) assert requirements_in.is_file @@ -303,15 +296,15 @@ def test_pipreqcheck_requirements(host, requirements_in_path, requirements_txt_p assert requirements_txt.mode == 0o640 -@pytest.mark.parametrize("pip_path, expected_packages", [ - ('/var/lib/pipreqcheck/virtualenv/bin/pip', [ - "click==7.1.2", - "pip-tools==5.5.0", - "pip==20.3.4", - "setuptools==44.1.1", - "wheel==0.37.1", - ]), - ('/var/lib/pipreqcheck/virtualenv-py3/bin/pip', [ +def test_pipreqcheck_virtualenv_packages(host): + """ + Tests if correct packages are installed in virtualenv used for pip + requirements checks.. + """ + + pip_path = '/var/lib/pipreqcheck/virtualenv/bin/pip' + + expected_packages = [ "build==1.0.3", "click==8.1.7", "importlib-metadata==6.7.0", @@ -324,13 +317,7 @@ def test_pipreqcheck_requirements(host, requirements_in_path, requirements_txt_p "typing_extensions==4.7.1", "wheel==0.41.3", "zipp==3.15.0", - ]), -]) -def test_pipreqcheck_virtualenv_packages(host, pip_path, expected_packages): - """ - Tests if correct packages are installed in virtualenv used for pip - requirements checks.. - """ + ] packages = host.run("sudo -u pipreqcheck %s freeze --all", pip_path) @@ -362,16 +349,14 @@ def test_pipreqcheck_script(host): assert pipreqcheck_script.mode == 0o755 -@pytest.mark.parametrize('crontab_path, virtualenv_path', [ - ('/etc/cron.d/check_pip_requirements', '/var/lib/pipreqcheck/virtualenv'), - ('/etc/cron.d/check_pip_requirements-py3', '/var/lib/pipreqcheck/virtualenv-py3'), -]) -def test_pipreqcheck_crontab(host, crontab_path, virtualenv_path): +def test_pipreqcheck_crontab(host): """ Tests if crontab entry is set-up correctly for running the pip requirements upgrade checks. """ + crontab_path = '/etc/cron.d/check_pip_requirements' + virtualenv_path = '/var/lib/pipreqcheck/virtualenv' crontab = host.file(crontab_path) assert crontab.is_file @@ -382,16 +367,15 @@ def test_pipreqcheck_crontab(host, crontab_path, virtualenv_path): assert virtualenv_path in crontab.content_string.split(" ") -@pytest.mark.parametrize('python_path, expected_major_version', [ - ('/var/lib/pipreqcheck/virtualenv/bin/python', '2'), - ('/var/lib/pipreqcheck/virtualenv-py3/bin/python', '3'), -]) -def test_pipreqcheck_virtualenv_python_version(host, python_path, expected_major_version): +def test_pipreqcheck_virtualenv_python_version(host): """ Tests if Python virtual environment for pipreqcheck has been set-up correctly. """ + python_path = '/var/lib/pipreqcheck/virtualenv/bin/python' + expected_major_version = '3' + with host.sudo('pipreqcheck'): major_version = host.run("%s -c %s", python_path, "import sys; print(sys.version_info.major)") @@ -399,37 +383,21 @@ def test_pipreqcheck_virtualenv_python_version(host, python_path, expected_major assert major_version.stdout.strip() == expected_major_version -@pytest.mark.parametrize('wrong_python_path', [ - '/var/lib/pipreqcheck/virtualenv/bin/python3', - '/var/lib/pipreqcheck/virtualenv-py3/bin/python2.7', -]) -def test_pipreqcheck_virtualenv_wrong_python_version_not_present(host, wrong_python_path): - """ - Tests if wrong version of Python 2 is absent or not. - """ - - with host.sudo(): - wrong_python_path_file = host.file(wrong_python_path) - - assert not wrong_python_path_file.exists - - -@pytest.mark.parametrize('environment,config_directory', [ - ('/var/lib/pipreqcheck/virtualenv', '/tmp/pip_check_requirements_upgrades'), - ('/var/lib/pipreqcheck/virtualenv-py3', '/tmp/pip_check_requirements_upgrades-py3'), -]) -def test_pipreqcheck_script_output(host, environment, config_directory): +def test_pipreqcheck_script_output(host): """ Tests if the pip_check_requirements_upgrades.sh script properly reports available updates or not. """ + virtualenv = '/var/lib/pipreqcheck/virtualenv' + config_directory = '/tmp/pip_check_requirements_upgrades' + expected_line_count = 9 expected_warning_message = "[WARN] Upgrades available for: %s/with_updates/requirements.txt" % config_directory expected_package_diff = "@@ -1 +1 @@\n-urllib3==1.24.2\n+urllib3==1.24.3" with host.sudo("pipreqcheck"): - report = host.run("/usr/local/bin/pip_check_requirements_upgrades.sh -q -V %s %s", environment, config_directory) + report = host.run("/usr/local/bin/pip_check_requirements_upgrades.sh -q -V %s %s", virtualenv, config_directory) # Clean-up the SSH warning from the beginning of stderr if # present. diff --git a/roles/common/molecule/default/tests/test_deprecated.py b/roles/common/molecule/default/tests/test_deprecated.py new file mode 100644 index 0000000000000000000000000000000000000000..a167b2edb407fb7d016194165e5b1459a89788f1 --- /dev/null +++ b/roles/common/molecule/default/tests/test_deprecated.py @@ -0,0 +1,23 @@ +import os + +import testinfra.utils.ansible_runner + +import pytest + + +testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner( + os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('parameters-*') + + +@pytest.mark.parametrize('path', [ + '/etc/pip_check_requirements_upgrades-py3/', + '/var/lib/pipreqcheck/virtualenv-py3/', + '/etc/cron.d/check_pip_requirements-py3', +]) +def test_deprecated_paths_are_absent(host, path): + """ + Tests if deprecated paths are absent. + """ + + with host.sudo(): + assert not host.file(path).exists diff --git a/roles/common/molecule/default/tests/test_parameters_mandatory.py b/roles/common/molecule/default/tests/test_parameters_mandatory.py index b3af54e64edfc0457db21ea78a1dfb7980aacbc6..925fcef85a450e87147351c271d4e8172523e6bd 100644 --- a/roles/common/molecule/default/tests/test_parameters_mandatory.py +++ b/roles/common/molecule/default/tests/test_parameters_mandatory.py @@ -5,8 +5,6 @@ import paramiko import testinfra.utils.ansible_runner -import pytest - testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner( os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('parameters-mandatory') @@ -135,27 +133,21 @@ def test_ntp_listening_interfaces(host): assert not host.socket('udp://:::123').is_listening -@pytest.mark.parametrize("requirements_path, expected_requirements", [ - ('/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in', [ - "pip", - "pip-tools", - "setuptools", - "wheel" - ]), - ('/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.in', [ - "pip", - "pip-tools", - "setuptools", - "wheel" - ]) -]) -def test_pipreqcheck_input_content(host, requirements_path, expected_requirements): +def test_pipreqcheck_input_content(host): """ Tests content of requirements input file used for virtual environment utilised by script that performs pip requirements upgrade checks. """ + requirements_path = '/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in' + expected_requirements = [ + "pip", + "pip-tools", + "setuptools", + "wheel" + ] + with host.sudo(): deployed_requirements = host.file(requirements_path).content_string diff --git a/roles/common/molecule/default/tests/test_parameters_optional.py b/roles/common/molecule/default/tests/test_parameters_optional.py index 32a93270c2d861d77d73cd01655599fddfef510d..d5a80b60923965682279c378592d56f1e618b4de 100644 --- a/roles/common/molecule/default/tests/test_parameters_optional.py +++ b/roles/common/molecule/default/tests/test_parameters_optional.py @@ -325,27 +325,21 @@ def test_ntp_listening_interfaces(host): assert host.socket('udp://:::123').is_listening -@pytest.mark.parametrize("requirements_path, expected_requirements", [ - ('/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in', [ - "pip >= 0.2.1", - "pip-tools >= 0.2.2", - "setuptools >= 0.2.3", - "wheel >= 0.2.4" - ]), - ('/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.in', [ - "pip >= 0.3.1", - "pip-tools >= 0.3.2", - "setuptools >= 0.3.3", - "wheel >= 0.3.4" - ]) -]) -def test_pipreqcheck_input_content(host, requirements_path, expected_requirements): +def test_pipreqcheck_input_content(host): """ Tests content of requirements input file used for virtual environment utilised by script that performs pip requirements upgrade checks. """ + requirements_path = '/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in' + expected_requirements = [ + "pip >= 0.3.1", + "pip-tools >= 0.3.2", + "setuptools >= 0.3.3", + "wheel >= 0.3.4" + ] + with host.sudo(): deployed_requirements = host.file(requirements_path).content_string diff --git a/roles/common/tasks/main.yml b/roles/common/tasks/main.yml index bc684a8ae7ade2cd0139dde8562ed8395e398580..248ea27ff22857fdd8b1a295abcd8cfbd6862478 100644 --- a/roles/common/tasks/main.yml +++ b/roles/common/tasks/main.yml @@ -1,5 +1,20 @@ --- +# Deprecation +# =========== + +- name: Drop deprecated directories and files + file: + path: "{{ item }}" + state: absent + with_items: + - "/etc/pip_check_requirements_upgrades-py3" + - "/var/lib/pipreqcheck/virtualenv-py3" + - "/etc/cron.d/check_pip_requirements-py3" + +# Main implementation +# =================== + - name: Enable use of proxy for retrieving system packages via apt template: src: "apt_proxy.j2" @@ -325,10 +340,8 @@ stat: path: "{{ item.offending_binary }}" with_items: - - offending_binary: "/var/lib/pipreqcheck/virtualenv/bin/python3" + - offending_binary: "/var/lib/pipreqcheck/virtualenv/bin/python2" virtualenv_directory: "/var/lib/pipreqcheck/virtualenv" - - offending_binary: "/var/lib/pipreqcheck/virtualenv-py3/bin/python2" - virtualenv_directory: "/var/lib/pipreqcheck/virtualenv-py3" register: "pipreqcheck_wrong_python_version_check" - name: Drop Python virtual environment if wrong Python version has been detected within @@ -346,8 +359,8 @@ group: pipreqcheck mode: 0750 with_items: + - "/var/lib/pipreqcheck" - "/var/lib/pipreqcheck/virtualenv" - - "/var/lib/pipreqcheck/virtualenv-py3" - name: Create Python virtual environment used for installing/running pip-tools command: "/usr/bin/virtualenv --python '{{ item.python_path }}' --prompt '({{ item.name }})' '{{ item.virtualenv_path }}'" @@ -358,12 +371,8 @@ with_items: - name: pipreqcheck virtualenv_path: "/var/lib/pipreqcheck/virtualenv" - python_path: "/usr/bin/python2.7" - creates: "/var/lib/pipreqcheck/virtualenv/bin/python2.7" - - name: pipreqcheck-py3 - virtualenv_path: "/var/lib/pipreqcheck/virtualenv-py3" python_path: "/usr/bin/python3" - creates: "/var/lib/pipreqcheck/virtualenv-py3/bin/python3" + creates: "/var/lib/pipreqcheck/virtualenv/bin/python3" - name: Create directory for storing pip requirements files file: @@ -374,7 +383,6 @@ mode: 0750 with_items: - "/etc/pip_check_requirements_upgrades" - - "/etc/pip_check_requirements_upgrades-py3" - name: Set-up directory for storing pip requirements file for pip-tools virtual environment itself file: @@ -385,7 +393,6 @@ mode: 0750 with_items: - "/etc/pip_check_requirements_upgrades/pipreqcheck" - - "/etc/pip_check_requirements_upgrades-py3/pipreqcheck" - name: Deploy .in file for pip requirements in pip-tools virtual environment template: @@ -397,8 +404,6 @@ with_items: - path: "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in" requirements: "{{ pip_check_requirements_in }}" - - path: "/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.in" - requirements: "{{ pip_check_requirements_py3_in }}" - name: Deploy requirements file for pipreqcheck virtual environment template: @@ -410,8 +415,6 @@ with_items: - file: "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt" requirements: "{{ pip_check_requirements }}" - - file: "/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt" - requirements: "{{ pip_check_requirements_py3 }}" - name: Install requirements in the pipreqcheck virtual environment pip: @@ -422,10 +425,8 @@ with_items: - virtualenv: "~pipreqcheck/virtualenv" requirements: "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt" - - virtualenv: "~pipreqcheck/virtualenv-py3" - requirements: "/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt" -- name: Synchronise pip-tools virtual environment via deployed requirements file (Python 2) +- name: Synchronise pip-tools virtual environment via deployed requirements file shell: "source ~pipreqcheck/virtualenv/bin/activate && pip-sync /etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt" args: executable: /bin/bash @@ -434,15 +435,6 @@ register: pipreqcheck_pip_sync changed_when: "pipreqcheck_pip_sync.stdout != 'Everything up-to-date'" -- name: Synchronise pip-tools virtual environment via deployed requirements file (Python 3) - shell: "source ~pipreqcheck/virtualenv-py3/bin/activate && pip-sync /etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt" - args: - executable: /bin/bash - become: true - become_user: "pipreqcheck" - register: pipreqcheck_pip_sync - changed_when: "pipreqcheck_pip_sync.stdout != 'Everything up-to-date'" - - name: Deploy script for checking available upgrades copy: src: "pip_check_requirements_upgrades.sh" @@ -459,14 +451,6 @@ group: root mode: 0644 -- name: Deploy crontab entry for checking pip requirements - copy: - src: "cron_check_pip_requirements-py3" - dest: "/etc/cron.d/check_pip_requirements-py3" - owner: root - group: root - mode: 0644 - - name: Install NTP packages apt: name: