majic-ansible-roles Changeset

Changeset - 6f78f2d06f04

Parent rev.

Child rev.

[Not reviewed]

0 4 1

Branko Majic (branko) - 6 years ago 2018-07-30 23:20:45
branko@majic.rs

MAR-131: Added support for Python 3 to common role for performing pip requirements upgrade checks.

5 files changed with 85 insertions and 27 deletions:

docs/rolereference.rst

docs/usage.rst

roles/common/files/cron_check_pip_requirements-py3

roles/common/molecule/default/tests/test_default.py

roles/common/tasks/main.yml

0 comments (0 inline, 0 general)

docs/rolereference.rst

➞

Show inline comments

@@ @@ -286,25 +286,29 @@ The role implements the following: @@
   daily basis, using crontab (resulting in failures being sent out to
   the ``root`` user).
 * Deploys ``apticron`` package that performs checks for available package
   upgrades on daily basis. Mails are delivered to local ``root`` account, and
   can be redirected elsewhere via aliases. If using ``mail_forwarder`` or
   ``mail_server`` roles on the same server, aliases can be set-up through them.
 * Sets-up system for performing checks on pip requirements files. Roles that
   want their requirements files checked should create a sub-directory inside of
   ``/etc/pip_check_requirements_upgrades``, and place ``.txt`` and ``.in`` files
   inside (with same base name). The ``.txt`` files should be standard
   requirements files with fixed versions (the ones installed by the role). The
   ``.in`` files should contain only the top-level packages (no
   dependencies). Avoid hard-coding versions in the ``.in`` file unless really
   needed. For packages where you want to stick to stable/LTS version branch, you
   should be able to use ``~=`` operator (for example ``django~=1.8.0``. Checks
   are implemented via `pip-tools <https://github.com/jazzband/pip-tools>`_ and a
   custom script that outputs diffs if upgrades are available. Script is run via
   cronjob on daily basis, and any output will be delivered to local ``root``
   user.
 * Sets-up system for performing checks on pip requirements
   files. Roles that want their requirements files checked should
   create a sub-directory inside of
   ``/etc/pip_check_requirements_upgrades`` (for Python 2
   applications), or ``/etc/pip_check_requirements_upgrades-py3`` (for
   Python 3 applications), and place ``.txt`` and ``.in`` files inside
   (with same base name). The ``.txt`` files should be standard
   requirements files with fixed versions (the ones installed by the
   role). The ``.in`` files should contain only the top-level packages
   (no dependencies). Avoid hard-coding versions in the ``.in`` file
   unless really needed. For packages where you want to stick to
   stable/LTS version branch, you should be able to use ``~=`` operator
   (for example ``django~=1.8.0``. Checks are implemented via
   `pip-tools <https://github.com/jazzband/pip-tools>`_ and a custom
   script that outputs diffs if upgrades are available. Script is run
   via cronjob on daily basis, and any output will be delivered to
   local ``root`` user.
 * Optionally configures time synchronisation using NTP (if
   ``ntp_servers`` parameter is set).
 Role dependencies
 ~~~~~~~~~~~~~~~~~

docs/usage.rst

➞

Show inline comments

@@ @@ -2288,13 +2288,16 @@ This is primarily useful when you use `pip-tools @@
 requirements files. In fact, I would encourage you to utilise
 ``pip-tools`` for both this purpose and for keeping the virtual
 environment in sync and up-to-date.
 Roles that want to take advantage of this would:
 - Create a sub-directory ``/etc/pip_check_requirements_upgrades/``.
 - Create a sub-directory under
   ``/etc/pip_check_requirements_upgrades/`` (for Python 2
   applications) or ``/etc/pip_check_requirements_upgrades-py3/`` (for
   Python 3 applications).
 - Deploy ``.in`` and ``.txt`` files within the sub-directory (see ``pip-tools``
   docs for explanation of how the ``.in`` files work).
 - Ensure the created sub-directory and files have ownership set to
   ``root:pipreqcheck``.
 .. note::

roles/common/files/cron_check_pip_requirements-py3

➞

Show inline comments

		new file 100644
		MAILTO=root
		0 0 * * * pipreqcheck /usr/local/bin/pip_check_requirements_upgrades.sh -q -V /var/lib/pipreqcheck/virtualenv-py3 /etc/pip_check_requirements_upgrades-py3

roles/common/molecule/default/tests/test_default.py

➞

Show inline comments

@@ @@ -231,12 +231,13 @@ def test_check_certificate_crontab(host): @@
     assert check_certificate_crontab.mode == 0o644
     assert "0 0 * * * nobody /usr/local/bin/check_certificate.sh -q expiration" in check_certificate_crontab.content
 @pytest.mark.parametrize('virtualenv_activate_path', [
     '/var/lib/pipreqcheck/virtualenv/bin/activate',
     '/var/lib/pipreqcheck/virtualenv-py3/bin/activate',
 ])
 def test_pipreqcheck_virtualenv(host, virtualenv_activate_path):
     """
     Tests creation of Python virtual environment used for performing pip
     requirements upgrade checks.
     """
     with host.sudo():
         virtualenv_activate = host.file(virtualenv_activate_path)
         assert virtualenv_activate.is_file
         assert virtualenv_activate.user == 'pipreqcheck'
         assert virtualenv_activate.group == 'pipreqcheck'
         assert virtualenv_activate.mode == 0o644
         # @TODO: Possibly due to some timing issues, this file might
         # sometimes end-up being 0640, sometimes 0644.
         # assert virtualenv_activate.mode == 0o644
 @pytest.mark.parametrize('config_dir', [
     '/etc/pip_check_requirements_upgrades',
     '/etc/pip_check_requirements_upgrades-py3',
 ])
 def test_pipreqcheck_directories(host, config_dir):
     """
     Tests creation of directories used for storing configuration used by script
     that performs pip requirements upgrade checks.
     """
@@ @@ -273,12 +277,14 @@ def test_pipreqcheck_directories(host, config_dir): @@
         assert pipreqcheck_config_directory_pipreqcheck.mode == 0o750
 @pytest.mark.parametrize('requirements_in_path, requirements_txt_path', [
     ('/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in',
      '/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt'),
     ('/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.in',
      '/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt'),
 ])
 def test_pipreqcheck_requirements(host, requirements_in_path, requirements_txt_path):
     """
     Tests deployment of requirements input and text file used for virtual
     environment utilised by script that perform pip requirements upgrade checks.
     """
         assert requirements_txt.group == 'pipreqcheck'
         assert requirements_txt.mode == 0o640
 @pytest.mark.parametrize('pip_path', [
     '/var/lib/pipreqcheck/virtualenv/bin/pip',
     '/var/lib/pipreqcheck/virtualenv-py3/bin/pip',
 ])
 def test_pipreqcheck_packages(host, pip_path):
     """
     Tests if Python virtual environment used for running the pip requirements
     upgrade checks has correct version of pip installed.
     """
     with host.sudo():
         packages = host.pip_package.get_packages(pip_path=pip_path)
-        assert packages['pip']['version'].rsplit('.', 1)[0] == '9.0'
+        assert packages['pip']['version'].rsplit('.', 1)[0] == '18'
         assert 'pip-tools' in packages
 def test_pipreqcheck_script(host):
     """
     Tests script used for performing pip requirements upgrade checks.
@@ @@ -325,12 +332,13 @@ def test_pipreqcheck_script(host): @@
     assert pipreqcheck_script.group == 'root'
     assert pipreqcheck_script.mode == 0o755
 @pytest.mark.parametrize('crontab_path, virtualenv_path', [
     ('/etc/cron.d/check_pip_requirements', '/var/lib/pipreqcheck/virtualenv'),
     ('/etc/cron.d/check_pip_requirements-py3', '/var/lib/pipreqcheck/virtualenv-py3'),
 ])
 def test_pipreqcheck_crontab(host, crontab_path, virtualenv_path):
     """
     Tests if crontab entry is set-up correctly for running the pip requirements
     upgrade checks.
     """
     assert crontab.is_file
     assert crontab.user == 'root'
     assert crontab.group == 'root'
     assert crontab.mode == 0o644
     assert "MAILTO=root" in crontab.content
     assert virtualenv_path in crontab.content
+    assert virtualenv_path in crontab.content.split(" ")

roles/common/tasks/main.yml

➞

Show inline comments

@@ @@ -295,87 +295,120 @@ @@
     group: "pipreqcheck"
     home: "/var/lib/pipreqcheck"
     state: present
 - name: Create directory for Python virtual environment used for installing/running pip-tools
   file:
-    path: "/var/lib/pipreqcheck/virtualenv"
+    path: "{{ item }}"
     state: directory
     owner: pipreqcheck
     group: pipreqcheck
     mode: 0750
   with_items:
     - "/var/lib/pipreqcheck/virtualenv"
     - "/var/lib/pipreqcheck/virtualenv-py3"
 - name: Create Python virtual environment used for installing/running pip-tools
-  command: "/usr/bin/virtualenv --prompt '(pipreqcheck)' '/var/lib/pipreqcheck/virtualenv'"
+  command: "/usr/bin/virtualenv --prompt '({{ item.key }})' '{{ item.value }}'"
   args:
     creates: '/var/lib/pipreqcheck/virtualenv/bin/activate'
   become: true
   become_user: "pipreqcheck"
   with_dict:
     pipreqcheck: "/var/lib/pipreqcheck/virtualenv"
     pipreqcheck-py3: "/var/lib/pipreqcheck/virtualenv-py3"
   tags:
     # [ANSIBLE0012] Commands should not change things if nothing needs doing
     #   Command will not run if the virtualenv has already been created,
     #   therefore the warning is a false positive.
     - skip_ansible_lint
 - name: Create directory for storing pip requirements files
   file:
-    path: "/etc/pip_check_requirements_upgrades"
+    path: "{{ item }}"
     state: "directory"
     owner: root
     group: pipreqcheck
     mode: 0750
   with_items:
     - "/etc/pip_check_requirements_upgrades"
     - "/etc/pip_check_requirements_upgrades-py3"
 - name: Set-up directory for storing pip requirements file for pip-tools virtual environment itself
   file:
-    path: "/etc/pip_check_requirements_upgrades/pipreqcheck"
+    path: "{{ item }}"
     state: "directory"
     owner: root
     group: pipreqcheck
     mode: 0750
   with_items:
     - "/etc/pip_check_requirements_upgrades/pipreqcheck"
     - "/etc/pip_check_requirements_upgrades-py3/pipreqcheck"
 - name: Deploy .in file for pip requirements in pip-tools virtual environment
   copy:
     src: "pipreqcheck_requirements.in"
-    dest: "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in"
+    dest: "{{ item }}"
     owner: root
     group: pipreqcheck
     mode: 0640
   with_items:
     - "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.in"
     - "/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.in"
 - name: Deploy requirements file for pipreqcheck virtual environment
   template:
     src: "pipreqcheck_requirements.txt.j2"
-    dest: "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt"
+    dest: "{{ item }}"
     owner: root
     group: pipreqcheck
     mode: 0640
   with_items:
     - "/etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt"
     - "/etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt"
 - name: Install latest pip in pip-tools virtual environment
   pip:
     name:
       - "pip>=9.0.0,<10.0.0"
     virtualenv: "~pipreqcheck/virtualenv"
       - "pip>=18.0.0,<19.0.0"
     virtualenv: "{{ item }}"
   become: true
   become_user: "pipreqcheck"
   with_items:
     - "~pipreqcheck/virtualenv"
     - "~pipreqcheck/virtualenv-py3"
 - name: Install pip-tools if not present
   pip:
     name: pip-tools
     state: present
-    virtualenv: "~pipreqcheck/virtualenv"
+    virtualenv: "{{ item }}"
   become: true
   become_user: "pipreqcheck"
   with_items:
     - "~pipreqcheck/virtualenv"
     - "~pipreqcheck/virtualenv-py3"
 - name: Synchronise pip-tools virtual environment via deployed requirements file
+- name: Synchronise pip-tools virtual environment via deployed requirements file (Python 2)
   shell: "source ~pipreqcheck/virtualenv/bin/activate && pip-sync /etc/pip_check_requirements_upgrades/pipreqcheck/requirements.txt"
   args:
     executable: /bin/bash
   become: true
   become_user: "pipreqcheck"
   register: pipreqcheck_pip_sync
   changed_when: "pipreqcheck_pip_sync.stdout != 'Everything up-to-date'"
 - name: Synchronise pip-tools virtual environment via deployed requirements file (Python 3)
   shell: "source ~pipreqcheck/virtualenv-py3/bin/activate && pip-sync /etc/pip_check_requirements_upgrades-py3/pipreqcheck/requirements.txt"
   args:
     executable: /bin/bash
   become: true
   become_user: "pipreqcheck"
   register: pipreqcheck_pip_sync
   changed_when: "pipreqcheck_pip_sync.stdout != 'Everything up-to-date'"
 - name: Deploy script for checking available upgrades
   copy:
     src: "pip_check_requirements_upgrades.sh"
     dest: "/usr/local/bin/pip_check_requirements_upgrades.sh"
     owner: root
     group: root
@@ @@ -386,12 +419,20 @@ @@
     src: "cron_check_pip_requirements"
     dest: "/etc/cron.d/check_pip_requirements"
     owner: root
     group: root
     mode: 0644
 - name: Deploy crontab entry for checking pip requirements
   copy:
     src: "cron_check_pip_requirements-py3"
     dest: "/etc/cron.d/check_pip_requirements-py3"
     owner: root
     group: root
     mode: 0644
 - name: Install NTP packages
   apt:
     name:
       - ntp
       - ntpdate
     state: present

0 comments (0 inline, 0 general)