Changeset - e3ebfa5942bd
[Not reviewed]
0 2 0
Branko Majic (branko) - 6 years ago 2018-07-23 12:47:43
branko@majic.rs
MAR-130: Don't output warnings when missing .txt file for .in file when performing pip requirements upgrade check.
2 files changed with 9 insertions and 3 deletions:
0 comments (0 inline, 0 general)
docs/releasenotes.rst
Show inline comments
 
@@ -9,48 +9,51 @@ Upgrade to Ansible 2.5.x.
 

	
 
Breaking changes:
 

	
 
* Switched to Ansible 2.5.x, removing support for older versions. All
 
  documentation has been updated.
 

	
 
* ``ldap_server`` role
 

	
 
  * Custom LDAP module ``m_ldap_entry`` has been removed. Role uses
 
    the official ``ldap_entry`` and ``ldap_attr`` modules.
 

	
 
  * The ``ldap_entries`` parameter now supports only the states
 
    supported by ``ldap_entry`` module (e.g. ``append`` is not
 
    supported any longer - since it came with custom LDAP module).
 

	
 
New features/improvements:
 

	
 
* Tests have been updated to work with latest Molecule/Testinfra as
 
  part of the Ansible upgrade process.
 

	
 
* ``common`` role
 

	
 
  * The ``pip`` requirements upgrade checks are now performed once per
 
    day instead of once per hour.
 
  * The ``pip`` requirements upgrade checks now do not output warning
 
    in case deployed ``.in`` file does not have a matching ``.txt``
 
    file.
 

	
 

	
 
2.0.0
 
-----
 

	
 
Upgrade to Ansible 2.3.x, minor bug fixes and updates needed for the upgrade.
 

	
 
Breaking changes:
 

	
 
* Switched to Ansible 2.3.x, removing support for Ansible 1.9.x. All
 
  documentation has been updated.
 

	
 
* Due to switch to Ansible 2.x which is more restrictive when deploying code on
 
  remote server, it is now necessary to use one of the methods listed in
 
  `Ansible documentation
 
  <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_
 
  if connecting to remote server as user other than ``root``. Easiest fix is to
 
  enable ``pipelining``. Tests have been already updated to take advantage of
 
  this.
 

	
 
* ``ldap_server`` role
 

	
 
   * Renamed ``ldap_entry`` module to ``m_ldap_entry`` to avoid collision with
 
     official module.
roles/common/files/pip_check_requirements_upgrades.sh
Show inline comments
 
@@ -23,50 +23,53 @@ program="pip_check_requirements_upgrades.sh"
 
function usage() {
 
    cat <<EOF
 
$program, a non-interactive utility for checking for available upgrades in
 
Python pip requirements files based on pip-tools
 

	
 
Usage: $program [OPTIONS] configuration_directory
 

	
 
$program a non-interactive utility for checking for available upgrades in Python
 
pip requirements files based on pip-tools. Utility is written specifically with
 
pip-tools in mind, and pip-tools must be available when running the utility.
 

	
 
If you are unfamiliar with pip-tools, please read its documentation first, and
 
make sure you fully understand how it works.
 

	
 
Utility accepts a single positionl argument - path to configuration
 
directory. Configuration directory should contain one or more sub-directories
 
where each sub-directory is treated as describing a separate Python virtual
 
environment. This allows the checks to be run for multiple virtual environments
 
in a modular manner.
 

	
 
Each sub-directory should contain one or more .in files with corresponding .txt
 
file. Base names must match (i.e. if you have production.in, the requirements
 
file must be called production.txt).
 

	
 
Utility iterates over each .in/.txt pair, calculates new requirements based on
 
the .in file, and diffs this against existing .txt file.
 
Utility iterates over each .in/.txt pair, calculates new requirements
 
based on the .in file, and diffs this against existing .txt file. If a
 
.in file does not have a corresponding .txt file, it is ignored
 
(making it possible to use the '-r base.in' syntax for including base
 
requirements etc).
 

	
 
Utility creates copy of existing requirements file, stripping it from all
 
comments, then calculates new requirements file (storing result in temporary
 
location as well), and runs a diff between them.
 

	
 
If newer pacakges are available that would satisfiy the provided .in file, a
 
diff is shown on standard output in addition to a warning message.
 

	
 
$program accepts the following options:
 

	
 
    -V virtualenv
 
        Path to virtual environment with pre-installed pip-tools. If specified,
 
        the virtual environment will be activated prior to running the utility.
 

	
 
    -q
 
        Quiet mode. Output a message only if newer packages are available.
 
    -d
 
        Enable debug mode.
 
    -v
 
        Show script licensing information.
 
    -h
 
        Show usage help.
 

	
 

	
 
@@ -224,49 +227,49 @@ fi
 
# stored.
 
tmp_current=$(mktemp)
 
tmp_new=$(mktemp)
 
tmp_new_sorted=$(mktemp)
 

	
 
# Process each environment.
 
for environment in "$config_dir"/*; do
 
    # Empty directory.
 
    if [[ ! -e $environment ]]; then
 
	error "Configuration directory is empty: $config_dir"
 
	exit "$ERROR_CONFIG_DIR"
 
    fi
 

	
 
    # Process each .in file.
 
    for req_in in "$environment"/*.in; do
 
	# Directory without .in files.
 
	if [[ ! -e $req_in ]]; then
 
	    error "No .in files in directory, skipping: $environment"
 
	    continue
 
	fi
 

	
 
	# Figure out if .txt file exists.
 
	req_txt="${req_in%.in}.txt"
 
	if [[ ! -f $req_txt ]]; then
 
	    warning "Missing .txt file for: $req_in"
 
	    [[ $quiet == 0 ]] && info "Skipping input file with missing .txt file: $req_in"
 
	    continue
 
	fi
 

	
 
	# Deploy the existing requirements file and the new one.
 
	sed -e 's/[[:blank:]]*#.*//' "$req_txt" | grep -v "^$" | sort -u > "$tmp_current" 
 
	if ! pip-compile --no-header --no-annotate --no-index --output-file "$tmp_new" --upgrade "$req_in" > /dev/null; then
 
	    error "Failed while running pip-compile command against (see error stack trace above): $req_in"
 
	    continue
 
	fi
 

	
 
        # Sort the new requirements file.
 
        sed -e 's/[[:blank:]]*#.*//' "$tmp_new" | grep -v "^$" | sort -u > "$tmp_new_sorted"
 

	
 
        debug "Current requirements:\n$(cat "$tmp_current")"
 
        debug "New requirements:\n$(cat "$tmp_new_sorted")"
 

	
 
	# Run diff, storing the output and result.
 
	diff=$(diff -u "$tmp_current" "$tmp_new_sorted")
 
	result="$?"
 

	
 
	# Show warning about available updates.
 
	if [[ $result == 0 ]]; then
 
	    [[ $quiet == 0 ]] && info "No upgrades available for: $req_txt"
 
	else
0 comments (0 inline, 0 general)