Release notes

=============

NEXT RELEASE

------------

Upgrade to Ansible 2.5.x.

Breaking changes:

* Switched to Ansible 2.5.x, removing support for older versions. All

  documentation has been updated.

* ``ldap_server`` role

  * Custom LDAP module ``m_ldap_entry`` has been removed. Role uses

    the official ``ldap_entry`` and ``ldap_attr`` modules.

  * The ``ldap_entries`` parameter now supports only the states

    supported by ``ldap_entry`` module (e.g. ``append`` is not

    supported any longer - since it came with custom LDAP module).

New features/improvements:

* Tests have been updated to work with latest Molecule/Testinfra as

  part of the Ansible upgrade process.

* ``common`` role

  * The ``pip`` requirements upgrade checks are now performed once per

    day instead of once per hour.

2.0.0

-----

Upgrade to Ansible 2.3.x, minor bug fixes and updates needed for the upgrade.

Breaking changes:

* Switched to Ansible 2.3.x, removing support for Ansible 1.9.x. All

  documentation has been updated.

* Due to switch to Ansible 2.x which is more restrictive when deploying code on

  remote server, it is now necessary to use one of the methods listed in

  `Ansible documentation

  <https://docs.ansible.com/ansible/latest/become.html#becoming-an-unprivileged-user>`_

  if connecting to remote server as user other than ``root``. Easiest fix is to

  enable ``pipelining``. Tests have been already updated to take advantage of

  this.

* ``ldap_server`` role

   * Renamed ``ldap_entry`` module to ``m_ldap_entry`` to avoid collision with

     official module.

   * Renamed ``ldap_permissions`` module to ``m_ldap_permissions`` to be

     consistent and to avoid potential future collisions with official module

     names.

* ``wsgi_website`` role

   * Removed handler with parametrised name used for restarting the web

     service. Dependent roles should instead define their own handlers from now

     on.

New features/improvements:

* Updated documentation to refer to Debian Jessie documentation where necessary.

* ``ldap_server`` role

  * Updated tests to be more resilient to ordering changes.

* ``mail_forwarder`` role

  * Updated tests to be more resilient to ordering changes and time races.

* ``backup_client`` role

  * Switched to using the ``file`` module when cleaning-up GnuPG backup keyring,

    which should make it more robust..

Bug-fixes:

* Updated pip requirements and its input file to include ``python-vagrant``

  (needed for Molecule tests).

* Updated handling of key ID extraction for OpenPGP keys in order for it to work

  with Ansible 2.x.

* Updated usage instructions in order to be able to install The Bug Genie via

  script, and added missing instructions for creating one of the directories for

  the demo wiki role (``handlers`` directory).

* All roles

  * Updated test playbooks to avoid idempotence test failures due to apt cache

    updates.

* ``backup_client`` role

  * Fixed scenario in which backup keys could not be properly replaced on Debian

    Stretch machines (due to more up-to-date version of GnuPG compared to Debian

    Jessie).

* ``common`` role

  * Fixed handling of complex version specifications when installing packages

    via pip. Needed for Ansible 2.x.

* ``wsgi_website`` role

  * Fixed handling of complex version specifications when installing packages

    via pip. Needed for Ansible 2.x.

  * Fixed erroneous calculation of adminstrator username in internal

    defaults parameter.

1.7.0

-----

Minor improvements for mail-related roles, internal refactoring of task syntax,

and improvements of tests.

New features/improvements:

* Documentation

  * Added new sub-section in development section describing some of the

    conventions used while developing the roles.

* All roles

  * Switched to using expanded syntax in all roles and cleaned-up the tasks a

    bit (mainly internal change).

  * Minor cleanups within tests and tasks to accomodate the syntax changes

    (mainly internal change).

* ``mail_forwarder`` role

  * Added parameter ``smtp_from_relay_allowed`` that controls if managed machine

    should accept incoming SMTP connections from the relay server. Useful for

    NAT'ed or laptop machines.

  * Added parameter ``smtp_relay_host_port`` that controls what port is used for

    connecting to the specified SMTP relay. Useful for machines behind

    restrictive ISPs.

* ``mail_server`` role

  * Introduced firewall rules to redirect from TCP port 27 to TCP port 25,

    useful for machines behind restrictive ISPs.

1.6.0

-----

Implemented full test suite with a plethora of smaller bug-fixes, and some minor

(internal) improvements.

Breaking changes:

* All roles

  * Previously a number of roles would modify permissions on the ``/srv``

    directory. This has now been fixed in order to prevent weird backup failures

    etc. Manual intervention is necessary on existing servers to fix the issue

    by changing the mode to ``0755`` (this is the usual default upon the OS

    installation).

* ``mail_server`` role

  * Since Postfix will now fall-back to using ``/etc/hosts`` if it cannot

    resovle a domain via DNS, some special care may be needed in case you have

    some unusual entries in ``/etc/hosts``. Normally this should not be an

    issue, though.

* ``backup_client`` role

  * Up to this point, if you had more than one additional encryption key

    specified in configuration, only the first one was taken into account. This

    is a major issue since it would render backups up to this point decryptable

    only with one of the keys. It is highly suggested to perform a full backup

    after upgrading to new version of Majic Ansible Roles via command::

      sudo duply main full

    This will ensure the most recent backup is decryptable with all additional

    keys!

New features/improvements:

* Added new documentation chapter dedicated to development.

* Added ``requirements.txt`` that can be used for installing the required

  packages in virtual environment (useful for development).

* Small fixes (mostly typos and such) throughout the documentation.

* All roles

  * Implemented tests using Molecule. See documentation for instructions on how

    to run tests.

  * Small internal refactorings to make things simpler and easier to maintain.

* ``common`` role

  * Added missing documentation for parameters ``pipreqcheck_uid`` and

    ``pipreqcheck_gid``.

* ``mail_server`` role

  * Updated Postfix configuration to fall-back to using ``/etc/hosts`` if DNS

    lookup fails. This allows for more flexibility when testing and deploying if

    proper DNS is not available.

Bug-fixes:

* All roles

  * Fixed how TLS key and certificate material is deployed in order to avoid

    mangling of tabs.

  * Fixed how file modes are specified in tasks to ensure correct permissions

    are applied.

  * Fixed missing ``become`` keyword in tasks that use ``become_user`` for

    consistent execution.

* ``backup_client`` role

  * Fixed configuration (and documentation) for specifying the backup server

    URI - previous implementation included too many forward slashes which could

    cause failures in case of custom SSH server being used for backup.

  * Fixed configuration of additional encryption keys to include all keys listed

    instead of just the first one.

  * Fixed issue with ``backup_server_port`` parameter being completely ignored

    in the configuration.

  * Fixed issue with missing ``/etc/duply/main/include`` configuration file in

    case no backup patterns are deployed.

* ``backup_server`` role

  * Fixed deployment of backup server SSH keys in order to avoid unusable

    ``ed25519`` keys.

* ``common`` role

  * Fixated version of ``pip`` installed for performing Pyhton requirements

    package upgrade checks.

  * Fixed incorrect documentation for parameter ``additional_groups``.

  * Fixed ownership setting for firewall configuration file.

  * Fixed script used for performing checks on pip requirementes files for

    availalbe package upgrades. False positives due to different sorting will

    not be reported anymore, and the script will actually make sure to check if

    upgrades are available (which was not the case before due ot missing

    paramter to pip-compile).

* ``ldap_server`` role

  * Fixed invalid configuration of LDAP server package via

    ``debconf-set-selections`` (wrong option was used for

    ``shared/organization``).

  * Fixed role documentation example for parameter ``ldap_entries`` (was using

    obsolete syntax of ``ldap_entry`` module).

  * When making changes to the LDAP server configuration, make sure to use Unix

    socket. This way the role does not depend on correct LDAP client

    configuration.

* ``mail_server`` role

  * Fixed Postfix main configuraiton file permissions set-up to be explicit.

  * Fixed issue where Postfix server is not restarted when the truststore (used

    for verifying the LDAP server certificate) is changed.

  * Fixed issue with Postfix configuration where the parameter ``mail_user`` was

    ignored when making deliveries to Dovecot (old implementation used fixed

    value of ``vmail`` instead of parameter).

* ``php_website`` role

  * Fixed Nginx configuration file to use correct parameter (``enforce_https``

    instead of ``default_enforce_https``) when configuring HSTS. Previously it

    was possible to set the parameter to ``no``, and still end-up with HSTS

    headers being set-up.

* ``wsgi_website`` role

  * Fixed Nginx configuration file to use correct parameter (``enforce_https``

    instead of ``default_enforce_https``) when configuring HSTS. Previously it

    was possible to set the parameter to ``no``, and still end-up with HSTS

    headers being set-up.

* ``xmpp_server`` role

  * Fixed invalid default value for paramerer ``xmpp_domains`` - it should be a

    list and not a simple string. Previously this would result in invalid domain

    set-up in Prosody configuration file.

  * Fixed issue with permissions not being set on Prosody configuration file,

    making it world-readable (the configuration file contains passwords).

1.5.1

-----

Small bug-fix release for misbehaving package upgrade checks.

Bug-fixes:

* ``common`` role

  * Fixed script used for performing checks on pip requirementes files for

    availalbe package upgrades. False positives due to different sorting will

    not be reported anymore, and the script will actually make sure to check if

    upgrades are available (which was not the case before due ot missing

    paramter to pip-compile).

1.5.0

-----

Minor bug-fixes, package upgrade checks, and better support for next Debian

stable release (Stretch).

New features/improvements:

* ``backup_client`` role

  * Implemented support for next Debian stable release (*Debian Stretch*). This

    was needed due to changes in duplicity parameters and their syntax.

* ``common`` role

  * Added parameter for configuring common backup patterns. Allows for better

    control over ``/root`` and ``/home`` directories. Backup of remaining

    directories is still hard-coded.

  * Added support for checking if package upgrades are available. Covers system

    packages out-of-the-box, and provides ability to perform checks on pip

    requirements files.

  * Added generic support for checking certificate expiration dates. Relevant

    roles need to deploy special configuration files to trigger the checks.

* ``ldap_server`` role

  * Updated role to perform certificate expiration date check on LDAP server

    certificate.

* ``mail_server`` role

  * Updated role to perform certificate expiration date check on all mail server

    certificates.

* ``php_website`` role

  * Updated role to perform certificate expiration date check on website server

    certificate.

* ``xmpp_server`` role

  * Updated role to perform certificate expiration date check on XMPP server

    certificate.

* ``web_server`` role

  * Updated role to perform certificate expiration date check on default web

    server certificate.

* ``wsgi_website`` role

  * Added alternative way to specify Gunicorn version to install in virtual

    environment (via separate parameter). If this parameter is in use, package

    upgrade checks will be done as well (against auto-assembled pip requirements

    file). See role reference documentation for details.

  * Updated role to perform certificate expiration date check on website server

    certificate.

Bug-fixes:

* ``mail_server`` role

  * Fixed incorrect mail name (FQDN) used for mails originating from the server.

* ``web_server`` role

  * Fixed configuration of available TLS versions on the Nginx web server.

Documentation:

* Added release procedures and related information.

* Added information about Debian release compatibility to role reference.

1.4.0

-----

Minor fixes and features allowing for more fine-tuning of installations.

New features/improvements:

* ``ldap_server`` role

  * TLS versions and ciphers supported by server are now configurable.

* ``mail_server`` role

  * TLS versions and ciphers supported by SMTP and IMAP server are now

    configurable.

  * Number of allowed concurent IMAP connections for a single user from a single

    IP address is now configurable.

* ``web_server`` role

  * TLS versions and ciphers supported by server are now configurable.

1.3.0

-----

IPv6 support in firewall rules, small bug fixes and improvements.

New features/improvements:

#!/bin/bash

#

# pip_check_requirements_upgrades.sh

#

# Copyright (C) 2017, Branko Majic <branko@majic.rs>

#

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

#

program="pip_check_requirements_upgrades.sh"

function usage() {

    cat <<EOF

$program, a non-interactive utility for checking for available upgrades in

Python pip requirements files based on pip-tools

Usage: $program [OPTIONS] configuration_directory

$program a non-interactive utility for checking for available upgrades in Python

pip requirements files based on pip-tools. Utility is written specifically with

pip-tools in mind, and pip-tools must be available when running the utility.

If you are unfamiliar with pip-tools, please read its documentation first, and

make sure you fully understand how it works.

Utility accepts a single positionl argument - path to configuration

directory. Configuration directory should contain one or more sub-directories

where each sub-directory is treated as describing a separate Python virtual

environment. This allows the checks to be run for multiple virtual environments

in a modular manner.

Each sub-directory should contain one or more .in files with corresponding .txt

file. Base names must match (i.e. if you have production.in, the requirements

file must be called production.txt).

Utility creates copy of existing requirements file, stripping it from all

comments, then calculates new requirements file (storing result in temporary

location as well), and runs a diff between them.

If newer pacakges are available that would satisfiy the provided .in file, a

diff is shown on standard output in addition to a warning message.

$program accepts the following options:

    -V virtualenv

        Path to virtual environment with pre-installed pip-tools. If specified,

        the virtual environment will be activated prior to running the utility.

    -q

        Quiet mode. Output a message only if newer packages are available.

    -d

        Enable debug mode.

    -v

        Show script licensing information.

    -h

        Show usage help.

Please report bugs and send feature requests to <branko@majic.rs>.

EOF

}

function version() {

    cat <<EOF

$program

+-----------------------------------------------------------------------+

| Copyright (C) 2017, Branko Majic <branko@majic.rs>                    |

|                                                                       |

| This program is free software: you can redistribute it and/or modify  |

| it under the terms of the GNU General Public License as published by  |

| the Free Software Foundation, either version 3 of the License, or     |

| (at your option) any later version.                                   |

|                                                                       |

| This program is distributed in the hope that it will be useful,       |

| but WITHOUT ANY WARRANTY; without even the implied warranty of        |

| MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the         |

| GNU General Public License for more details.                          |

|                                                                       |

| You should have received a copy of the GNU General Public License     |

| along with this program.  If not, see <http://www.gnu.org/licenses/>. |

+-----------------------------------------------------------------------+

EOF

}

# Set-up colours for message printing if we're not piping and terminal is

# capable of outputting the colors.

_color_terminal=$(tput colors 2>&1)

if [[ -t 1 ]] && (( ${_color_terminal} > 0 )); then

    _text_bold=$(tput bold)

    _text_white=$(tput setaf 7)

    _text_blue=$(tput setaf 6)

    _text_green=$(tput setaf 2)

    _text_yellow=$(tput setaf 3)

    _text_red=$(tput setaf 1)

    _text_reset=$(tput sgr0)

else

    _text_bold=""

    _text_white=""

    _text_blue=""

    _text_green=""

    _text_yellow=""

    _text_red=""

    _text_reset=""

fi

# Set-up functions for printing coloured messages.

function debug() {

    if [[ $debug != 0 ]]; then

        echo -e "${_text_bold}${_text_blue}[DEBUG]${_text_reset}" "$@"

    fi

}

function info() {

    echo -e "${_text_bold}${_text_white}[INFO] ${_text_reset}" "$@"

}

function success() {

    echo -e "${_text_bold}${_text_green}[OK]   ${_text_reset}" "$@"

}

function warning() {

    echo -e "${_text_bold}${_text_yellow}[WARN] ${_text_reset}" "$@"

}

function error() {

    echo -e "${_text_bold}${_text_red}[ERROR]${_text_reset}" "$@" >&2

}

# Clean-up command for temporary files.

function on_exit() {

    debug "Cleaning-up temporary file: $tmp_current"

    [[ -f $tmp_current ]] && rm "$tmp_current"

    debug "Cleaning-up temporary file: $tmp_new"

    [[ -f $tmp_new ]] && rm "$tmp_new"

    debug "Cleaning-up temporary file: $tmp_new_sorted"

    [[ -f $tmp_new_sorted ]] && rm "$tmp_new_sorted"

}

trap on_exit EXIT

# Define error codes.

SUCCESS=0

ERROR_ARGUMENTS=1

ERROR_CONFIG_DIR=2

ERROR_PIP_TOOLS_MISSING=3

# Disable debug and quiet modes by default.

debug=0

quiet=0

# Set-up defaults.

virtualenv=""

# If no arguments were given, just show usage help.

if [[ -z $1 ]]; then

    usage

    exit "$SUCCESS"

fi

# Parse the arguments

while getopts "V:qdvh" opt; do

    case "$opt" in

	V) virtualenv="$OPTARG";;

	q) quiet=1;;

	d) debug=1;;

        v) version

           exit "$SUCCESS";;

        h) usage

           exit "$SUCCESS";;

        *) usage

           exit "$ERROR_ARGUMENTS";;

    esac

done

i=$OPTIND

shift $(($i-1))

if [[ ${#@} == 0 ]]; then

    error "Must pass configuration directory."

    usage

    exit "$ERROR_ARGUMENTS"

fi

# Verfiy positional arguments.

config_dir="$1"

if [[ ! -d $config_dir ]]; then

    error "No such directory: $config_dir"

    exit "$ERROR_ARGUMENTS"

fi

# Make sure virtual environment exists.

if [[ -n $virtualenv && ! -f $virtualenv/bin/activate ]]; then

    error "Invalid virtual environment specified: $virtualenv"

    exit "$ERROR_ARGUMENTS"

fi

# Activate the virtual environment if it was specified.

[[ -n $virtualenv ]] && source "$virtualenv/bin/activate"

# Verify pip-compile is available.

if ! which pip-compile >/dev/null 2>&1; then

    error "Could not find command pip-compile from packagep pip-tools. Package might not be available in the virtual environment."

    exit "$ERROR_PIP_TOOLS_MISSING"

fi

# Create temporary files where files where "normalised" outputs will be

# stored.

tmp_current=$(mktemp)

tmp_new=$(mktemp)

tmp_new_sorted=$(mktemp)

# Process each environment.

for environment in "$config_dir"/*; do

    # Empty directory.

    if [[ ! -e $environment ]]; then

	error "Configuration directory is empty: $config_dir"

	exit "$ERROR_CONFIG_DIR"

    fi

    # Process each .in file.

    for req_in in "$environment"/*.in; do

	# Directory without .in files.

	if [[ ! -e $req_in ]]; then

	    error "No .in files in directory, skipping: $environment"

	    continue

	fi

	# Figure out if .txt file exists.

	req_txt="${req_in%.in}.txt"

	if [[ ! -f $req_txt ]]; then

	    continue

	fi

	# Deploy the existing requirements file and the new one.

	sed -e 's/[[:blank:]]*#.*//' "$req_txt" | grep -v "^$" | sort -u > "$tmp_current" 

	if ! pip-compile --no-header --no-annotate --no-index --output-file "$tmp_new" --upgrade "$req_in" > /dev/null; then

	    error "Failed while running pip-compile command against (see error stack trace above): $req_in"

	    continue

	fi

        # Sort the new requirements file.

        sed -e 's/[[:blank:]]*#.*//' "$tmp_new" | grep -v "^$" | sort -u > "$tmp_new_sorted"

        debug "Current requirements:\n$(cat "$tmp_current")"

        debug "New requirements:\n$(cat "$tmp_new_sorted")"

	# Run diff, storing the output and result.

	diff=$(diff -u "$tmp_current" "$tmp_new_sorted")

	result="$?"

	# Show warning about available updates.

	if [[ $result == 0 ]]; then

	    [[ $quiet == 0 ]] && info "No upgrades available for: $req_txt"

	else

	    warning "Upgrades available for: $req_txt"

	    echo

	    echo "$diff"

	    echo

	fi

    done

done