* Backup server, where the backups will be stored at.

     pip install -U pip

     pip install 'ansible~=1.9'

   :file:`~/mysite/ansible.cfg`

   ::

   :file:`~/mysite/hosts`

   ::

4. Create a number of directories for storing playbooks, group variables, SSH

   keys, and GnuPG keyring (we'll get to this later)::

     mkdir ~/mysite/ssh/

     mkdir ~/mysite/gnupg/

5. Before moving ahead, we should also create SSH private/public key pair that

   :file:`~/mysite/playbooks/preseed.yml`

   ::

   :file:`~/mysite/group_vars/preseed.yml`

   ::

benefit of having the authorized_keys set-up for the root operating system user,

   :file:`~/mysite/playbooks/bootstrap.yml`

   ::

      - hosts: [communications, web, backup]

      changes (especially the user passwords/info). Backups of LDAP directory on

      regular basis are important. We will get to that at a later point.

Setting-up mail relaying from web and backup servers

----------------------------------------------------

server on web and backup servers to relay mails via the communications

server. This way we can make sure that mail that gets sent via local SMTP to

external addresses on those two servers goes through our anti-virus scanner.

1. Update the list of roles for web and backup server to include the mail

   forwarder role.

          - ldap_client

   :file:`~/mysite/playbooks/backup.yml`

      - hosts: backup

        remote_user: ansible

        sudo: yes

        roles:

          - common

          - mail_forwarder

2. The next thing is to set-up the configuration for the new role. We can define

   this globally for all servers

   :file:`~/mysite/group_vars/all.yml`

   ::

3. Although we have told our web and backup servers to use the communications

   server as relay for non-local mail, the communications server is not aware of

      # We want to allow relaying of mails from our web and backup servers

      # here.Beware the IP spoofing, though! Don't forget to change the bellow

      # IP for your server ;)

         - 10.32.64.23

5. After this you may want to test out sending mail via web or backup server's

   local SMTP to the root user (to see if the aliasing works), and to some

   external mail address to check if forwarding works correctly too. Run some

   something similar to the following on your web server::

of the other servers.

Let us first define what we want to deploy on the web server. Here is the plan:

   values. But let us spicen up things a bit nevertheless. No configuration has

   been deployed before for the web server, so we will be creating a new file.

      ---

      from django import setup

      setup()

Backups, backups, backups!

--------------------------

As it is well known, everyone has backups of their important data. Right?

Riiiiight?

There are two Ansible roles that implement backup functionality. One is the

``backup_server`` role, while the other is ``backup_client`` role. Backup is

based around the use of `Duplicity <http://duplicity.nongnu.org/>`_ and its

convenience wrapper, `Duply <http://duply.net>`_. Due to this selection, it

should be noted that the backup clients are the ones making connection to the

backup server (not the other way around).

Backups are encrypted and signed using GnuPG before being stored on the backup

server. Private key used for encryption and signing is therefore stored on the

client side. This key should not be encrypted in order to allow for unattended

backups.

Although not necessary, it is highly recommended to have separation between

different backup clients and the keys used for encryption and

signing. I.e. stick to a separate encryption/signing key for each backup

client. It should be noted that it is also possible to specify additional

*public* keys to encrypt with. This lets you have a backup decryptable with some

other, "master" key.

The backups are transferred to the backup server via SFTP - the

``backup_server`` role sets-up a dedicated OpenSSH server instance that limits

the connecting clients to a SFTP chroot.

All backups are stored within directory ``/srv/backups`` (on the backup

server). Within this directory, every client server has a dedicated

sub-directory, and within this sub-directory another sub-directory called

``duplicity``, where the actual *Duplicity* backups are stored. So, for example,

the directory where backups for ``www.example.com`` are stored at would be

``/srv/backups/www.example.com/duplicity``.

When backups are configured, they are set-up to be running every morning at

02:00. Before the backup run it is possible to run a preparation task, and a lot

of roles do this in order to create database dumps etc.

Setting-up the backup server

----------------------------

With the overview of backups out of the way, it is time to set-up the backup

server itself first. This is a farily simple task to perform, so let's get

straight to it:

1. Update the playbook for backup server to include the backup server role.

   :file:`~/mysite/playbooks/backup.yml`

   ::

      ---

      - hosts: backup

        remote_user: ansible

        sudo: yes

        roles:

          - common

          - mail_forwarder

          - backup_server

2. There is just one mandatory parameter for the role - OpenSSH server keys to

   be used for backup-dedicated instance:

   :file:`~/mysite/group_vars/backup.yml`

   ::

      ---

      backup_host_ssh_private_keys:

        dsa: "{{ lookup('file', inventory_dir + '/ssh/bak_dsa_key') }}"

        rsa: "{{ lookup('file', inventory_dir + '/ssh/bak_rsa_key') }}"

        ed25519: "{{ lookup('file', inventory_dir + '/ssh/bak_ed25519_key') }}"

        ecdsa: "{{ lookup('file', inventory_dir + '/ssh/bak_ecdsa_key') }}"

3. Since we have decided to specify the keys above through file lookup, the

   above-listed keys now need to be generated::

     ssh-keygen -f ~/mysite/ssh/bak_dsa_key -N '' -t dsa

     ssh-keygen -f ~/mysite/ssh/bak_rsa_key -N '' -t rsa

     ssh-keygen -f ~/mysite/ssh/bak_ed25519_key -N '' -t ed25519

     ssh-keygen -f ~/mysite/ssh/bak_ecdsa_key -N '' -t ecdsa

Adding backup clients

---------------------

Well, that was all nice and dandy, but it does look like something is

missing... Ah, we haven't really configured any backup clients, right?

Surprisingly enough, specifying backup clients is optional, but that won't get

you far.

Luckily for you, all relevant *Majic Ansible Roles* are *backup-aware*. In other

words, all the roles have been implemented with support for doing back-ups - it

is just that by default this functionality is disabled (since you might be

relying on some other schema to back things up - LVM snapshots or what-not).

All that is needed is to enable the backups for each role, and provide some

extra variables required by the ``backup_client`` role.

For this a set of GnuPG private keys are necessary. These need to be provided as

ASCII-armoured GnuPG-exported files. For simplicity sake, this example documents

use of GnuPG keyring in conjunction with Ansible's ``pipe`` lookup.

So, back to the business:

1. Update the backup server configuration - each client needs to be explicitly

   registered:

   :file:`~/mysite/group_vars/backup.yml`

   ::

      backup_clients:

        - server: comms.example.com

          public_key: "{{ lookup('file', inventory_dir + '/ssh/comms.example.com.pub') }}"

          ip: 10.32.64.19

        - server: www.example.com

          public_key: "{{ lookup('file', inventory_dir + '/ssh/www.example.com.pub') }}"

          ip: 10.32.64.20

        # Ah, this is a bit interesting - we can back-up the backup server

        # itself! Don't worry, though, this is mainly so the logs and home

        # directories are preserved, so it won't take too much space ;)

        - server: bak.example.com

          public_key: "{{ lookup('file', inventory_dir + '/ssh/bak.example.com.pub') }}"

          ip: 127.0.0.1

2. And now to configure backup clients for all servers:

   :file:`~/mysite/group_vars/all.yml`

   ::

      enable_backup: yes

      backup_encryption_key: "{{ lookup('pipe', 'gpg2 --homedir ~/mysite/gnupg/ --armour --export-secret-keys ' + ansible_fqdn ) }}"

      backup_server: bak.example.com

      backup_server_host_ssh_public_keys:

        - "{{ lookup('file', inventory_dir + '/ssh/bak_dsa_key.pub') }}"

        - "{{ lookup('file', inventory_dir + '/ssh/bak_rsa_key.pub') }}"

        - "{{ lookup('file', inventory_dir + '/ssh/bak_ed25519_key.pub') }}"

        - "{{ lookup('file', inventory_dir + '/ssh/bak_ecdsa_key.pub') }}"

      backup_ssh_key: "{{ lookup('file', inventory_dir + '/ssh/' + ansible_fqdn) }}"

3. So, looking at the configuration up there, there is a couple of file lookups

   for getting the variable values, as well as one pipe lookup for fetching the

   encryption keys. For start, let's create the SSH private keys used for client

   log-ins to backup server::

     ssh-keygen -f ~/mysite/ssh/comms.example.com -N ''

     ssh-keygen -f ~/mysite/ssh/www.example.com -N ''

     ssh-keygen -f ~/mysite/ssh/bak.example.com -N ''

4. Next off, a GnuPG keyring needs to be populated with private keys that will

   be used for backup encryption and signing. In total, we need three keys, one

   for each server. The keys should not be encrypted, and they should be named

   after the respective server's FQDN. For simplicity sake, here is a nice

   copy-pastable batch version for doing so:

   .. note:: Key generation requires a lot of entropy. If you are running this

             command on a VM, you may want to ``apt-get install haveged`` to

             speed this up. Please do read up on haveged if deploying to a real

             system, though! Don't trust it blindly!

   ::

     chmod 700 ~/mysite/gnupg

     cat << EOF | gpg2 --homedir ~/mysite/gnupg --batch --gen-key

     Key-Type:RSA

     Key-Length:1024

     Name-Real:comms.example.com

     Expire-Date:0

     %commit

     Key-Type:RSA

     Key-Length:1024

     Name-Real:www.example.com

     Expire-Date:0

     %commit

     Key-Type:RSA

     Key-Length:1024

     Name-Real:bak.example.com

     Expire-Date:0

     %commit

     EOF

5. And... Apply::

     workon mysite && ansible-playbook playbooks/site.yml

6. At this point the backup roles have been set-up everywhere, and the backups

   will be running every day at 02:00 in the morning. Of course, you may want to

   test out a backup yourself immediatelly by running the following command on

   servers::

     duply main backup

   .. note:: For more information on available commands and how to work with

             backup tool, please have look at `Official Duply Pages

             <http://duply.net/>`_.

Adding backup support to custom roles

-------------------------------------

As mentioned before, all of the supplied roles coming with *Majic Ansible Roles*

include backup support. What gets backed-up depends on the role implementation

(see role reference for details). What about backup support for custom roles?

This is something that has to be done by hand. However, it is quite simple to do

so.

Backup integration will be demonstrated with the previously implemented

``tbg`` role.

*The Bug Genie* stores most of its data in database, but thanks to the

``database`` role its backup is already handled for us. As a side-note, just

before every backup run the database is dumped and stored in location

``/srv/backup/tbg.sql``. That file is subsequently backed-up via *Duply* run.

What is not backed-up for us, though, are the files uploaded to *The Bug

Genie*. So let's fix that one.

1. Add the backup client role to list of dependencies.

   :file:`~/mysite/roles/tbg/meta/main.yml`

   ::

        - role: backup_client

2. Add task to the ``tbg`` role that deploys file containing additional patterns

   that should be backed-up from the disk during backup runs:

   :file:`~/mysite/roles/tbg/tasks/main.yml`

   ::

      - name: Configure back-up of The Bug Genie uploaded files

        copy: content="/var/www/tbg.example.com/files" dest="/etc/duply/main/patterns/tbg"

              owner="root" group="root" mode=600

        notify:

          - Assemble Duply include patterns

   .. warning:: Don't forget to notify the ``Assemble Duply include patterns``

                handler! This handler will collect all the different patterns

                deployed by various roles in a single place for backup software

                to pick-up.

3. Apply the changes::

     workon mysite && ansible-playbook playbooks/site.yml

4. Now rerun the backup on server ``www.example.com`` (as root). If you haven't

   uploaded any files, you may want to do so before testing to make sure

   something is backed-up.

   ::

     duply main backup

5. Verify that the files have been backed-up:

   ::

      duply main list

.. note:: If you wanted to run a script prior to backup run, you would simply

          deploy a shell script with desired content to

          ``/etc/duply/main/pre.d/``. Just make sure the permissions for it are

          ok (it has to be executable by the root user).