SHIELD Documentation

Plugins Reference

Target Plugins

Target plugins are used to perform backup and restore operations against actual data systems like PostgreSQL, MySQL, and Consul.

Local Filesystem

The fs plugin lets you back up and restore files and directories on locally attached (or locally-accessible) disks. It performs backups by creating a POSIX tar archive of the selected file set, which it unpacks on restore. No compression is used directly by the plugin.

Note: In previous versions of SHIELD (notably 7.x and below), the fs plugin could also be used as a storage plugin. This caused large amounts of confusion, and often lead to data loss. As of SHIELD 8.x, this behavior has been removed outright.

Configuration Options

Matching Files for include and exclude

The include and exclude options let you selectively include and exclude files from the backup archive. These options each take a single shell glob which will be matched against each file name.

Shell globs have the following syntax:

*      Match zero or more characters.
?      Match zero or one character.
[a-z]  Match a range of characters, in this example,
       lowercase alphabetic characters.
[^a-z] Match anything that doesn't match [a-z]
[xaby] Match either 'x', 'a', 'b', or 'y'
\*     Match a literal asterisk.
       Also works for '?', '[', and ']'

For example, the file name /var/data/index/foo.db would match all of the following globs:

  1. foo.*
  2. * (but everything matches this so...)
  3. *.db
  4. foo?.[a-f]*

It does not match the following globs:

  1. FOO.* - globs are case sensitive.
  2. */index/* - matching is not applied to the path.
  3. (foo|bar).* - globs are not regular expressions.

The fs plugin evaluates exclusions and then inclusions. This makes it easy to handle the 80% case of back up everything except the *.tmp files. By default, nothing is excluded and everything is included, so if you know specifically what you want to back up, you can specify just the include pattern.

Here's a few illustrative examples (base_dir assumed present, but not shown):

PostgreSQL Database

The postgres plugin lets you back up and restore PostgreSQL databases. It uses standard PostgreSQL tooling for backups: pg_dump and pg_dumpall for backups, and psql to restore.

During restore, connected clients will be forcibly disconnected so that the databases they were using can be dropped and recreated. This happens transparently, but does require that client software be able to gracefully reconnect as needed.

Configuration Options

Co-locating the SHIELD PostgreSQL Addon

If you are running the SHIELD agent on a different machine than your PostgreSQL database server, you will need to install the SHIELD PostgreSQL Addon to get the pg_dump, pg_dumpall and psql tools.

Add the release to the top-level releases: section in your agent deployment manifest:

releases:
  - name:    shield-addon-postgres
    version: latest   # or a specific version

Then add the job that corresponds to the version of PostgreSQL you want to backup:

instance_groups:
  - name: your-shield-agent
    jobs:
      - release: shield-addon-postgres
        name:    shield-addon-postgres-10.1

Check the add-on's README for full details of available versions.

MySQL / MariaDB

Backing Up Via mysqldump

The mysql plugin lets you back up and restore MySQL / MariaDB databases, using the pure-SQL mysqldump utility. For very large database, the time spent serializing database records into SQL commands may be prohibitive, and you may want to investigate use of the xtrabackup plugin instead.

Restores are handled by feeding the SQL backup through the standard mysql utility. Connected clients will remain connected during restore.

Configuration Options

Backing Up Via xtrabackup

The xtrabackup plugin lets you back up and restore your MySQL / MariaDB databases using a filesystem-based approach leveraging Percona's XtraBackup utility. This method is often faster, if less portable, than the pure SQL approach taken by the mysql plugin.

Note: Since xtrabackup requires access to the MySQL / MariaDB data directory, it can only be run from the database host itself.

Configuration Options

Co-locating the SHIELD MariaDB Addon

If your MySQL installation does not include xtrabackup, you will need to install the SHIELD MariaDB Addon.

Add the release to the top-level releases: section in your agent deployment manifest:

releases:
  - name:    shield-addon-mysql
    version: latest   # or a specific version

Then add the job that corresponds to the version of MySQL or MariaDB you want to backup:

instance_groups:
  - name: your-shield-agent
    jobs:
      - release: shield-addon-mysql
        #name:   shield-addon-xtrabackup-2.4

Check the add-on's README for full details of available versions.

MongoDB NoSQL

The mongo plugin lets you back up and restore your MongoDB databases. It relies on the mongodump and mongorestore utilities, installed on the agent host.

Configuration Options

Co-locating the SHIELD MongoDB Add-on

If you are running the SHIELD agent on a different machine than your MongoDB database server, you will need to install the SHIELD MongoDB Add-on to get the mongodump and monogrestore tools.

Add the release to the top-level releases: section in your agent deployment manifest:

releases:
  - name:    shield-addon-mongodb
    version: latest   # or a specific version

Then add the job that corresponds to the version of MongoDB you want to backup:

instance_groups:
  - name: your-shield-agent
    jobs:
      - release: shield-addon-mongodb
        name:    shield-addon-mongo-tools-3.4

Check the add-on's README for full details of available versions.

Consul Key-Value Store

The consul plugin lets you back up a Consul key-value store. It works by walking the entire store and capturing the value of every key in the hierarchy.

On restore, keys from the backup archive will be put back into the running key-value store, but existing keys will not be removed.

Configuration Options

CF Service Brokers

Dockerized PostgreSQL CF Service Broker

The docker-postgres plugin lets you back up data in a Dockerized PostgreSQL service broker that uses the cf-containers-broker provided by the docker-boshrelease (https://github.com/cloudfoundry-incubator/docker-boshrelease).

This is a highly specific configuration, and this plugin does not work with other PostgreSQL + Docker combinations.

Note: This plugin must be executed on the system that runs the container broker, which requires a co-located SHIELD agent in almost all cases.

Configuration Options

This plugin has no configuration options.

How It Works (Implementation Details)

Backups are performed by connecting to the local Docker daemon, and finding all the running containers. It then iterates over that list, compiles some metadata about port mappings (from Docker), and executes a pg_dump backup on the running PostgreSQL instance. All of this information is then stored in a POSIX tar archive.

Restore operations work through the tar archive, bringing up a new Docker container for each entry, remapping ports, and restoring the database. This is a service-affecting operation, since any existing container images will be destroyed, and any connected clients will be disconnected.

Cloud Foundry RabbitMQ Service Broker

The rabbitmq-broker plugin lets you back up the configuration of the Cloud Foundry RabbitMQ Service Broker (https://github.com/pivotal-cf/cf-rabbitmq-release). However, given that it only uses stock RabbitMQ API calls, it may work with other configurations.

Note: RabbitMQ is usually used as a non-durable message queue and dispatch / routing system. As such, this plugin only backs up the metadata of a RabbitMQ brokered installation, not the actual messages.

This plugin uses the RabbitMQ Management API, which requires that the rabbitmq_management plugin be loaded into your RabbitMQ installation.

Configuration Options

Cloud Foundry Redis Service Broker

The redis-broker plugin lets you back up the configuration of the Cloud Foundry Redis Service Broker (https://github.com/pivotal-cf/cf-redis-release). It is not a general-purpose Redis backup plugin, and will not work with stock Redis installations.

Note: Since Redis does not allow backups across the network, any target using this plugin must execute on a co-located SHIELD agent, on the Redis VM.

Configuration Options
How It Works (Implementation Details)

A Redis Service Broker deployment features two types of VMs: shared and dedicated. A shared VM runs multiple Redis processes, each bound on their own port, with their own append-only (AOF) file. A dedicated VM runs a single Redis process (usually on the standard port).

The plugin backs up all data in /var/vcap/store, which works regardless of the type of Redis VM being targeted.

How the plugin restores data depends on the type of Redis VM.

For shared VMs, a restore will stop the service broker process, terminate all Redis instances, and extract the backup archive back into /var/vcap/store. Then, it validates the AOF and resolves any corruption issues that may have occurred during backup (like a mid-write snapshot). Finally, it restarts the service broker process, which will then re-launch all of the configured Redis instances.

For dedicated VMs, a restore will stop the service broker's agent process, and the Redis instance itself, and extract the backup archive back into /var/vcap/store. Then it validates the AOF and resolves any corruption issues that may have occurred during backup (like a mid-write snapshot). Finally, it restarts the agent and Redis process.

Restore operations are service-impacting, as the Redis instances are shut down for the duration of the restore. Additionally, the service broker apparatus is disabled, to prevent creation of new service instances during the restoration.

BOSH Backup / Restore

BOSH Backup / Restore (for deployments)

The bbr-deployment plugin lets you back up a BOSH deployment using BBR.

Configuration Options
Co-locating the SHIELD BBR Addon

If you are running the SHIELD agent on a machine that does not provide the bbr executable, you will need to install the SHIELD BBR Addon.

Add the release to the top-level releases: section in your agent deployment manifest:

releases:
  - name:    shield-addon-bbr
    version: latest   # or a specific version

Then add the job to install bbr:

instance_groups:
  - name: your-shield-agent
    jobs:
      - release: shield-addon-bbr
        name:    bbr

Check the add-on's README for more information.

BOSH Backup / Restore (for directors)

The bbr-director plugin lets you back up a BOSH director using BBR.

Configuration Options
Co-locating the SHIELD BBR Addon

If you are running the SHIELD agent on a machine that does not provide the bbr executable, you will need to install the SHIELD BBR Addon.

Add the release to the top-level releases: section in your agent deployment manifest:

releases:
  - name:    shield-addon-bbr
    version: latest   # or a specific version

Then add the job to install bbr:

instance_groups:
  - name: your-shield-agent
    jobs:
      - release: shield-addon-bbr
        name:    bbr

Check the add-on's README for more information.

Storage Plugins

Amazon S3

The s3 plugin lets you store backup archives in an Amazon AWS Simple Storage Service bucket. In theory, this plugin should also work with other implementations, not from Amazon, which we term S3 work-alikes.

Backup archives will be stored in a file name / path that encodes the date and time of the backup operation, to make it easier to track down a specific archive later:

$prefix/YYYY/MM/DD/YYYY-MM-DD-HHmmSS-$uuid

for example, given a prefix of "prod/backups", a backup might be stored at:

prod/backups/2018/07/12/2018-07-12-134255-f3b564f2-ef62-4e38-9d94-ba17c37abf09

Configuration Options

Google Cloud Storage

The google plugin lets you store backup archives in Google Cloud's blobstore storage system, which conceptually behaves a lot like Amazon's S3.

Backup archives will be stored in a file name / path that encodes the date and time of the backup operation, to make it easier to track down a specific archive later:

$prefix/YYYY/MM/DD/YYYY-MM-DD-HHmmSS-$uuid

for example, given a prefix of "prod/backups", a backup might be stored at:

prod/backups/2018/07/12/2018-07-12-134255-f3b564f2-ef62-4e38-9d94-ba17c37abf09

Configuration Options

Microsoft Azure

The azure plugin lets you store backup archives in Microsoft Azure's Blobstore, in a storage container.

Backup archives will be stored in a file name / path that encodes the date and time of the backup operation, to make it easier to track down a specific archive later:

$prefix/YYYY-MM-DD-HHmmSS-$uuid

for example, given a prefix of "prod/backups", a backup might be stored at:

prod/backups/2018-07-12-134255-f3b564f2-ef62-4e38-9d94-ba17c37abf09

Configuration Options

OpenStack Swift

The swift plugin lets you store backup archives in an OpenStack Swift Blobstore.

Backup archives will be stored in a file name / path that encodes the date and time of the backup operation, to make it easier to track down a specific archive later:

$prefix/YYYY/MM/DD/HHmmSS-$uuid

for example, given a prefix of "prod/backups", a backup might be stored at:

prod/backups/2018/07/12/134255-f3b564f2-ef62-4e38-9d94-ba17c37abf09

Configuration Options

WebDAV Filesystem

The webdav plugin lets you store backup archives in any WebDAV server that complies with RFC 2518. This includes Apache and Nginx.

Configuration Options