From 9c7586c7c3a235672ec6490d1a8bc44a222ce5d1 Mon Sep 17 00:00:00 2001 From: Colin Wilk Date: Tue, 1 Sep 2026 21:01:05 +0200 Subject: Add borg prune and compact jobs Run repository retention either after a successful backup or from a dedicated systemd timer. Clean up script generation with templates and expand molecule test coverage. BREAKING CHANGE: Aggregate backup scripts are no longer managed, and state=preset now requires at least one readable included directory. --- CHANGELOG.md | 105 +++++---- README.md | 124 +++++----- defaults/main.yml | 93 +++++++- meta/argument_specs.yml | 149 ++++++++++++ molecule/default/converge.yml | 164 +++++++++++++ molecule/default/molecule.yml | 17 ++ molecule/default/side_effect.yml | 167 ++++++++++++++ molecule/default/tests/test_client_setup.py | 329 +++++++++++++++++++++------ molecule/default/tests/test_manual_backup.py | 122 ++++++++++ molecule/default/tests/test_server_setup.py | 2 + molecule/default/tests/test_systemd.py | 84 ++++++- molecule/delete/converge.yml | 8 + molecule/delete/prepare.yml | 4 + molecule/delete/tests/test_delete.py | 34 ++- tasks/absent.yml | 74 +++--- tasks/client_create_scripts_each.yml | 34 --- tasks/client_setup.yml | 135 +++++++---- tasks/main.yml | 3 + tasks/validate.yml | 8 + tasks/validate_absent.yml | 10 + tasks/validate_present.yml | 104 +++++++++ templates/borg_backup.service.j2 | 5 +- templates/borg_backup_script.j2 | 15 ++ templates/borg_prune.service.j2 | 17 ++ templates/borg_prune.timer.j2 | 11 + templates/borg_prune_script.j2 | 71 ++++++ 26 files changed, 1585 insertions(+), 304 deletions(-) create mode 100644 molecule/default/side_effect.yml delete mode 100644 tasks/client_create_scripts_each.yml create mode 100644 tasks/validate.yml create mode 100644 tasks/validate_absent.yml create mode 100644 tasks/validate_present.yml create mode 100644 templates/borg_backup_script.j2 create mode 100644 templates/borg_prune.service.j2 create mode 100644 templates/borg_prune.timer.j2 create mode 100644 templates/borg_prune_script.j2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 653cbcb..9b09d12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ All notable changes to this project will be documented in this file. ### Breaking Changes -#### Decryption keys file format changed +#### Decryption keys file format changed (`a22ff18`) **Am I affected?** You have a `decryption_keys.yml` file with entries from previous versions. @@ -39,74 +39,79 @@ format. --- -#### Backup script block markers changed +#### Aggregate backup and prune scripts are no longer managed **Am I affected?** -Yes. +You manually run the unsuffixed `run_borg_backup` or `run_borg_prune` script +while `borg_backup_argument` is not empty. **What changed?** -Block markers in the backup script now include repository name to support -multiple repos per host. +The role now creates one complete script per repository at +`run_borg_backup@ARGUMENT` and, whenever pruning is enabled, +`run_borg_prune@ARGUMENT`. Systemd and manual runs use these repository-specific +scripts. Scripts are rendered atomically from templates instead of assembled +with `blockinfile`. -Old: - -```bash -## BEGIN ANSIBLE MANAGED BLOCK for server: backup-server -``` - -New: - -```bash -## BEGIN ANSIBLE MANAGED BLOCK for backup-server/my-repo -``` +Existing unsuffixed scripts are left untouched when `borg_backup_argument` is +not empty because the same path may belong to another repository configured +with an empty argument. If `borg_backup_argument` is empty, that repository's +script continues to use the unsuffixed path and is managed normally. **Migration:** - - - -Delete the script and re-run the role: +Run each repository-specific script explicitly, or invoke the corresponding +systemd service. For example: ```bash -rm /usr/local/bin/run_borg_backup -# Then run your playbook +/usr/local/bin/run_borg_backup@ARGUMENT +systemctl start borg_backup@ARGUMENT.service ``` +After confirming that no repository uses an empty `borg_backup_argument`, you +may manually remove legacy aggregate scripts. + --- -#### Default backup argument +#### Backup source validation is now enforced **Am I affected?** -You are using the default value of `borg_backup_argument`. +You use `state: present` with an empty `borg_included_dirs` list, or one of its +paths is missing or unreadable by `borg_client_user`. -**What will change?** -Default will change from `{{ borg_server_host_url }}` to -`{{ borg_server_host_url }}-{{ borg_repo_name }}`. +**What changed?** +The role now rejects empty `borg_included_dirs`, missing included paths, and +paths that `borg_client_user` cannot read. Previously, an empty list was +accepted. **Migration:** - - - -Systemd unit names will change. Manually migrate: - -```bash -# Stop old units -systemctl stop borg_backup@OLD-VALUE.timer -systemctl disable borg_backup@OLD-VALUE.timer - -# Run role to create new units -# Then enable new units -systemctl enable borg_backup@NEW-VALUE.timer -systemctl start borg_backup@NEW-VALUE.timer -``` +Configure at least one existing path that `borg_client_user` can read in +`borg_included_dirs` whenever using `state: present`. ### Added - Multi-instance backup support (multiple repositories per client host) -- Non-root backup user support via `borg_client_user` variable -- Configurable SSH key type (`borg_ssh_key_type`) with support for - ed25519, rsa, and ecdsa -- Per-repo SSH key support (`borg_ssh_key_per_repo`) for independent keys per repository -- Storage quota support (`borg_storage_quota`) to limit repository size on server -- Comprehensive test suite including disaster recovery scenarios -- Negative security tests for cross-host repository isolation -- Appendix-only repository mode (`borg_mode_append_only`) + (`a22ff18`) +- Non-root backup user support via `borg_client_user` variable (`cce7d2d`) +- Configurable SSH key type (`borg_ssh_key_type`) with support for ed25519, + rsa, and ecdsa (`cce7d2d`) +- Per-repo SSH key support (`borg_ssh_key_per_repo`) for independent keys per + repository (`8d87206`) +- Storage quota support (`borg_storage_quota`) to limit repository size on + server (`8b0175c`) +- Option to disable decryption key export by setting + `borg_decryption_keys_yaml_path` to an empty string (`785e6c8`) +- Configurable server-side borg user via `borg_server_user` and + `borg_server_user_create` (`7d96c1a`) +- Custom `borg create` arguments via `borg_create_additional_arguments` + (`66e46df`) +- Repository removal support via `state: absent` and + `borg_dangerously_delete_backups` (`50b914e`) +- Automatic repository retention with `borg prune`, optional `borg compact`. + +### Changed + +- Declared `community.crypto` dependency in role metadata (`fa137a9`) + +### Fixed + +- Read `getent` user data from `ansible_facts`, restoring compatibility with + current Ansible fact injection behavior (`1143a27`). diff --git a/README.md b/README.md index 1eab45d..f5690f5 100644 --- a/README.md +++ b/README.md @@ -160,7 +160,8 @@ your Borg backups, but it is not used by default. `borg_included_dirs` and `borg_excluded_dirs` finally picks what directories -will be backed up by Borg. +will be backed up by Borg. `borg_included_dirs` must not be empty when +`state: present`. ```yaml borg_compression: zstd # -C argument @@ -182,6 +183,66 @@ borg_create_additional_arguments: "--stats --list --filter=AME" borg_create_additional_arguments: "--one-file-system --exclude-caches" ``` +### Prune and Compact Configuration + +The role can manage Borg retention with `borg prune` and optionally run `borg +compact` afterwards to actually free repository disk space. + +Enable pruning with `borg_prune_enabled`. Pruning always runs in a dedicated +systemd service. By default, `borg_prune_trigger` is `after_backup`, so a +successful backup service starts the prune service. The backup script itself +only runs `borg create`; the prune script runs `borg prune` and optionally +`borg compact`. + +If you want prune on its own schedule, set `borg_prune_trigger: timer`. That +creates a dedicated prune timer using `borg_prune_timer_name`, +`borg_prune_systemd_oncalendar`, and `borg_prune_systemd_accuracysec` to start +the same prune service. + +You can set an archive filter via `borg_prune_glob_archives` so prune only +touches the archive series for this backup job. + +```yaml +borg_prune_enabled: true +borg_prune_trigger: after_backup +borg_prune_glob_archives: "{hostname}-*" + +borg_prune_keep_daily: "7" +borg_prune_keep_weekly: "4" +borg_prune_keep_monthly: "6" +borg_prune_keep_yearly: "1" + +borg_prune_compact_enabled: true +borg_compact_threshold: 10 +``` + +Example with a separate weekly prune timer: + +```yaml +borg_prune_enabled: true +borg_prune_trigger: timer +borg_prune_systemd_oncalendar: "Sun *-*-* 04:00:00" +borg_prune_systemd_accuracysec: 60min +borg_prune_glob_archives: "{hostname}-*" +borg_prune_keep_daily: "7" +borg_prune_keep_weekly: "8" +borg_prune_keep_monthly: "12" +borg_prune_compact_enabled: true +borg_compact_threshold: 1 +``` + +For the full list of supported variables, see: +[`defaults/main.yml`](defaults/main.yml) as well as the relevant Borg +documentation: + +- +- + +> [!INFO] +> Prune and compact are not compatible with append-only repositories from the +> client side. The role will fail early if you enable both +> `borg_prune_enabled: true` and `borg_mode_append_only: true`. + To decrypt your backups without the client, we store the decryption keys in a YAML file in your Ansible repository. You require the decryption keys as well as access to the repository files on the Borg server to access the backups. @@ -211,10 +272,12 @@ The names of the systemd service and timer are: `{{ borg_backup_timer_name }}{{ borg_backup_argument }}.service` and `{{ borg_backup_timer_name }}{{ borg_backup_argument }}.timer`. -For the backup scripts we add `{{ borg_backup_script_location }}` for creating a -backup on all specified targets and -`{{ borg_backup_script_location }}{{ borg_backup_argument }}` for backing up to -each target. +Each target gets exactly one repository-specific backup script at +`{{ borg_backup_script_location }}@{{ borg_backup_argument }}`. Systemd executes +that script, and the same script can be called manually. The role does not +create an aggregate script that runs all configured targets. If +`borg_backup_argument` is empty, the script uses the unsuffixed +`borg_backup_script_location` path. To configure the backup schedule, we offer `borg_systemd_oncalendar` and `borg_systemd_accuracysec`, which map to the corresponding systemd options, @@ -523,66 +586,21 @@ To remove a specific repository (applies to both single and multi-instance setup rm /usr/local/bin/run_borg_backup@ARGUMENT ``` -3. _(Multi-instance only)_ Edit the base backup script to remove the repository block: - - ```bash - # Edit /usr/local/bin/run_borg_backup - # Remove the ANSIBLE MANAGED BLOCK for the deleted repository - ``` - -4. _(If other repositories remain)_ Update `authorized_keys` on the Borg server +3. _(If other repositories remain)_ Update `authorized_keys` on the Borg server to remove the repository restriction. Edit `/opt/borg/.ssh/authorized_keys` and remove the `--restrict-to-repository /opt/borg/REPONAME` from the appropriate line. If this is the last repository for the host, remove the entire line instead. -5. Delete the repository on the Borg server: +4. Delete the repository on the Borg server: ```bash ssh borg@SERVER "borg delete /opt/borg/REPONAME" # Or simply: ssh borg@SERVER "rm -rf /opt/borg/REPONAME" ``` -6. Remove the decryption key entry from your `decryption_keys.yml`. - -## Deprovisioning a Complete Host - -To remove all backup configuration for a host: - -1. Stop and disable all systemd timers and services: - - ```bash - systemctl list-units --type=timer --all | grep borg_backup - # For each relevant timer: - systemctl stop borg_backup@*.timer - systemctl disable borg_backup@*.timer - ``` - -2. Remove all backup scripts: - - ```bash - rm -f /usr/local/bin/run_borg_backup* - ``` - -3. Remove the SSH key from the Borg server's `authorized_keys`: - - ```bash - # On the Borg server, edit /opt/borg/.ssh/authorized_keys - # Remove the line containing root@HOSTNAME - ``` - -4. Delete all repositories for the host on the Borg server: - - ```bash - ssh borg@SERVER "rm -rf /opt/borg/HOSTNAME" - # For multi-instance: - ssh borg@SERVER "rm -rf /opt/borg/HOSTNAME_repo1" - ssh borg@SERVER "rm -rf /opt/borg/HOSTNAME_repo2" - ``` - -5. Remove all decryption key entries for the host from your - `decryption_keys.yml`. +5. Remove the decryption key entry from your `decryption_keys.yml`. ## Dependencies diff --git a/defaults/main.yml b/defaults/main.yml index 1c9d05d..1872e8b 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -144,8 +144,67 @@ borg_compression: zstd # See: https://borgbackup.readthedocs.io/en/stable/usage/create.html borg_create_additional_arguments: "" +################################################################################ +# Borg Prune / Compact Configuration +# See: https://borgbackup.readthedocs.io/en/stable/usage/prune.html +# See: https://borgbackup.readthedocs.io/en/stable/usage/compact.html +################################################################################ + +# Enable repository retention management for this backup job. +borg_prune_enabled: false + +# How prune is triggered: +# - after_backup: start the dedicated prune service after a successful backup +# - timer: start the dedicated prune service from its own timer +borg_prune_trigger: after_backup + +# Additional arguments passed to `borg prune`. +borg_prune_additional_arguments: "" + +# Restrict prune to a subset of archives within the repository. +borg_prune_glob_archives: "{hostname}-*" + +# Force pruning of corrupted archives. +borg_prune_force: false + +# Print deletion statistics after prune. +borg_prune_stats: true + +# Print verbose keep/prune output. +borg_prune_list: false + +# Work slower but use less space while pruning. +borg_prune_save_space: false + +# Write checkpoint every N seconds while pruning. +borg_prune_checkpoint_interval: 1800 + +# Borg retention rules. Set to empty string to omit an option. +borg_prune_keep_within: "" +borg_prune_keep_last: "" +borg_prune_keep_minutely: "" +borg_prune_keep_hourly: "" +borg_prune_keep_daily: "" +borg_prune_keep_weekly: "" +borg_prune_keep_monthly: "" +borg_prune_keep_13weekly: "" +borg_prune_keep_3monthly: "" +borg_prune_keep_yearly: "" + +# Run `borg compact` after prune. This is needed to actually free disk space. +borg_prune_compact_enabled: true + +# Additional arguments passed to `borg compact`. +borg_compact_additional_arguments: "" + +# Remove old 17-byte commit-only segments before compaction. +borg_compact_cleanup_commits: false + +# Minimum saved-space threshold in percent for compaction. +borg_compact_threshold: 10 + # This is a list of files and directories to be backed up in the systemd job. -# In case you leave this empty, the role will not create an automatic backup job +# This must not be empty when state=present. borg_included_dirs: [] # This is a list of files and directories that you wish to have excluded from @@ -177,17 +236,16 @@ borg_passphrase: "" # Set to empty string to disable exporting decryption keys entirely. borg_decryption_keys_yaml_path: "{{ inventory_dir }}/decryption_keys.yml" -# The role creates a script for backing up with the configured parameters that -# the regular systemd service then executes. This specifies the default location -# and name where the script is stored. By default, we store it as -# `/usr/local/bin/run_borg_backup` so that you can run `run_borg_backup` from -# your shell to create manual backups. -# When you use multiple backups, this script will trigger all of them. You can -# trigger them individually by calling -# {{ borg_backup_script_location }}@{{ borg_backup_argument }}. +# Base path for repository-specific backup scripts. Unless +# borg_backup_argument is empty, the role appends @{{ borg_backup_argument }}. +# Each script contains exactly one backup job and is used by its systemd service. # See: `borg_backup_argument` variable. borg_backup_script_location: /usr/local/bin/run_borg_backup +# Base path for repository-specific prune/compact scripts. Unless +# borg_backup_argument is empty, the role appends @{{ borg_backup_argument }}. +borg_prune_script_location: /usr/local/bin/run_borg_prune + ################################################################################ # Borg Backup SystemD configuration ################################################################################ @@ -202,6 +260,12 @@ borg_backup_timer_name: borg_backup # will be called {{ borg_backup_service_name }}@{{ borg_backup_argument }} borg_backup_service_name: borg_backup +# Name of the systemd timer that is created when borg_prune_trigger=timer. +borg_prune_timer_name: borg_prune + +# Name of the systemd service created when pruning is enabled. +borg_prune_service_name: borg_prune + # Includes a list of successful exit codes that are accepted as a successful # backup and not throw a systemd failure in addition to exit code 0. # @@ -220,6 +284,11 @@ borg_backup_service_name: borg_backup # https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#SuccessExitStatus= borg_backup_service_successful_exit_status: [] +# Includes a list of successful exit codes accepted by the dedicated prune +# service in addition to exit code 0. This has the same format and behavior as +# borg_backup_service_successful_exit_status. +borg_prune_service_successful_exit_status: [] + # The backup argument is appended to systemd timer / systemd service and the # backup script. It is used to distinguish backup targets from one another # meaning it should be unique per target. @@ -236,3 +305,9 @@ borg_systemd_oncalendar: "*-*-* 02:00:00" # to distribute the load on the backup server. For more information on how to # configure this see: systemd.timer(5) borg_systemd_accuracysec: 60min + +# Schedule for the prune timer when borg_prune_trigger=timer. +borg_prune_systemd_oncalendar: "*-*-* 03:30:00" + +# Accuracy for the prune timer when borg_prune_trigger=timer. +borg_prune_systemd_accuracysec: 60min diff --git a/meta/argument_specs.yml b/meta/argument_specs.yml index 52e85cb..984b1ca 100644 --- a/meta/argument_specs.yml +++ b/meta/argument_specs.yml @@ -102,6 +102,124 @@ argument_specs: required: false default: "" + borg_prune_enabled: + type: bool + required: false + default: false + + borg_prune_trigger: + type: str + required: false + default: after_backup + choices: + - after_backup + - timer + + borg_prune_additional_arguments: + type: str + required: false + default: "" + + borg_prune_glob_archives: + type: str + required: false + default: "{hostname}-*" + + borg_prune_force: + type: bool + required: false + default: false + + borg_prune_stats: + type: bool + required: false + default: true + + borg_prune_list: + type: bool + required: false + default: false + + borg_prune_save_space: + type: bool + required: false + default: false + + borg_prune_checkpoint_interval: + type: int + required: false + default: 1800 + + borg_prune_keep_within: + type: str + required: false + default: "" + + borg_prune_keep_last: + type: str + required: false + default: "" + + borg_prune_keep_minutely: + type: str + required: false + default: "" + + borg_prune_keep_hourly: + type: str + required: false + default: "" + + borg_prune_keep_daily: + type: str + required: false + default: "" + + borg_prune_keep_weekly: + type: str + required: false + default: "" + + borg_prune_keep_monthly: + type: str + required: false + default: "" + + borg_prune_keep_13weekly: + type: str + required: false + default: "" + + borg_prune_keep_3monthly: + type: str + required: false + default: "" + + borg_prune_keep_yearly: + type: str + required: false + default: "" + + borg_prune_compact_enabled: + type: bool + required: false + default: true + + borg_compact_additional_arguments: + type: str + required: false + default: "" + + borg_compact_cleanup_commits: + type: bool + required: false + default: false + + borg_compact_threshold: + type: int + required: false + default: 10 + borg_included_dirs: type: list elements: str @@ -126,6 +244,11 @@ argument_specs: required: false default: /usr/local/bin/run_borg_backup + borg_prune_script_location: + type: str + required: false + default: /usr/local/bin/run_borg_prune + borg_backup_timer_name: type: str required: false @@ -136,6 +259,16 @@ argument_specs: required: false default: borg_backup + borg_prune_timer_name: + type: str + required: false + default: borg_prune + + borg_prune_service_name: + type: str + required: false + default: borg_prune + borg_backup_argument: type: str required: false @@ -147,6 +280,12 @@ argument_specs: required: false default: [] + borg_prune_service_successful_exit_status: + type: list + elements: str + required: false + default: [] + borg_systemd_oncalendar: type: str required: false @@ -156,3 +295,13 @@ argument_specs: type: str required: false default: 60min + + borg_prune_systemd_oncalendar: + type: str + required: false + default: "*-*-* 03:30:00" + + borg_prune_systemd_accuracysec: + type: str + required: false + default: 60min diff --git a/molecule/default/converge.yml b/molecule/default/converge.yml index 90b3258..0e9fbf0 100644 --- a/molecule/default/converge.yml +++ b/molecule/default/converge.yml @@ -6,6 +6,8 @@ - borg-client-multi - borg-client-nonroot - borg-client-multi-keys + - borg-client-transition + - borg-client-validation vars: borg_server_host: borg-server @@ -64,6 +66,11 @@ - /opt - /var - /reee reeee + borg_prune_enabled: true + borg_prune_keep_last: "2" + borg_prune_glob_archives: "{hostname}-*" + borg_prune_compact_enabled: true + borg_compact_threshold: 1 - name: Converge - borg-client-2 (custom server user) hosts: borg-client-2 @@ -93,6 +100,17 @@ - /opt - /var - /reee reeee + borg_prune_enabled: true + borg_prune_trigger: timer + borg_prune_keep_daily: "7" + borg_prune_keep_weekly: "4" + borg_prune_glob_archives: "{hostname}-*" + borg_prune_compact_enabled: true + borg_compact_threshold: 10 + borg_prune_service_successful_exit_status: + - 1 + - TEMPFAIL + borg_prune_systemd_oncalendar: "*-*-* 05:00:00" - name: Converge - Multi-instance backup (same host, different repos) hosts: borg-client-multi @@ -190,3 +208,149 @@ borg_excluded_dirs: - /home/*/.cache borg_systemd_oncalendar: "*-*-* 04:00:00" + +- name: Converge - Transition host in stable after_backup state + hosts: borg-client-transition + + pre_tasks: + - name: Seed legacy aggregate backup script + ansible.builtin.copy: + dest: /usr/local/bin/run_borg_backup + content: | + #!/bin/bash + ## BEGIN ANSIBLE MANAGED BLOCK for borg-server/transition-repo + echo legacy-backup-block + ## END ANSIBLE MANAGED BLOCK for borg-server/transition-repo + owner: root + group: root + mode: "0711" + force: false + become: true + + roles: + - role: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: transition-repo + borg_backup_argument: transition-repo + borg_included_dirs: + - /etc + borg_excluded_dirs: [] + borg_prune_enabled: true + borg_prune_trigger: after_backup + borg_prune_keep_last: "2" + borg_prune_glob_archives: "{hostname}-*" + +- name: Converge - Validation guard coverage + hosts: borg-client-validation + + tasks: + - name: Verify prune with append-only fails validation + block: + - name: Run role with incompatible append-only pruning + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-append-only + borg_backup_argument: validation-append-only + borg_included_dirs: + - /etc + borg_prune_enabled: true + borg_mode_append_only: true + borg_prune_keep_last: "2" + borg_prune_glob_archives: "{hostname}-*" + + - name: Fail when append-only prune validation unexpectedly passes + ansible.builtin.fail: + msg: Append-only prune validation unexpectedly passed + rescue: + - name: Assert append-only prune validation failed as expected + ansible.builtin.assert: + that: + - >- + 'incompatible with borg_mode_append_only' + in (ansible_failed_result.msg | default('')) + + - name: Verify prune without a retention policy fails validation + block: + - name: Run role without a prune retention policy + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-no-retention + borg_backup_argument: validation-no-retention + borg_included_dirs: + - /etc + borg_prune_enabled: true + borg_prune_glob_archives: "{hostname}-*" + + - name: Fail when missing retention validation unexpectedly passes + ansible.builtin.fail: + msg: Missing retention validation unexpectedly passed + rescue: + - name: Assert missing retention validation failed as expected + ansible.builtin.assert: + that: + - >- + 'Prune requires at least one retention rule' + in (ansible_failed_result.msg | default('')) + + - name: Prune with retention only in additional arguments should succeed + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-additional-args + borg_backup_argument: validation-additional-args + borg_included_dirs: + - /etc + borg_prune_enabled: true + borg_prune_glob_archives: "{hostname}-*" + borg_prune_additional_arguments: --keep-last 2 + register: prune_additional_args_result + + - name: Assert additional-arguments prune validation passes + ansible.builtin.assert: + that: + - prune_additional_args_result is succeeded + + - name: Converge disabled pruning baseline + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-disable-prune + borg_backup_argument: validation-disable-prune + borg_included_dirs: + - /etc + borg_prune_enabled: false + + - name: Absent state with empty prune script path should succeed + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + state: absent + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-absent-empty-prune-path + borg_backup_argument: validation-absent-empty-prune-path + borg_prune_script_location: "" + register: absent_empty_prune_path_result + + - name: Assert absent with empty prune script path passes + ansible.builtin.assert: + that: + - absent_empty_prune_path_result is succeeded diff --git a/molecule/default/molecule.yml b/molecule/default/molecule.yml index 0ea5c96..68dcf4a 100644 --- a/molecule/default/molecule.yml +++ b/molecule/default/molecule.yml @@ -51,6 +51,22 @@ platforms: networks: - name: molecule-container-net + - name: borg-client-transition + image: ${MOLECULE_DISTRO_CLIENT:-debian:12} + dockerfile: ../Dockerfile.j2 + pre_build_image: false + privileged: true + networks: + - name: molecule-container-net + + - name: borg-client-validation + image: ${MOLECULE_DISTRO_CLIENT:-debian:12} + dockerfile: ../Dockerfile.j2 + pre_build_image: false + privileged: true + networks: + - name: molecule-container-net + - name: borg-server image: ${MOLECULE_DISTRO_SERVER:-debian:12} dockerfile: ../Dockerfile.j2 @@ -71,6 +87,7 @@ provisioner: name: ansible playbooks: converge: ${MOLECULE_PLAYBOOK:-converge.yml} + side_effect: side_effect.yml verifier: name: testinfra diff --git a/molecule/default/side_effect.yml b/molecule/default/side_effect.yml new file mode 100644 index 0000000..63a10e5 --- /dev/null +++ b/molecule/default/side_effect.yml @@ -0,0 +1,167 @@ +--- +- name: Side effect - Configure SSH access for transition tests + hosts: + - borg-client-transition + - borg-client-validation + + tasks: + - name: Start ssh on borg-server + ansible.builtin.systemd: + name: sshd + state: started + become: true + delegate_to: borg-server + + - name: Fetch ssh key for borg-server + ansible.builtin.command: >- + ssh-keyscan -t rsa borg-server | sed "s/^[^ ]* //" + register: borg_server_ssh_keyscan + changed_when: false + + - name: Set ssh key for borg-server + ansible.builtin.set_fact: + borg_server_host_ssh_key: >- + {{ borg_server_ssh_keyscan.stdout + | split(" ") + | reject("search", "borg-server") + | join(" ") }} + +- name: Side effect - Transition prune from timer to after_backup + hosts: borg-client-transition + + tasks: + - name: Configure timer-triggered pruning + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: transition-repo + borg_backup_argument: transition-repo + borg_included_dirs: + - /etc + borg_excluded_dirs: [] + borg_prune_enabled: true + borg_prune_trigger: timer + borg_prune_keep_daily: "2" + borg_prune_glob_archives: "{hostname}-*" + borg_prune_systemd_oncalendar: "*-*-* 05:15:00" + + - name: Check timer-triggered prune resources + ansible.builtin.stat: + path: "{{ item }}" + loop: + - /usr/local/bin/run_borg_prune@transition-repo + - /etc/systemd/system/borg_prune@transition-repo.service + - /etc/systemd/system/borg_prune@transition-repo.timer + register: transition_timer_resources + + - name: Check timer-triggered prune timer is enabled + ansible.builtin.systemd: + name: borg_prune@transition-repo.timer + register: transition_timer_status + become: true + + - name: Assert timer-triggered pruning was configured + ansible.builtin.assert: + that: + - transition_timer_resources.results | map(attribute='stat.exists') | list == [true, true, true] + - transition_timer_status.status.UnitFileState == 'enabled' + + - name: Reconfigure pruning to run after backup + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: transition-repo + borg_backup_argument: transition-repo + borg_included_dirs: + - /etc + borg_excluded_dirs: [] + borg_prune_enabled: true + borg_prune_trigger: after_backup + borg_prune_keep_last: "2" + borg_prune_glob_archives: "{hostname}-*" + + - name: Check stale prune timer was removed + ansible.builtin.stat: + path: /etc/systemd/system/borg_prune@transition-repo.timer + register: stale_transition_timer + + - name: Read transition backup service + ansible.builtin.slurp: + src: /etc/systemd/system/borg_backup@transition-repo.service + register: transition_backup_service + become: true + + - name: Assert after_backup transition completed + ansible.builtin.assert: + that: + - not stale_transition_timer.stat.exists + - >- + 'OnSuccess=borg_prune@transition-repo.service' + in (transition_backup_service.content | b64decode) + +- name: Side effect - Disable timer-triggered pruning + hosts: borg-client-validation + + tasks: + - name: Configure timer-triggered pruning before disabling it + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-disable-prune + borg_backup_argument: validation-disable-prune + borg_included_dirs: + - /etc + borg_prune_enabled: true + borg_prune_trigger: timer + borg_prune_keep_last: "2" + + - name: Check prune resources before disabling + ansible.builtin.stat: + path: "{{ item }}" + loop: + - /usr/local/bin/run_borg_prune@validation-disable-prune + - /etc/systemd/system/borg_prune@validation-disable-prune.service + - /etc/systemd/system/borg_prune@validation-disable-prune.timer + register: enabled_prune_resources + + - name: Assert prune resources were created + ansible.builtin.assert: + that: + - enabled_prune_resources.results | map(attribute='stat.exists') | list == [true, true, true] + + - name: Disable timer-triggered pruning + ansible.builtin.include_role: + name: kliwniloc.borgbackup + vars: + borg_server_host: borg-server + borg_server_user_home: /opt/borg + borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml" + borg_repo_name: validation-disable-prune + borg_backup_argument: validation-disable-prune + borg_included_dirs: + - /etc + borg_prune_enabled: false + + - name: Check that disabled prune resources were removed + ansible.builtin.stat: + path: "{{ item }}" + loop: + - /usr/local/bin/run_borg_prune + - /usr/local/bin/run_borg_prune@validation-disable-prune + - /etc/systemd/system/borg_prune@validation-disable-prune.service + - /etc/systemd/system/borg_prune@validation-disable-prune.timer + register: disabled_prune_resources + + - name: Assert disabled prune resources were removed + ansible.builtin.assert: + that: + - disabled_prune_resources.results | map(attribute='stat.exists') | select | list | length == 0 diff --git a/molecule/default/tests/test_client_setup.py b/molecule/default/tests/test_client_setup.py index 78e751a..231b946 100644 --- a/molecule/default/tests/test_client_setup.py +++ b/molecule/default/tests/test_client_setup.py @@ -1,5 +1,7 @@ """Tests for client setup configuration""" +import shlex + import pytest testinfra_hosts = [ @@ -8,6 +10,7 @@ testinfra_hosts = [ "borg-client-multi", "borg-client-nonroot", "borg-client-multi-keys", + "borg-client-transition", ] @@ -17,6 +20,7 @@ CLIENT_USER_MAP = { "borg-client-multi": "root", "borg-client-nonroot": "backupuser", "borg-client-multi-keys": "root", + "borg-client-transition": "root", } CLIENT_SSH_KEY_TYPE_MAP = { @@ -25,6 +29,7 @@ CLIENT_SSH_KEY_TYPE_MAP = { "borg-client-multi": "rsa", "borg-client-nonroot": "rsa", "borg-client-multi-keys": "ed25519", + "borg-client-transition": "rsa", } CLIENT_SSH_KEY_PER_REPO_MAP = { @@ -33,6 +38,7 @@ CLIENT_SSH_KEY_PER_REPO_MAP = { "borg-client-multi": False, "borg-client-nonroot": False, "borg-client-multi-keys": True, + "borg-client-transition": False, } @@ -56,6 +62,76 @@ def get_client_home(host): return f"/home/{user}" if user != "root" else "/root" +def get_backup_script_paths(hostname): + if hostname == "borg-client-multi": + return [ + "/usr/local/bin/run_borg_backup@configs", + "/usr/local/bin/run_borg_backup@home-data", + ] + if hostname == "borg-client-multi-keys": + return [ + "/usr/local/bin/run_borg_backup@configs-keys", + "/usr/local/bin/run_borg_backup@home-data-keys", + ] + if hostname == "borg-client-nonroot": + return ["/usr/local/bin/run_borg_backup@borg-server"] + if hostname == "borg-client": + return ["/usr/local/bin/run_borg_backup@borg-server"] + if hostname == "borg-client-2": + return ["/usr/local/bin/run_borg_backup@borg-server-2"] + if hostname == "borg-client-transition": + return ["/usr/local/bin/run_borg_backup@transition-repo"] + return [] + + +def get_prune_script_paths(hostname): + if hostname == "borg-client": + return ["/usr/local/bin/run_borg_prune@borg-server"] + if hostname == "borg-client-2": + return ["/usr/local/bin/run_borg_prune@borg-server-2"] + if hostname == "borg-client-transition": + return ["/usr/local/bin/run_borg_prune@transition-repo"] + return [] + + +def run_script_with_fake_borg(host, script_path, failed_command, failed_status): + """Run a generated script with a fake borg and return its result and calls.""" + temp_dir_result = host.run("mktemp -d /tmp/borg-exit-test.XXXXXX") + assert temp_dir_result.rc == 0 + temp_dir = temp_dir_result.stdout.strip() + fake_borg_path = f"{temp_dir}/borg" + log_path = f"{temp_dir}/calls" + + fake_borg = """#!/bin/bash +printf '%s\\n' "$1" >> "${BORG_TEST_LOG}" +if [ "$1" = "${BORG_FAIL_COMMAND}" ]; then + exit "${BORG_FAIL_STATUS}" +fi +exit 0 +""" + setup = host.run( + f"cat > {shlex.quote(fake_borg_path)} <<'EOF'\n" + f"{fake_borg}" + "EOF\n" + f"chmod 0755 {shlex.quote(fake_borg_path)}" + ) + assert setup.rc == 0 + + try: + result = host.run( + "env " + f"BORG_FAIL_COMMAND={shlex.quote(failed_command)} " + f"BORG_FAIL_STATUS={failed_status} " + f"BORG_TEST_LOG={shlex.quote(log_path)} " + f"PATH={shlex.quote(temp_dir)}:$PATH " + f"{shlex.quote(script_path)}" + ) + calls = host.file(log_path).content_string.splitlines() + return result, calls + finally: + host.run(f"rm -rf {shlex.quote(temp_dir)}") + + class TestSSHSetup: def test_ssh_directory_exists(self, host): client_home = get_client_home(host) @@ -162,8 +238,8 @@ class TestBackupScript: assert script.user == client_user assert script.group == client_user assert script.mode == 0o711 - elif hostname in ("borg-client", "borg-client-2"): - script = host.file("/usr/local/bin/run_borg_backup") + elif hostname in ("borg-client", "borg-client-2", "borg-client-transition"): + script = host.file(get_backup_script_paths(hostname)[0]) assert script.exists assert script.user == client_user assert script.group == client_user @@ -187,12 +263,156 @@ class TestBackupScript: elif hostname == "borg-client-nonroot": script = host.file("/usr/local/bin/run_borg_backup@borg-server") assert script.contains("borg create") - elif hostname in ("borg-client", "borg-client-2"): - script = host.file("/usr/local/bin/run_borg_backup") + elif hostname in ("borg-client", "borg-client-2", "borg-client-transition"): + script = host.file(get_backup_script_paths(hostname)[0]) assert script.contains("borg create") + assert not script.contains("borg prune") + assert not script.contains("borg compact") else: pytest.fail(f"Unexpected hostname: {hostname}") + +class TestPruneScript: + def test_prune_script_created_when_pruning_enabled(self, host): + hostname = host.backend.get_hostname() + if hostname not in ("borg-client", "borg-client-2", "borg-client-transition"): + return + + for script_path in get_prune_script_paths(hostname): + script = host.file(script_path) + assert script.exists + assert script.mode == 0o711 + assert script.contains("borg prune") + assert script.contains("borg compact") + + def test_scripts_have_valid_shell_syntax(self, host): + hostname = host.backend.get_hostname() + + for script_path in get_backup_script_paths(hostname) + get_prune_script_paths( + hostname + ): + result = host.run(f"bash -n {script_path}") + assert result.rc == 0 + assert "ANSIBLE MANAGED BLOCK" not in host.file(script_path).content_string + + def test_after_backup_prune_script_flow_and_options(self, host): + hostname = host.backend.get_hostname() + if hostname != "borg-client": + return + + backup = host.file("/usr/local/bin/run_borg_backup@borg-server") + prune = host.file("/usr/local/bin/run_borg_prune@borg-server") + content = prune.content_string + + assert "borg create" in backup.content_string + assert "borg prune" not in backup.content_string + assert "borg compact" not in backup.content_string + + prune_index = content.index("borg prune") + prune_status_index = content.index("borg_prune_exit=$?") + prune_guard_index = content.index('if [ "${borg_prune_exit}" -ne 0 ]; then') + compact_index = content.index("borg compact") + + assert prune_index < prune_status_index < prune_guard_index < compact_index + assert "--checkpoint-interval 1800" in content + assert "--glob-archives '{hostname}-*'" in content + assert "--keep-last 2" in content + assert "--stats" in content + assert "--threshold 1" in content + + def test_timer_prune_script_flow_and_options(self, host): + hostname = host.backend.get_hostname() + if hostname != "borg-client-2": + return + + backup_script = host.file("/usr/local/bin/run_borg_backup@borg-server-2") + prune_script = host.file("/usr/local/bin/run_borg_prune@borg-server-2") + prune_content = prune_script.content_string + + assert "borg prune" not in backup_script.content_string + assert "borg compact" not in backup_script.content_string + + prune_index = prune_content.index("borg prune") + prune_status_index = prune_content.index("borg_prune_exit=$?") + prune_guard_index = prune_content.index( + 'if [ "${borg_prune_exit}" -ne 0 ]; then' + ) + compact_index = prune_content.index("borg compact") + + assert prune_index < prune_status_index < prune_guard_index < compact_index + assert "--checkpoint-interval 1800" in prune_content + assert "--glob-archives '{hostname}-*'" in prune_content + assert "--keep-daily 7" in prune_content + assert "--keep-weekly 4" in prune_content + assert "--stats" in prune_content + assert "--threshold 10" in prune_content + + def test_transition_host_uses_separate_prune_script(self, host): + hostname = host.backend.get_hostname() + if hostname != "borg-client-transition": + return + + backup = host.file("/usr/local/bin/run_borg_backup@transition-repo") + prune = host.file("/usr/local/bin/run_borg_prune@transition-repo") + + assert "borg create" in backup.content_string + assert "borg prune" not in backup.content_string + assert "borg prune" in prune.content_string + assert "borg compact" in prune.content_string + assert "--keep-last 2" in prune.content_string + assert "--glob-archives '{hostname}-*'" in prune.content_string + + def test_backup_propagates_create_failure_status(self, host): + if host.backend.get_hostname() != "borg-client": + return + + result, calls = run_script_with_fake_borg( + host, + "/usr/local/bin/run_borg_backup@borg-server", + "create", + 42, + ) + + assert result.rc == 42 + assert calls == ["create"] + + def test_after_backup_preserves_borg_warning_status(self, host): + if host.backend.get_hostname() != "borg-client": + return + + result, calls = run_script_with_fake_borg( + host, + "/usr/local/bin/run_borg_backup@borg-server", + "create", + 1, + ) + + assert result.rc == 1 + assert calls == ["create"] + + @pytest.mark.parametrize( + ("failed_command", "failed_status", "expected_calls"), + [ + ("prune", 43, ["prune"]), + ("compact", 44, ["prune", "compact"]), + ], + ) + def test_timer_prune_propagates_failure_status( + self, host, failed_command, failed_status, expected_calls + ): + if host.backend.get_hostname() != "borg-client-2": + return + + result, calls = run_script_with_fake_borg( + host, + "/usr/local/bin/run_borg_prune@borg-server-2", + failed_command, + failed_status, + ) + + assert result.rc == failed_status + assert calls == expected_calls + def test_backup_script_contains_compression(self, host): hostname = host.backend.get_hostname() @@ -206,18 +426,15 @@ class TestBackupScript: script2 = host.file("/usr/local/bin/run_borg_backup@home-data-keys") assert script1.contains("-C zstd") assert script2.contains("-C lz4") - elif hostname == "borg-client-2": - script = host.file("/usr/local/bin/run_borg_backup") - assert script.contains("-C") - assert script.contains("lz4") - elif hostname in ("borg-client", "borg-client-nonroot"): - script = ( - host.file("/usr/local/bin/run_borg_backup") - if hostname == "borg-client" - else host.file("/usr/local/bin/run_borg_backup@borg-server") - ) + elif hostname in ( + "borg-client", + "borg-client-2", + "borg-client-nonroot", + "borg-client-transition", + ): + script = host.file(get_backup_script_paths(hostname)[0]) assert script.contains("-C") - assert script.contains("zstd") + assert script.contains("lz4" if hostname == "borg-client-2" else "zstd") else: pytest.fail(f"Unexpected hostname: {hostname}") @@ -243,13 +460,17 @@ class TestBackupScript: assert script.contains("borg@borg-server") assert script.contains("/opt/borg") elif hostname == "borg-client": - script = host.file("/usr/local/bin/run_borg_backup") + script = host.file(get_backup_script_paths(hostname)[0]) assert script.contains("borg@borg-server") assert script.contains("/opt/borg") elif hostname == "borg-client-2": - script = host.file("/usr/local/bin/run_borg_backup") + script = host.file(get_backup_script_paths(hostname)[0]) assert script.contains("backupserver@borg-server-2") assert script.contains("/var/backups") + elif hostname == "borg-client-transition": + script = host.file("/usr/local/bin/run_borg_backup@transition-repo") + assert script.contains("borg@borg-server") + assert script.contains("/opt/borg/transition-repo") else: pytest.fail(f"Unexpected hostname: {hostname}") @@ -272,8 +493,8 @@ class TestBackupScript: script = host.file("/usr/local/bin/run_borg_backup@borg-server") content = script.content_string assert "/etc" in content - elif hostname in ("borg-client", "borg-client-2"): - script = host.file("/usr/local/bin/run_borg_backup") + elif hostname in ("borg-client", "borg-client-2", "borg-client-transition"): + script = host.file(get_backup_script_paths(hostname)[0]) content = script.content_string assert "/etc" in content or "/home" in content else: @@ -295,8 +516,8 @@ class TestBackupScript: elif hostname == "borg-client-nonroot": script = host.file("/usr/local/bin/run_borg_backup@borg-server") assert script.mode == 0o711 - elif hostname in ("borg-client", "borg-client-2"): - script = host.file("/usr/local/bin/run_borg_backup") + elif hostname in ("borg-client", "borg-client-2", "borg-client-transition"): + script = host.file(get_backup_script_paths(hostname)[0]) assert script.mode == 0o711 else: pytest.fail(f"Unexpected hostname: {hostname}") @@ -313,7 +534,7 @@ class TestBackupScript: assert "--one-file-system" in script2.content_string assert "--exclude-caches" in script2.content_string elif hostname == "borg-client-2": - script = host.file("/usr/local/bin/run_borg_backup") + script = host.file(get_backup_script_paths(hostname)[0]) content = script.content_string assert "-C zlib,6" in content @@ -322,7 +543,7 @@ class TestBackupScript: if hostname != "borg-client-2": return - script = host.file("/usr/local/bin/run_borg_backup") + script = host.file(get_backup_script_paths(hostname)[0]) content = script.content_string assert "-C lz4" in content @@ -344,57 +565,15 @@ class TestBackupScript: assert "ssh -i" in script2.content_string -class TestMultiInstanceBaseScript: - def test_base_script_exists(self, host): - hostname = host.backend.get_hostname() - if hostname != "borg-client-multi": - return - - client_user = get_client_user(host) - base_script = host.file("/usr/local/bin/run_borg_backup") - assert base_script.exists - assert base_script.user == client_user - assert base_script.mode == 0o711 - - def test_base_script_contains_both_blocks(self, host): - hostname = host.backend.get_hostname() - if hostname != "borg-client-multi": - return - - base_script = host.file("/usr/local/bin/run_borg_backup") - content = base_script.content_string +class TestAggregateScripts: + def test_legacy_aggregate_backup_script_is_preserved(self, host): + aggregate = host.file("/usr/local/bin/run_borg_backup") - assert "borg-server/configs" in content - assert "borg-server/home-data" in content - - def test_base_script_contains_both_repos(self, host): - hostname = host.backend.get_hostname() - if hostname != "borg-client-multi": - return - - base_script = host.file("/usr/local/bin/run_borg_backup") - content = base_script.content_string - - assert "/opt/borg/configs" in content - assert "/opt/borg/home-data" in content - - def test_base_script_contains_both_compressions(self, host): - hostname = host.backend.get_hostname() - if hostname != "borg-client-multi": - return - - base_script = host.file("/usr/local/bin/run_borg_backup") - content = base_script.content_string - - assert "-C zstd" in content - assert "-C lz4" in content - - def test_base_script_two_borg_create_commands(self, host): - hostname = host.backend.get_hostname() - if hostname != "borg-client-multi": - return - - base_script = host.file("/usr/local/bin/run_borg_backup") - content = base_script.content_string + if host.backend.get_hostname() == "borg-client-transition": + assert aggregate.exists + assert "legacy-backup-block" in aggregate.content_string + else: + assert not aggregate.exists - assert content.count("borg create") == 2 + def test_aggregate_prune_script_is_not_created(self, host): + assert not host.file("/usr/local/bin/run_borg_prune").exists diff --git a/molecule/default/tests/test_manual_backup.py b/molecule/default/tests/test_manual_backup.py index 8cd044c..ea48655 100644 --- a/molecule/default/tests/test_manual_backup.py +++ b/molecule/default/tests/test_manual_backup.py @@ -84,3 +84,125 @@ def test_backup_restore(host, compression): assert c.rc == 0 assert c.stdout == "" assert c.stderr == "" + + +def test_after_backup_scripts_prune_old_archives(host): + hostname = host.backend.get_hostname() + if hostname != "borg-client": + return + + server_user, server_host, server_path = get_server_info(hostname) + backup_script_path = "/usr/local/bin/run_borg_backup@borg-server" + prune_script_path = "/usr/local/bin/run_borg_prune@borg-server" + + original_backup_script = host.file(backup_script_path).content_string + original_prune_script = host.file(prune_script_path).content_string + prefix = f"testprune-{datetime.now().strftime('%Y%m%d%H%M%S%f')}" + + modified_backup_script = original_backup_script.replace( + "::{hostname}-{now:%Y-%m-%dT%H:%M:%S}", + f"::{prefix}-${{BORG_TEST_ITERATION}}-{{now:%Y-%m-%dT%H:%M:%S}}", + ) + modified_prune_script = original_prune_script.replace( + "{hostname}-*", + f"{prefix}-*", + ) + + rewrite = host.run( + "python3 - <<'PY'\n" + "from pathlib import Path\n" + f"Path({backup_script_path!r}).write_text({modified_backup_script!r})\n" + f"Path({prune_script_path!r}).write_text({modified_prune_script!r})\n" + "PY" + ) + assert rewrite.rc == 0 + assert host.run(f"bash -n {backup_script_path}").rc == 0 + assert host.run(f"bash -n {prune_script_path}").rc == 0 + + try: + for iteration in range(4): + backup = host.run(f"BORG_TEST_ITERATION={iteration} {backup_script_path}") + assert backup.rc == 0 + prune = host.run(prune_script_path) + assert prune.rc == 0 + + archives = host.run( + f"borg list {server_user}@{server_host}:{server_path}/{hostname} --glob-archives '{prefix}-*'" + ) + assert archives.rc == 0 + + archive_lines = [line for line in archives.stdout.splitlines() if line.strip()] + assert len(archive_lines) == 2 + finally: + restore = host.run( + "python3 - <<'PY'\n" + "from pathlib import Path\n" + f"Path({backup_script_path!r}).write_text({original_backup_script!r})\n" + f"Path({prune_script_path!r}).write_text({original_prune_script!r})\n" + "PY" + ) + assert restore.rc == 0 + + +def test_manual_prune_script_prunes_old_archives(host): + hostname = host.backend.get_hostname() + if hostname != "borg-client-2": + return + + server_user, server_host, server_path = get_server_info(hostname) + prefix = f"testprune-timer-{datetime.now().strftime('%Y%m%d%H%M%S%f')}" + script_path = "/usr/local/bin/run_borg_prune@borg-server-2" + + for iteration in range(4): + create = host.run( + f"borg create -C lz4 {server_user}@{server_host}:{server_path}/{hostname}::{prefix}-{iteration}-{{now:%Y-%m-%dT%H:%M:%S}} /etc" + ) + assert create.rc == 0 + + original_script = host.file(script_path).content_string + assert "--keep-daily 7" in original_script + assert "--keep-weekly 4" in original_script + + modified_script = ( + original_script.replace("{hostname}-*", f"{prefix}-*") + .replace("--keep-daily 7", "--keep-last 2") + .replace("--keep-weekly 4", "") + ) + assert "--keep-last 2" in modified_script + assert "--keep-daily 7" not in modified_script + assert "--keep-weekly 4" not in modified_script + + rewrite = host.run( + "python3 - <<'PY'\n" + "from pathlib import Path\n" + f"Path({script_path!r}).write_text({modified_script!r})\n" + "PY" + ) + assert rewrite.rc == 0 + syntax_check = host.run(f"bash -n {script_path}") + assert syntax_check.rc == 0 + + try: + result = host.run(script_path) + assert result.rc == 0 + + archives = host.run( + f"borg list {server_user}@{server_host}:{server_path}/{hostname} --glob-archives '{prefix}-*'" + ) + assert archives.rc == 0 + + archive_lines = [line for line in archives.stdout.splitlines() if line.strip()] + assert len(archive_lines) == 2 + finally: + restore = host.run( + "python3 - <<'PY'\n" + "from pathlib import Path\n" + f"Path({script_path!r}).write_text({original_script!r})\n" + "PY" + ) + assert restore.rc == 0 + + cleanup = host.run( + f"borg delete --glob-archives '{prefix}-*' {server_user}@{server_host}:{server_path}/{hostname}" + ) + assert cleanup.rc == 0 diff --git a/molecule/default/tests/test_server_setup.py b/molecule/default/tests/test_server_setup.py index 93c319d..9a5156b 100644 --- a/molecule/default/tests/test_server_setup.py +++ b/molecule/default/tests/test_server_setup.py @@ -73,6 +73,8 @@ class TestBorgSSHSetup: "borg-client-multi", "borg-client-nonroot", "borg-client-multi-keys", + "borg-client-transition", + "borg-client-validation", ) for line in content.split("\n"): if not line.strip(): diff --git a/molecule/default/tests/test_systemd.py b/molecule/default/tests/test_systemd.py index e68e232..297ae0d 100644 --- a/molecule/default/tests/test_systemd.py +++ b/molecule/default/tests/test_systemd.py @@ -8,6 +8,7 @@ testinfra_hosts = [ "borg-client-multi", "borg-client-nonroot", "borg-client-multi-keys", + "borg-client-transition", ] CLIENT_CONFIGS = { @@ -16,18 +17,33 @@ CLIENT_CONFIGS = { "server": "borg-server", "schedule": "*-*-* 02:00:00", "success_exit_status": False, + "prune_enabled": True, + "prune_timer": False, }, "borg-client-2": { "user": "root", "server": "borg-server-2", "schedule": "*-*-* 03:00:00", "success_exit_status": True, + "prune_enabled": True, + "prune_timer": True, + "prune_schedule": "*-*-* 05:00:00", }, "borg-client-nonroot": { "user": "backupuser", "server": "borg-server", "schedule": "*-*-* 02:00:00", "success_exit_status": False, + "prune_enabled": False, + "prune_timer": False, + }, + "borg-client-transition": { + "user": "root", + "server": "transition-repo", + "schedule": "*-*-* 02:00:00", + "success_exit_status": False, + "prune_enabled": True, + "prune_timer": False, }, } @@ -85,7 +101,19 @@ class TestSystemdServiceFile: assert service.contains("[Service]") assert service.contains("[Install]") assert service.contains("Type=oneshot") - assert service.contains("ExecStart=/usr/local/bin/run_borg_backup") + assert service.contains( + f"ExecStart=/usr/local/bin/run_borg_backup@{config['server']}" + ) + assert ( + "ExecStart=/usr/local/bin/run_borg_backup\n" + not in service.content_string + ) + if config["prune_enabled"] and not config["prune_timer"]: + assert service.contains( + f"OnSuccess=borg_prune@{config['server']}.service" + ) + else: + assert not service.contains("OnSuccess=") else: for repo in config["repos"]: service = host.file(f"/etc/systemd/system/borg_backup@{repo}.service") @@ -93,6 +121,9 @@ class TestSystemdServiceFile: assert service.contains("[Service]") assert service.contains("[Install]") assert service.contains("Type=oneshot") + assert service.contains( + f"ExecStart=/usr/local/bin/run_borg_backup@{repo}" + ) def test_service_user(self, host, config): if config["type"] == "single": @@ -193,3 +224,54 @@ class TestSystemdState: def test_daemon_reload_ok(self, host, config): c = host.run("systemctl daemon-reload") assert c.rc == 0 + + +class TestPruneSystemd: + def test_prune_service_created_when_pruning_enabled(self, host, config): + if config["type"] != "single": + return + + service = host.file( + f"/etc/systemd/system/borg_prune@{config['server']}.service" + ) + assert service.exists == config["prune_enabled"] + if config["prune_enabled"]: + assert ( + f"ExecStart=/usr/local/bin/run_borg_prune@{config['server']}" + in service.content_string + ) + + def test_prune_timer_created_when_configured(self, host, config): + if config["type"] != "single" or not config.get("prune_timer"): + return + + timer = host.file(f"/etc/systemd/system/borg_prune@{config['server']}.timer") + service = host.file( + f"/etc/systemd/system/borg_prune@{config['server']}.service" + ) + assert timer.exists + assert service.exists + assert f"OnCalendar={config['prune_schedule']}" in timer.content_string + assert ( + f"ExecStart=/usr/local/bin/run_borg_prune@{config['server']}" + in service.content_string + ) + assert "ExecStart=/usr/local/bin/run_borg_prune\n" not in service.content_string + assert "SuccessExitStatus=1 TEMPFAIL" in service.content_string + + def test_prune_timer_not_created_for_after_backup(self, host, config): + if config["type"] != "single" or config.get("prune_timer"): + return + + timer = host.file(f"/etc/systemd/system/borg_prune@{config['server']}.timer") + assert not timer.exists + + def test_transition_host_stale_prune_timer_removed(self, host): + hostname = host.backend.get_hostname() + if hostname != "borg-client-transition": + return + + timer = host.file("/etc/systemd/system/borg_prune@transition-repo.timer") + service = host.file("/etc/systemd/system/borg_prune@transition-repo.service") + assert not timer.exists + assert service.exists diff --git a/molecule/delete/converge.yml b/molecule/delete/converge.yml index 00de93d..037e56a 100644 --- a/molecule/delete/converge.yml +++ b/molecule/delete/converge.yml @@ -35,6 +35,10 @@ borg_backup_argument: single-backup borg_dangerously_delete_backups: true borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys_delete.yml" + borg_prune_enabled: true + borg_prune_trigger: timer + borg_prune_keep_last: "2" + borg_prune_glob_archives: "{hostname}-*" - name: Delete one repo from multi-instance (keep data) hosts: borg-client-multi-delete @@ -135,6 +139,10 @@ borg_ssh_key_type: "{{ 'ed25519' if inventory_hostname == 'borg-client-per-repo-delete' else 'rsa' }}" borg_dangerously_delete_backups: >- {{ inventory_hostname in ['borg-client-single-delete', 'borg-client-per-repo-delete'] }} + borg_prune_enabled: "{{ inventory_hostname == 'borg-client-single-delete' }}" + borg_prune_trigger: "{{ 'timer' if inventory_hostname == 'borg-client-single-delete' else 'after_backup' }}" + borg_prune_keep_last: "{{ '2' if inventory_hostname == 'borg-client-single-delete' else '' }}" + borg_prune_glob_archives: "{{ '{hostname}-*' if inventory_hostname == 'borg-client-single-delete' else '{hostname}-*' }}" borg_decryption_keys_yaml_path: >- {{ (playbook_dir ~ '/decryption_keys_empty_delete.yml') diff --git a/molecule/delete/prepare.yml b/molecule/delete/prepare.yml index 98a94a1..430387d 100644 --- a/molecule/delete/prepare.yml +++ b/molecule/delete/prepare.yml @@ -44,6 +44,10 @@ borg_included_dirs: - /etc borg_excluded_dirs: [] + borg_prune_enabled: true + borg_prune_trigger: timer + borg_prune_keep_last: "2" + borg_prune_glob_archives: "{hostname}-*" - name: Create multi-instance backup (shared key) hosts: borg-client-multi-delete diff --git a/molecule/delete/tests/test_delete.py b/molecule/delete/tests/test_delete.py index 4c0d859..3fc3189 100644 --- a/molecule/delete/tests/test_delete.py +++ b/molecule/delete/tests/test_delete.py @@ -37,6 +37,16 @@ class TestSingleRepoDelete: assert not timer.exists assert not service.exists + def test_prune_systemd_units_removed(self, host): + hostname = host.backend.get_hostname() + if hostname != "borg-client-single-delete": + return + + timer = host.file("/etc/systemd/system/borg_prune@single-backup.timer") + service = host.file("/etc/systemd/system/borg_prune@single-backup.service") + assert not timer.exists + assert not service.exists + def test_backup_scripts_removed(self, host): hostname = host.backend.get_hostname() if hostname != "borg-client-single-delete": @@ -47,6 +57,16 @@ class TestSingleRepoDelete: assert not repo_script.exists assert not base_script.exists + def test_prune_scripts_removed(self, host): + hostname = host.backend.get_hostname() + if hostname != "borg-client-single-delete": + return + + repo_script = host.file("/usr/local/bin/run_borg_prune@single-backup") + base_script = host.file("/usr/local/bin/run_borg_prune") + assert not repo_script.exists + assert not base_script.exists + def test_shared_ssh_key_kept(self, host): hostname = host.backend.get_hostname() if hostname != "borg-client-single-delete": @@ -98,17 +118,12 @@ class TestMultiInstanceDelete: assert service.exists assert script.exists - def test_base_script_keeps_only_remaining_block(self, host): + def test_aggregate_script_is_absent(self, host): hostname = host.backend.get_hostname() if hostname != "borg-client-multi-delete": return - base_script = host.file("/usr/local/bin/run_borg_backup") - assert base_script.exists - content = base_script.content_string - assert "/opt/borg/multi-repo-b" in content - assert "/opt/borg/multi-repo-a" not in content - assert content.count("borg create") == 1 + assert not host.file("/usr/local/bin/run_borg_backup").exists def test_shared_ssh_key_kept(self, host): hostname = host.backend.get_hostname() @@ -168,13 +183,10 @@ class TestPerRepoKeyDelete: timer = host.file("/etc/systemd/system/borg_backup@per-repo-b.timer") service = host.file("/etc/systemd/system/borg_backup@per-repo-b.service") script = host.file("/usr/local/bin/run_borg_backup@per-repo-b") - base_script = host.file("/usr/local/bin/run_borg_backup") assert timer.exists assert service.exists assert script.exists - assert base_script.exists - assert "/opt/borg/per-repo-b" in base_script.content_string - assert "/opt/borg/per-repo-a" not in base_script.content_string + assert not host.file("/usr/local/bin/run_borg_backup").exists def test_per_repo_ssh_key_removed(self, host): hostname = host.backend.get_hostname() diff --git a/tasks/absent.yml b/tasks/absent.yml index b1a5592..6afb7f3 100644 --- a/tasks/absent.yml +++ b/tasks/absent.yml @@ -38,6 +38,12 @@ register: timer_stat become: true +- name: Check if prune systemd timer exists + ansible.builtin.stat: + path: /etc/systemd/system/{{ borg_prune_timer_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.timer + register: prune_timer_stat + become: true + - name: Stop systemd timer ansible.builtin.systemd: name: "{{ borg_backup_timer_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.timer" @@ -46,12 +52,26 @@ become: true when: timer_stat.stat.exists +- name: Stop prune systemd timer + ansible.builtin.systemd: + name: "{{ borg_prune_timer_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.timer" + state: stopped + enabled: false + become: true + when: prune_timer_stat.stat.exists + - name: Check if systemd service exists ansible.builtin.stat: path: /etc/systemd/system/{{ borg_backup_service_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.service register: service_stat become: true +- name: Check if prune systemd service exists + ansible.builtin.stat: + path: /etc/systemd/system/{{ borg_prune_service_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.service + register: prune_service_stat + become: true + - name: Stop systemd service ansible.builtin.systemd: name: "{{ borg_backup_service_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.service" @@ -60,59 +80,55 @@ become: true when: service_stat.stat.exists +- name: Stop prune systemd service + ansible.builtin.systemd: + name: "{{ borg_prune_service_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.service" + state: stopped + enabled: false + become: true + when: prune_service_stat.stat.exists + - name: Remove systemd timer file ansible.builtin.file: path: /etc/systemd/system/{{ borg_backup_timer_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.timer state: absent become: true +- name: Remove prune systemd timer file + ansible.builtin.file: + path: /etc/systemd/system/{{ borg_prune_timer_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.timer + state: absent + become: true + - name: Remove systemd service file ansible.builtin.file: path: /etc/systemd/system/{{ borg_backup_service_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.service state: absent become: true +- name: Remove prune systemd service file + ansible.builtin.file: + path: /etc/systemd/system/{{ borg_prune_service_name }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}.service + state: absent + become: true + - name: Reload systemd daemon ansible.builtin.systemd: daemon_reload: true become: true -- name: Check if base backup script exists - ansible.builtin.stat: - path: "{{ borg_backup_script_location }}" - register: base_script_stat - become: true - -- name: Remove repo-specific backup script +- name: Remove repository-specific backup script ansible.builtin.file: - path: "{{ borg_backup_script_location }}{{ '@' if borg_backup_argument != '' }}{{ borg_backup_argument }}" - state: absent - become: true - when: borg_backup_argument | length > 0 - -- name: Remove block from base backup script - ansible.builtin.blockinfile: - path: "{{ borg_backup_script_location }}" - marker: "## {mark} ANSIBLE MANAGED BLOCK for {{ borg_server_host_url }}/{{ borg_repo_name }}" + path: "{{ borg_backup_script_location }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}" state: absent become: true - when: base_script_stat.stat.exists -- name: Read base script content - ansible.builtin.slurp: - src: "{{ borg_backup_script_location }}" - register: base_script_content - become: true - when: base_script_stat.stat.exists - -- name: Remove empty base script +- name: Remove repository-specific prune script ansible.builtin.file: - path: "{{ borg_backup_script_location }}" + path: "{{ borg_prune_script_location }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}" state: absent become: true - when: - - base_script_stat.stat.exists - - ('ANSIBLE MANAGED BLOCK' not in (base_script_content.content | b64decode)) + when: borg_prune_script_location | length > 0 - name: Remove per-repo SSH private key ansible.builtin.file: diff --git a/tasks/client_create_scripts_each.yml b/tasks/client_create_scripts_each.yml deleted file mode 100644 index 6056b81..0000000 --- a/tasks/client_create_scripts_each.yml +++ /dev/null @@ -1,34 +0,0 @@ ---- -- name: Create script for automatic borg backup - ansible.builtin.file: - dest: "{{ script_location }}" - state: touch - owner: "{{ borg_client_user }}" - group: "{{ borg_client_user }}" - modification_time: preserve - access_time: preserve - mode: "0711" - become: true - -- name: Insert shebang into backup script - ansible.builtin.lineinfile: - path: "{{ script_location }}" - line: "#!/bin/bash" - insertbefore: BOF - state: present - become: true - -- name: Insert Backup job block into scripts - ansible.builtin.blockinfile: - path: "{{ script_location }}" - marker: "## {mark} ANSIBLE MANAGED BLOCK for {{ borg_server_host_url }}/{{ borg_repo_name }}" - block: | - export BORG_PASSPHRASE={{ borg_passphrase | quote }} - {% if borg_ssh_key_per_repo %} - export BORG_RSH="ssh -i {{ borg_ssh_key_path }}" - {% endif %} - borg create -C {{ borg_compression }}{% if borg_create_additional_arguments %} {{ borg_create_additional_arguments }}{% endif %} \ - {{ borg_server_user }}@{{ borg_server_host_url }}:{{ borg_server_user_home }}/{{ borg_repo_name }}::{{ borg_backup_name_format }} \ - {{ borg_included_dirs | map('quote') | join(' ') }} \ - {% for e in (borg_excluded_dirs | map('quote')) %} --exclude {{ e }} {% endfor %} - become: true diff --git a/tasks/client_setup.yml b/tasks/client_setup.yml index ab09926..2a1389a 100644 --- a/tasks/client_setup.yml +++ b/tasks/client_setup.yml @@ -1,37 +1,3 @@ ---- -- name: Ensure borg_client_user exists - ansible.builtin.getent: - database: passwd - key: "{{ borg_client_user }}" - become: true - -- name: Compute borg_client_user_home if not set - ansible.builtin.set_fact: - borg_client_user_home: "{{ ansible_facts.getent_passwd[borg_client_user][4] }}" - when: borg_client_user_home is not defined - -- name: Validate borg_client_user home exists - ansible.builtin.stat: - path: "{{ borg_client_user_home }}" - register: user_home_stat - become: true - -- name: Fail if borg_client_user home missing - ansible.builtin.fail: - msg: | - Home directory {{ borg_client_user_home }} for user {{ borg_client_user }} does not exist. - Please ensure the user has a valid home directory before running this role. - when: not user_home_stat.stat.exists - -- name: Check readability of included paths - ansible.builtin.stat: - path: "{{ item }}" - loop: "{{ borg_included_dirs }}" - register: included_paths_stat - become: true - become_user: "{{ borg_client_user }}" - when: borg_included_dirs | length > 0 - - name: Compute SSH key identifier ansible.builtin.set_fact: borg_ssh_key_identifier: "{{ (borg_server_host_url ~ '_' ~ borg_repo_name) | regex_replace('[^a-zA-Z0-9]', '_') }}" @@ -230,13 +196,24 @@ delegate_to: localhost become: false -- name: Create backup scripts - ansible.builtin.include_tasks: client_create_scripts_each.yml - loop: - - "{{ borg_backup_script_location }}" - - "{{ borg_backup_script_location }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}" - loop_control: - loop_var: script_location +- name: Create repository-specific backup script + ansible.builtin.template: + src: borg_backup_script.j2 + dest: "{{ borg_backup_script_location }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}" + owner: "{{ borg_client_user }}" + group: "{{ borg_client_user }}" + mode: "0711" + become: true + +- name: Create repository-specific prune script + ansible.builtin.template: + src: borg_prune_script.j2 + dest: "{{ borg_prune_script_location }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}" + owner: "{{ borg_client_user }}" + group: "{{ borg_client_user }}" + mode: "0711" + become: true + when: borg_prune_enabled - name: Configure systemd borg_backup service ansible.builtin.template: @@ -258,6 +235,72 @@ notify: Reload systemd become: true +- name: Configure systemd borg_prune service + ansible.builtin.template: + src: borg_prune.service.j2 + dest: /etc/systemd/system/{{ borg_prune_service_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.service + mode: "0644" + owner: root + group: root + notify: Reload systemd + become: true + when: borg_prune_enabled + +- name: Configure systemd borg_prune timer + ansible.builtin.template: + src: borg_prune.timer.j2 + dest: /etc/systemd/system/{{ borg_prune_timer_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.timer + mode: "0644" + owner: root + group: root + notify: Reload systemd + become: true + when: + - borg_prune_enabled + - borg_prune_trigger == 'timer' + +- name: Check if stale borg_prune timer exists + ansible.builtin.stat: + path: /etc/systemd/system/{{ borg_prune_timer_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.timer + register: stale_prune_timer_stat + become: true + when: not (borg_prune_enabled and borg_prune_trigger == 'timer') + +- name: Disable stale borg_prune timer + ansible.builtin.systemd: + name: "{{ borg_prune_timer_name }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}.timer" + state: stopped + enabled: false + become: true + when: + - not (borg_prune_enabled and borg_prune_trigger == 'timer') + - stale_prune_timer_stat.stat.exists + +- name: Remove stale borg_prune timer file + ansible.builtin.file: + path: /etc/systemd/system/{{ borg_prune_timer_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.timer + state: absent + notify: Reload systemd + become: true + when: not (borg_prune_enabled and borg_prune_trigger == 'timer') + +- name: Remove stale borg_prune service file + ansible.builtin.file: + path: /etc/systemd/system/{{ borg_prune_service_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.service + state: absent + notify: Reload systemd + become: true + when: not borg_prune_enabled + +- name: Remove stale repository-specific prune script + ansible.builtin.file: + path: "{{ borg_prune_script_location }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}" + state: absent + become: true + when: + - not borg_prune_enabled + - borg_prune_script_location | length > 0 + - name: Reload systemd now before enabling services ansible.builtin.meta: flush_handlers @@ -267,3 +310,13 @@ state: started enabled: true become: true + +- name: Enable borg_prune systemd timer + ansible.builtin.systemd: + name: "{{ borg_prune_timer_name }}{{ '@' if borg_backup_argument != '' else '' }}{{ borg_backup_argument }}.timer" + state: started + enabled: true + become: true + when: + - borg_prune_enabled + - borg_prune_trigger == 'timer' diff --git a/tasks/main.yml b/tasks/main.yml index 6feca79..47c45a7 100644 --- a/tasks/main.yml +++ b/tasks/main.yml @@ -1,4 +1,7 @@ --- +- name: Validate role configuration + ansible.builtin.include_tasks: validate.yml + - name: Install dependencies ansible.builtin.include_tasks: installation.yml when: state == "present" diff --git a/tasks/validate.yml b/tasks/validate.yml new file mode 100644 index 0000000..04e7401 --- /dev/null +++ b/tasks/validate.yml @@ -0,0 +1,8 @@ +--- +- name: Validate present-state configuration + ansible.builtin.include_tasks: validate_present.yml + when: state == "present" + +- name: Validate absent-state configuration + ansible.builtin.include_tasks: validate_absent.yml + when: state == "absent" diff --git a/tasks/validate_absent.yml b/tasks/validate_absent.yml new file mode 100644 index 0000000..b234107 --- /dev/null +++ b/tasks/validate_absent.yml @@ -0,0 +1,10 @@ +--- +- name: Validate absent-state variables + ansible.builtin.assert: + that: + - borg_repo_name | length > 0 + - borg_server_user_home | length > 0 + - borg_backup_script_location | length > 0 + fail_msg: >- + Invalid configuration for state=absent. Ensure borg_repo_name, + borg_server_user_home, and borg_backup_script_location are set. diff --git a/tasks/validate_present.yml b/tasks/validate_present.yml new file mode 100644 index 0000000..fafa807 --- /dev/null +++ b/tasks/validate_present.yml @@ -0,0 +1,104 @@ +--- +- name: Validate required present-state variables + ansible.builtin.assert: + that: + - borg_repo_name | length > 0 + - borg_server_user | length > 0 + - borg_server_user_home | length > 0 + - borg_backup_script_location | length > 0 + - borg_included_dirs | length > 0 + fail_msg: >- + Invalid configuration for state=present. Ensure borg_repo_name, + borg_server_user, borg_server_user_home, borg_backup_script_location are + set and borg_included_dirs is not empty. + +- name: Validate prune configuration + ansible.builtin.assert: + that: + - not borg_prune_enabled or not borg_mode_append_only + - not borg_prune_enabled or borg_prune_glob_archives | length > 0 + - not borg_prune_enabled or borg_prune_trigger in ['after_backup', 'timer'] + - not borg_prune_enabled or borg_compact_threshold >= 0 + - not borg_prune_enabled or borg_compact_threshold <= 100 + - not borg_prune_enabled or ( + borg_prune_keep_within | length > 0 or + borg_prune_keep_last | length > 0 or + borg_prune_keep_minutely | length > 0 or + borg_prune_keep_hourly | length > 0 or + borg_prune_keep_daily | length > 0 or + borg_prune_keep_weekly | length > 0 or + borg_prune_keep_monthly | length > 0 or + borg_prune_keep_13weekly | length > 0 or + borg_prune_keep_3monthly | length > 0 or + borg_prune_keep_yearly | length > 0 or + borg_prune_additional_arguments | length > 0 + ) + - not borg_prune_enabled or borg_prune_service_name | length > 0 + - not borg_prune_enabled or borg_prune_script_location | length > 0 + - borg_prune_trigger != 'timer' or borg_prune_timer_name | length > 0 + - borg_prune_trigger != 'timer' or borg_prune_systemd_oncalendar | length > 0 + - borg_prune_trigger != 'timer' or borg_prune_systemd_accuracysec | length > 0 + fail_msg: >- + Invalid prune configuration. Prune requires at least one retention rule + from borg_prune_keep_* / borg_prune_keep_within or + borg_prune_additional_arguments, a non-empty archive glob, compact + threshold between 0 and 100, and it is incompatible with + borg_mode_append_only. + +- name: Ensure borg_client_user exists + ansible.builtin.getent: + database: passwd + key: "{{ borg_client_user }}" + become: true + +- name: Compute borg_client_user_home if not set + ansible.builtin.set_fact: + borg_client_user_home: "{{ ansible_facts.getent_passwd[borg_client_user][4] }}" + when: borg_client_user_home is not defined + +- name: Validate borg_client_user home exists + ansible.builtin.stat: + path: "{{ borg_client_user_home }}" + register: user_home_stat + become: true + +- name: Fail if borg_client_user home missing + ansible.builtin.fail: + msg: | + Home directory {{ borg_client_user_home }} for user {{ borg_client_user }} does not exist. + Please ensure the user has a valid home directory before running this role. + when: not user_home_stat.stat.exists + +- name: Check readability of included paths + ansible.builtin.stat: + path: "{{ item }}" + loop: "{{ borg_included_dirs }}" + register: included_paths_stat + become: true + become_user: "{{ borg_client_user }}" + +- name: Fail if included path is unreadable or missing + ansible.builtin.fail: + msg: >- + Included path {{ item.item }} is missing or not accessible by + {{ borg_client_user }}. + when: + - not item.stat.exists or not item.stat.readable + loop: "{{ included_paths_stat.results }}" + +- name: Ensure borg_server_user exists when auto-create disabled + ansible.builtin.getent: + database: passwd + key: "{{ borg_server_user }}" + become: true + delegate_to: "{{ borg_server_host }}" + when: not borg_server_user_create + +- name: Fail if borg_server_user does not exist when auto-create disabled + ansible.builtin.fail: + msg: | + User {{ borg_server_user }} does not exist on {{ borg_server_host }}. + Please create the user before running this role or set borg_server_user_create: true. + when: + - not borg_server_user_create + - ansible_facts.getent_passwd[borg_server_user] is not defined diff --git a/templates/borg_backup.service.j2 b/templates/borg_backup.service.j2 index 04f2a49..c4901c3 100644 --- a/templates/borg_backup.service.j2 +++ b/templates/borg_backup.service.j2 @@ -1,10 +1,13 @@ [Unit] Description=Runs borgbackup to create application level backup (ANSIBLE MANAGED) Wants={{ borg_backup_timer_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.timer +{% if borg_prune_enabled and borg_prune_trigger == 'after_backup' %} +OnSuccess={{ borg_prune_service_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.service +{% endif %} [Service] Type=oneshot -ExecStart={{ borg_backup_script_location }} +ExecStart={{ borg_backup_script_location }}{{ "@" if borg_backup_argument != "" else "" }}{{ borg_backup_argument }} User={{ borg_client_user }} Group={{ borg_client_user }} {% if borg_backup_service_successful_exit_status | length > 0 %} diff --git a/templates/borg_backup_script.j2 b/templates/borg_backup_script.j2 new file mode 100644 index 0000000..0ef22ff --- /dev/null +++ b/templates/borg_backup_script.j2 @@ -0,0 +1,15 @@ +#!/bin/bash +export BORG_PASSPHRASE={{ borg_passphrase | quote }} +{% if borg_ssh_key_per_repo %} +export BORG_RSH="ssh -i {{ borg_ssh_key_path }}" +{% endif %} +borg create \ + -C {{ borg_compression }} \ +{% for excluded_dir in borg_excluded_dirs %} + --exclude {{ excluded_dir | quote }} \ +{% endfor %} +{% if borg_create_additional_arguments %} + {{ borg_create_additional_arguments }} \ +{% endif %} + {{ borg_server_user }}@{{ borg_server_host_url }}:{{ borg_server_user_home }}/{{ borg_repo_name }}::{{ borg_backup_name_format }} \ + {{ borg_included_dirs | map('quote') | join(' ') }} diff --git a/templates/borg_prune.service.j2 b/templates/borg_prune.service.j2 new file mode 100644 index 0000000..ce8e528 --- /dev/null +++ b/templates/borg_prune.service.j2 @@ -0,0 +1,17 @@ +[Unit] +Description=Runs borgbackup prune/compact retention job (ANSIBLE MANAGED) +{% if borg_prune_trigger == 'timer' %} +Wants={{ borg_prune_timer_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.timer +{% endif %} + +[Service] +Type=oneshot +ExecStart={{ borg_prune_script_location }}{{ "@" if borg_backup_argument != "" else "" }}{{ borg_backup_argument }} +User={{ borg_client_user }} +Group={{ borg_client_user }} +{% if borg_prune_service_successful_exit_status | length > 0 %} +SuccessExitStatus={{ borg_prune_service_successful_exit_status | join(' ') }} +{% endif %} + +[Install] +WantedBy=multi-user.target diff --git a/templates/borg_prune.timer.j2 b/templates/borg_prune.timer.j2 new file mode 100644 index 0000000..caf3406 --- /dev/null +++ b/templates/borg_prune.timer.j2 @@ -0,0 +1,11 @@ +[Unit] +Description=Runs borgbackup prune/compact retention job (ANSIBLE MANAGED) +Requires={{ borg_prune_service_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.service + +[Timer] +Unit={{ borg_prune_service_name }}{{ "@" if borg_backup_argument != "" }}{{ borg_backup_argument }}.service +OnCalendar={{ borg_prune_systemd_oncalendar }} +AccuracySec={{ borg_prune_systemd_accuracysec }} + +[Install] +WantedBy=timers.target diff --git a/templates/borg_prune_script.j2 b/templates/borg_prune_script.j2 new file mode 100644 index 0000000..bc1f5b9 --- /dev/null +++ b/templates/borg_prune_script.j2 @@ -0,0 +1,71 @@ +#!/bin/bash +export BORG_PASSPHRASE={{ borg_passphrase | quote }} +{% if borg_ssh_key_per_repo %} +export BORG_RSH="ssh -i {{ borg_ssh_key_path }}" +{% endif %} + +borg prune \ + {% if borg_prune_force %} +--force \ + {% endif %} +{% if borg_prune_stats %} +--stats \ + {% endif %} +{% if borg_prune_list %} +--list \ + {% endif %} +{% if borg_prune_save_space %} +--save-space \ + {% endif %} +--checkpoint-interval {{ borg_prune_checkpoint_interval }} \ + --glob-archives {{ borg_prune_glob_archives | quote }} \ + {% if borg_prune_keep_within %} +--keep-within {{ borg_prune_keep_within | quote }} \ + {% endif %} +{% if borg_prune_keep_last %} +--keep-last {{ borg_prune_keep_last | quote }} \ + {% endif %} +{% if borg_prune_keep_minutely %} +--keep-minutely {{ borg_prune_keep_minutely | quote }} \ + {% endif %} +{% if borg_prune_keep_hourly %} +--keep-hourly {{ borg_prune_keep_hourly | quote }} \ + {% endif %} +{% if borg_prune_keep_daily %} +--keep-daily {{ borg_prune_keep_daily | quote }} \ + {% endif %} +{% if borg_prune_keep_weekly %} +--keep-weekly {{ borg_prune_keep_weekly | quote }} \ + {% endif %} +{% if borg_prune_keep_monthly %} +--keep-monthly {{ borg_prune_keep_monthly | quote }} \ + {% endif %} +{% if borg_prune_keep_13weekly %} +--keep-13weekly {{ borg_prune_keep_13weekly | quote }} \ + {% endif %} +{% if borg_prune_keep_3monthly %} +--keep-3monthly {{ borg_prune_keep_3monthly | quote }} \ + {% endif %} +{% if borg_prune_keep_yearly %} +--keep-yearly {{ borg_prune_keep_yearly | quote }} \ + {% endif %} +{% if borg_prune_additional_arguments %} +{{ borg_prune_additional_arguments }} \ + {% endif %} +{{ borg_server_user }}@{{ borg_server_host_url }}:{{ borg_server_user_home }}/{{ borg_repo_name }} +borg_prune_exit=$? +if [ "${borg_prune_exit}" -ne 0 ]; then + exit "${borg_prune_exit}" +fi +{% if borg_prune_compact_enabled %} + +borg compact \ + {% if borg_compact_cleanup_commits %} +--cleanup-commits \ + {% endif %} +--threshold {{ borg_compact_threshold }} \ + {% if borg_compact_additional_arguments %} +{{ borg_compact_additional_arguments }} \ + {% endif %} +{{ borg_server_user }}@{{ borg_server_host_url }}:{{ borg_server_user_home }}/{{ borg_repo_name }} +{% endif %} -- cgit v1.2.3