borg!: support multiple backups to same target

This brings support for multiple backups pointing to the same borg
server, different repositories, from the same client. We support this by
parsing the SSH authorized key file first and appending allowed
repositories to allow independent definitions of the borg targets.

So that the decryption_keys do not clash we include the repository name
as well as the host in the key. This will lead to new keys being created
for existing hosts in the new format.

    Old Format: {{ combine({inventory_hostname: borg_keys.stdout}) }}
    New Format: {{ combine({(inventory_hostname ~ '_' ~ borg_repo_name): borg_keys.stdout}) }}

If upgrading from an older version that used just the hostname as the
key, your existing `decryption_keys.yml` can be manually removed once
the new format is also added.

4 files changed, 310 insertions, 83 deletions

diff --git a/README.md b/README.md
index d556818..5281a01 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,4 @@
-Ansible Role: BorgBackup
-========================
+# Ansible Role: BorgBackup
 
 An Ansible role that installs and configures a systemd service for BorgBackup on
 a client and (delegated) server.
@@ -9,8 +8,7 @@ The documentation assumes basic knowledge about BorgBackup. You can get up to
 speed with Borg by reading their excellent documentation on
 <https://borgbackup.readthedocs.io>.
 
-How it works
-------------
+## How it works
 
 ```mermaid
 sequenceDiagram
@@ -27,8 +25,7 @@ borg-client->>Ansible: Store decryption keys
 borg-client-->borg-server: Do Backup
 ```
 
-Installation
-------------
+## Installation
 
 via ansible-galaxy
 
@@ -54,8 +51,7 @@ ansible-galaxy role install git+https://github.com/kliwniloc/ansible-role-borgba
   name: kliwniloc.borgbackup
 ```
 
-Role Variables
---------------
+## Role Variables
 
 For more details, see: [`defaults/main.yml`](defaults/main.yml)
 
@@ -76,7 +72,7 @@ will default to `borg_server_host`.
 ```yaml
 borg_server_host: "" # Required
 borg_server_host_ssh_key: "" # Required
-borg_server_host_url: {{ borg_server_host }} # Optional
+borg_server_host_url: { { borg_server_host } } # Optional
 ```
 
 The role creates a user for the Borg repositories on the Borg server. You can
@@ -121,6 +117,8 @@ your Borg backups, but it is not used by default.
 > If you want true end-to-end encryption, you should configure a passphrase
 > Read more: [BorgBackup Security](https://borgbackup.readthedocs.io/en/stable/internals/security.html#offline-key-security)
 
+<!-- TODO: Include keyfile backup mode -->
+
 `borg_included_dirs` and `borg_excluded_dirs` finally picks what directories
 will be backed up by Borg.
 
@@ -167,18 +165,17 @@ see man-page systemd.timer(5).
 borg_backup_script_location: /usr/local/bin/run_borg_backup
 borg_backup_timer_name: borg_backup
 borg_backup_service_name: borg_backup
-borg_backup_argument: '{{ borg_server_host_url }}'
-borg_systemd_oncalendar: '*-*-* 02:00:00'
+borg_backup_argument: "{{ borg_server_host_url }}"
+borg_systemd_oncalendar: "*-*-* 02:00:00"
 borg_systemd_accuracysec: 60min
 ```
 
-Example Playbooks
------------------
+## Example Playbooks
 
-Simple Playbook
+### Simple Playbook
 
 ```yaml
-- name: >
+- name: >-
     Add Borg backup to the borg-client and save the
     decryption keys alongside the playbook
   hosts: borg-client
@@ -192,10 +189,10 @@ Simple Playbook
         - /opt/project
 ```
 
-Multiple Backup servers
+### Multiple Backup Servers
 
 ```yaml
-- name: >
+- name: >-
     Add Borg backup with multiple borg servers to the borg-client and
     save the decryption keys alongside the playbook
   hosts: borg-client
@@ -212,12 +209,202 @@ Multiple Backup servers
       borg_server_host_ssh_key: ssh-rsa BBBBBBBB...
 ```
 
-Dependencies
-------------
+### Multiple Repositories on Single Server, Single Client (Multi-Instance)
+
+> [!NOTE]
+> **This is considered a beta feature**
+> This feature is still being developed and has some quirks around it.
+> We plan to release this properly on the next major version with some breaking
+> changes to make this less of a footgun. Until then, read this section
+> carefully.
+
+When a single client host needs to back up different data sets to the same Borg
+server (e.g., system configs and user data), you can run the role multiple times
+with different `borg_repo_name` values.
+
+**Important:** Each invocation must have a unique `borg_backup_argument` to create
+separate systemd timer units.
+
+<!-- TODO: (next major version): Change default borg_backup_argument to include
+     borg_repo_name for uniqueness:
+     "{{ borg_server_host_url }}-{{ borg_repo_name }}".
+     This will be a breaking change requiring migration of existing systemd
+     units. -->
+
+```yaml
+- name: Configure multi-instance backup (same host, different repos)
+  hosts: borg-client
+  vars:
+    borg_server_host: borg-server
+    borg_server_host_ssh_key: ssh-rsa AAAAAAAA...
+    borg_decryption_keys_yaml_path: "{{ playbook_dir }}/decryption_keys.yml"
+
+  roles:
+    - role: kliwniloc.borgbackup
+      vars:
+        borg_repo_name: configs
+        borg_backup_argument: configs
+        borg_compression: zstd
+        borg_included_dirs:
+          - /etc
+        borg_systemd_oncalendar: "*-*-* 02:00:00"
+
+    - role: kliwniloc.borgbackup
+      vars:
+        borg_repo_name: home-data
+        borg_backup_argument: home-data
+        borg_compression: lz4
+        borg_included_dirs:
+          - /home
+        borg_excluded_dirs:
+          - /home/*/.cache
+        borg_systemd_oncalendar: "*-*-* 04:00:00"
+```
+
+This creates:
+
+- Two repositories on the Borg server: `/opt/borg/configs` and `/opt/borg/home-data`
+- Two systemd timers: `borg_backup@configs.timer` and `borg_backup@home-data.timer`
+- A single SSH keypair shared between both repositories
+- A single `authorized_keys` entry with both repositories in `--restrict-to-repository`
+
+## Important Behaviors and Limitations
+
+### Consistent `--append-only` Setting Required
+
+All repositories for a given host must use the same `borg_mode_append_only`
+setting. The role will fail with an error if you attempt to configure
+repositories with conflicting `--append-only` settings for the same host.
+
+This is because the SSH `authorized_keys` entry uses a single `--append-only`
+flag that applies to all repositories accessible via that key.
+
+```yaml
+# This will FAIL - conflicting append-only settings
+roles:
+  - role: kliwniloc.borgbackup
+    vars:
+      borg_repo_name: configs
+      borg_mode_append_only: false # ERROR: inconsistent
+
+  - role: kliwniloc.borgbackup
+    vars:
+      borg_repo_name: home-data
+      borg_mode_append_only: true # ERROR: inconsistent
+```
+
+### Single SSH Key per Host\*\*
+
+The role generates one SSH keypair per client host. All repositories for that
+host share the same SSH key for authentication to the Borg server.
+
+### Decryption Keys Storage Format
+
+Decryption keys are stored in the `decryption_keys.yml` file using the format:
+
+```yaml
+hostname_repo_name: |
+  To restore key use borg key import --paper /path/to/repo
+  BORG PAPER KEY v1
+  ...
+```
+
+For hosts with multiple repos, each repository has its own entry:
+
+- `borg-client_configs`
+- `borg-client_home-data`
+
+### Deprovisioning Borg Repositories
+
+This role does not currently handle deprovisioning or removing repositories.
+Deprovisioning must be done manually or via separate playbooks.
+
+### Removing a Repository
+
+To remove a specific repository (applies to both single and multi-instance setups):
+
+1. Stop and disable the systemd timer and service:
+
+   ```bash
+   systemctl stop borg_backup@ARGUMENT.timer
+   systemctl disable borg_backup@ARGUMENT.timer
+   systemctl stop borg_backup@ARGUMENT.service
+   systemctl disable borg_backup@ARGUMENT.service
+   ```
+
+   Replace `ARGUMENT` with the `borg_backup_argument` value for that repository.
+
+2. Remove the backup script:
+
+   ```bash
+   rm /usr/local/bin/run_borg_backup@ARGUMENT
+   ```
+
+3. _(Multi-instance only)_ Edit the base backup script to remove the repository block:
+
+   ```bash
+   # Edit /usr/local/bin/run_borg_backup
+   # Remove the ANSIBLE MANAGED BLOCK for the deleted repository
+   ```
+
+4. _(If other repositories remain)_ Update `authorized_keys` on the Borg server
+   to remove the repository restriction. Edit `/opt/borg/.ssh/authorized_keys`
+   and remove the `--restrict-to-repository /opt/borg/REPONAME` from the
+   appropriate line.
+
+   If this is the last repository for the host, remove the entire line instead.
+
+5. Delete the repository on the Borg server:
+
+   ```bash
+   ssh borg@SERVER "borg delete /opt/borg/REPONAME"
+   # Or simply: ssh borg@SERVER "rm -rf /opt/borg/REPONAME"
+   ```
+
+6. Remove the decryption key entry from your `decryption_keys.yml`.
+
+## Deprovisioning a Complete Host
+
+To remove all backup configuration for a host:
+
+1. Stop and disable all systemd timers and services:
+
+   ```bash
+   systemctl list-units --type=timer --all | grep borg_backup
+   # For each relevant timer:
+   systemctl stop borg_backup@*.timer
+   systemctl disable borg_backup@*.timer
+   ```
+
+2. Remove all backup scripts:
+
+   ```bash
+   rm -f /usr/local/bin/run_borg_backup*
+   ```
+
+3. Remove the SSH key from the Borg server's `authorized_keys`:
+
+   ```bash
+   # On the Borg server, edit /opt/borg/.ssh/authorized_keys
+   # Remove the line containing root@HOSTNAME
+   ```
+
+4. Delete all repositories for the host on the Borg server:
+
+   ```bash
+   ssh borg@SERVER "rm -rf /opt/borg/HOSTNAME"
+   # For multi-instance:
+   ssh borg@SERVER "rm -rf /opt/borg/HOSTNAME_repo1"
+   ssh borg@SERVER "rm -rf /opt/borg/HOSTNAME_repo2"
+   ```
+
+5. Remove all decryption key entries for the host from your
+   `decryption_keys.yml`.
+
+## Dependencies
 
 None.
 
-License
--------
+## License
 
 MIT
diff --git a/molecule/default/tests/test_decryption_keys.py b/molecule/default/tests/test_decryption_keys.py
index 227baf0..c772477 100644
--- a/molecule/default/tests/test_decryption_keys.py
+++ b/molecule/default/tests/test_decryption_keys.py
@@ -6,27 +6,27 @@ import pytest
 
 def _get_keys_path():
     """Get decryption_keys.yml path from molecule environment"""
-    scenario_dir = os.environ.get("MOLECULE_SCENARIO_DIRECTORY")
+    scenario_dir = os.environ.get('MOLECULE_SCENARIO_DIRECTORY')
     if scenario_dir:
-        return os.path.join(scenario_dir, "decryption_keys.yml")
-    
+        return os.path.join(scenario_dir, 'decryption_keys.yml')
+
     test_dir = os.path.dirname(os.path.abspath(__file__))
-    return os.path.join(test_dir, "..", "decryption_keys.yml")
+    return os.path.join(test_dir, '..', 'decryption_keys.yml')
 
 
 def test_decryption_keys_file_exists():
     """Test that decryption_keys.yml exists"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
+        pytest.skip('decryption_keys.yml not yet generated')
 
 
 def test_decryption_keys_file_permissions():
     """Test that decryption_keys.yml has secure permissions"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
-    
+        pytest.skip('decryption_keys.yml not yet generated')
+
     file_stat = os.stat(keys_path)
     file_mode = stat.S_IMODE(file_stat.st_mode)
     assert file_mode == 0o600, (
@@ -38,15 +38,15 @@ def test_decryption_keys_structure_single_repo():
     """Test single-repo hosts have correct key format"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
-    
-    with open(keys_path, "r") as f:
+        pytest.skip('decryption_keys.yml not yet generated')
+
+    with open(keys_path, 'r') as f:
         content = f.read()
-    
-    assert "borg-client_borg-client:" in content, (
+
+    assert 'borg-client_borg-client:' in content, (
         "Single repo host should have key named 'hostname_repo_name'"
     )
-    assert "borg-client-2_borg-client-2:" in content, (
+    assert 'borg-client-2_borg-client-2:' in content, (
         "Second single repo host should have key named 'hostname_repo_name'"
     )
 
@@ -55,15 +55,15 @@ def test_decryption_keys_structure_multi_repo():
     """Test multi-instance hosts have correct key format"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
-    
-    with open(keys_path, "r") as f:
+        pytest.skip('decryption_keys.yml not yet generated')
+
+    with open(keys_path, 'r') as f:
         content = f.read()
-    
-    assert "borg-client-multi_configs:" in content, (
+
+    assert 'borg-client-multi_configs:' in content, (
         "Multi-instance host should have key for 'configs' repo"
     )
-    assert "borg-client-multi_home-data:" in content, (
+    assert 'borg-client-multi_home-data:' in content, (
         "Multi-instance host should have key for 'home-data' repo"
     )
 
@@ -72,14 +72,14 @@ def test_decryption_keys_multi_instance_separate_entries():
     """Test multi-instance hosts have separate keys for each repo"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
-    
-    with open(keys_path, "r") as f:
+        pytest.skip('decryption_keys.yml not yet generated')
+
+    with open(keys_path, 'r') as f:
         content = f.read()
-    
-    configs_count = content.count("borg-client-multi_configs:")
-    home_data_count = content.count("borg-client-multi_home-data:")
-    
+
+    configs_count = content.count('borg-client-multi_configs:')
+    home_data_count = content.count('borg-client-multi_home-data:')
+
     assert configs_count == 1, (
         f"configs key should appear exactly once, found {configs_count}"
     )
@@ -92,13 +92,13 @@ def test_decryption_keys_contain_paper_key_format():
     """Test that decryption keys use borg paper key format"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
-    
-    with open(keys_path, "r") as f:
+        pytest.skip('decryption_keys.yml not yet generated')
+
+    with open(keys_path, 'r') as f:
         content = f.read()
-    
-    assert "BORG PAPER KEY" in content, (
-        "Decryption keys should contain borg paper key format"
+
+    assert 'BORG PAPER KEY' in content, (
+        'Decryption keys should contain borg paper key format'
     )
 
 
@@ -106,17 +106,17 @@ def test_decryption_keys_all_hosts_present():
     """Test that all expected hosts have keys"""
     keys_path = _get_keys_path()
     if not os.path.exists(keys_path):
-        pytest.skip("decryption_keys.yml not yet generated")
-    
-    with open(keys_path, "r") as f:
+        pytest.skip('decryption_keys.yml not yet generated')
+
+    with open(keys_path, 'r') as f:
         content = f.read()
-    
+
     expected_keys = [
-        "borg-client_borg-client:",
-        "borg-client-2_borg-client-2:",
-        "borg-client-multi_configs:",
-        "borg-client-multi_home-data:",
+        'borg-client_borg-client:',
+        'borg-client-2_borg-client-2:',
+        'borg-client-multi_configs:',
+        'borg-client-multi_home-data:',
     ]
-    
+
     for key in expected_keys:
         assert key in content, f"Expected key {key} not found in decryption_keys.yml"
diff --git a/tasks/client_setup.yml b/tasks/client_setup.yml
index 1eae105..8769852 100644
--- a/tasks/client_setup.yml
+++ b/tasks/client_setup.yml
@@ -26,15 +26,67 @@
   become: true
   register: ssh_key
 
-- name: Deploy Keys to Borg server
+- name: Ensure authorized_keys file exists on borg server
+  ansible.builtin.file:
+    path: "{{ borg_server_user_home }}/.ssh/authorized_keys"
+    state: touch
+    owner: borg
+    group: borg
+    mode: "0600"
+    access_time: preserve
+    modification_time: preserve
+  become: true
+  delegate_to: "{{ borg_server_host }}"
+
+- name: Read existing authorized_keys content
+  ansible.builtin.slurp:
+    src: "{{ borg_server_user_home }}/.ssh/authorized_keys"
+  become: true
+  delegate_to: "{{ borg_server_host }}"
+  register: auth_keys_content
+
+- name: Find existing authorized_keys line for this host
+  ansible.builtin.set_fact:
+    existing_line: >-
+      {{
+        (auth_keys_content.content | b64decode).splitlines()
+        | select("search", ssh_key.public_key | trim | regex_escape)
+        | first
+        | default("")
+      }}
+
+- name: Detect --append-only setting from existing entry
+  ansible.builtin.set_fact:
+    existing_append_only: "{{ existing_line is search('--append-only') }}"
+  when: existing_line | length > 0
+
+- name: Fail if --append-only setting differs from existing entry
+  ansible.builtin.fail:
+    msg: |
+      Inconsistent --append-only setting for host {{ inventory_hostname }}.
+      Existing entry has --append-only={{ existing_append_only }}, but current invocation uses --append-only={{ borg_mode_append_only }}.
+      All repositories for a host must have the same --append-only setting.
+  when:
+    - existing_line | length > 0
+    - existing_append_only != borg_mode_append_only
+
+- name: Compute all repos for this host
+  ansible.builtin.set_fact:
+    all_repos: >-
+      {{
+        (
+          (existing_line | regex_findall('--restrict-to-repository ([^\s"]+)'))
+          + [borg_server_user_home + '/' + borg_repo_name]
+        ) | unique | sort }}
+
+- name: Update authorized_keys entry for this host
   ansible.builtin.lineinfile:
     path: "{{ borg_server_user_home }}/.ssh/authorized_keys"
-    line: >
-      restrict,command="borg serve
-      {{ "--append-only" if borg_mode_append_only }}
-      --restrict-to-repository {{ borg_repo_name }}"
-      {{ ssh_key.public_key }} root@{{ inventory_hostname }}
-    search_string: --restrict-to-repository {{ borg_repo_name }}" {{ ssh_key.public_key }}
+    search_string: "{{ ssh_key.public_key | trim }}"
+    line: >-
+      restrict,command="borg serve{{ " --append-only" if borg_mode_append_only }}
+      {{ all_repos | map('regex_replace', '^', '--restrict-to-repository ') | join(' ') }}"
+      {{ ssh_key.public_key | trim }} root@{{ inventory_hostname }}
     state: present
   become: true
   delegate_to: "{{ borg_server_host }}"
@@ -68,7 +120,7 @@
   register: local
 
 - name: Add repository encryption keys to ansible repo
-  when: not inventory_hostname in local.ansible_facts
+  when: not (inventory_hostname ~ '_' ~ borg_repo_name) in local.ansible_facts
   throttle: 1
   block:
     - name: If host new read encryption keys
@@ -81,7 +133,7 @@
 
     - name: If host new add encryption keys to vars
       ansible.builtin.set_fact:
-        decryption_keys: "{{ local.ansible_facts | combine({inventory_hostname: borg_keys.stdout}) }}"
+        decryption_keys: "{{ local.ansible_facts | combine({(inventory_hostname ~ '_' ~ borg_repo_name): borg_keys.stdout}) }}"
 
 - name: Update encryption vars
   ansible.builtin.copy:
diff --git a/tasks/server_setup.yml b/tasks/server_setup.yml
index 603dd3d..e47d012 100644
--- a/tasks/server_setup.yml
+++ b/tasks/server_setup.yml
@@ -8,15 +8,3 @@
     generate_ssh_key: true
   become: true
   delegate_to: "{{ borg_server_host }}"
-
-- name: Make sure authorized keys exists
-  ansible.builtin.file:
-    path: "{{ borg_server_user_home }}/.ssh/authorized_keys"
-    state: touch
-    owner: borg
-    group: borg
-    mode: "644"
-    access_time: preserve
-    modification_time: preserve
-  become: true
-  delegate_to: "{{ borg_server_host }}"