Skip to main content

Server Backup Maintenance

This guide covers essential maintenance tasks for full server backup plans to keep your disaster recovery capability healthy and optimized.

Cleanup / Prune Old Backups​

Pruning removes old backup snapshots based on your retention policy, freeing up storage space.

When to Prune​

  • Automatic: Pruning happens automatically based on your retention settings
  • Manual: Run prune when you need to immediately reclaim storage space
  • After changing retention: When you reduce the "Snapshots to Keep" setting
  • Storage optimization: When storage space is running low

How Pruning Works​

Your server backup plan maintains a specified number of recent snapshots (configured as "Snapshots to Keep"). When you run prune:

  • Snapshots exceeding your retention limit are removed
  • Associated ISO images may be removed if no snapshots reference them
  • Only the data blocks are deleted from storage
  • Referenced data blocks (used by kept snapshots) remain intact
  • Storage space is freed gradually as unreferenced blocks are removed

Important for Server Backups:

  • Each snapshot includes both filesystem data and ISO reference
  • Pruning doesn't delete ISO if newer snapshots reference it
  • ISO images are only removed when all referencing snapshots are pruned
  • Consider disaster recovery needs when setting retention

Running Manual Prune​

  1. Navigate to your server backup plan details page
  2. Click the More Options (⋮) button in the header
  3. Select "Clean Up"
  4. Review the prune summary:
    • Current snapshot count
    • Number of snapshots to be removed
    • Estimated space to be reclaimed
    • Retention policy settings
  5. Click "Yes, Remove Old Backups" to confirm

What You'll See:

  • If snapshots exceed your retention limit: Shows how many will be removed
  • If within retention limit: "There are no excess snapshots to clean up"
  • List of snapshots that will be deleted with their ages

Important Notes:

  • Pruning cannot be undone
  • The process may take several minutes for large backups
  • Active backups are never pruned
  • You cannot prune while a backup is in progress
  • Latest ISO is always preserved if any recent snapshots exist

Unlock Repository​

Unlocking removes stale locks from the backup repository that may prevent new backups from running.

When to Unlock​

Unlock when:

  • A backup appears stuck or won't start
  • You see errors about "repository is already locked"
  • A backup was forcefully terminated (power loss, crash)
  • Previous backup failed with locking errors
  • Server was rebooted during backup operation

What Are Repository Locks?​

Restic creates locks to prevent multiple simultaneous backups to the same repository (which could corrupt data). Normally, locks are automatically released when backups complete. However, if a backup crashes or is interrupted, stale locks may remain.

Running Unlock​

  1. Navigate to your server backup plan details page
  2. Click the More Options (⋮) button in the header
  3. Select "Unlock"
  4. Read the confirmation message:
    • Explains that unlock removes unused locks from failed backups
    • Helps start new backups when stuck
  5. Click "Yes, Unlock" to confirm

What Happens:

  • Restic scans the repository for locks
  • Identifies and removes stale/inactive locks
  • Active locks (from running backups) are preserved
  • A success message confirms removal

Important Notes:

  • Safe to run - doesn't affect backup data or snapshots
  • Only removes locks not associated with active processes
  • Run this only when backups are stuck, not routinely
  • Cannot unlock while a backup is actively running

Troubleshooting After Unlock​

If unlocking doesn't resolve the issue:

  • Check storage connectivity (verify in Storages page)
  • Ensure source device is accessible
  • Review plan logs for other errors
  • Verify sufficient storage space available
  • Check ReaR configuration integrity
  • Try pausing and resuming the plan

View Plan Logs​

Plan logs provide detailed information about all backup operations, useful for troubleshooting and monitoring.

Accessing Logs​

  1. Navigate to your server backup plan details page
  2. Click the More Options (⋮) button in the header
  3. Select "View Logs"
  4. A side panel opens with the complete log history

Understanding Log Entries​

Logs display chronologically with newest entries at the top. Each entry includes:

Timestamp: Exact date and time of the event

Level: Log severity

  • Info - Normal operations (backup started, completed)
  • Warn - Non-critical issues (retries, warnings)
  • Error - Failures requiring attention

Message: Description of what happened

Common Log Events for Server Backups:

  • Server Backup Plan created - Plan initialized
  • Backup started - Backup execution began
  • Disk layout check: unchanged - Incremental backup will run
  • Disk layout changed - Full backup with ISO creation will run
  • ISO creation started - Bootable image generation began
  • ISO creation complete - ISO successfully created
  • Data backup phase started - Filesystem backup began
  • Backup completed successfully - Backup finished normally
  • Backup failed - Backup encountered errors
  • Plan paused/resumed - Schedule changes
  • Server Backup Plan updated - Configuration modified
  • Repository unlocked - Stale locks removed
  • Backup Plan prune successfully performed - Cleanup completed

Server Backup Specific Log Information​

ISO Creation Logs:

  • ReaR system analysis output
  • Bootloader detection and configuration
  • Driver inclusion information
  • ISO generation progress
  • ISO upload to storage

Data Backup Logs:

  • Filesystem scanning progress
  • Files processed and backed up
  • Data transfer statistics
  • Snapshot creation confirmation

Layout Change Logs:

  • Disk layout comparison results
  • Reasons for layout change detection
  • Filesystem additions or removals
  • Partition modifications

Filtering and Searching Logs​

  • Use the search box to find specific events or errors
  • Filter by log level (Info, Warn, Error)
  • Search for specific filesystem names
  • Click on individual entries to expand full details
  • Scroll through history to track patterns

Downloading Logs​

To save logs for offline review or support:

  1. While viewing logs, click "Download Logs" button
  2. Logs download as a .log file
  3. Open with any text editor
  4. Share with support if needed for troubleshooting

Downloaded Log Format:

  • JSON format for structured analysis
  • Each line is a complete log entry
  • Includes all metadata (timestamps, levels, context)
  • Contains ReaR and Restic output

Using Logs for Troubleshooting​

Common Issues and Log Indicators:

"Permission denied" errors:

  • Check source paths have proper access rights
  • Verify service account has root privileges
  • Check filesystem mount permissions

"Connection refused/timeout":

  • Storage connectivity issues
  • Check network and storage configuration
  • Verify credentials are valid

"Repository locked" errors:

  • Another backup may be running
  • Stale lock needs removal (use Unlock)
  • Previous backup didn't complete cleanly

"Out of space" errors:

  • Storage quota exceeded
  • Run prune or add storage capacity
  • Check if ISO images are consuming space

"ISO creation failed":

  • ReaR configuration issues
  • Bootloader not accessible
  • Missing system components
  • Check ReaR logs for details

"Filesystem not found":

  • Filesystem was unmounted
  • Disk configuration changed
  • Update plan filesystem selection

"ReaR not found":

  • ReaR not installed or not in PATH
  • Check ReaR installation
  • Verify binary accessibility

Log Retention​

  • Logs are stored in data/logs/ directory
  • Log file named plan-{planId}.log
  • Logs persist until plan is deleted
  • Large log files may be rotated automatically
  • No automatic cleanup - manage manually if needed

Disaster Recovery & Restoration​

Restore your complete system from server backups.

Restoration Overview​

Server backup restoration is a complete system recovery process:

  1. Boot from ISO recovery media
  2. ReaR recovery environment loads
  3. Select restore option
  4. ReaR recreates disk layout
  5. Restic restores filesystem data
  6. System configuration is applied
  7. Bootloader is reinstalled
  8. System reboots into recovered OS

Preparing for Recovery​

Before Disaster:

  1. Download ISO Image

    • Access storage destination
    • Download latest ISO image
    • Burn to USB drive or CD
    • Test ISO boots successfully

  2. Document Recovery Information

    • Storage destination details
    • Access credentials
    • ISO encryption password (if set)
    • Network configuration needed
    • Latest snapshot ID

  3. Test Recovery Process

    • In virtual machine or test hardware
    • Verify ISO boots correctly
    • Confirm recovery menu appears
    • Validate procedure understanding

After Disaster:

  1. Assess the Situation

    • Determine extent of failure
    • Identify available hardware
    • Gather recovery documentation
    • Prepare network access

  2. Prepare Recovery Hardware

    • Boot target system from ISO
    • Connect to network (if needed for restore)
    • Ensure sufficient disk space
    • Have credentials ready

Recovery Process​

Step 1: Boot from ISO​

  1. Insert recovery USB or mount ISO
  2. Configure BIOS/UEFI to boot from media
  3. Boot system from ISO
  4. ReaR recovery environment loads
  5. Recovery menu appears

Step 2: Verify Network Access​

If restoring from remote storage:

  1. Configure network settings in recovery environment
  2. Test connectivity to storage
  3. Verify DNS resolution
  4. Confirm authentication

Step 3: Select Recovery Option​

  1. From ReaR menu, select "Recover System"
  2. Confirm disk layout recreation
  3. Review target disk configuration
  4. Proceed with layout recreation

Step 4: ReaR Disk Layout Restoration​

  1. ReaR analyzes target disks
  2. Partition tables are created
  3. Filesystems are formatted
  4. Mount points are configured
  5. Confirmation prompt appears

Step 5: Data Restoration​

  1. ReaR initiates Restic restore
  2. Filesystem data is downloaded from storage
  3. Files are restored to recreated filesystems
  4. Progress is displayed
  5. Restoration completes

Step 6: System Configuration​

  1. ReaR applies system configuration
  2. Bootloader (GRUB/UEFI) is installed
  3. Network configuration is restored
  4. System services are configured
  5. Final checks are performed

Step 7: Complete Recovery​

  1. Remove recovery media
  2. System reboots
  3. Recovered OS boots normally
  4. Verify system functionality
  5. Validate critical services

Recovery Considerations​

Hardware Differences:

  • ReaR handles different hardware well
  • Disk size must be equal or larger
  • Partition scheme is adapted automatically
  • Network interfaces may need reconfiguration
  • Drivers are included in ISO for common hardware

Network Storage:

  • Requires network access during recovery
  • Have network configuration details ready
  • Storage credentials must be available
  • Consider slow network recovery time

Disk Layout Changes:

  • Target disks must accommodate original layout
  • Larger disks are fine (partitions adapted)
  • Smaller disks may fail restoration
  • Multiple disks can be consolidated or split

Time Requirements:

  • Full recovery can take several hours
  • Depends on data size and network speed
  • ISO boot and layout: 10-30 minutes
  • Data restoration: Hours for large systems
  • Configuration and finalization: 10-20 minutes

Verify Backup Integrity​

Periodically verify that your backups can be successfully restored.

Manual Integrity Check​

  1. Navigate to your server backup plan details page
  2. Click "More Options (⋮)"
  3. Select "Check Integrity" (if available)
  4. Restic verifies repository integrity
  5. Results show any corruption or issues

Scheduled Integrity Checks​

Configure automatic integrity verification:

  1. Edit the server backup plan
  2. Go to Advanced Settings
  3. Configure Integrity settings:
    • Enable scheduled checks
    • Set check frequency (weekly/monthly recommended)
    • Choose verification method (full or partial)
  4. Save changes

Integrity Check Methods:

  • Full: Reads all backup data for verification (slow but thorough)
  • Partial (10%, 25%, 50%): Samples data blocks (faster, less comprehensive)
  • Metadata only: Verifies repository structure (fastest)

Testing Recovery​

The only way to truly verify backups is to test recovery:

In Test Environment:

  1. Create virtual machine or use spare hardware
  2. Boot from ISO image
  3. Perform actual system restoration
  4. Verify recovered system functionality
  5. Document any issues encountered

Testing Frequency:

  • Initially: Test first backup immediately
  • Regular: Quarterly or semi-annually
  • After changes: When system configuration changes significantly
  • Before critical events: Prior to major migrations or updates

Storage Management​

Monitor and manage storage used by server backups.

Viewing Storage Usage​

  1. Navigate to Storages page
  2. Find the storage used by your plan
  3. View storage statistics:
    • Total space used
    • Available space
    • Number of files (includes ISOs and backup data)

Components of Server Backup Storage​

ISO Images:

  • Each full backup creates an ISO (300MB - 2GB typically)
  • Old ISOs may remain if referenced by snapshots
  • Multiple ISOs if layout changed multiple times

Restic Repository:

  • Snapshot metadata
  • Filesystem data blocks
  • Index files
  • Configuration

Total Storage:

  • Sum of all ISO images
  • Plus Restic repository size
  • Grows with each incremental backup
  • Reduced by pruning old snapshots

Managing Storage Space​

If Running Low on Space:

  1. Prune old snapshots - Clean up retention exceeding policy
  2. Reduce retention - Keep fewer historical snapshots
  3. Remove old ISOs manually - Delete ISOs not needed (carefully)
  4. Upgrade storage - Add more capacity if needed
  5. Review filesystem selection - Exclude non-essential filesystems

Preventing Space Issues:

  • Set appropriate snapshot retention
  • Monitor storage usage regularly
  • Prune excess snapshots proactively
  • Plan for storage growth
  • Consider storage costs vs. retention needs

Maintenance Schedule​

Daily Monitoring (Critical Servers)​

  • Check latest backup completed successfully
  • Review any error notifications
  • Verify backup status in dashboard

Weekly Maintenance​

  • Review backup history for past week
  • Check for any warnings or errors in logs
  • Verify storage usage is within limits
  • Confirm schedule is running as expected

Monthly Maintenance​

  • Full log review and analysis
  • Verify snapshot retention is appropriate
  • Prune old snapshots if needed
  • Check ISO image availability
  • Test ISO boots successfully
  • Review disaster recovery documentation

Quarterly Maintenance​

  • Comprehensive integrity check (full verification)
  • Test recovery in test environment
  • Update disaster recovery procedures
  • Review and optimize retention policies
  • Audit server backup coverage
  • Update recovery documentation
  • Test recovery on different hardware (if possible)

Annual Maintenance​

  • Full disaster recovery test
  • Review entire backup strategy
  • Update and test all documentation
  • Train team on recovery procedures
  • Evaluate storage requirements
  • Consider system changes and updates

Troubleshooting​

Backups Consistently Failing​

Check Logs for:

  • Specific error messages
  • Phase where failure occurs
  • Consistent patterns

Common Causes:

  • Storage connectivity issues
  • Insufficient storage space
  • Filesystem access problems
  • ReaR configuration errors
  • Network instability

Solutions:

  • Address specific errors from logs
  • Verify storage accessibility
  • Check filesystem mounts
  • Validate ReaR installation
  • Test network stability

ISO Creation Always Failing​

Symptoms: Full backups fail during ISO creation phase

Solutions:

  • Check ReaR installation and version
  • Verify bootloader accessibility
  • Ensure /boot is included in filesystems
  • Review ReaR configuration file
  • Check system compatibility with ReaR
  • Try ReaR update or reinstallation

Incremental Backups Becoming Slow​

Symptoms: Backups taking progressively longer

Causes:

  • Repository growth over time
  • Large number of files changing
  • Network performance degradation

Solutions:

  • Prune old snapshots to optimize repository
  • Check network performance
  • Review what files are changing
  • Consider performance setting adjustments
  • Monitor system resources during backup

Cannot Unlock Repository​

Symptoms: Unlock operation fails

Solutions:

  • Verify storage connectivity
  • Check repository integrity
  • Try manual Restic unlock command
  • Check for active processes
  • Review filesystem permissions
  • Consider repository repair (advanced)

Recovery Boot Fails​

Symptoms: ISO doesn't boot or boots incorrectly

Causes:

  • Corrupted ISO image
  • BIOS/UEFI boot configuration
  • Hardware incompatibility
  • USB/media issues

Solutions:

  • Re-download ISO from storage
  • Verify ISO file integrity
  • Try different boot media (USB vs CD)
  • Check BIOS/UEFI boot settings
  • Test on known-compatible hardware
  • Recreate ISO with new full backup

Maintenance Checklist​

Pre-Disaster Checklist​

  • Latest backup completed successfully
  • ISO image downloaded and accessible
  • ISO boots successfully (tested)
  • Recovery documentation up-to-date
  • Storage credentials documented
  • ISO encryption password documented (if used)
  • Team trained on recovery procedure
  • Test hardware available for recovery testing

Post-Disaster Checklist​

  • Disaster extent assessed
  • Recovery hardware prepared
  • ISO media available
  • Network access configured
  • Storage credentials available
  • Recovery procedure reviewed
  • Team notified and ready
  • Backup verification performed

Regular Maintenance Checklist​

  • Recent backups completed successfully
  • No persistent errors in logs
  • Storage usage within acceptable limits
  • Snapshot count aligns with retention policy
  • ISO images accessible and bootable
  • Recovery documentation current
  • Team familiar with procedures
  • Test recovery performed (quarterly)

Next Steps​

Regular maintenance and testing ensure your server backups provide reliable disaster recovery when you need it most.