SOWN Backup Cloud
The SOWN Backup Cloud project is designed as a means of backing up important content and configuration from all SOWN servers, so if there is a failure the server can be rebuilt without losing anything important. It is a cloud as we currently have two different servers that hold backups:
Currently, the backup service runs once a day, rsync-ing all the files onto a central location (Sown-auth2) and then (gzip) tarballing all these files before scp-ing this tarball onto the two backup servers. Once on these backup servers, all tarballs from the previous 7 days are retained along with all weekly (Sunday) tarballs from the past 30 days and all monthly (1st of the month) tarballs. Periodically older monthly tarballs are manually deleted when the servers start to run out of space.
As rsync is only incremental old files will not be deleted even if they are deleted from their original location. Therefore there is a tidy-backups script that runs on Sown-auth2 to deal with directories where this is a particular issue, (e.g. the remote syslogs captured off nodes stored on Sown-auth2 under /srv/www/sisyphus/docs/).
Planned New Backup Setup
Rather than collating all the files on Sown-auth2 and then generating a tarball that can be copied to the two backup servers, a better solution would be to rsync to the backup servers themselves and then use ZFS snapshots. Scripts to perform this process have been written and put on GitHub.