SOWN Backup Cloud
The SOWN Backup Cloud project is designed as a means of backing up important content and configuration from all SOWN servers, so if there is a failure the server can be rebuilt without losing anything important. It is a cloud as we currently have three different servers that hold backups:
- Sown-gw - In Zepler (building 59) level 1 server room
- Backup2 - In building 16 level 2 server room
- Backup3 - In building 32 level 3 south server room
Currently, the backup service runs once a day, rsync-ing all the files onto a central location Sown-auth2 and then rsync-ing all these files onto the three backup servers. Once on these backup servers, a tarball is created of those files on that particular day. All tarballs from the previous 7 days are retained along with all weekly (Sunday) tarballs from the past 30 days and all monthly (1st ofthe month) tarballs. Periodically older monthly tarballs are manually deleted when the servers start to run out of space.
As rsync is only incremental old files will not be deleted even if they are deleted from their original location. Therefore there is a tidy-backups script that runs on each server to deal with directories where this is a particular issue, (e.g. the remote syslogs captured off nodes stored on Sown-auth2 under /srv/www/sisyphus/docs/).
Proposed changes to backups
Backup2 and Backup3 are in need of upgrading as there disks are really too small (< 100GB). As part of this process there has been a review of how backups should be managed. There are currently two suggested changes:
- Tarball the daily backup on the central server (Sown-auth2) and then SCP this to each backup servers rather than getting each server to do the tarball. Backup2 and Backup3 have rather slow CPUs and this dramatically increases the time it takes to do a daily backup.
- Run rsync in a mode where old files are deleted if they have been deleted from their original location, so the backups are a more accurate representation of what is currently on the servers being backed up.
meshach that is one of the 1U quarter depth black boxes like Backup2 and Backup3 (abednego and shadrach) is nearly ready to be deployed as a replacement for Backup3. meshach has a 1GB SATA drive rather than the smaller IDE disks currently on Backup2 and Backup3. It is running a 32-bit non-pae version of Ubuntu 14.04.