$ sudo apt-get install backupninja
# # |\_ # B A C K U P N I N J A /()/ # `\| # main configuration file # # how verbose to make the logs # 5 -- Debugging messages (and below) # 4 -- Informational messages (and below) # 3 -- Warnings (and below) # 2 -- Errors (and below) # 1 -- Fatal errors (only) loglevel = 4 # send a summary of the backup status to # this email address: reportemail = root # if set to 'yes', a report email will be generated # even if all modules reported success. (default = yes) reportsuccess = yes # if set to 'yes', a report email will be generated # even if there was no error. (default = yes) reportwarning = yes # if set to 'yes', disk space usage will be included in # the backup email report reportspace = no # where to rsync the backupninja.log to be aggregated in # a ninjareport reporthost = # what user to connect to reporthost to sync the # backupninja.log reportuser = ninja # where on the reporthost should the report go # NOTE: the name of the log will be used in the report, # use a globally unique name, preferably the hostname reportdirectory = /var/lib/backupninja/reports # set to the administration group that is allowed to # read/write configuration files in /etc/backup.d admingroup = root ####################################################### # for most installations, the defaults below are good # ####################################################### # where to log: logfile = /var/log/backupninja.log # directory where all the backup configuration files live configdirectory = /etc/backup.d # where backupninja helper scripts are found scriptdirectory = /usr/share/backupninja # where backupninja libs are found libdirectory = /usr/lib/backupninja # whether to use colors in the log file usecolors = yes # default value for 'when' when = everyday at 01:00 # if running vservers, set to yes vservers = no # programs paths # SLAPCAT=/usr/sbin/slapcat # LDAPSEARCH=/usr/bin/ldapsearch # RDIFFBACKUP=/usr/bin/rdiff-backup # CSTREAM=/usr/bin/cstream # MYSQL=/usr/bin/mysql # MYSQLHOTCOPY=/usr/bin/mysqlhotcopy # MYSQLDUMP=/usr/bin/mysqldump # PGSQLDUMP=/usr/bin/pg_dump # PGSQLDUMPALL=/usr/bin/pg_dumpall # GZIP=/bin/gzip # RSYNC=/usr/bin/rsync # VSERVERINFO=/usr/sbin/vserver-info # VSERVER=/usr/sbin/vserver # VROOTDIR=/var/lib/vservers
To preform the actual backup actions, backupninja processes each configuration file in /etc/backup.d according to the file's suffix: Databases:
Sufix (extensió) del fitxer:
Source Control:
Altres:
Remote backup:
* .rsync: backup action for incremental backups with rsync + hard links. * .rdiff: backup action for rdiff-backup (incremental backups) * .dup: backup action for duplicity (encrypted backups). * .wget: backup action for incremental backups with wget + hard links.
Support for additional configuration types can be added by dropping bash scripts with the name of the suffix into /usr/share/backupninja.
$ sudo ls -la /etc/backup.d/ total 8 drwxrwx--- 2 root root 4096 2008-12-10 01:45 . drwxr-xr-x 100 root root 4096 2010-12-04 16:16 ..
$ sudo cat /etc/backup.d/90.rdiff # options = --force # when = everyday at 02 [source] type = local keep = 60D # A few notes about includes and excludes: # 1. include, exclude and vsinclude statements support globbing with '*' # 2. Symlinks are not dereferenced. Moreover, an include line whose path # contains, at any level, a symlink to a directory, will only have the # symlink backed-up, not the target directory's content. Yes, you have to # dereference yourself the symlinks, or to use 'mount --bind' instead. # Example: let's say /home is a symlink to /mnt/crypt/home ; the following # line will only backup a "/home" symlink ; neither /home/user nor # /home/user/Mail will be backed-up : # include = /home/user/Mail # A workaround is to 'mount --bind /mnt/crypt/home /home' ; another one is to # write : # include = /mnt/crypt/home/user/Mail # 3. All the excludes come after all the includes. The order is not otherwise # taken into account. # files to include in the backup include = /var/spool/cron/crontabs include = /var/backups include = /etc include = /root include = /home include = /usr/local/*bin include = /var/lib/dpkg/status* include = /samba include = /opt exclude = /home/*/.gnupg exclude = /home/*/.local/share/Trash exclude = /home/*/.Trash exclude = /home/*/.thumbnails exclude = /home/*/.beagle exclude = /home/*/.aMule exclude = /home/*/gtk-gnutella-downloads ###################################################### ## destination section ## (where the files are copied to) [dest] type = remote directory = /backup1/caro host = 192.168.0.254 user = caro
Requereix dels paquets:
Si no teniu instal·lats els paquets els us instal·larà al executar l'assistent.
Exemple de fitxer de configuració:
$ sudo cat /etc/backup.d/10.sys packages = yes partitions = yes dosfdisk = yes hardware = yes luksheaders = no lvm = yes # packagesfile = /var/backups/dpkg-selections.txt # selectionsfile = /var/backups/debconfsel.txt # partitionsfile = /var/backups/partitions.__star__.txt # hardwarefile = /var/backups/hardware.txt # luksheadersfile = /var/backups/luksheader.__star__.bin # If vservers = yes in /etc/backupninja.conf then the following variables can # be used: # vsnames = all | <vserver1> <vserver2> ... (default = all)
La còpia de seguretat de la informació es guarda a /var/backups:
$ ls /var/backups/ apt.extended_states.0 aptitude.pkgstates.0 dpkg-selections.txt dpkg.status.2.gz hardware.txt partitions.sdb.txt shadow.bak apt.extended_states.1.gz aptitude.pkgstates.1.gz dpkg.status.0 group.bak lvm partitions.sdc.txt sysreport.txt apt.extended_states.2.gz debconfsel.txt dpkg.status.1.gz gshadow.bak partitions.sda.txt passwd.bak
Un exemple de fitxers de configuració:
$ sudo cat /etc/backup.d/20.mysql ### backupninja MySQL config file ### # hotcopy = < yes | no > (default = no) # make a backup of the actual database binary files using mysqlhotcopy. hotcopy = no # sqldump = < yes | no > (default = no) # make a backup using mysqldump. this creates text files with sql commands # sufficient to recontruct the database. # sqldump = yes # sqldumpoptions = <options> # (default = --lock-tables --complete-insert --add-drop-table --quick --quote-names) # arguments to pass to mysqldump # sqldumpoptions = --add-drop-table --quick --quote-names # compress = < yes | no > (default = yes) # if yes, compress the sqldump output. compress = yes # dbhost = <host> (default = localhost) # backupdir = <dir> (default: /var/backups/mysql) # where to dump the backups. hotcopy backups will be in a subdirectory # 'hotcopy' and sqldump backups will be in a subdirectory 'sqldump' backupdir = /var/backups/mysql # databases = <all | db1 db2 db3 > (default = all) # which databases to backup. should either be the word 'all' or a # space separated list of database names. databases = all configfile = /etc/mysql/debian.cnf
Les còpies de les bases de dades es fan a /var/backups/mysql/sqldump:
$ sudo ls -la /var/backups/mysql/sqldump total 152 drwx------ 2 root root 4096 2011-06-06 10:30 . drwx------ 3 root root 4096 2011-06-06 10:30 .. -rw------- 1 root root 371 2011-06-06 10:30 information_schema.sql.gz -rw------- 1 root root 135640 2011-06-06 10:30 mysql.sql.gz -rw------- 1 root root 1771 2011-06-06 10:30 phpmyadmin.sql.gz
L'assistent d'aquesta tasca s'encarrega de:
Que necessiteu?:
Exemple dels passos seguits:
$ sudo apt-get install rdiff-backup
Crear un usuari. Jo li solc posar el mateix nom que té la màquina origen, a l'exemple cop:
$ sudo adduser cop
Crear una carpeta per a l'usuari:
$ sudo mkdir -p /backup/cop.iesmontsia.org $ sudo chown cop:cop -R /backup/cop.iesmontsia.org
Modificar la home de l'usuari a la nova carpeta, es pot fer editant el fitxer /etc/passwd:
$ sudo joe /etc/passwd
Esborrar l'antiga home:
$ sudo rm -rf /home/cop
I ara ja podeu executar l'assistent a l'origen.
IMPORTANT: Com a carpeta de destinació cal indicar una carpeta que no existeixi! A l'exemple: /backup/cop.iesmontsia.org/cop.iesmontsia.org
Un exemple de fitxer és:
$ sudo cat /etc/backup.d/90.rdiff # options = --force # when = everyday at 02 [source] type = local keep = 60D # A few notes about includes and excludes: # 1. include, exclude and vsinclude statements support globbing with '*' # 2. Symlinks are not dereferenced. Moreover, an include line whose path # contains, at any level, a symlink to a directory, will only have the # symlink backed-up, not the target directory's content. Yes, you have to # dereference yourself the symlinks, or to use 'mount --bind' instead. # Example: let's say /home is a symlink to /mnt/crypt/home ; the following # line will only backup a "/home" symlink ; neither /home/user nor # /home/user/Mail will be backed-up : # include = /home/user/Mail # A workaround is to 'mount --bind /mnt/crypt/home /home' ; another one is to # write : # include = /mnt/crypt/home/user/Mail # 3. All the excludes come after all the includes. The order is not otherwise # taken into account. # files to include in the backup include = /var/spool/cron/crontabs include = /var/backups include = /etc include = /root include = /home include = /usr/local/*bin include = /var/lib/dpkg/status* include = /var exclude = /home/*/.gnupg exclude = /home/*/.local/share/Trash exclude = /home/*/.Trash exclude = /home/*/.thumbnails exclude = /home/*/.beagle exclude = /home/*/.aMule exclude = /home/*/gtk-gnutella-downloads ###################################################### ## destination section ## (where the files are copied to) [dest] type = remote directory = /backup/cop.iesmontsia.org/cop.iesmontsia.org host = 192.168.0.2 user = cop
Es pot fer consultant el fitxer de log:
$ sudo tail -f --lines=500 /var/log/backupninja.log
La instal·lació des dels repositoris configura el següent fitxer de cron:
$ cat /etc/cron.d/backupninja # /etc/cron.d/backupninja -- cron tab entry for package backupninja PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # run backupninja every hour on the hour 0 * * * * root if [ -x /usr/sbin/backupninja ]; then /usr/sbin/backupninja; fi
Com podeu veure s'executa al minut zero de cada hora de tots els dies. Cal però tenir en compte que després té una configuració interna https://labs.riseup.net/code/projects/backupninja/wiki/Configuration():
IMPORTANT: Per defecte, cada fitxer de configuració és processat a les 01:00 (1AM).
$ sudo cat /etc/backupninja.conf ... # default value for 'when' when = everyday at 01:00
Això es pot canviar especificant l'opció when. Per exemple:
when = sundays at 02:00 when = 30th at 22 when = 30 at 22:00 when = everyday at 01 <-- the default when = Tuesday at 05:00 when = hourly
IMPORTANT: Si varies configuracions s'ha d'executar al mateix temps aleshores s'executaran en l'ordre del prefix (número) indicat a cada fitxer de configuració. El when ha d'aparèixer abans de les seccions
Altres exemples:
when = tuesday at 05:30 when = TUESDAYS at 05
valors NO vàlids:
when = tuesday at 2am when = tuesday at 2 when = tues at 02
Al fitxer de configuració principal indiqueu a qui enviar els correus electrònics:
$ sudo cat /etc/backupninja.conf ... # send a summary of the backup status to # this email address: reportemail = root # if set to 'yes', a report email will be generated # even if all modules reported success. (default = yes) reportsuccess = yes # if set to 'yes', a report email will be generated # even if there was no error. (default = yes) reportwarning = yes # if set to 'yes', disk space usage will be included in # the backup email report reportspace = no
Podeu utilitzar l'ordre mail per veure els correus electrònics locals:
# mail Mail version 8.1.2 01/15/2001. Type ? for help. "/var/mail/root": 8 messages 8 new >N 1 [email protected] Sat Dec 4 21:00 15/494 backupninja: linux N 2 [email protected] Sat Dec 4 21:04 15/494 backupninja: linux N 3 [email protected] Sun Dec 5 14:45 22/834 backupninja: linux WARNING N 4 [email protected] Sun Dec 5 18:20 15/492 backupninja: linux N 5 [email protected] Sun Dec 5 18:20 15/494 backupninja: linux N 6 [email protected] Sun Dec 5 18:54 15/494 backupninja: linux N 7 [email protected] Mon Dec 6 02:19 17/560 backupninja: linux N 8 [email protected] Tue Dec 7 02:22 17/560 backupninja: linux & 8 Message 8: From [email protected] Tue Dec 7 02:22:22 2010 X-Original-To: root To: [email protected] Subject: backupninja: linux Date: Tue, 7 Dec 2010 02:22:22 +0100 (CET) From: [email protected] (root)
success -- /etc/backup.d/10.sys success -- /etc/backup.d/20.mysql success -- /etc/backup.d/90.rdiff
&
IMPORTANT: No funciona
backuninja no funciona amb el nou format (LDIF) de fitxer de configuració de Ldap. Ha calgut modificar el codi del helper de Ldap.
Els fitxers han quedat de la següent manera:
$ sudo cat /usr/share/backupninja/ldap
Acacha Wiki# -*- mode: sh; sh-basic-offset: 3; indent-tabs-mode: nil; -*-
#
# openldap backup handler script for backupninja
#
getconf backupdir /var/backups/ldap
getconf conf /etc/ldap/slapd.conf
getconf databases all
getconf compress yes
getconf ldif yes
getconf restart no
getconf method ldapsearch
getconf passwordfile
getconf binddn
getconf ldaphost
getconf ssl yes
getconf tls no
if [ $ssl = 'yes' ]; then
URLBASE="ldaps"
else
URLBASE="ldap"
fi
status="ok"
# Modified by Sergi Tur
# /etc/ldap/slapd.conf doesn' exists
# ignore error
#[ -f $conf ] || fatal "slapd config file ($conf) not found"
[ -d $backupdir ] || mkdir -p $backupdir
[ -d $backupdir ] || fatal "Backup directory '$backupdir'"
# Modified by Sergi Tur
# /etc/ldap/slapd.conf doesn' exists
# ignore error
#dbsuffixes=(`/usr/bin/awk 'BEGIN {OFS=":"} /[:space:]*^database[:space:]*\w*/ {db=$2}; /^[:space:]*suffix[:space:]*\w*/ {if (db=="bdb"||db=="hdb"||db="ldbm") print db,$2}' $conf|/bin/sed -e 's/[" ]//g'`)
## LDIF DUMP
if [ "$ldif" == "yes" ]; then
dumpdir="$backupdir"
[ -d $dumpdir ] || mkdir -p $dumpdir
if [ "$databases" == 'all' ]; then
dbcount=`grep '^database' $conf | wc -l`
let "dbcount = dbcount - 1"
databases=`seq 0 $dbcount`;
fi
for db in $databases; do
if [ `expr index "$db" "="` == "0" ]; then
# db is a number, get the suffix.
dbsuffix=${dbsuffixes[$db]/*:/}
else
dbsuffix=$db
fi
# some databases don't have suffix (like monitor), skip these
if [ "$dbsuffix" == "" ]; then
continue;
fi
if [ "$method" == "slapcat" ]; then
execstr="$SLAPCAT -f $conf -b $dbsuffix"
else
LDAPARGS=""
if [ "$tls" == "yes" ]; then
LDAPARGS="-ZZ"
fi
if [ -n "$ldaphost" ]; then
execstr="$LDAPSEARCH $LDAPARGS -H $URLBASE://$ldaphost -x -L -b ""$dbsuffix"" -D ""$binddn"" -y $passwordfile"
else
execstr="$LDAPSEARCH -H $URLBASE://$ldaphost -x -L -b ""$dbsuffix"" -D ""$binddn"" -y $passwordfile"
fi
[ -f "$passwordfile" ] || fatal "Password file $passwordfile not found. When method is set to ldapsearch, you must also specify a password file."
debug "$execstr"
fi
if [ ! $test ]; then
if [ "$restart" == "yes" ]; then
debug "Shutting down ldap server..."
/etc/init.d/slapd stop
fi
ext=
if [ "$compress" == "yes" ]; then
ext=".gz"
fi
touch $dumpdir/$dbsuffix.ldif$ext
if [ ! -f $dumpdir/$dbsuffix.ldif$ext ]; then
fatal "Couldn't create ldif dump file: $dumpdir/$dbsuffix.ldif$ext"
fi
if [ "$compress" == "yes" ]; then
execstr="$execstr | $GZIP > $dumpdir/$dbsuffix.ldif.gz"
else
execstr="$execstr > $dumpdir/$dbsuffix.ldif"
fi
debug "$execstr"
output=`su root -c "$execstr" 2>&1`
code=$?
if [ "$code" == "0" ]; then
debug $output
info "Successfully finished ldif export of $dbsuffix"
else
warning $output
warning "Failed ldif export of $dbsuffix"
fi
if [ "$restart" == "yes" ]; then
debug "Starting ldap server..."
/etc/init.d/slapd start
fi
fi
done
fi
return 0
$ sudo cat /usr/share/backupninja/ldap.helper
# -*- mode: sh; sh-basic-offset: 3; indent-tabs-mode: nil; -*-
HELPERS="$HELPERS ldap:ldap_database_backup"
ldap_create_file() {
while true; do
checkBox "ldap action wizard" "check options (slapcat OR ldapsearch)" \
"slapcat" "export ldif using slapcat" yes \
"ldapsearch" "export ldif using ldapsearch" no \
"compress" "compress the ldif output files" yes \
"ssl" "use SSL (deprecated)" no \
"tls" "use TLS extended operations (RFC2246, RFC2830)" yes
status=$?
compress="compress = no"
method="method = <unset>"
restart="restart = no"
binddn=""
passwordfile=""
ssl="ssl = no"
tls="tls = no"
[ $status = 1 ] && return;
result="$REPLY"
for opt in $result; do
case $opt in
'"compress"') compress="compress = yes";;
'"slapcat"')
method="method = slapcat"
[ "$_RESTART" == "yes" ] && restart="restart = yes"
;;
'"ldapsearch"')
method="method = ldapsearch"
inputBox "ldap action wizard" "ldapsearch requires authentication. Specify here what password file to use. It must have the password with no trailing return and it should not be world readable."
[ $? = 1 ] && return
passwordfile="passwordfile = $REPLY"
inputBox "ldap action wizard" "ldapsearch requires authentication. Specify here what DN to bind as:"
[ $? = 1 ] && return
binddn="binddn = $REPLY"
require_packages ldap-utils
;;
'"ssl"') ssl="ssl = yes";;
'"tls"') tls="tls = yes";;
esac
done
get_next_filename $configdirectory/30.ldap
cat > $next_filename <<EOF
$method
$compress
$restart
$binddn
$passwordfile
$ssl
$tls
# backupdir = /var/backups/ldap
# conf = /etc/ldap/slapd.conf
# databases = all
EOF
chmod 600 $next_filename
return
done
}
ldap_wizard() {
bdb=no
hdb=no
ldbm=no
# Added by Sergi Tur
# /etc/ldap/slapd.conf doesn't exists with new slapd configuration: /etc/ldap/slapd.d
#
for backend in `grep -e "^backend" /etc/ldap/slapd.conf | /usr/bin/awk '{print $2}'`; do
if [ "$backend" == "bdb" ]; then
bdb=yes
elif [ "$backend" == "hdb" ]; then
hdb=yes
elif [ "$backend" == "ldbm" ]; then
ldbm=yes
fi
done
# Force backend to hdb
hdb=yes
if [ "$bdb" == "yes" -o "$hdb" == "yes" ]; then
if [ "$ldbm" == "no" ]; then
msgBox "ldap action wizard" "It looks like the backend in your slapd.conf is set to BDB or HDB. If this is not the case, exit this wizard! From this point on, we will assume BDB or HDB backend, which might have disasterious consequences if this is incorrect."
_RESTART=no
ldap_create_file
fi
elif [ "$ldbm" == "yes" ]; then
msgBox "ldap action wizard" "It looks like the backend in your slapd.conf is set to LDBM. Because of this, you will have less options (because it is not safe to use slapcat while slapd is running LDBM)."
_RESTART=yes
ldap_create_file
else
msgBox "ldap action wizard" "I couldn't find any supported backend in your slapd.conf. Bailing out."
return
fi
}
$ sudo cat /etc/backup.d/30.ldap
method = ldapsearch compress = yes restart = no binddn = cn=admin,dc=iesebre,dc=com passwordfile = /etc/backup.d/.password.ldap ssl = no tls = no # backupdir = /var/backups/ldap # conf = /etc/ldap/slapd.conf # databases = all databases = dc=iesebre,dc=com
Observeu el fitxer amb la paraula de pas:
$ sudo ls -la /etc/backup.d/.password.ldap -rw------- 1 root root 13 2010-12-05 17:07 /etc/backup.d/.password.ldap
Per seguretat:
$ sudo chmod 600 /etc/backup.d/.password.ldap $ sudo chmod 600 /etc/backup.d/30.ldap
TODO
execution
backups get orchestrated by backupninja. i use it just a wrapper for my bash scripts - to have single backup log. each script starts with
function handle {
echo Error
error problem occured
}
set -e
trap handle ERR
so i get error in logs whenever any of the commands [ eg mysqldump or rsync ] fails.
all backups end up in rdiff repository so i have n days of increments.
all backups are transmitted using rsync to central storage server.
on storage server all backups are verified daily and after successful verification of data on local disk they get copied to external usb drive.
verification
backupninja.log on all servers is monitored by nagios. i check if they contain only DEBUG and INFO messages. anything else triggers alert.
every backup 'touches' a test file, presence and freshness of which is monitored on central backup repository server with nagios.
additionally more critical sql dumps get checked for their size [not just freshness] and completeness [eg at the end of mysql dumps i expect fresh timestamp in
-- Dump completed on 2010-04-22 23:21:02
all rdiff archives are verified daily before data gets synced to USB drive and then again after they get synced. so even if nightly transfer is interrupted i will have consistent repository just on USB disk. result of checking is logged to file which content and freshness is checked by nagios.
usb disks get rotated weekly and are stored offline, just in case. this might be overkill for bigger amounts of data, but works fine for ~300GB of slowly changing files/dumps.
trends
i use simple custom munin plugin to plot size of diff/data for each rdiff repository.
time it takes to execute can be checked in backupninja logs but for now i dont bother about it.
ninjabackup utilitza logrotate per fer rotar els fitxers de log:
$ cat /etc/logrotate.d/backupninja
/var/log/backupninja.log {
rotate 6
monthly
compress
missingok
}
En aquest fitxer es registren les accions portades a terme per l'aplicació backupninja i pot servir per comprovar si funciona o no correctament.
Si us dona l'error:
Fatal Error: Destination directory /backup/cop.iesmontsia.org exists, but does not look like a rdiff-backup directory. Running rdiff-backup like this could mess up what is currently in it. If you want to update or overwrite it, run rdiff-backup with the --force option. Fatal Error: Lost connection to the remote system Warning: Failed backup up source Warning: <<<< finished action /etc/backup.d/90.rdiff: WARNING
Consulteu:
Rdiff-backup#Destination_directory_XXX_exists.2C_but_does_not_look_like_a_rdiff-backup_directory._Running_rdiff-backup_like_this_could_mess_up_what_is_currently_in_it