BackupPC

Current status

This guide is incomplete. It hasn't been tested very thoroughly. (Some updates were made as the guide was being created.) The guide is believed to only be complete enough to get a successful “test” backup, although scheduling might not be fully resolved yet...

Configuration of the “client” systems is only minimally documented, at this time.

CGI support is not fully operational at this time.

Overview
[#bkupcxpt]: Key data transport method(s)

A computer, called the “server”, keeps track of backups and initiates the backups on the “client” machines. (Yes, that does feel a bit backwards, in the sense that the server makes the request, and then the client fulfills it. If it helps, think of the server as the powerful machine, able to store data that might be found on multiple “client” machines.)

The client machine also needs to respond to a way to transfer files. The available methods are:

  • SMB (the backup server will run samba's smbclient)
  • rsyncd
  • The server uses ssh to run ssh, or the server runs rsh to run rsync
  • ssh/rsh/nfs piped through tar
    • Documentation indicates this needs to be GNU tar, although the Mac OSX tar may work? (And Mac OSX tar might be similar to OpenBSD's?)

Each “transport method” just listed is also mentioned by BackupPC FAQ: How to backup a client, first sub-section, BackupPC FAQ: Configuration file: How to backup a client: Transfer method.

(Which transport method gets used will depend on the specified configuration. This is demonstrated in more detail in the upcoming small section about BackupPC Configuration: XferMethod.)

Requirements

The "client" machines (which have the data that needs to be backed up) need to respond to ping (per http://backuppc.sourceforge.net/faq/BackupPC.html#BackupPC-operation ).

Also, one of the “key data transfer method(s)” needs to be supported by the client.

The server needs to support hard linking. Basically, the current general expectation is that the server is using a Unix-like operating system. Although a machine using Microsoft Windows may work for a client, that is not supported as a server the last time this was checked.

Limitations

In addition to the need to fulfill the requirements, there may be some additional limitations worth knowing about.

http://backuppc.sourceforge.net/faq/limitations.html

Some additional resources

Knowing the directories available on the computer being used can also be helpful. The following example shows the results fromt he OpenBSD package. e.g.:

$ grep -i ^\$Conf{ /etc/backuppc/config.pl | grep -i \Dir}
$Conf{TopDir}      = '/var/db/backuppc';
$Conf{ConfDir}     = '/etc/backuppc';
$Conf{LogDir}      = '/var/log/backuppc';
$Conf{InstallDir}  = '/usr/local';
$Conf{CgiDir}      = '/usr/local/bin';
$CgiImageDir} = '/usr/local/share/backuppc';
$
Installation
time sudo -Hi pkg_add -ivv backuppc
Next steps
Supporting a new user
Adding a user

This step is optional, but is recommended for security. If this is not done, some configuration options may need to be changed from defaults.

This guide recommends naming the user “_backuppc”

See: backing up the user database, and adding a user

Do NOT add the user to unnecessary privileged groups such as “wheel” or “_sshok”. The point of this user account is to run some software. Actual administration will be done with a different account. The user will be given some escalated privileges, but only enough to run software that is related to backups.

Providing access
sudo ls -lRa /var/db/backuppc /var/log/backuppc | tee -a ~/backuppc.dir
sudo chown _backuppc /var/db/backuppc/
sudo chown _backuppc /var/db/backuppc/pc/
sudo chmod g+x /var/db/backuppc/
sudo chown _backuppc /var/db/backuppc/cpool/
sudo chown _backuppc /var/db/backuppc/trash/
sudo chown _backuppc /var/log/backuppc/
[ -f /var/log/backuppc/LOG ]&&sudo chown _backuppc /var/log/backuppc/LOG

(Yes, that could have been done with as few as two commands. That was seperated out as multiple commands because multiple commands are probably easier for anyone who needs to manually enter the commands, but who can readily use command line “recall”/editing.)

The next step is to provide this new user with escalated privileges, but only enough to run BackupPC command.

cpytobak /etc/sudoers
sudo -E visudo

(Yes, that could have been done with as few as two commands. That was seperated out as multiple commands because multiple commands are probably easier for anyone who is needed to manually type.)

Add:

_backuppc ALL=(ALL) NOPASSWD:/usr/local/bin/BackupPC
Configuration file(s)
Test need for config backup
ls -l /etc/backuppc/
ls -l /usr/local/share/examples/backuppc/
diff -s /etc/backuppc/config.pl /usr/local/share/examples/backuppc/config.pl
echo ${?}
diff -s /etc/backuppc/hosts /usr/local/share/examples/backuppc/hosts

If output says:

Files /etc/backuppc/config.pl and /usr/local/share/examples/backuppc/config.pl a re identical

Then there is no need to back up one of the files (e.g., /etc/backuppc/) as long as the other file remains unmodified.

Specify a system with data to back up

Go ahead and start adding support for a system/computer/device that has data that will be backed up.

Customize this variable, to include the name of the system that has data that is going to be backed up.

NEEDBKUP=sysname

Make the host be recognized. (If the file needs to be backed up, do that first. The following example makes some sort of backup.)

sudo cpytobak /etc/backuppc/hosts
echo ${NEEDBKUP} 1 root| sudo -n tee -a /etc/backuppc/hosts
Per-PC configuration file

The next configuration steps go to a different file.

According to BackupPC FAQ on configuration files, the Per-PC configuration file locations vary based on operating system. There are two possible names.

This guide uses a variable that points to the BackupPC Per-PC configuration file.

If BackupPC is known to be using the configuration for FHS-compliant platforms
BKPCPRPC=/etc/backuppc/pc/${NEEDBKUP}.pl
If BackupPC is known to be using the configuration for platforms that are not FHS-compliant

That has not been tested by the author of this text. See BackupPC FAQ on configuration files and determine what file to set BKPCPRPC to.

Don't worry too terribly much about picking the wrong filename. If the wrong filename is chosen, then the configuration file will have no effect. In that case, update the BKPCPRPC variable and move the file.

If there is uncertainty about whether BackupPC is using FHS-compliant mode

This guide recommends starting by trying to follow the earlier instructions for the FHS-compliant BackupPC. If that doesn't work, try the other instructions.

Make sure any required directory exists.

ls -l /etc/backuppc/
sudo mkdir /etc/backuppc/pc/
[#bkupcxfm]: BackupPC Configuration: XferMethod

A transport method needs to be specified.

echo \# \$Conf{XferMethod} = \'smb\'\;| sudo -n tee -a ${BKPCPRPC}
echo \$Conf{XferMethod} = \'rsync\'\;| sudo -n tee -a ${BKPCPRPC}

...

echo \$Conf{XferLogLevel} = 2\;| sudo -n tee -a ${BKPCPRPC}

Specify...

echo \$Conf{BackupFilesOnly} = \[\'/etc\', \'/home\'\]\;| sudo -n tee -a ${BKPCPRPC}

By default, backups older than this may be deleted, unless the minimum number of backups hasn't been reached yet.

Current documentation commentary: unknown how to disable this, and unknown max, so 999 was selected (2.736986301369863 years)

echo \$Conf{FullAgeMax} = 999\;| sudo -n tee -a ${BKPCPRPC}

If a full backup is scheduled in less than this time, the software will basically ignore the request. This is the "Minimum period in days between full backups."

echo \$Conf{FullPeriod} = .02\;| sudo -n tee -a ${BKPCPRPC}

Adjust number of full backups to keep before old ones get rotated away. Default is 1, per:
grep Conf{FullKeepCnt} /etc/backuppc/config.pl | grep -v ^\#

...

echo \$Conf{FullKeepCnt} = 3\;| sudo -n tee -a ${BKPCPRPC}

Adjust number of incremental backups to keep before old ones get rotated away. Default is 6, per: grep Conf{IncrKeepCnt}

grep Conf{IncrKeepCnt} /etc/backuppc/config.pl | grep -v ^\#

6 might be a fine default; this configuration example shows a higher value of 99, which was selected just because it was a value that might be nicer during testing.

echo \$Conf{IncrKeepCnt} = 99\;| sudo -n tee -a ${BKPCPRPC}

6 might be a fine default; this configuration example shows a higher value of 99, which was selected just because it was a value that might be nicer during testing.

e.g.:

$ grep -i ^\$Conf{RsyncClientCmd} /etc/backuppc/config.pl
$Conf{RsyncClientCmd} = '$sshPath -q -x -l root $host $rsyncPath $argList+';
$

adjust that, customizing this as sensible:

echo \$Conf{RsyncClientCmd} = \'\$sshPath -q -x -l _bakslave -p 22 -i /home/_backuppc/keys/$(hostname -s)/$(hostname -s) \$host \$rsyncPath \$argList+\'\;| sudo -n tee -a ${BKPCPRPC}
echo ${VISUAL}
sudoedit ${BKPCPRPC}
  • If you want to copy and paste that directly into a text editor, rather than running it from the command line, remove all of the backslashes before pasting, because the backslashes don't end up being part of the text that goes into the file. (Also, of course, remove the very beginning, and the end, of the command line.)
  • The main reason for adding this is so that some things can be customized. Customize the user's login name (if needed), the TCP port number (if needed, which is a rather good idea in the opinion of this text's author), and the location of the private key file (which this example shows named after systems).
  • At the time of this writing, using ~ instead of /home/ has not been fully tested. It'll probably work, but some programs don't like to have such tilde characters be expanded like a shell prompt. To minimize risks, this example is showing the full path to the key.
  • (Official documentation related to this option: BackupPC FAQ: Configuration option: RsyncClientCmd)

Adjust the path that should be run on the remote system.

e.g.:

$ grep -i ^\$Conf{RsyncClientPath} /etc/backuppc/config.pl
$Conf{RsyncClientPath} = '/usr/local/bin/rsync'
$

adjust that, customizing this as sensible:

echo \$Conf{RsyncClientPath} = \'/usr/bin/sudo -n /usr/local/bin/rsync\'\;| sudo -n tee -a ${BKPCPRPC}
echo ${VISUAL}
sudoedit ${BKPCPRPC}
  • The reason for this change is that the default configuration logged in as root, who could potentially run any other command. This method involves logging in as a less-privileged user, and then escalating privileges. The idea is that this less-privileged user will only be able to escalate privileges by running rsync, which will substantially limit the user account's ability to run commands in an escalated fashion.
  • (Official documentation related to this option: BackupPC FAQ: Configuration option: RsyncClientPath)

Specifying the schedule:

Default: Every hour except midnight

This is the default anyway, so there is no need to do this. However, it is being shown as an example so that people can easily see what it would look like, and can choose to customize it:

echo \$Conf{WakeupSchedule} = \[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,| sudo -n tee -a ${BKPCPRPC}
echo 15, 16, 17, 18, 19, 20, 21, 22, 23\]\;| sudo -n tee -a ${BKPCPRPC}
Half-hour increments

This may be a better schedule when testing small backups.

echo \$Conf{WakeupSchedule} = \[1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5,| sudo -n tee -a ${BKPCPRPC}
echo 7, 7.5, 8, 8.5, 9, 9.5, 10, 10.5, 11, 11.5, 12, 12.5, 13, 13.5, 14, 14.5,| sudo -n tee -a ${BKPCPRPC}
echo 15, 15.5, 16, 16.5, 17, 17.5, 18, 18.5, 19, 19.5, 20, 20.5, 21, 21.5, 22,| sudo -n tee -a ${BKPCPRPC}
echo 23, 23.5\]\;| sudo -n tee -a ${BKPCPRPC}
echo ${VISUAL}
sudoedit ${BKPCPRPC}

Whew... how about reviewing that file that got created?

echo ${VISUAL}
sudoedit ${BKPCPRPC}
Making the key

An earlier step involved setting RsyncClientCmd to create a key file. So, that file needs to be created. Creating keys is discussed at generating SSH keys.

sudo -u _backuppc mkdir ~_backuppc/keys/
sudo -u _backuppc mkdir ~_backuppc/keys/$(hostname -s)/
sudo -u _backuppc ssh-keygen -t rsa -N "" -C "privKeyName=$(hostname -s)" -f ~_backuppc/keys/$(hostname -s)/$(hostname -s)
echo ${VISUAL}
sudoedit ~_backuppc/keys/$(hostname -s)/$(hostname -s).pub

The reason to edit the public key file is to follow the advice about the public key file. Recommended further modification: SSH public key file's comment. (Actually, this ought to have been done with the sample command that creates the key files, but going into the editor is just done as a way to manually check the created comment.)

sudoedit ~_backuppc/keys/$(hostname -s)/$(hostname -s).pub
Configuring the client system
HTML clean-up recommended here...
ssh -p 22 -i ... -l admin-account system-name

sudo adduser
_bakslave
Slavefor Backups
_sshok (not wheel)

sudo ksh
cd ~_bakslave

# echo -n 'command="/usr/bin/sudo -n /usr/local/bin/rsync ",no-agent-forwarding,no-port-forwarding,no-X11-forwarding '|sudo -n tee -a ~_bakslave/.ssh/authorized_keys
	# don't do that; it isn't allowing passed parameters as intended.  Might be able to pass parameters via variable, as described by 
http://cybermashup.com/2013/05/14/restrict-ssh-logins-to-a-single-command/


cat >> .ssh/authorized_keys
sudoedit .ssh/authorized_keys

### Actually, don't do this:
at the start of the line that was just added, add:
command="echo Say \\"Hello\\" cheerfully daily.",no-agent-forwarding,no-port-forwarding,no-X11-forwarding 
(Note: a space at the end)


pkg_add -ivv rsync--iconv

visudo
_bakslave ALL=(ALL) NOPASSWD:/usr/local/bin/rsync *
Some other/misc notes:
ls -ld /var/db/backuppc/pc
ls -ld /var/db/backuppc/cpool
Starting the server
Initial test
time sudo -u _backuppc BackupPC -d
echo ${?}
grep -i \$Conf{LogDir} /etc/backuppc/config.pl

Then, when running:

time sudo -u _backuppc BackupPC_Servermesg status info
echo ${?}

The following is possible output that, unsurprisingly, is not desirable:

Can't connect to server (unix connect: Connection refused)

Better output is something like:

Got reply: %Info = ("startTime" => "1446844461","DUlastValue" => 8,"HostsModTime" => "1446841799","DUDailyMax" => 8,"ConfigLTime" => "1446844461","DUDailyMaxTime" => "1446844461","pid" => 24733,"nextWakeup" => "1446847200","DUlastValueTime" => 1446844461,"ConfigModTime" => 1446838641,"Version" => "3.3.0");
Longer term approach

A key reason for doing that is just to see that the server works well. If not, some useful error messages may be provided. Another technique that may be worthwhile is closing any running copy of the server, and instead running “BackupPC” (without the -d (to see if that shows some error messages), although that will probably be best to do from another command prompt. (Being in the habit of using a terminal multiplexer is highly encouraged.)

When all is well, try the operating system's normal approach.

On systems other than OpenBSD, the following may need to be customized due to ps implementation differences. Also, the process of starting up a service may be different on different platforms.

ps -auxww | grep -i BackupPC | grep -v grep
sudo pkill -f BackupPC
ps -auxww | grep -i BackupPC | grep -v grep
sudo /etc/rc.d/backuppc check
echo ${?}
sudo /etc/rc.d/backuppc restart
echo ${?}
  • It is entirely okay for the check to report “failed”, and to provide a return code of 1. That indicates that BackupPC isn't running, which is the result that was intended by using pkill.
Some other/misc notes
#seems the /var/log/backuppc/status.pl may get created with every run? So if _backuppc owns the log directory, you won't get: #couldn't do /var/log/backuppc/status.pl: Permission denied If you've got that working, quit the server. That can be done with Ctrl-C, or SIGINT Then, try: sudo /etc/rc.d/backuppc start (which wil end up running BackupPC with the -d parameter)
Additional checking

The “client” host should show up with this output:

sudo -u _backuppc BackupPC_serverMesg status hosts

Also, there are a couple of commands that specify the “client” system's host name, and which provide details about a specific “client” host. Customize the following examples by specifying the name of the client system:

time sudo -u _backuppc BackupPC_serverMesg "status client"
  • e.g.:
    time sudo -u _backuppc BackupPC_serverMesg "status ${NEEDBKUP}"

The desired results from that command are:

Got reply: ok

The other command is:

time sudo -u _backuppc BackupPC_serverMesg "status host(client)"
  • e.g.:
    time sudo -u _backuppc BackupPC_serverMesg "status host(${NEEDBKUP})"

The desired result from that command is to receive some info. Instead of just “Got reply: ok” (which, despite the word “ok”, is actually not okay/good), there should be some more details like “Got reply: %StatusHost = ("activeJob" => 0,"state" => "Status_idle","CmdQueueOn" => undef,"UserQueueOn" => undef,"BgQueueOn" => undef);

Firewall rules

Make sure that traffic is permitted if it matches the BackupPC Configuration: XferMethod. (That configuration specifies which of the key/main data transport method(s) gets used.) Of course, more specific rules can be used, like only permitting traffic involving an authorized host. (If a reminder is needed about what hosts need to be supported right away, use “ cat /etc/backuppc/hosts ”.

Using OpenBSD's pf
cpytobak /etc/pf.conf
echo ${VISUAL}
sudoedit /etc/pf.conf

(Because there are multiple possible transport methods, no specific rule is currently a clear recommendation. This text does not currently have an example rule.)

sudo pfctl -nf /etc/pf.conf&&sudo pfctl -ef /etc/pf.conf
Performing a backup
[#bkupcdmp]: To back up (with monitoring)
time sudo -u _backuppc BackupPC_dump -v -f client
echo ${?}
  • e.g.:
    time sudo -u _backuppc BackupPC_dump -v -f ${NEEDBKUP}
    echo ${?}
/// sudo -u _backuppc /usr/bin/ssh -q -x -l _bakslave -p 22 -i /home/_backuppc/keys/THISHOST/THATHOST/THATHOST THATHOST /usr/bin/sudo\ -n\ /usr/local/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . / sudo -u _backuppc /usr/bin/ssh -q -x -l _bakslave -p 22 -i /home/ _backuppc/keys/THISHOST/THATHOST/THATHOST THATHOST /usr/bin/sudo\ -n\ /usr/local/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
Alternate method...
Starting a full backup

Example:

$ time sudo -u _backuppc BackupPC_serverMesg backup client client _backuppc 1
Got reply: ok: requested backup of client (doFull)
$ echo ${?}
0
$ sudo tail -1 /var/log/backuppc/LOG
YYYY-MM-DD hh:mm:ss User _backuppc requested backup of client (client)
$

but then... it doesn't happen... BackupPC FAQ: FullPeriod describes a configuration option as, “Minimum period in days between full backups. A full dump will only be done if at least this much time has elapsed since the last full dump, and at least $Conf{IncrPeriod} days has elapsed since the last successful dump.”

If you really want to make another backup when there is already a sufficient backup, then either adjust the backup's configuration, or see: “To back up (with monitoring)

Seeing results
The backups file

Upon a successful backup, a new entry should be made in the /var/db/backuppc/pc/client/backups file.

e.g.:

$ ls -l /var/db/backuppc/pc/client/
$ cat /var/db/backuppc/pc/client/backups
0 full 1447464996 1447465034 491 4523622 4 0
0 0 0 0 3 0 0
1 rsync 0 3.3.0
1 full 1447469643 1447469330 491 4523622 458 4510889
0 0 0 0 31500524 0
1 rsync 0 3.3.0

Look for the job type (e.g. “full), and to the left is the number that corresponds to where the data got stored.

(HTML clean-up needed here... 0 full 1447464996 1447465034 491 4523622 4 0 0 0 0 0 3 0 0 1 rsync 0 3.3.0 1 full 1447469643 1447469662 491 4523622 458 4510889 0 0 0 0 3 1500524 0 1 rsync 0 3.3.0 1 full 1446851681 1446851700 491 4522445 399 4400714 0 0 0 0 3 1466986 0 1 rsync 0 3.3.0 2 full 1446893741 1446893760 491 4522445 457 4506644 0 0 0 0 3 1499258 0 1 rsync 0 3.3.0
ls -l /var/db/backuppc/pc/client/
ls -l /var/db/backuppc/pc/client/1/f%2f/fetc/

Note that those are not necessarily direct copies of the files. (For details about accessing a copy of the actual file data, see instructions about restoring the data.)

The number “1/” may change. The number related to the most recent full copy may be found in /var/db/backuppc/pc/client/backups

Working with a running job
sudo -u _backuppc BackupPC_serverMesg status queues | sed 's/,/,\n/g' | ${PAGER}
https://adsm.org/lists/html/BackupPC-users/2010-12/msg00074.html
sudo -u _backuppc BackupPC_serverMesg stop [client]
sudo -u _backuppc BackupPC_serverMesg status jobs
Watching active progress
Watching active progress: http://sourceforge.net/p/backuppc/mailman/message/27377911/ http://sysadminnotebook.blogspot.com/2011/09/watch-backuppc-progress.html clean-up desired...
	ls -l /var/db/backuppc/pc/
	ls -l /var/db/backuppc/pc/nulifdhc
	/usr/local/bin/BackupPC_zcat /var/db/backuppc/pc/nulifdhc/XferLOG.*.z


Should view nightly jobs...

After backup:

ls -l /var/db/backuppc/pc/nulifdhc/LOG.*

And, repeating some of the commands mentioned earlier:

time sudo -u _backuppc BackupPC_serverMesg "status client"
time sudo -u _backuppc BackupPC_serverMesg "status host(client)"
sudo -u _backuppc BackupPC_serverMesg status hosts
time sudo -u _backuppc BackupPC_Servermesg status info
Supporting the CGI interface
  1. Get the WWW server to respond to standard HTTP
  2. Get CGI working. (See: ...
  3. Get a test backup working in BackupPC. (This might not be absolutely required to do before working further on CGI, but it is one way that works.)
  4. Using OpenBSD httpd

    This guide is not complete at this time. If this guide is followed, the server will still have an issue (resulting in the web server responding by providing a return code of 500) becuase of missing files. A complete guide is not yet available...

    ls -l /usr/local/bin/BackupPC_Admin*
    ls -l /var/www/bin/
    sudo cp /usr/local/bin/BackupPC_Admin /var/www/sites/$(hostname -s)/cgi-bin/
    Using Apache

    Probably want to look at /usr/local/share/examples/backuppc/httpd-backuppc.conf

    Perhaps see also: Forum posting

Misc notes
Restore: http://backuppc.sourceforge.net/faq/BackupPC.html#CGI-restore-options has a "direct restore" option http://backuppc.sourceforge.net/faq/BackupPC.html#Command-line-restore-options does not. A copy of the file can be retrieved. Apparently the person who runs the command will be responsible for replacing the original file. sudo /etc/rc.d/backuppc restart Setting up the web server/CGI I beieve the following is true: The web server needs to be able to access at least some of BackupPC's files. The web server runs the perl script called BackupPC_Admin. One option provided by According to BackupPC mailing list posting archive about adding CGI, the web server can “mount” the location “where BackupPC is installed (eg: /usr/local/BackupPC) on the other machine”. A problem: With OpenBSD, the BackupPC package merges files into the system's heirarchy. e.g., there may be 20 different /usr/local/bin/BackupPC* executables, and /usr/local/bin/ is not specific to BackupPC. In such a setup, using a “mount” command to permitt he web server to access just BackupPC's data is probably not going to be feasible, since mounting will typically involve sharing at least a directory (as SMB supports), if not an entire filesystem (like NFS is designed for). So, with the “mount” option unavailable, the other supported configuration is to have both the web server and a copy of BackupPC installed to the same machine. Then the web server can run its own copy of BackupPC, which can load a config.pl file, which can optionally have $Conf{ServerHost} and other settings point to another machine that has BackupPC running. The first step in the process is to get the web server running, and to have BackupPC installed. Then, getting CGI working is likely to be the next step. Testing this will require a CGI program. That could be the BackupPC_Amdin from BackupPC, although using a simple program to test CGI might not be a bad approach. The second step is to have the web server direct requests to http://serverName/cgi-bin/BackupPC_Admin to run perl and have perl run the BackupPCAdmin script that came from the BackupPC package. Fortunately, the BackupPC_Admin script starts with #!/usr/bin/perl so typically this just means the web server needs to try to "run" the BackupPC_Admin script, and that first line will cause PERL to be run.