From Open Source Ecology
Jump to: navigation, search

Backups Retention Policy

  1. Daily: 3 days
  2. Weekly (Mondays): 31 days
  3. Monthly (first of month): 365 days
  4. Yearly (Jan 1): Indefinite


  • Aug 2019 monthly cost (Storage only, no downloads) is at $2.57 for a total of 160 GB with ~ 20 GB added monthly
  • 2018-12 estimate: $100-$200 per year.

Useful commands

Here we will list useful commands for our backups on B2

List Backup Contents

The following command will output a list of files stored on our 'ose-server-backups' bucket in our b2 account.

[root@opensourceecology ~]# sudo su - b2user
Last login: Sat Aug  3 05:57:40 UTC 2019 on pts/0
[b2user@opensourceecology ~]$ ~/virtualenv/bin/b2 ls ose-server-backups
[b2user@opensourceecology ~]$ 

Install CLI

Backblaze has an API for B2, and they maintain a python CLI tool for interacting with your storage buckets. Unfortunately, this is not in the centos yum repositories, and the suggested way to install it via easy_install may break other python packages, such as certbot. The result of breaking certbot is that our website becomes inaccessible because our https certificates fail to autorenew. Therefore, it's critical that b2 is installed with care using a virtualenv.

sudo su -

# we must install the b2 command in a virtualenv from the binary. if we do it via easy_install, it will
# break other critical python packages, such as certbot (yielding our websites down when the https certs
# fail to auto-renew properly) 
# for more info:
yum install python-virtualenv

# and install other depends
yum install python-setuptools git

adduser b2user
sudo su - b2user
mkdir virtualenv
pushd virtualenv/
virtualenv .
source ~/virtualenv/bin/activate
mkdir sandbox
pushd sandbox
git clone
pushd B2_Command_Line_Tool/
python install

# test it
 ~/virtualenv/bin/b2 version

For more information about the b2 command, please see the backblaze docs for the b2 command-line tool

Update CLI

As stated above, it's critical that this is done in a python virtualenv to prevent these steps from breaking the OS and therefore the websites, etc.

sudo su - b2user
source ~/virtualenv/bin/activate
cd ~/sandbox/B2_Command_Line_Tool/
git pull
python install
~/virtualenv/bin/b2 version

Restore from backups

This section will describe how to extract data from an encrypted backup file stored in Backblaze B2.

Download from WUI

One of the significant benefits of Backblaze over similar cloud storage solutions is how easy it is to download archives. Indeed, you can simply do it from their Web User Interface.

To download one of or encrypted backup files from Backblaze B2, click on the "Upload/Download" button for the "ose-server-backups" bucket on the screen immediately after logging into the wui (see the OSE shared Keepass for the credentials)

BackblazeB2BackupsDownloadFromWui 01 clickUploadDownload.png

Next, determine which file you need and click on it.

Note that, while storage is relatively cheap for B2, download bandwidth can be pricy. Before downloading a ton of files from our Backblaze B2 account, do your due diligence to determine _which_ archive you need from _which_ day to prevent excessive fees.

BackblazeB2BackupsDownloadFromWui 02 clickFile.png

In the file's Details pop-up, click the "Download" button.

BackblazeB2BackupsDownloadFromWui 03 clickDownload.png

HintLightbulb.png Hint: Because [a] it can take a _very_ long time to download these full backups and [b] an intermittent internet connection on your computer could cause the download to fail and [c] download bandwidth from B2 is not cheap, it may be wise to download from the cli instead

Download from CLI (Command Line Interface)

You can extract data from a backup on Backblaze B2 using their command line utility `b2`.

Because b2 wasn't installed from a trusted OS repository, we execute it as an unprivliged user `b2user`. The following commands shows how to become the `b2user`, list the backups available on the Backblaze B2 `ose-server-backups` bucket, and download the encrypted backup file.

Note: Don't download any archives within the /home/ directory tree because the entire '/home/' directory is backed-up. Instead, download to someplace in '/var/tmp/' as shown below.

[maltfield@hetzner2 backblaze]$ sudo su - b2user
[sudo] password for maltfield: 
Last login: Sat Nov 24 13:10:42 UTC 2018 on pts/124
[b2user@hetzner2 ~]$ ~/virtualenv/bin/b2 ls ose-server-backups
[b2user@hetzner2 tmp]$ tmpDir="/var/tmp/backblazeRestore_`date +%Y%m%d_%H%M%S`"
[b2user@hetzner2 tmp]$ cd $tmpDir
[b2user@hetzner2 backblazeRestore_20181223_135712]$ ~/virtualenv/bin/b2 download-file-by-name ose-server-backups daily_hetzner2_20181223_111501.tar.gpg daily_hetzner2_20181223_111501.tar.gpg
daily_hetzner2_20181223_111501.tar.gpg: 100%|| 17.5G/17.5G [19:29<00:00, 15.0MB/s]
File name:    daily_hetzner2_20181223_111501.tar.gpg
File id:      4_z5605817c251dadb96e4d0118_f206daef4188682f6_d20181223_m113926_c001_v0001106_t0057
File size:    17509827199
Content type: application/octet-stream
Content sha1: none
INFO src_last_modified_millis: 1545565162536
[b2user@hetzner2 backblazeRestore_20181223_135712]$ du -sh *
17G     daily_hetzner2_20181223_111501.tar.gpg
[b2user@hetzner2 backblazeRestore_20181223_135712]$ 

For more information about the b2 command, please see the backblaze docs for the b2 command-line tool


OSE's backup data holds very sensitive content (ie; passwords, logs, etc), so they're encrypted before being uploaded to 3rd parties.

Use gpg and the 4K 'ose-backups-cron.key' keyfile (which lives in '/root/backups/' on the server and can also be found in the shared ose keepass) to decrypt this data as follows:

Note: Depending on the version of `gpg` installed, you may need to omit the '--batch' option.

[maltfield@hetzner2 backblaze]$ sudo su -
[sudo] password for maltfield: 
Last login: Sun Dec 23 12:09:45 UTC 2018 on pts/113
[root@hetzner2 ~]# cd /var/tmp/back
backblaze/                           backblazeRestore_20181223_135712/    backups_for_migration_from_hetzner1/
[root@hetzner2 ~]# cd /var/tmp/backblazeRestore_20181223_135712/
[root@hetzner2 backblazeRestore_20181223_135712]# du -sh *
17G     daily_hetzner2_20181223_111501.tar.gpg
[root@hetzner2 backblazeRestore_20181223_135712]# gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --output daily_hetzner2_20181223_111501.tar daily_hetzner2_20181223_111501.tar.gpg
gpg: AES256 encrypted data
gpg: encrypted with 1 passphrase
[root@hetzner2 backblazeRestore_20181223_135712]# du -sh *
17G     daily_hetzner2_20181223_111501.tar
17G     daily_hetzner2_20181223_111501.tar.gpg
[root@hetzner2 backblazeRestore_20181223_135712]# 

There should now be a decrypted file. You can extract it to view the contents using `tar`.

See Also