Backblaze

Backups Retention Policy

1. Daily: 3 days
2. Weekly (Mondays): 31 days
3. Monthly (first of month): 365 days
4. Yearly (Jan 1): Indefinite

Cost

• Aug 2019 monthly cost (Storage only, no downloads) is at $2.57 for a total of 160 GB with ~ 20 GB added monthly
• 2018-12 estimate: $100-$200 per year.

Useful commands

Here we will list useful commands for our backups on B2

List Backup Contents

The following command will output a list of files stored on our 'ose-server-backups' bucket in our b2 account.

[root@opensourceecology backups]# rclone --b2-versions ls b2:ose-server-backups
21812118477 daily_hetzner2_20240921_072001.tar.gpg
21349753244 monthly_hetzner2_20231001_072001.tar.gpg
21360808568 monthly_hetzner2_20231101_072001.tar.gpg
21360301269 monthly_hetzner2_20231201_072001.tar.gpg
21820017340 monthly_hetzner2_20240201_072001.tar.gpg
21683700909 monthly_hetzner2_20240301_072001.tar.gpg
21660296728 monthly_hetzner2_20240401_072001.tar.gpg
21790035424 monthly_hetzner2_20240501_072001.tar.gpg
21603737883 monthly_hetzner2_20240601_072001.tar.gpg
21663769333 monthly_hetzner2_20240701_072001.tar.gpg
21991147307 monthly_hetzner2_20240801_072001.tar.gpg
21896377523 monthly_hetzner2_20240901_072001.tar.gpg
21942660432 weekly_hetzner2_20240826_072001.tar.gpg
21902006508 weekly_hetzner2_20240902_072001.tar.gpg
21873908566 weekly_hetzner2_20240909_072001.tar.gpg
21830987241 weekly_hetzner2_20240916_072001.tar.gpg
17516124812 yearly_hetzner2_20190101_111520.tar.gpg
18872422001 yearly_hetzner2_20200101_072001.tar.gpg
19827971632 yearly_hetzner2_20210101_072001.tar.gpg
21079942509 yearly_hetzner2_20230101_072001.tar.gpg
21541199047 yearly_hetzner2_20240101_072001.tar.gpg
[root@opensourceecology backups]# 

Note: The API keys will need the listFiles permission in order to execute the above command successfully. This permission is given to the root user, but not to the b2user user.

Install

OSE uses the backblaze-b2 CLI tool to manage API keys and rclone to upload files to backblaze from the server.

Note that, to protect our server and its data from randsomware, it's critical that our server cannot delete its own backups. For that reason, we should only ever put "append-only" keys on the server (keys which can upload but cannot delete data from rclone). For this reason, we don't install backblaze-b2 on the server, and we should never upload any keys that can create other keys on the server -- that should be done locally on your dev laptop, using a temporary key created in the BackBlaze B2 WUI.

You can install rclone on the server with

sudo apt-get install rclone

Create Key

The OSE Server should have two keys:

1. append-only
2. list-and-append-only

The less privileged b2user user on the server uses the append-only key, which only has access to the ose-server-backups bucket and only has the following capabilities: readFiles, writeFiles

The root user on the server uses the list-and-append-only key, which only has access to the ose-server-backups bucket and only has the following capabilities: listFiles, readFiles, writeFiles.

The root user includes listFiles for the purpose of executing /root/backups/backupReport.sh, which lists the backups that have been successfully uploaded to Backblaze, and it generates an email to let us know if any backups are missing (it also sends an email on the 2nd of every month if everything is OK, so we know that the backupReport itself is operating correctly).

Warning: To provide protection from ransomeware, it's critically important that the server cannot delete files from backblaze. This is why we only generate append-only keys with very limited permissions for use on the server.

Never put a "master key" on the server! If you need a key with more permissions (eg a key to create keys), then you should create a new master key and only use it locally on your laptop. And delete the master key when you're done using it.

At the time of writing, it's not possible to create a new append-only key from the Backblaze WUI. You have to use the B2 CLI.

In order to use the B2 CLI, you'll first need to create a (temporary) master key in the Backblaze WUI.

1. Login to the Backblaze WUI (see shared OSE keepass for credentials under "Backblaze B2" entry)
2. Click on "Application Keys" in the left-hand navigation
3. Click on the "Add a New Application Key" button
4. Name the key master-<YYYY>-<MM (eg master-2024-09) with full permissions

On your Debian laptop, install the backblaze-b2 package

Warning: Never put a "master key" on the server! Use your personal laptop.

sudo apt-get install backblaze-b2

Configure your local backblaze-b2 app with the new master key credentials

user@disp3202:~$ backblaze-b2 authorize-account
Using https://api.backblazeb2.com
Backblaze account ID: REDACTED
Backblaze application key: 
user@disp3202:~$ 

Create an append-only key

user@disp3202:~$ backblaze-b2 create-key --bucket 'ose-server-backups' 'hetzner3-append-only-2024-09' 'readFiles, writeFiles'
OBFUSCATED OBFUSCATED
user@disp3202:~$ 

Create a list-and-append-only key

user@disp3202:~$ backblaze-b2 create-key --bucket 'ose-server-backups' 'hetzner3-list-and-append-only-2024-09' 'listFiles, readFiles, writeFiles'
OBFUSCATED OBFUSCATED
user@disp3202:~$ 

Finally, delete the master key created above from the Backblaze B2 WUI

Configure rclone

Execute rclone config to enter the API key credentials that were outputted when creating them above

root@mail ~/backups # rclone config
2024/09/22 20:31:36 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> b2

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 1 / 1Fichier
   \ (fichier)
 2 / Akamai NetStorage
   \ (netstorage)
 3 / Alias for an existing remote
   \ (alias)
 4 / Amazon Drive
   \ (amazon cloud drive)
 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
   \ (s3)
 6 / Backblaze B2
   \ (b2)
 7 / Better checksums for other remotes
   \ (hasher)
 8 / Box
   \ (box)
 9 / Cache a remote
   \ (cache)
10 / Citrix Sharefile
   \ (sharefile)
11 / Combine several remotes into one
   \ (combine)
12 / Compress a remote
   \ (compress)
13 / Dropbox
   \ (dropbox)
14 / Encrypt/Decrypt a remote
   \ (crypt)
15 / Enterprise File Fabric
   \ (filefabric)
16 / FTP
   \ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
   \ (google cloud storage)
18 / Google Drive
   \ (drive)
19 / Google Photos
   \ (google photos)
20 / HTTP
   \ (http)
21 / Hadoop distributed file system
   \ (hdfs)
22 / HiDrive
   \ (hidrive)
23 / In memory object storage system.
   \ (memory)
24 / Internet Archive
   \ (internetarchive)
25 / Jottacloud
   \ (jottacloud)
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
   \ (koofr)
27 / Local Disk
   \ (local)
28 / Mail.ru Cloud
   \ (mailru)
29 / Microsoft Azure Blob Storage
   \ (azureblob)
30 / Microsoft OneDrive
   \ (onedrive)
31 / OpenDrive
   \ (opendrive)
32 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ (swift)
33 / Pcloud
   \ (pcloud)
34 / Put.io
   \ (putio)
35 / SMB / CIFS
   \ (smb)
36 / SSH/SFTP
   \ (sftp)
37 / Sia Decentralized Cloud
   \ (sia)
38 / Sugarsync
   \ (sugarsync)
39 / Transparently chunk/split large files
   \ (chunker)
40 / Union merges the contents of several upstream fs
   \ (union)
41 / Uptobox
   \ (uptobox)
42 / WebDAV
   \ (webdav)
43 / Yandex Disk
   \ (yandex)
44 / Zoho
   \ (zoho)
45 / premiumize.me
   \ (premiumizeme)
46 / seafile
   \ (seafile)
Storage> b2

Option account.
Account ID or Application Key ID.
Enter a value.
account> OBFUSCATED

Option key.
Application Key.
Enter a value.
key> OBFUSCATED

Option hard_delete.
Permanently delete files on remote removal, otherwise hide files.
Enter a boolean value (true or false). Press Enter for the default (false).
hard_delete> true

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:
- type: b2
- account: OBFUSCATED
- key: OBFUSCATED
- hard_delete: true
Keep this "b2" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Current remotes:

Name                 Type
====                 ====
b2                   b2

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
root@mail ~/backups #

You can test that this works by attempting to list the files in the bucket:

rclone --b2-versions ls b2:ose-server-backups

And do the same process for the b2user user.

sudo su - b2user
rclone config

Restore from backups

This section will describe how to extract data from an encrypted backup file stored in Backblaze B2.

Download from WUI

One of the significant benefits of Backblaze over similar cloud storage solutions is how easy it is to download archives. Indeed, you can simply do it from their Web User Interface.

To download one of or encrypted backup files from Backblaze B2, click on the "Upload/Download" button for the "ose-server-backups" bucket on the screen immediately after logging into the wui (see the OSE shared Keepass for the credentials)

Next, determine which file you need and click on it.

Note that, while storage is relatively cheap for B2, download bandwidth can be pricy. Before downloading a ton of files from our Backblaze B2 account, do your due diligence to determine _which_ archive you need from _which_ day to prevent excessive fees.

In the file's Details pop-up, click the "Download" button.

Hint: Because [a] it can take a _very_ long time to download these full backups and [b] an intermittent internet connection on your computer could cause the download to fail and [c] download bandwidth from B2 is not cheap, it may be wise to download from the cli instead

Download from CLI (Command Line Interface)

You can extract data from a backup on Backblaze B2 using their command line utility `b2`.

Because b2 wasn't installed from a trusted OS repository, we execute it as an unprivliged user `b2user`. The following commands shows how to become the `b2user`, list the backups available on the Backblaze B2 `ose-server-backups` bucket, and download the encrypted backup file.

Note: Don't download any archives within the /home/ directory tree because the entire '/home/' directory is backed-up. Instead, download to someplace in '/var/tmp/' as shown below.

[maltfield@hetzner2 backblaze]$ sudo su - b2user
[sudo] password for maltfield: 
Last login: Sat Nov 24 13:10:42 UTC 2018 on pts/124
[b2user@hetzner2 ~]$ ~/virtualenv/bin/b2 ls ose-server-backups
daily_hetzner2_20181221_111534.tar.gpg
daily_hetzner2_20181222_111516.tar.gpg
daily_hetzner2_20181223_111501.tar.gpg
monthly_hetzner2_20181001_091809.tar.gpg
monthly_hetzner2_20181101_091810.tar.gpg
monthly_hetzner2_20181201_091759.tar.gpg
weekly_hetzner2_20181126_091805.tar.gpg
weekly_hetzner2_20181203_091823.tar.gpg
weekly_hetzner2_20181210_091822.tar.gpg
weekly_hetzner2_20181217_111545.tar.gpg
[b2user@hetzner2 tmp]$ tmpDir="/var/tmp/backblazeRestore_`date +%Y%m%d_%H%M%S`"
[b2user@hetzner2 tmp]$ cd $tmpDir
[b2user@hetzner2 backblazeRestore_20181223_135712]$ ~/virtualenv/bin/b2 download-file-by-name ose-server-backups daily_hetzner2_20181223_111501.tar.gpg daily_hetzner2_20181223_111501.tar.gpg
daily_hetzner2_20181223_111501.tar.gpg: 100%|| 17.5G/17.5G [19:29<00:00, 15.0MB/s]
File name:    daily_hetzner2_20181223_111501.tar.gpg
File id:      4_z5605817c251dadb96e4d0118_f206daef4188682f6_d20181223_m113926_c001_v0001106_t0057
File size:    17509827199
Content type: application/octet-stream
Content sha1: none
INFO src_last_modified_millis: 1545565162536
[b2user@hetzner2 backblazeRestore_20181223_135712]$ du -sh *
17G     daily_hetzner2_20181223_111501.tar.gpg
[b2user@hetzner2 backblazeRestore_20181223_135712]$ 

For more information about the b2 command, please see the backblaze docs for the b2 command-line tool

• https://www.backblaze.com/b2/docs/quick_command_line.html

Decrypt

OSE's backup data holds very sensitive content (ie; passwords, logs, etc), so they're encrypted before being uploaded to 3rd parties.

Use gpg and the 4K 'ose-backups-cron.key' keyfile (which lives in '/root/backups/' on the server and can also be found in the shared ose keepass) to decrypt this data as follows:

Note: Depending on the version of `gpg` installed, you may need to omit the '--batch' option.

[maltfield@hetzner2 backblaze]$ sudo su -
[sudo] password for maltfield: 
Last login: Sun Dec 23 12:09:45 UTC 2018 on pts/113
[root@hetzner2 ~]# cd /var/tmp/back
backblaze/                           backblazeRestore_20181223_135712/    backups_for_migration_from_hetzner1/
[root@hetzner2 ~]# cd /var/tmp/backblazeRestore_20181223_135712/
[root@hetzner2 backblazeRestore_20181223_135712]# du -sh *
17G     daily_hetzner2_20181223_111501.tar.gpg
[root@hetzner2 backblazeRestore_20181223_135712]# gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --output daily_hetzner2_20181223_111501.tar daily_hetzner2_20181223_111501.tar.gpg
gpg: AES256 encrypted data
gpg: encrypted with 1 passphrase
[root@hetzner2 backblazeRestore_20181223_135712]# du -sh *
17G     daily_hetzner2_20181223_111501.tar
17G     daily_hetzner2_20181223_111501.tar.gpg
[root@hetzner2 backblazeRestore_20181223_135712]# 

There should now be a decrypted file. You can extract it to view the contents using `tar`.

See Also

• OSE Server#Backups
• Amazon Glacier
• Maltfield_Log/2018_Q3#Sat_Jul_28.2C_2018
• Maltfield_Log/2022#Fri_October_28.2C_2022
• wikipedia:Comparison_of_file_hosting_services