Maltfield Log/2022
My work log from the year 2022. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
See Also
Wed November 02, 2022
- The monthly backup report came-out today
ATTENTION: BACKUPS MISSING! WARNING: First of last month's backup (20221001) is missing! See below for the contents of the backblaze b2 bucket = ose-server-backups 21039187167 daily_hetzner2_20221102_072001.tar.gpg 21012267893 monthly_hetzner2_20221101_072001.tar.gpg 21016856935 weekly_hetzner2_20221031_072001.tar.gpg 17516124812 yearly_hetzner2_20190101_111520.tar.gpg 18872422001 yearly_hetzner2_20200101_072001.tar.gpg 19827971632 yearly_hetzner2_20210101_072001.tar.gpg --- Note: This report was generated on 20221103_042001 UTC by script '/root/backups/backupReport.sh' This script was triggered by '/etc/cron.d/backup_to_backblaze' For more information about OSE backups, please see the relevant documentation pages on the wiki: * https://wiki.opensourceecology.org/wiki/Backblaze * https://wiki.opensourceecology.org/wiki/OSE_Server#Backups
- It doesn't complain about yesterday's backups nor the first-of-this-month. The one from the first of last month is missing of-course, but that warning should go away in next month's backup report
- I sent an email to Marcin about the fix
Hey Marcin, I fixed OSE's server backups. The most-recent backup that you had was from 2021-11-01. So if your server died, you would have lost about a year's worth of data. The fix was switching the tool to do the uploads from a very old version of Backblaze's CLI tool to another backup tool (that supports Backblaze and many other cloud storage providers) called rclone. The root cause of the issue was an outdated library that couldn't be updated due to its dependency on a newer version of python. Your prod server is running CentOS 7.7 (that's a free/open-source clone of Red Hat Enterprise Linux 7.7). RHEL's core tools (eg yum) heavily depend on a very old version of python v2.7.5. Python 3.7 is not available in the yum repos, and if you're not careful with upgrading python on RHEL machines, you can completely break the OS. In my prior work at OSE, I've done my best to design your systems to be robust and "well oiled" so that they would run for as long as possible with as little maintenance as possible. However, code rots over time, and there's only so long you can hold-off before things fall apart. Python 2.7.5 was End-of-Life'd on 2020-01-01, and it no longer receives any updates. * https://en.wikipedia.org/wiki/History_of_Python CentOS 7.7 was released 2019-09-17. "Full Updates" stopped 2020-08-06, and it will no longer receive any maintenance updates after 2024-06-30. * https://wiki.centos.org/About/Product At some point, you're going to want to migrate to a new server with a new OS. I strongly recommend initiating this project before 2024. As you can see from the latest monthly backup report, yesterday's backup is fine. The day before that is fine. Of course, you're missing the backup from the first of last month, but the warnings should clear for next month's report.
Sat October 29, 2022
- the backup I kicked-off yesterday finally finished, but it took magnitudes longer than the previous one
[b2user@opensourceecology ~]$ time rclone -v --progress copy sync/daily_hetzner2_20221028_072001.tar.gpg b2:ose-server-backups 2022-10-28 23:48:14 INFO : daily_hetzner2_20221028_072001.tar.gpg: Copied (new) Transferred: 19.571G / 19.571 GBytes, 100%, 27.552 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 12m8.5s 2022/10/28 23:48:14 INFO : Transferred: 19.571G / 19.571 GBytes, 100%, 27.552 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 12m8.5s real 12m8.591s user 2m49.455s sys 0m39.219s [b2user@opensourceecology ~]$ [b2user@opensourceecology ~]$ time rclone -v --progress copy sync.old/daily_hetzner2_20221027_072001.tar.gpg b2:ose-server-backups Transferred: 10.863G / 19.573 GBytes, 56%, 4.528 MBytes/s, ETA 32m49s 2022-10-29 02:05:41 INFO : daily_hetzner2_20221027_072001.tar.gpg: Copied (new) Transferred: 19.948G / 19.948 GBytes, 100%, 2.529 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 2h14m38.9s 2022/10/29 02:05:41 INFO : Transferred: 19.948G / 19.948 GBytes, 100%, 2.529 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 2h14m38.9s real 134m38.987s user 2m54.695s sys 0m46.883s [b2user@opensourceecology ~]$
- I re-ran the backup report. It no longer complains about yesterday's backup missing
[root@opensourceecology ~]# /root/backups/backupReport.sh INFO: email body below ATTENTION: BACKUPS MISSING! WARNING: First of this month's backup (20221001) is missing! WARNING: First of last month's backup (20220901) is missing! See below for the contents of the backblaze b2 bucket = ose-server-backups 21016856790 daily_hetzner2_20221027_072001.tar.gpg 21014547058 daily_hetzner2_20221028_072001.tar.gpg 21016238617 daily_hetzner2_20221029_072001.tar.gpg 20619278434 monthly_hetzner2_20211101_072001.tar.gpg 50 test.20221028 50 test.20221028b 50 test.20221028c 50 test5/test.20221028 17516124812 yearly_hetzner2_20190101_111520.tar.gpg 18872422001 yearly_hetzner2_20200101_072001.tar.gpg 19827971632 yearly_hetzner2_20210101_072001.tar.gpg --- Note: This report was generated on 20221029_203609 UTC by script '/root/backups/backupReport.sh' This script was triggered by '/etc/cron.d/backup_to_backblaze' For more information about OSE backups, please see the relevant documentation pages on the wiki: * https://wiki.opensourceecology.org/wiki/Backblaze * https://wiki.opensourceecology.org/wiki/OSE_Server#Backups [root@opensourceecology ~]#
- I deleted the test files from the wui, and I re-ran the backup report
[root@opensourceecology ~]# /root/backups/backupReport.sh
INFO: email body below
ATTENTION: BACKUPS MISSING!
WARNING: First of this month's backup (20221001) is missing!
WARNING: First of last month's backup (20220901) is missing!
See below for the contents of the backblaze b2 bucket = ose-server-backups
21016856790 daily_hetzner2_20221027_072001.tar.gpg
21014547058 daily_hetzner2_20221028_072001.tar.gpg
21016238617 daily_hetzner2_20221029_072001.tar.gpg
20619278434 monthly_hetzner2_20211101_072001.tar.gpg
17516124812 yearly_hetzner2_20190101_111520.tar.gpg
18872422001 yearly_hetzner2_20200101_072001.tar.gpg
19827971632 yearly_hetzner2_20210101_072001.tar.gpg
---
Note: This report was generated on 20221029_204210 UTC by script '/root/backups/backupReport.sh'
This script was triggered by '/etc/cron.d/backup_to_backblaze'
For more information about OSE backups, please see the relevant documentation pages on the wiki:
* https://wiki.opensourceecology.org/wiki/Backblaze
* https://wiki.opensourceecology.org/wiki/OSE_Server#Backups
[root@opensourceecology ~]#
- the aboe ouput also shows that the backup from the 29th that was kicked-off by the cron job was uploaded successfully, and the WUI confirms it's 21 GB, so that looks good
21016238617 daily_hetzner2_20221029_072001.tar.gpg
- the backup logs (/var/log/backups/backup.log) also confirmed this
+ echo 'INFO: Beginning upload to backblaze b2' INFO: Beginning upload to backblaze b2 + /bin/sudo -u b2user /bin/rclone -v --bwlimit 3M --progress copy /home/b2user/sync/daily_hetzne r2_20221029_072001.tar.gpg b2:ose-server-backups 2022/10/29 07:48:14 INFO : Starting bandwidth limiter at 3MBytes/s Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Elapsed time: 1.1sTransferred: 0 / 19.573 GBytes, 0%, 0 Bytes/s, ETA - Transferred: 0 / 1, 0% Elapsed time: 1.6s Transferring: * daily_hetzner2_20221029_072001.tar.gpg: 0% /19.573G, 0/s, - ... Transferring: * daily_hetzner2_20221029_072001.tar.gpg:100% /19.573G, 2.721M/s, -Transferred: 19.573G / 19.573 GBytes, 100%, 2.690 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 2h4m10.7s 2022/10/29 09:52:25 INFO : Transferred: 19.573G / 19.573 GBytes, 100%, 2.690 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 2h4m10.7s real 124m10.764s user 5m40.221s sys 2m21.000s + echo ================================================================================ ================================================================================ ++ date -u +%Y%m%d_%H%M%S + echo 'INFO: Finished Backup Run at 20221029_095225' INFO: Finished Backup Run at 20221029_095225 + echo ================================================================================ ================================================================================ + exit 0
Fri October 28, 2022
- today I went to investigate why backups had been failing
- the first email we received indicating that there was an issue with backups was on 2021-08-02
ATTENTION: BACKUPS MISSING! WARNING: First of this month's backup (20210801) is missing! WARNING: The day before yesterday's backup (20210801) is missing! See below for the contents of the backblaze b2 bucket = ose-server-backups daily_hetzner2_20210731_072001.tar.gpg monthly_hetzner2_20200901_072001.tar.gpg monthly_hetzner2_20201001_072001.tar.gpg monthly_hetzner2_20201101_072001.tar.gpg monthly_hetzner2_20201201_072001.tar.gpg monthly_hetzner2_20210201_072001.tar.gpg monthly_hetzner2_20210301_072001.tar.gpg monthly_hetzner2_20210401_072001.tar.gpg monthly_hetzner2_20210501_072001.tar.gpg monthly_hetzner2_20210601_072001.tar.gpg monthly_hetzner2_20210701_072001.tar.gpg weekly_hetzner2_20210705_072001.tar.gpg weekly_hetzner2_20210712_072001.tar.gpg weekly_hetzner2_20210719_072001.tar.gpg weekly_hetzner2_20210802_072002.tar.gpg yearly_hetzner2_20190101_111520.tar.gpg yearly_hetzner2_20200101_072001.tar.gpg yearly_hetzner2_20210101_072001.tar.gpg --- Note: This report was generated on 20210803_042001 UTC by script '/root/backups/backupReport.sh' This script was triggered by '/etc/cron.d/backup_to_backblaze' For more information about OSE backups, please see the relevant documentation pages on the wiki: * https://wiki.opensourceecology.org/wiki/Backblaze * https://wiki.opensourceecology.org/wiki/OSE_Server#Backups
- the most recent backup report was 2022-10-02. Here we can't even connect to the backblaze backend at all
ATTENTION: BACKUPS MISSING! ERROR: Connection error: HTTPSConnectionPool(host='api001.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_list_buckets (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) WARNING: First of this month's backup (20221001) is missing! WARNING: First of last month's backup (20220901) is missing! WARNING: Yesterday's backup (20221002) is missing! WARNING: The day before yesterday's backup (20221001) is missing! See below for the contents of the backblaze b2 bucket = ose-server-backups ERROR: Connection error: HTTPSConnectionPool(host='api001.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_list_buckets (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) --- Note: This report was generated on 20221003_042001 UTC by script '/root/backups/backupReport.sh' This script was triggered by '/etc/cron.d/backup_to_backblaze' For more information about OSE backups, please see the relevant documentation pages on the wiki: * https://wiki.opensourceecology.org/wiki/Backblaze * https://wiki.opensourceecology.org/wiki/OSE_Server#Backups
- I was able to log into the backblaze wui using the shared ose keepass. I didn't get any complaints about billing. The last bill of $0.90 was paid on 2022-10-18
- In 'Browse Files' I see we have two buckets
- ose-dev-server-backups (currently 89 MB), and
- ose-server-backups (currently 76.8 GB)
- Unfortunaetly, in this bucket we only have 5x files (and one looks corrupt; it's 0 bytes).
- monthly_hetzner2_20211101_072001.tar.gpg
- yearly_hetzner2_20190101_111520.tar.gpg
- yearly_hetzner2_20200101_072001.tar.gpg
- yearly_hetzner2_20210101_072001.tar.gpg
- Unfortunaetly, in this bucket we only have 5x files (and one looks corrupt; it's 0 bytes).
- so it looks like we're missing the yearly from this year, and our mots-recent backup is basically 1 year old D:
- on the server, I still had a screen open. I was tailing the logs, and I saw this (the file was probably already rotated since long ago)
[root@opensourceecology ~]# tail -f /var/log/backups/backup.log "size": 19356681423, "uploadTimestamp": 1589010425000 } real 139m39.520s user 5m23.227s sys 1m5.762s ================================================================================ INFO: Finished Backup Run at 20200509_100641 ================================================================================
- Here's the end of the log now
... /bin/tar: Removing leading `/' from member names /root/backups/sync/daily_hetzner2_20221028_072001/etc/ /root/backups/sync/daily_hetzner2_20221028_072001/etc/etc.20221028_072001.tar.gz /root/backups/sync/daily_hetzner2_20221028_072001/home/ /root/backups/sync/daily_hetzner2_20221028_072001/home/home.20221028_072001.tar.gz /root/backups/sync/daily_hetzner2_20221028_072001/log/ /root/backups/sync/daily_hetzner2_20221028_072001/log/log.20221028_072001.tar.gz /root/backups/sync/daily_hetzner2_20221028_072001/mysqldump/ /root/backups/sync/daily_hetzner2_20221028_072001/mysqldump/mysqldump.20221028_072001.sql.gz /root/backups/sync/daily_hetzner2_20221028_072001/root/ /root/backups/sync/daily_hetzner2_20221028_072001/root/root.20221028_072001.tar.gz /root/backups/sync/daily_hetzner2_20221028_072001/www/ /root/backups/sync/daily_hetzner2_20221028_072001/www/www.20221028_072001.tar.gz real 2m29.614s user 0m0.761s sys 0m22.953s INFO: Encrypting the single-file tarball real 10m21.757s user 9m29.237s sys 0m14.868s INFO: Deleting unencrypted backup archive INFO: moving encrypted backup file to b2user's sync dir INFO: Beginning upload to backblaze b2 ERROR: Connection error: HTTPSConnectionPool(host='api001.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_list_buckets (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) real 0m10.246s user 0m0.290s sys 0m0.044s ================================================================================ INFO: Finished Backup Run at 20221028_075029 ================================================================================
- this smells like an outdated root certificate store. I tried to hit that URL with curl
[root@opensourceecology ~]# curl -i https://api001.backblaze2.com curl: (60) Peer's Certificate has expired. More details here: http://curl.haxx.se/docs/sslcerts.html curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). If the default bundle file isn't adequate, you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option. [root@opensourceecology ~]#
- I loaded the same URL in my browser so I could inspect the certificate. But, umm, firefox said "We can’t connect to the server at api001.backblaze2.com"
- my local machine can't resolve dns for that IP
user@disp6070:~$ dig @8.8.8.8 api001.backblaze2.com ; <<>> DiG 9.16.33-Debian <<>> @8.8.8.8 api001.backblaze2.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 27417 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: a42d4745f035b6bc901a169d635c1767ac42e6ce5d46a5f6 (good) ;; QUESTION SECTION: ;api001.backblaze2.com. IN A ;; AUTHORITY SECTION: backblaze2.com. 514 IN SOA ns23.domaincontrol.com. dns.jomax.net. 2022032400 28800 7200 604800 600 ;; Query time: 196 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 28 12:54:45 CDT 2022 ;; MSG SIZE rcvd: 146 user@disp6070:~$
- actually, the server can't even resolve an IP for that dns!
[root@opensourceecology ~]# dig api001.backblaze2.com ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.2 <<>> api001.backblaze2.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 26656 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;api001.backblaze2.com. IN A ;; AUTHORITY SECTION: backblaze2.com. 600 IN SOA ns23.domaincontrol.com. dns.jomax.net. 2022032400 28800 7200 604800 600 ;; Query time: 34 msec ;; SERVER: 213.133.98.98#53(213.133.98.98) ;; WHEN: Fri Oct 28 17:56:32 UTC 2022 ;; MSG SIZE rcvd: 118 [root@opensourceecology ~]#
- somehow curl is able to resolve this IP, but I really don't understand how
[root@opensourceecology ~]# curl -viL api001.backblaze2.com * About to connect() to api001.backblaze2.com port 80 (#0) * Trying 34.206.39.153... * Connected to api001.backblaze2.com (34.206.39.153) port 80 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: api001.backblaze2.com > Accept: */* >
- I tried to execute the b2 command directly (which should be done only as the less-privliged b2 user)
[maltfield@opensourceecology ~]$ sudo su - b2user [sudo] password for maltfield: Last login: Thu May 7 08:23:26 UTC 2020 on pts/0 [b2user@opensourceecology ~]$ ls hetzner2_20180727_072001.tar.gpg sandbox sync sync.old tmp virtualenv [b2user@opensourceecology ~]$ [b2user@opensourceecology ~]$ /home/b2user/virtualenv/bin/b2 This program provides command-line access to the B2 service. ... [b2user@opensourceecology ~]$
- executing 'list-buckets' clearly fails
[b2user@opensourceecology ~]$ /home/b2user/virtualenv/bin/b2 list-buckets ERROR: Connection error: HTTPSConnectionPool(host='api001.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_list_buckets (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) [b2user@opensourceecology ~]$
- I followed the guide on the wiki to update the CLI, but it failed https://wiki.opensourceecology.org/wiki/Backblaze#Update_CLI
[b2user@opensourceecology ~]$ du -sh virtualenv
16M virtualenv
[b2user@opensourceecology ~]$ time rsync -a virtualenv virtualenv.20221028
real 0m0.237s
user 0m0.092s
sys 0m0.111s
[b2user@opensourceecology ~]$ du -sh virtualenv*
16M virtualenv
16M virtualenv.20221028
[b2user@opensourceecology ~]$
[b2user@opensourceecology ~]$ source ~/virtualenv/bin/activate
(virtualenv) [b2user@opensourceecology ~]$ cd ~/sandbox/B2_Command_Line_Tool/
(virtualenv) [b2user@opensourceecology B2_Command_Line_Tool]$ git pull
remote: Enumerating objects: 2759, done.
remote: Counting objects: 100% (2020/2020), done.
remote: Compressing objects: 100% (637/637), done.
remote: Total 2759 (delta 1487), reused 1794 (delta 1380), pack-reused 739
Receiving objects: 100% (2759/2759), 643.29 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1909/1909), completed with 32 local objects.
From https://github.com/Backblaze/B2_Command_Line_Tool
4b8c653..25c13d3 master -> origin/master
...
delete mode 100644 test_b2_command_line.py
(virtualenv) [b2user@opensourceecology B2_Command_Line_Tool]$
(virtualenv) [b2user@opensourceecology B2_Command_Line_Tool]$ python setup.py install
Couldn't find index page for 'setuptools_scm' (maybe misspelled?)
No local packages or working download links found for setuptools_scm<6.0
Traceback (most recent call last):
File "setup.py", line 133, in <module>
'console_scripts': ['b2=b2.console_tool:main'],
File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/home/b2user/virtualenv/lib/python2.7/site-packages/setuptools/dist.py", line 315, in init
self.fetch_build_eggs(attrs['setup_requires'])
File "/home/b2user/virtualenv/lib/python2.7/site-packages/setuptools/dist.py", line 361, in fetch_build_eggs
replace_conflicting=True,
File "/home/b2user/virtualenv/lib/python2.7/site-packages/pkg_resources/init.py", line 850, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
File "/home/b2user/virtualenv/lib/python2.7/site-packages/pkg_resources/init.py", line 1122, in best_match
return self.obtain(req, installer)
File "/home/b2user/virtualenv/lib/python2.7/site-packages/pkg_resources/init.py", line 1134, in obtain
return installer(requirement)
File "/home/b2user/virtualenv/lib/python2.7/site-packages/setuptools/dist.py", line 429, in fetch_build_egg
return cmd.easy_install(req)
File "/home/b2user/virtualenv/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 659, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('setuptools_scm<6.0')
(virtualenv) [b2user@opensourceecology B2_Command_Line_Tool]$
- I guess 'setuptools_scm' is a new req? We don't have it installed.
[root@opensourceecology backups]# yum search setuptools Loaded plugins: fastestmirror, replace Loading mirror speeds from cached hostfile * base: de.mirrors.clouvider.net * epel: ftp.plusline.net * extras: mirror.checkdomain.de * updates: mirror.checkdomain.de N/S matched: setuptools ======================= python-setuptools.noarch : Easily build and distribute Python packages python3-setuptools.noarch : Easily build and distribute Python 3 packages python34-setuptools.noarch : Easily build and distribute Python 3.4 packages python34-setuptools_scm.noarch : The blessed package to manage your versions by scm tags python36-setuptools_scm.noarch : The blessed package to manage your versions by scm tags Name and summary matches only, use "search all" for everything. [root@opensourceecology backups]# rpm -qa | grep -i setuptools python-setuptools-0.9.8-7.el7.noarch [root@opensourceecology backups]#
- ugh, this is cent7, and the 'setuptools_cm' package appears to only be available for python3. But it looks like the version that we're using the b2 virtualenv is python 2.7.5
(virtualenv) [b2user@opensourceecology B2_Command_Line_Tool]$ python --version Python 2.7.5 (virtualenv) [b2user@opensourceecology B2_Command_Line_Tool]$
- according to the b2 documentation, you should have python >=3.7 https://www.backblaze.com/b2/docs/quick_command_line.html
If you'd like to run the Python version of the B2 command-line tool, it has been packaged in Python modules and published on the Python Package Index (PyPI). The easiest way to get the Python version of the B2 command-line Tool is using the standard Python pip3 installation tool. Your first step is to make sure that you have Python 3 (3.7 or later) installed.
- unfortunately, it looks like we don't have python3 installed at all
[maltfield@opensourceecology ~]$ rpm -qa | grep -i python-3 [maltfield@opensourceecology ~]$
- apparently it was very difficult to install python3 on <cent7.7. Fortuntately we have 7.8 https://www.liquidweb.com/kb/how-to-install-python-3-on-centos-7/
[maltfield@opensourceecology ~]$ cat /etc/redhat-release CentOS Linux release 7.8.2003 (Core) [maltfield@opensourceecology ~]$
- my biggest fear in upgrading python is breaking the base install of the OS. Tools like yum would get totally fucked by doing this in the past. I think this may have been what was fixed in cent7.7. this guide also suggests installing centos-release-scl, but I think that just adds a repo and the guide was actually written pre-7.7 https://linuxize.com/post/how-to-install-python-3-on-centos-7/
- fortunately we have a staging sever. unforunately, I couldn't connect to it. For one reason, my cert had expired
user@ose:~/openvpn$ sudo openvpn client.conf 2022-10-28 13:31:36 WARNING: file '/home/user/openvpn/username.txt' is group or others accessible 2022-10-28 13:31:36 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 14 2021 2022-10-28 13:31:36 library versions: OpenSSL 1.1.1n 15 Mar 2022, LZO 2.10 🔐 Enter Auth Password: ****** 2022-10-28 13:31:43 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts 🔐 Enter Private Key Password: *************** 2022-10-28 13:31:46 WARNING: Your certificate has expired! 2022-10-28 13:31:46 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication 2022-10-28 13:31:46 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication 2022-10-28 13:31:46 TCP/UDP: Preserving recently used remote address: [AF_INET]195.201.233.113:1194 2022-10-28 13:31:46 Socket Buffers: R=[212992->212992] S=[212992->212992] 2022-10-28 13:31:46 UDP link local: (not bound) 2022-10-28 13:31:46 UDP link remote: [AF_INET]195.201.233.113:1194 2022-10-28 13:32:46 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity) 2022-10-28 13:32:46 TLS Error: TLS handshake failed 2022-10-28 13:32:46 SIGUSR1[soft,tls-error] received, process restarting 2022-10-28 13:32:46 Restart pause, 5 second(s) 2022-10-28 13:32:51 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
- the staging server is actually a container running on our dev server in the hetzner cloud. so I logged into hetzner cloud, grabbed the IP, and ssh'd into the dev server directly
user@ose:~$ ssh maltfield@195.201.233.113 -p 32415 Last login: Sat Aug 14 19:21:00 2021 from dynamic-XYZ.pool.telefonica.de [maltfield@osedev1 ~]$
- and I was able to get into the staging container this way
[maltfield@osedev1 ~]$ sudo lxc-attach --name osestaging1 [root@osestaging1 maltfield]# hostname osestaging1 [root@osestaging1 maltfield]#
- I tested the install of python3 on the staging server to make sure it wouldn't break yum
- first we that python3 isn't installed
[root@osestaging1 maltfield]# rpm -qa | grep -i python3 [root@osestaging1 maltfield]# rpm -qa | grep -i python-3 [root@osestaging1 maltfield]#
- I went to confirm that yum was working, but it was blocked by some process that was started 2 years ago?
[root@osestaging1 maltfield]# yum search python3 Loaded plugins: fastestmirror, replace Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Existing lock /var/run/yum.pid: another copy is running as pid 28043. Another app is currently holding the yum lock; waiting for it to exit... The other application is: sed Memory : 140 k RSS ( 13 MB VSZ) Started: Sun Dec 6 06:23:29 2020 - 691 day(s) 12:22:08 ago State : Sleeping, pid: 28043 Another app is currently holding the yum lock; waiting for it to exit... The other application is: sed Memory : 140 k RSS ( 13 MB VSZ) Started: Sun Dec 6 06:23:29 2020 - 691 day(s) 12:22:10 ago State : Sleeping, pid: 28043 ^C Exiting on user cancel. [root@osestaging1 maltfield]# ^C [root@osestaging1 maltfield]# [root@osestaging1 maltfield]# ps -ef | grep -i yum root 12952 12018 0 18:45 ? 00:00:00 grep --color=auto -i yum root 28040 28020 0 2020 ? 00:00:00 sh -c /bin/yum check-update 2>/dev/null | tr ' ' '#' | sed -e 's/# / /g' | tr '#' ' ' 2>/dev/null root 28041 28040 0 2020 ? 1-13:26:59 /usr/bin/python /bin/yum check-update [root@osestaging1 maltfield]# ## I killed those processes and tried again. It failed <pre> [root@osestaging1 maltfield]# yum search python3 Loaded plugins: fastestmirror, replace Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Existing lock /var/run/yum.pid: another copy is running as pid 28043. Another app is currently holding the yum lock; waiting for it to exit... The other application is: sed Memory : 140 k RSS ( 13 MB VSZ) Started: Sun Dec 6 06:23:29 2020 - 691 day(s) 12:22:08 ago State : Sleeping, pid: 28043 Another app is currently holding the yum lock; waiting for it to exit... The other application is: sed Memory : 140 k RSS ( 13 MB VSZ) Started: Sun Dec 6 06:23:29 2020 - 691 day(s) 12:22:10 ago State : Sleeping, pid: 28043 ^C Exiting on user cancel. [root@osestaging1 maltfield]# ^C [root@osestaging1 maltfield]# ^C [root@osestaging1 maltfield]# ps -ef | grep -i yum root 12952 12018 0 18:45 ? 00:00:00 grep --color=auto -i yum root 28040 28020 0 2020 ? 00:00:00 sh -c /bin/yum check-update 2>/dev/null | tr ' ' '#' | sed -e 's/# / /g' | tr '#' ' ' 2>/dev/null root 28041 28040 0 2020 ? 1-13:26:59 /usr/bin/python /bin/yum check-update [root@osestaging1 maltfield]#
- ah, the disk on the staging server is full
[root@osestaging1 maltfield]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/ose_dev_volume_1 125G 119G 0 100% / devtmpfs 873M 0 873M 0% /dev tmpfs 896M 0 896M 0% /dev/shm tmpfs 896M 120M 776M 14% /run tmpfs 896M 0 896M 0% /sys/fs/cgroup overlay 125G 119G 0 100% /var/lib/docker/overlay2/091141fe6d851626be2979681c474fac9a7e1a1645469a502ab14718c6799d98/merged tmpfs 180M 0 180M 0% /run/user/1005 tmpfs 180M 0 180M 0% /run/user/0 tmpfs 180M 0 180M 0% /run/user/991 tmpfs 180M 0 180M 0% /run/user/48 [root@osestaging1 maltfield]# ## looks like docker filled the disks. should I be surprised?
[root@osestaging1 maltfield]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0f16a84fb05f local_discourse/discourse_ose "/sbin/boot" 2 years ago Up 2 years 0.0.0.0:8020->80/tcp discourse_ose [root@osestaging1 maltfield]# </pre>
- I ran my script to cleanup old docker crap (though it should be on cron), it didn't do anything
[root@osestaging1 maltfield]# /usr/local/bin/docker-purge.sh + NICE=/bin/nice + DOCKER=/bin/docker + DATE=/bin/date ++ /bin/date -u +%Y%m%d_%H%M%S + stamp=20221028_184935 + echo ================================================================================ ================================================================================ + echo 'INFO: Beginning docker pruning on 20221028_184935' INFO: Beginning docker pruning on 20221028_184935 + /bin/nice /bin/docker container prune --force --filter until=672h Total reclaimed space: 0B real 0m0.148s user 0m0.057s sys 0m0.060s + /bin/nice /bin/docker image prune --force --all --filter until=672h Total reclaimed space: 0B real 0m0.451s user 0m0.043s sys 0m0.030s + /bin/nice /bin/docker system prune --force --all --filter until=672h Total reclaimed space: 0B real 0m0.318s user 0m0.042s sys 0m0.039s + exit 0 [root@osestaging1 maltfield]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/ose_dev_volume_1 125G 119G 0 100% / devtmpfs 873M 0 873M 0% /dev tmpfs 896M 0 896M 0% /dev/shm tmpfs 896M 120M 776M 14% /run tmpfs 896M 0 896M 0% /sys/fs/cgroup overlay 125G 119G 0 100% /var/lib/docker/overlay2/091141fe6d851626be2979681c474fac9a7e1a1645469a502ab14718c6799d98/merged tmpfs 180M 0 180M 0% /run/user/1005 tmpfs 180M 0 180M 0% /run/user/0 tmpfs 180M 0 180M 0% /run/user/991 tmpfs 180M 0 180M 0% /run/user/48 [root@osestaging1 maltfield]#
- looks like discourse had been backing itself up once daily since 2020-09-28
[root@osestaging1 maltfield]# ls /var/discourse/shared/standalone/tmp/backups/default | head 2020-09-28-033055 2020-09-29-033122 2020-09-30-033830 2020-10-01-033408 2020-10-02-033910 2020-10-03-033705 2020-10-04-033715 2020-10-05-033942 2020-10-06-033709 2020-10-07-033055 [root@osestaging1 maltfield]# ls /var/discourse/shared/standalone/tmp/backups/default | tail 2022-10-15-034044 2022-10-16-034459 2022-10-17-033136 2022-10-18-034128 2022-10-19-033339 2022-10-20-033840 2022-10-21-033633 2022-10-22-033735 2022-10-23-052235 2022-10-24-033323 [root@osestaging1 maltfield]# du -sh /var/discourse/shared/standalone/tmp/backups/default 46G /var/discourse/shared/standalone/tmp/backups/default [root@osestaging1 maltfield]#
- I deleted all but a few of the backups
[root@osestaging1 maltfield]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/ose_dev_volume_1 125G 73G 46G 62% / devtmpfs 873M 0 873M 0% /dev tmpfs 896M 0 896M 0% /dev/shm tmpfs 896M 120M 776M 14% /run tmpfs 896M 0 896M 0% /sys/fs/cgroup overlay 125G 73G 46G 62% /var/lib/docker/overlay2/091141fe6d851626be2979681c474fac9a7e1a1645469a502ab14718c6799d98/merged tmpfs 180M 0 180M 0% /run/user/1005 tmpfs 180M 0 180M 0% /run/user/0 tmpfs 180M 0 180M 0% /run/user/991 tmpfs 180M 0 180M 0% /run/user/48 [root@osestaging1 maltfield]# ls /var/discourse/shared/standalone/tmp/backups/default 2020-09-28-033055 2020-11-01-033827 2022-01-01-033309 2022-10-24-033323 2020-10-01-033408 2021-01-01-033533 2022-10-01-033823 [root@osestaging1 maltfield]#
- I tried to test that yum is working again; it's still failing
[root@osestaging1 maltfield]# yum search python3 error: rpmdb: BDB0113 Thread/process 28043/139752552937280 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm CRITICAL:yum.main: Error: rpmdb open failed [root@osestaging1 maltfield]#
- I gave the staging server a reboot. when it came up, I tried again
[root@osestaging1 ~]# yum search python3 Loaded plugins: fastestmirror, replace Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Determining fastest mirrors * base: mirror.fra10.de.leaseweb.net * epel: mirrors.n-ix.net * extras: mirror.softaculous.com * updates: mirror1.hs-esslingen.de ============================== N/S matched: python3 =============================== boost-python36-static.x86_64 : The Python3 Boost C++ static development libraries ... znc-modpython.x86_64 : Python3 module for ZNC Name and summary matches only, use "search all" for everything. [root@osestaging1 ~]#
- ok, so yum is working. and we confirm python3 is not available
[root@osestaging1 ~]# which python /bin/python [root@osestaging1 ~]# which python2 /bin/python2 [root@osestaging1 ~]# which python3 /usr/bin/which: no python3 in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin) [root@osestaging1 ~]#
- now let's try to install python3
[root@osestaging1 ~]# yum install python3 ... Dependencies Resolved =================================================================================== Package Arch Version Repository Size =================================================================================== Installing: python3 x86_64 3.6.8-18.el7 updates 70 k Installing for dependencies: libtirpc x86_64 0.2.4-0.16.el7 base 89 k python3-libs x86_64 3.6.8-18.el7 updates 6.9 M python3-pip noarch 9.0.3-8.el7 base 1.6 M python3-setuptools noarch 39.2.0-10.el7 base 629 k Transaction Summary =================================================================================== Install 1 Package (+4 Dependent packages) Total download size: 9.3 M Installed size: 48 M Is this ok [y/d/N]: y ... Installed: python3.x86_64 0:3.6.8-18.el7 Dependency Installed: libtirpc.x86_64 0:0.2.4-0.16.el7 python3-libs.x86_64 0:3.6.8-18.el7 python3-pip.noarch 0:9.0.3-8.el7 python3-setuptools.noarch 0:39.2.0-10.el7 Complete! [root@osestaging1 ~]#
- that only gave us python 3.6.8 :/
[root@osestaging1 ~]# which python /bin/python [root@osestaging1 ~]# which python2 /bin/python2 [root@osestaging1 ~]# which python3 /bin/python3 [root@osestaging1 ~]# python --version Python 2.7.5 [root@osestaging1 ~]# python3 --version Python 3.6.8 [root@osestaging1 ~]#
- looks like python37 isn't in the repos. and this also confirms that yum python isn't broken. yay!
[root@osestaging1 ~]# yum search python37 Loaded plugins: fastestmirror, replace Loading mirror speeds from cached hostfile * base: mirror.fra10.de.leaseweb.net * epel: mirror.karneval.cz * extras: mirror.softaculous.com * updates: mirror1.hs-esslingen.de Warning: No matches found for: python37 No matches found [root@osestaging1 ~]#
- anyway, I reproduced the issue on staging; let's see if we can fix it there
[b2user@osestaging1 ~]$ /home/b2user/virtualenv/bin/b2 list-buckets ERROR: Connection error: HTTPSConnectionPool(host='api001.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_list_buckets (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) [b2user@osestaging1 ~]$ [b2user@osestaging1 ~]$ ls hetzner2_20180727_072001.tar.gpg sandbox sync sync.old tmp virtualenv [b2user@osestaging1 ~]$ time rsync -a virtualenv virtualenv.20221028 real 0m1.788s user 0m0.173s sys 0m0.394s [b2user@osestaging1 ~]$ mv virtualenv virtualenv.old [b2user@osestaging1 ~]$
[root@osestaging1 virtualenv]# yum install python3-virtualenv
- I did have to install virtualenv for python3
Installed: python36-virtualenv.noarch 0:15.1.0-5.el7 Dependency Installed: python3-devel.x86_64 0:3.6.8-18.el7 python3-rpm-generators.noarch 0:6-2.el7 python3-rpm-macros.noarch 0:3-34.el7 Complete! [root@osestaging1 virtualenv]#
- And I tried the install again
[b2user@osestaging1 ~]$ mkdir virtualenv [b2user@osestaging1 ~]$ pushd virtualenv/ ~/virtualenv ~ [b2user@osestaging1 virtualenv]$ python3 -m virtualenv . Using base prefix '/usr' New python executable in /home/b2user/virtualenv/bin/python3 Also creating executable in /home/b2user/virtualenv/bin/python Installing setuptools, pip, wheel...done. [b2user@osestaging1 virtualenv]$ source ~/virtualenv/bin/activate (virtualenv) [b2user@osestaging1 virtualenv]$ popd ~ (virtualenv) [b2user@osestaging1 ~]$ mkdir sandbox mkdir: cannot create directory ‘sandbox’: File exists (virtualenv) [b2user@osestaging1 ~]$ pushd sandbox ~/sandbox ~ (virtualenv) [b2user@osestaging1 sandbox]$ ls B2_Command_Line_Tool (virtualenv) [b2user@osestaging1 sandbox]$ cd B2_Command_Line_Tool/ (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ git pull remote: Enumerating objects: 2759, done. remote: Counting objects: 100% (2020/2020), done. remote: Compressing objects: 100% (629/629), done. remote: Total 2759 (delta 1485), reused 1808 (delta 1388), pack-reused 739 Receiving objects: 100% (2759/2759), 644.64 KiB | 0 bytes/s, done. Resolving deltas: 100% (1907/1907), completed with 32 local objects. From https://github.com/Backblaze/B2_Command_Line_Tool 4b8c653..25c13d3 master -> origin/master ... delete mode 100644 test_b2_command_line.py (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ python setup.py install ... Installed /home/b2user/virtualenv/lib/python3.6/site-packages/six-1.16.0-py3.6.egg Finished processing dependencies for b2==3.6.1.dev1+g25c13d3 (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$
- I tried `list-buckets` again, and I got missing reqs issues
(virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ /home/b2user/virtualenv/bin/b2 list-buckets Traceback (most recent call last): File "/home/b2user/virtualenv/bin/b2", line 10, in <module> from importlib.metadata import distribution ModuleNotFoundError: No module named 'importlib.metadata' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/b2user/virtualenv/lib/python3.6/site-packages/importlib_metadata-5.0.0-py3.6.egg/importlib_metadata/_compat.py", line 9, in <module> from typing import Protocol ImportError: cannot import name 'Protocol' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/b2user/virtualenv/bin/b2", line 13, in <module> from importlib_metadata import distribution File "/home/b2user/virtualenv/lib/python3.6/site-packages/importlib_metadata-5.0.0-py3.6.egg/importlib_metadata/init.py", line 17, in <module> from . import _adapters, _meta, _py39compat File "/home/b2user/virtualenv/lib/python3.6/site-packages/importlib_metadata-5.0.0-py3.6.egg/importlib_metadata/_meta.py", line 1, in <module> from ._compat import Protocol File "/home/b2user/virtualenv/lib/python3.6/site-packages/importlib_metadata-5.0.0-py3.6.egg/importlib_metadata/_compat.py", line 12, in <module> from typing_extensions import Protocol # type: ignore File "/home/b2user/virtualenv/lib/python3.6/site-packages/typing_extensions-4.4.0-py3.6.egg/typing_extensions.py", line 193, in <module> class _FinalForm(typing._SpecialForm, _root=True): AttributeError: module 'typing' has no attribute '_SpecialForm' (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$
- welp, looks like typing is available in the repos, but importlib is not
(virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ yum search python3 | grep -i protocol python36-josepy.noarch : JOSE protocol implementation in Python python36-ncclient.noarch : Python library for the NETCONF protocol python36-paramiko.noarch : SSH2 protocol library for python python36-vxi11.noarch : Python implementation of the VXI-11 protocol (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ yum search python3 | grep -i importlib (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ yum search python3 | grep -i typing python36-typing.noarch : Typing defines a standard notation for type annotations (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$
- I try to avoid installing depends with pip if at all possible because it's not a secure method to obtain software https://security.stackexchange.com/questions/232855/does-pythons-pip-provide-cryptographic-authentication-and-integrity-validation
- but there's many other requirements, and I'm afraid we have no choice here but to use pip :(
(virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ cat requirements.txt arrow>=1.0.2,<2.0.0 b2sdk>=1.18.0 docutils==0.19 idna>=2.2.0; platform_system == 'Java' importlib-metadata>=3.3.0; python_version < '3.8' phx-class-registry==3.0.5 rst2ansi==0.1.5 tabulate==0.8.10 (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$
- there's a couple alternatives
- use rclone to push to b2, which is in the repos
(virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$ yum search rclone Loaded plugins: fastestmirror, replace Loading mirror speeds from cached hostfile * base: mirror.fra10.de.leaseweb.net * epel: mirror.de.leaseweb.net * extras: mirror.softaculous.com * updates: mirror.fra10.de.leaseweb.net =============================== N/S matched: rclone =============================== rclone.x86_64 : Rsync for cloud storage Name and summary matches only, use "search all" for everything. (virtualenv) [b2user@osestaging1 B2_Command_Line_Tool]$
- download the self-contained 'b2' binary release for linux from github, which is not signed https://github.com/Backblaze/B2_Command_Line_Tool/releases
- looks like I opened a ticket asking them to sign their releases in 2021-08; it's still an open ticket :( https://github.com/Backblaze/B2_Command_Line_Tool/issues/744
- download the self-contained 'b2' binary release for linux from github, which is not signed https://github.com/Backblaze/B2_Command_Line_Tool/releases
- rclone seems like the best option. I installed it from the repos. that gave us rclone v1.55
[root@osestaging1 ~]# yum install rclone ... Installed: rclone.x86_64 0:1.55.1-1.el7 Complete! [root@osestaging1 ~]# [root@osestaging1 ~]# rclone --version 2022/10/28 21:34:50 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults rclone v1.55.1-DEV - os/type: linux - os/arch: amd64 - go/version: go1.15.5 - go/linking: dynamic - go/tags: none [root@osestaging1 ~]#
- I went to configure rclone, but when it asked me for the API keys, I couldn't find them
- I went to create a new set of API keys, but you actually can't create append-only API keys from within the B2 WUI. For that, you need the `b2` command line. Well, now we're in a chicken-and-egg problem
- I searched the existing prod system for the API creds, and I found them in a binary file. The head of that file indicated it's an sqlite db
[root@osestaging1 b2user]# grep -irl 'OBFUSCATED_KEY_ID' .b2* .b2_account_info.append-only [root@osestaging1 b2user]# head .b2_account_info.append-only SQLite format 3@ ������tableaccountaccountCREATE TABLE account ( account_id TEXT NOT NULL, application_key TEXT NOT NULL, account_auth_token TEXT NOT NULL, api_url TEXT NOT NULL, download_url TEXT NOT NULL, minimum_part_size INT NOT NULL, realm TEXT NOT NULL , allowed TEXT, account_id_or_app_key_id TEXT)�\//�gtablebucket_upload_urlbucket_upload_urlCREATE TABLE bucket_upload_url ( bucket_id TEXT NOT NULL, [root@osestaging1 b2user]# [root@osestaging1 b2user]# sqlite3 .b2_account_info.append-only SQLite version 3.7.17 2013-05-20 00:56:22 Enter ".help" for instructions Enter SQL statements terminated with a ";" sqlite>
- I sifted through the db to extract the creds
sqlite> .schema CREATE TABLE update_done ( update_number INT NOT NULL ); CREATE TABLE account ( account_id TEXT NOT NULL, application_key TEXT NOT NULL, account_auth_token TEXT NOT NULL, api_url TEXT NOT NULL, download_url TEXT NOT NULL, minimum_part_size INT NOT NULL, realm TEXT NOT NULL , allowed TEXT, account_id_or_app_key_id TEXT); CREATE TABLE bucket ( bucket_name TEXT NOT NULL, bucket_id TEXT NOT NULL ); CREATE TABLE bucket_upload_url ( bucket_id TEXT NOT NULL, upload_url TEXT NOT NULL, upload_auth_token TEXT NOT NULL ); sqlite> select * from account;
- as the 'b2user', I configured rclone to use b2
[root@osestaging1 ~]# sudo su - b2user
Last login: Fri Oct 28 21:17:13 UTC 2022
[b2user@osestaging1 ~]$ rclone config
2022/10/28 21:37:15 NOTICE: Config file "/home/b2user/.config/rclone/rclone.conf" n
ot found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> b2
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
Storage> 5
See help for b2 backend at: https://rclone.org/b2/
Account ID or Application Key ID
Enter a string value. Press Enter for the default ("").
account>
account> OBFUSCATED_KEY_ID
Application Key
Enter a string value. Press Enter for the default ("").
key> OBFUSCATED_SECRET_KEY
Permanently delete files on remote removal, otherwise hide files.
Enter a boolean value (true or false). Press Enter for the default ("false").
hard_delete> true
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[b2]
type = b2
account = OBFUSCATED_KEY_ID
key = OBFUSCATED_SECRET_KEY
hard_delete = true
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
### =
b2 b2
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[b2user@osestaging1 ~]$
- the new config works! I was able to list the bucket (and, good, only the one bucket that our key is allowed to access)
<pr> [b2user@osestaging1 ~]$ rclone listremotes b2: [b2user@osestaging1 ~]$ rclone lsd b2: -1 2022-10-28 21:56:11 -1 ose-server-backups [b2user@osestaging1 ~]$ </pre>
- I tried to upload a test file, but unfortunately that failed
[b2user@osestaging1 ~]$ echo "test file created to test rclone from osestaging1" > test.20221028
[b2user@osestaging1 ~]$ rclone --progress copy test.20221028 b2:ose-server-backups/2022-10-28 21:59:03 ERROR : Attempt 1/3 failed with 1 errors and: failed to HEAD for download: Unknown 401 (401 unknown)
2022-10-28 21:59:06 ERROR : Attempt 2/3 failed with 1 errors and: failed to HEAD for download: Unknown 401 (401 unknown)
2022-10-28 21:59:10 ERROR : Attempt 3/3 failed with 1 errors and: failed to HEAD for download: Unknown 401 (401 unknown)
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 11.0s
2022/10/28 21:59:10 Failed to copy: failed to HEAD for download: Unknown 401 (401 unknown)
[b2user@osestaging1 ~]$
- I tried to copy it into the other bucket, but I got an error (as expected)
[b2user@osestaging1 ~]$ rclone --progress copy test.20221028 b2:ose-dev-server-backups/test.20221028 2022/10/28 22:15:39 Failed to create file system for "b2:ose-dev-server-backups/test.20221028": you must use bucket "ose-server-backups" with this application key [b2user@osestaging1 ~]$
- ah, I've encountered this before. With rcone, you need to give the key the 'listFiles' permission. It shouldn't be strictly necessary, but maybe rclone always attempts to list before upload to do some dedup or something, idk. Anyway, the existing key has 'listBuckets, writeFiles'
- I'll solve the chicken-and-egg problem by creating a new API key on my laptop, where I can get the b2 cli from apt in Debian
user@ose:~/openvpn$ sudo apt-cache search b2 | grep -i backblaze backblaze-b2 - Command Line Tool for Backblaze B2 golang-github-kurin-blazer-dev - Go library for Backblaze's B2 user@ose:~/openvpn$
- I had to create a new set of master keys in the WUI. That *should* be fine because we shouldn't have any actual applications using the master keys..
user@ose:~/openvpn$ sudo apt-get install backblaze-b2 ... user@ose:~/openvpn$ user@ose:~/openvpn$ backblaze-b2 authorize-account Using https://api.backblazeb2.com Backblaze account ID: OBFUSCATED Backblaze application key: OBFUSCATED user@ose:~/openvpn$ user@ose:~/openvpn$ backblaze-b2 list-buckets OBFUSCATED allPrivate ose-dev-server-backups OBFUSCATED allPrivate ose-server-backups user@ose:~/openvpn$ user@ose:~/openvpn$ backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10' 'listBuckets,writeFiles,listFiles' OBFUSCATED OBFUSCATED user@ose:~/openvpn$
- back on the staging node, I deleted the existing account and added this new one
[b2user@osestaging1 ~]$ rclone config
Current remotes:
Name Type
==== ====
b2 b2
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> d
Choose a number from below, or type in an existing value
1 > b2
remote> 1
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> b2
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
account> OBFUSCATED
Application Key
Enter a string value. Press Enter for the default ("").
key> OBFUSCATED
Permanently delete files on remote removal, otherwise hide files.
Enter a boolean value (true or false). Press Enter for the default ("false").
hard_delete> true
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[b2]
type = b2
account = OBFUSCATED
key = OBFUSCATED
hard_delete = true
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
b2 b2
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[b2user@osestaging1 ~]$
- it works!
[b2user@osestaging1 ~]$ rclone listremotes
b2:
[b2user@osestaging1 ~]$ rclone lsd b2:
-1 2022-10-28 22:35:31 -1 ose-server-backups
[b2user@osestaging1 ~]$
- this time the 'ls' works, but the uploads still fail :(
[b2user@osestaging1 ~]$ rclone ls b2:ose-server-backups 20619278434 monthly_hetzner2_20211101_072001.tar.gpg 17516124812 yearly_hetzner2_20190101_111520.tar.gpg 18872422001 yearly_hetzner2_20200101_072001.tar.gpg 19827971632 yearly_hetzner2_20210101_072001.tar.gpg [b2user@osestaging1 ~]$ [b2user@osestaging1 ~]$ rclone --progress copy test.20221028 b2:ose-server-backups/ 2022-10-28 22:36:17 ERROR : Attempt 1/3 failed with 1 errors and: failed to HEAD for download: Unknown 401 (401 unknown) 2022-10-28 22:36:21 ERROR : Attempt 2/3 failed with 1 errors and: failed to HEAD for download: Unknown 401 (401 unknown) 2022-10-28 22:36:24 ERROR : Attempt 3/3 failed with 1 errors and: failed to HEAD for download: Unknown 401 (401 unknown) Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 1 (retrying may help) Elapsed time: 11.4s 2022/10/28 22:36:24 Failed to copy: failed to HEAD for download: Unknown 401 (401 unknown) [b2user@osestaging1 ~]$
- so it looks like backblaze has added some new permissions since I last set this up https://www.backblaze.com/b2/docs/b2_create_key.html
listAllBucketNames, listBuckets, readBuckets, readBucketEncryption, writeBucketEncryption, readBucketRetentions, writeBucketRetentions, listFiles, readFiles, shareFiles, writeFiles, deleteFiles, readFileLegalHolds, writeFileLegalHolds, readFileRetentions, writeFileRetentions, and bypassGovernance.
- I experiemented by adding a new (temp) key with a lot of the ones I'm not familar with
user@ose:~/openvpn$ backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10b' 'listAllBucketNames, listBuckets, readBuckets, readBucketEncryption, writeBucketEncryption, readBucketRetentions, writeBucketRetentions, listFiles, readFiles, writeFiles' OBFUSCATED OBFUSCATED user@ose:~/openvpn$
- using the new keys (just by editing .config/rclone/rclone.conf directly), the copy worked
[b2user@osestaging1 ~]$ vim .config/rclone/rclone.conf [b2user@osestaging1 ~]$ [b2user@osestaging1 ~]$ rclone --progress copy test.20221028 b2:ose-server-backups Transferred: 50 / 50 Bytes, 100%, 35 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 2.6s
- now we need to just determine what exactly is the minimum set of permissions
- I removed the encrypted ones; the copy still worked
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10c' 'listAllBucketNames, listBuckets, readBuckets, readBucketRetentions, writeBucketRetentions, listFiles, readFiles, writeFiles'
- I removed the "read" ones; the copy still worked
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10d' 'listAllBucketNames, listBuckets, writeBucketRetentions, listFiles, readFiles, writeFiles'
- I removed 'listAllBucketNames' and it still worked
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10e' 'listBuckets, writeBucketRetentions, listFiles, readFiles, writeFiles'
- I removed 'writeBucketRetentions' and it still worked
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10f' 'listBuckets, listFiles, readFiles, writeFiles'
- But when I removed 'readFiles', it broke again
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10g' 'listBuckets, listFiles, writeFiles'
- I tried adding 'readFiles' but without 'listFiles'; it worked!
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10h' 'listBuckets, readFiles, writeFiles'
- so with just 'listBuckets, readFiles, writeFiles', I can write but not list. Cool
[b2user@osestaging1 ~]$ rclone --progress copy test.20221028 b2:ose-server-backups^C [b2user@osestaging1 ~]$ trclone ls b2:ose-server-backups -bash: trclone: command not found [b2user@osestaging1 ~]$ rclone ls b2:ose-server-backups 2022/10/28 23:07:39 Failed to ls: Unknown 401 (401 unauthorized) [b2user@osestaging1 ~]$ rclone --progress copy test.20221028 b2:ose-server-backups Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Checks: 1 / 1, 100% Elapsed time: 1.1s [b2user@osestaging1 ~]$
- I also tried removing 'listBuckets', and it still worked!
backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10i' 'readFiles, writeFiles'
- therefore, it appears that the actual minimum perimssion for our production append-only backblaze b2 key (since backblaze added this new 'readFiles' permission) is just 'readFiles, writeFiles'
- I tried to delete all the keys that I created today. I made it through all but the first one, when the WUI told me ERROR "too many requests" :(
- ok, after ~30 seconds I was able to delete the last one
- I created the new key with the actual min permissions
user@ose:~/openvpn$ backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-append-only-2022-10' 'readFiles, writeFiles' OBFUSCATED OBFUSCATED user@ose:~/openvpn$
- back on the prod server, I installed `rclone` and configured it with this new key
[root@opensourceecology ~]# yum install rclone
Loaded plugins: fastestmirror, replace
Loading mirror speeds from cached hostfile
* base: mirror.checkdomain.de
* epel: ftp.plusline.net
* extras: centosmirror.netcup.net
* updates: de.mirrors.clouvider.net
Resolving Dependencies
--> Running transaction check
---> Package rclone.x86_64 0:1.55.1-1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================
Package Arch Version Repository Size
=====================================================================================================
Installing:
rclone x86_64 1.55.1-1.el7 epel 17 M
Transaction Summary
=====================================================================================================
Install 1 Package
Total download size: 17 M
Installed size: 68 M
Is this ok [y/d/N]: y
Downloading packages:
rclone-1.55.1-1.el7.x86_64.rpm | 17 MB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : rclone-1.55.1-1.el7.x86_64 1/1
Verifying : rclone-1.55.1-1.el7.x86_64 1/1
Installed:
rclone.x86_64 0:1.55.1-1.el7
Complete!
[root@opensourceecology ~]#
[root@opensourceecology ~]# sudo su - b2user
Last login: Fri Oct 28 18:03:55 UTC 2022 on pts/24
[b2user@opensourceecology ~]$
[b2user@opensourceecology ~]$ rclone config
2022/10/28 23:17:25 NOTICE: Config file "/home/b2user/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> b2
...
\ "seafile"
Storage> 5
** See help for b2 backend at: https://rclone.org/b2/ **
Account ID or Application Key ID
Enter a string value. Press Enter for the default ("").
account> OBFUSCATED
Application Key
Enter a string value. Press Enter for the default ("").
key> OBFUSCATED
Permanently delete files on remote removal, otherwise hide files.
Enter a boolean value (true or false). Press Enter for the default ("false").
hard_delete> true
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[b2]
type = b2
account = OBFUSCATED
key = OBFUSCATED
hard_delete = true
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
b2 b2
e) Edit existing remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
b2 b2
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[b2user@opensourceecology ~]$
[b2user@opensourceecology ~]$ echo "test file created to test rclone from prod server" > test.20221028
[b2user@opensourceecology ~]$ rclone copy test.20221028 b2:ose-server-backups/test4
[b2user@opensourceecology ~]$ date
Fri Oct 28 23:20:34 UTC 2022
[b2user@opensourceecology ~]$ rclone copy test.20221028 b2:ose-server-backups/test5
[b2user@opensourceecology ~]$
- I confirmed from the WUI that these new files were uploaded successfully
- I updated the backup scripts to use `rclone` instead of `b2` (this hasn't been tested)
[root@opensourceecology backups]# diff backup.sh.20221028 backup.sh
148c148,149
< time $SUDO -u ${b2UserName} $TRICKLE -s -u 3000 $B2 upload-file --noProgress --threads 1 ${B2_BUCKET_NAME} "${b2EncryptedArchiveFilePath}" "${encryptedArchiveFile}"
---
> #time $SUDO -u ${b2UserName} $TRICKLE -s -u 3000 $B2 upload-file --noProgress --threads 1 ${B2_BUCKET_NAME} "${b2EncryptedArchiveFilePath}" "${encryptedArchiveFile}"
> time $SUDO -u ${b2UserName} ${RCLONE} -v --bwlimit 3M --progress copy "${b2EncryptedArchiveFilePath}" "b2:${B2_BUCKET_NAME}"
[root@opensourceecology backups]#
[root@opensourceecology backups]# diff backup.settings.20221028 backup.settings
37a38
> RCLONE='/bin/rclone'
[root@opensourceecology backups]#
- before I try to kickoff a new backup, I manually uploaded the most-recent one
- with the normal bandwidth limit of 3 Mbps, it was going to take over 2 hours, so I removed that for the sake of time
[b2user@opensourceecology ~]$ time rclone -v --bwlimit 3M --progress copy sync/daily_hetzner2_20221028_072001.tar.gpg b2:ose-server-backups 2022/10/28 23:34:22 INFO : Starting bandwidth limiter at 3MBytes/s 2022-10-28 23:35:55 INFO : Signal received: interrupt Transferred: 203.156M / 19.571 GBytes, 1%, 2.231 MBytes/s, ETA 2h28m10s Transferred: 0 / 1, 0% Elapsed time: 1m32.1s Transferring: * daily_hetzner2_20221028_072001.tar.gpg: 1% /19.571G, 2.960M/s, 1h51m42s^C real 1m32.390s user 0m26.399s sys 0m7.641s [b2user@opensourceecology ~]$ ^C
- speeds were actually 20-40 Mbps without caps
[b2user@opensourceecology ~]$ time rclone -v --progress copy sync/daily_hetzner2_20221028_072001.tar.gpg b2:ose-server-backups 2022-10-28 23:48:14 INFO : daily_hetzner2_20221028_072001.tar.gpg: Copied (new) Transferred: 19.571G / 19.571 GBytes, 100%, 27.552 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 12m8.5s 2022/10/28 23:48:14 INFO : Transferred: 19.571G / 19.571 GBytes, 100%, 27.552 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 12m8.5s real 12m8.591s user 2m49.455s sys 0m39.219s [b2user@opensourceecology ~]$
- I also went ahead and uploaded the previous backup too
- the other problem is the backup reports. this will actually fail with our existing key because it can't list. But this runs as root, so we should create a different key that has permissions to list, but still not delete
- Note: it's super important that all the API keys stored on the server are append-only. That is, they should be able to upload backups, but they should never have the capability to delete backups. This is super-important to protect us from, for example, ransomware https://en.wikipedia.org/wiki/Append-only
- We also wanted to reduce the permissions as much as possible for the keys that the backup script had, and to make it run as an unprivliged user, mainly because I didn't trust the `b2` code that we insecurely fetched from github without any cryptographic verification of its authenticity (eg to protect from release infrastructure comprimise). But since we're now getting `rclone` from the much-safer method of `yum install`, that concern is largely allevaited. Still, principle of least privlige is best
[root@opensourceecology backups]# cat /etc/cron.d/backup_to_backblaze 20 07 * * * root time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log 20 04 03 * * root time /bin/nice /root/backups/backupReport.sh [root@opensourceecology backups]#
- Anyway, since this cron runs as root, I'll create a new set of keys called and add that to root's rclone config
- first on my laptop, create the keys
- Anyway, since this cron runs as root, I'll create a new set of keys called and add that to root's rclone config
user@ose:~/openvpn$ backblaze-b2 create-key --bucket 'ose-server-backups' 'prod-list-and-append-only-2022-10' 'listFiles, readFiles, writeFiles' OBFUSCATED OBFUSCATED user@ose:~/openvpn$
- next on the server, as root
root@opensourceecology ~]# rclone config
2022/10/28 23:46:11 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> b2
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
Storage> 5
** See help for b2 backend at: https://rclone.org/b2/ **
Account ID or Application Key ID
Enter a string value. Press Enter for the default ("").
account> OBFUSCATED
Application Key
Enter a string value. Press Enter for the default ("").
key> OBFUSCATED
Permanently delete files on remote removal, otherwise hide files.
Enter a boolean value (true or false). Press Enter for the default ("false").
hard_delete> true
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[b2]
type = b2
account = OBFUSCATED
key = OBFUSCATED
hard_delete = true
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
b2 b2
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
[root@opensourceecology ~]#
- a test: list works, delete does not. Perfect
[root@opensourceecology ~]# rclone ls b2:ose-server-backups
21014547058 daily_hetzner2_20221028_072001.tar.gpg
20619278434 monthly_hetzner2_20211101_072001.tar.gpg
50 test.20221028
50 test.20221028b
50 test.20221028c
50 test5/test.20221028
17516124812 yearly_hetzner2_20190101_111520.tar.gpg
18872422001 yearly_hetzner2_20200101_072001.tar.gpg
19827971632 yearly_hetzner2_20210101_072001.tar.gpg
[root@opensourceecology ~]# rclone delete b2:ose-server-backups/test.20221028
2022/10/28 23:49:40 ERROR : test.20221028: Couldn't delete: failed to delete "test.20221028": Unknown 401 (401 unauthorized)
2022/10/28 23:49:40 ERROR : Attempt 1/3 failed with 2 errors and: failed to delete 1 files
2022/10/28 23:49:44 ERROR : test.20221028: Couldn't delete: failed to delete "test.20221028": Unknown 401 (401 unauthorized)
2022/10/28 23:49:44 ERROR : Attempt 2/3 failed with 2 errors and: failed to delete 1 files
2022/10/28 23:49:47 ERROR : test.20221028: Couldn't delete: failed to delete "test.20221028": Unknown 401 (401 unauthorized)
2022/10/28 23:49:47 ERROR : Attempt 3/3 failed with 2 errors and: failed to delete 1 files
2022/10/28 23:49:47 Failed to delete with 2 errors: last error was: failed to delete 1 files
[root@opensourceecology ~]#
- Fixing the backupReport.sh script was incredibly simple. The first test got all the files, but it's still complaining because yesterday's backup is still upoading (I only finished the one for today above so far)
[root@opensourceecology backups]# diff backupReport.sh.20221028 backupReport.sh
27c28,29
< backupsInCloud=`$SUDO -u ${b2UserName} $B2 ls ${B2_BUCKET_NAME} 2>&1`
---
> #backupsInCloud=`$SUDO -u ${b2UserName} $B2 ls ${B2_BUCKET_NAME} 2>&1`
> backupsInCloud=`${RCLONE} ls "b2:${B2_BUCKET_NAME}" 2>&1`
[root@opensourceecology backups]#
