Maltfield Log/2019 Q1

My work log from the year 2019 Quarter 1. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

1. Maltfield_Log
2. User:Maltfield
3. Special:Contributions/Maltfield

Wed Jan 16, 2019

1. I logged into the server and found that the backup was currently uploading today's nightly backup to backblaze b2
2. The logs are being hella spammed with the upload progress bar (the backup log containing just today's backup was already 14M), so I updated the backup.sh script to use the --noProgress argument.

[root@hetzner2 ~]# du -sh /var/log/backups/*
6.2M    /var/log/backups/backup.log
48K     /var/log/backups/backup.log-20181206.gz
96K     /var/log/backups/backup.log-20181207.gz
48K     /var/log/backups/backup.log-20181209.gz
48K     /var/log/backups/backup.log-20181210.gz
96K     /var/log/backups/backup.log-20181211.gz
348K    /var/log/backups/backup.log-20181213.gz
1012K   /var/log/backups/backup.log-20181215.gz
340K    /var/log/backups/backup.log-20181216.gz
356K    /var/log/backups/backup.log-20181218.gz
748K    /var/log/backups/backup.log-20181219.gz
364K    /var/log/backups/backup.log-20181221.gz
372K    /var/log/backups/backup.log-20181222.gz
716K    /var/log/backups/backup.log-20181223.gz
712K    /var/log/backups/backup.log-20181225.gz
352K    /var/log/backups/backup.log-20181226.gz
392K    /var/log/backups/backup.log-20181227.gz
352K    /var/log/backups/backup.log-20181228.gz
352K    /var/log/backups/backup.log-20181230.gz
356K    /var/log/backups/backup.log-20181231.gz
360K    /var/log/backups/backup.log-20190101.gz
360K    /var/log/backups/backup.log-20190102.gz
700K    /var/log/backups/backup.log-20190103.gz
352K    /var/log/backups/backup.log-20190105.gz
352K    /var/log/backups/backup.log-20190106.gz
364K    /var/log/backups/backup.log-20190107.gz
388K    /var/log/backups/backup.log-20190108.gz
408K    /var/log/backups/backup.log-20190109.gz
360K    /var/log/backups/backup.log-20190110.gz
488K    /var/log/backups/backup.log-20190111.gz
352K    /var/log/backups/backup.log-20190113.gz
384K    /var/log/backups/backup.log-20190114.gz
14M     /var/log/backups/backup.log-20190115
[root@hetzner2 ~]# 
79M     .
[root@hetzner2 ~]# 

1. I also added many INFO echos to the script to make the logs more useful
2. I ssh'd into our dreamhost server. Our ose_marcin user's home directly is currently using 304G of disk space. Hetzner1 backups are at 13G & hetzner2 backups are at 288G

hancock% pwd
/home/marcin_ose
hancock% date
Wed Jan 16 02:04:22 PST 2019
hancock% du -sh .
304G	.
hancock% du -sh hetzner1
13G	hetzner1
hancock% du -sh hetzner2
288G	hetzner2
hancock% 

1. I confirmed that the backup from today (20190116) was not present on the dreamhost server. success!

hancock% du -sh hetzner1/*
12G	hetzner1/20180501-052002
259M	hetzner1/20180502-052001
464M	hetzner1/20180602-052001
464M	hetzner1/20180702-052001
hancock% du -sh hetzner2/*
15G	hetzner2/20180501-072001
15G	hetzner2/20180601_072001
15G	hetzner2/20180701_072001
16G	hetzner2/20180801_072001
17G	hetzner2/20180901_072001
17G	hetzner2/20181001_072001
17G	hetzner2/20181101_072001
17G	hetzner2/20181202_072001
33G	hetzner2/20190101_072001
33G	hetzner2/20190112_072001
33G	hetzner2/20190113_072001
33G	hetzner2/20190114_072001
33G	hetzner2/20190115_072001
hancock% 

Tue Jan 15, 2019

1. I renamed the /root/backups/backup.sh script to be 'backup.old.20180115.sh' and I renamed backup2.sh to be 'backup.sh'
2. I also renamed /etc/cron.d/backup_to_dreamhost to be 'backup_to_backblaze'
3. The above has the effect of disabling backups being sent to dreamhost. All backups will now only go to backblaze b2 going forward.

Thr Jan 10, 2019

1. checked backblaze wui
   1. total size of bucket is currently 157.7G
   2. I confirmed that th yearly backup file exists at 17.5G = yearly_hetzner2_20190101_111520.tar.gpg
   3. I checked our billing; in all of 2018 it was $0.02. In 2019, it's $0.63 so far in 2019. (which is actually the billing period between Dec 16 through Jan 10).
      1. I should do a test downloading the 2019-01-01 file and document the process (with screenshots) on our wiki after 2019-01-17--since we get a free quota of downloads once per month, and I used some of it for testing/documenting the download process via the cli in this billing period (which cost $0.61 to do)

Wed Jan 09, 2019

1. marked time
2. still waiting on Marcin to unblock me for phplist