CHG-2018-07-06 hetzner1 deprecation
This article describes the change for deprecating our hetzner1 server after all its functions were migrated to the cheaper & more powerful hetzner2 server.
Status
2018-08-22 02:22 UTC
Marcin responded, authorizing ending this contract.
I replied stating that I'll pull the trigger on Thursday 2018-08-23, so there's at least 24 hours for him to tell me to abort. After all, there is no undo button after we terminate this contract!
2018-08-21
I finally confirmed today that all the (encrypted) backup archives from hetzner1 have been uploaded to our aws glacier account. Maltfield_Log/2018_Q3#Tue_Aug_21.2C_2018
[root@hetzner2 ~]# aws glacier initiate-job --account-id - --vault-name deleteMeIn2020 --job-parameters '{"Type": "inventory-retrieval"}'{
"location": "/099400651767/vaults/deleteMeIn2020/jobs/RHHVDhGRgJpG2t-ne49vKPZcZzGRSobq5XVw1pw4x7HuT1iWT9FFHC3xSl6bga_my22IEO8C5EbbxtbWBwUIslsHpYiC",
"jobId": "RHHVDhGRgJpG2t-ne49vKPZcZzGRSobq5XVw1pw4x7HuT1iWT9FFHC3xSl6bga_my22IEO8C5EbbxtbWBwUIslsHpYiC"
}
[root@hetzner2 ~]#
...
[root@hetzner2 ~]# aws glacier get-job-output --account-id - --vault-name deleteMeIn2020 --job-id 'RHHVDhGRgJpG2t-ne49vKPZcZzGRSobq5XVw1pw4x7HuT1iWT9FFHC3xSl6bga_my22IEO8C5EbbxtbWBwUIslsHpYiC' output.json
{
"status": 200,
"acceptRanges": "bytes",
"contentType": "application/json"
}
[root@hetzner2 ~]#
...
user@ose:/tmp$ cat glacierInventory.20180821.txt | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | grep -EiA 2 -B1 'microft|oseblog|osecivi|oseforum|osemain|osesurv' | grep -vi CreationDate
"ArchiveId":"NX074yaGa7FGL_QH5W9-mZ9TVmi0T2428B1lW8aEck6Ydjk3H3W6pgQTisOqE9B7azs1jykJ_IL-fdbkLzhAmrpWNGJBq5hVjfMNSP-Msm976Mf7mnXe6Z6QDkO5PVXaFsNZ1EzNyw"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_osesurv.tar.gpg"
"Size":258332
--
"ArchiveId":"uHk-GTBb6LVulxOkgs_ZYdF-cvKubUpvdP7hoS9Cqduw8YPInJaHB4LbBHpIxOL1idfYoMm-h4YI_Jq8qN3EnOBHiAjqUEwJAstagfMEvk2E38IlNLu_5J_09E0JM7MZXc4RSEZfNA"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_osesurv.fileList.txt.bz2.gpg"
"Size":7277
...
"ArchiveId":"T01giTZdzpQVhhijB47T97HEtIYDHTG7sVy5mpfUbaxBaGq5fU5C1aKleXpwTKOz7_aTiWAlkeM5rM3Lg_SS3qMI1JBeZR7l8M4W5a4JmFw3MVneRYZIC9JIuTO46F91SIa8JH1vgw"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_microft.fileList.txt.bz2.gpg"
"Size":829141
--
"ArchiveId":"GPXAjVjuNoyBjKU_Zx5wAxcjtLhsHwHXxKPuKDugGK3-jxNezXUG27MnJ5yDLay6yVJhZ_h3gCwlkd2y2gokIre6CK2wf2Ms3fk_m0BVkGI_Qx1PDRb7RL6P5l7yYeL1HWMhUfLFuw"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_oseforum.fileList.txt.bz2.gpg"
"Size":104307
--
"ArchiveId":"o2o3n7hTDoRKwTwD2IN9bHRT7Ox1K1A3ZaauheIlvQyhycJgy4mSqRusieHvYihijY9hqWaIXXDLQMAn6xa55idBgAWkuLe_Px1xNeaae7uy3nNUOb5GgPoDr8YJ9lollj3cKd_iNw"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_oseforum.tar.gpg"
"Size":1002685782
--
"ArchiveId":"EM59cEYkyc2dTk6AZjPOWDG4ftxqTxXIM5RAMCgB8xP_wMaawcz8TY8ojij-zF9qve7Ae0grQqxe1R74HLA6Yh3R7UHMueMPThlUhpW_r2atTntGZTOkJVjevCoyCkG3P23wDckMXA"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_osemain.fileList.txt.bz2.gpg"
"Size":190625
--
"ArchiveId":"vgOBLcSW8orHNlOyfPHT071fxpWtCu27wjyHoNx1Lq8V727HYmLX7JZyRXEBpszEYSKIdSU-X1DT1kzDlUeb5amFbcBU3E0s4qSja8fXz769bM89SwSNQ4gWYYgiqUqar6EbJZS2-A"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_osecivi.fileList.txt.bz2.gpg"
"Size":101912
--
"ArchiveId":"rmnC1D8xwBN4PI90pLKD-Och9gluScd7c_3tVF9dLOlEPB8Lp_f2Y0m6YwGnmQkpkc43hrPwoaYTzQWOJRMBbhN0vdLc4RT1DRhfCE68HrQM7YzsEYY7ANf4h_lfFAE7mx1JGv16Cg"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_osecivi.tar.gpg"
"Size":15254072
--
"ArchiveId":"DBhGz8QPdNcS4Zp-jLpSOfE4sOk7tiLCQsA_rCJs9nQ1722YiaXeOLSThCvFn9RaUqfRj0UmomPE9_A2bdXbxuNi2Re3Q1uxav8HHR7RkJKdNMzYeTWbbpaDSNGyJZgYVKujqrT8TQ"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_oseblog.fileList.txt.bz2.gpg"
"Size":4160970
--
"ArchiveId":"8lqDSeF3uiA9PcGb3W4hC1LwUBwT1oPLGColzsnKBq-0RpKZ4aVBMqcpKXlu5oYDGSrM4KjXnEk6ksgRkAtLuimSOWiMKUf-Nq34aDAVG5e1gmRIs8fgE2ghDaa6ToGi3os5-Q48zg"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_microft.tar.gpg"
"Size":6641139501
--
"ArchiveId":"Q9FdpUiXA32lWVEa1Xdr0vigTynWsUX5nLkzvg6QCP7LsrWpOykIHrzSZIRdSubWKlJkZ5JR6eZgln7DnPV_Wso5JcFjRMP2L0TgpAMPoOSPsrP9uet3pLQPWr_ZP7aWISR7XLjhdg"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_oseblog.tar.gpg"
"Size":4702431197
--
"ArchiveId":"e2bM8CyOWaQWtrZXk3OahrVeNpateHCkkyBd4UkBPuaJPz-HNdlnrVMA9M4nZdhqTdNMVfpsLK0HEIWxBT8zVJaHKMRdDWKSs9rb86gxyDjyIX6m4oSlik3I6EC1_ZMFhmpYKlMrPQ"
"ArchiveDescription":"hetzner1_final_backup_before_hetzner1_deprecation_osemain.tar.gpg"
"Size":3522007765
user@ose:/tmp$
I sent an email to Marcin asking if he has noticed any issues since the hetzner1 server was shut down over a month ago. If not, I gave him my blessing to cancel the contract.
2018-07-19
Per our support request, Hetzner shut off our server. Though it was not requested, they rebooted it into a "live-rescue" system. Therefore, our server still responds to pings. I did a nmap scan and found ssh to be running, albeit with a different private key (hopefully they didn't do something stupid like leave PermitRootLogin enabled with a bad default password). Anyway, all our sites on hetzner1 now appear to be inaccessible as desired.
I emailed Marcin (CCing Catarina & Tom) for notification of this server going offline & asking everyone to keep a keen eye out for anything broken. If we don't discover anything is broken in a few weeks, I think it will be safe to cancel this contract.
2018-07-16
I (Michael Altfield) finished creating backup tarballs of everything from hetzner1 & scp'd it to hetzner2 where it is being uploaded to aws glacier.
Here is a listing of the files (before encrypting & creating metadata fileList files by the uploadToGlacier.sh script):
[root@hetzner2 deprecateHetzner1]# date Mon Jul 16 18:51:55 UTC 2018 [root@hetzner2 deprecateHetzner1]# pwd /var/tmp/deprecateHetzner1 [root@hetzner2 deprecateHetzner1]# ls -lahR .: total 32K drwx------ 8 root root 4.0K Jul 16 18:29 . drwxrwxrwt. 52 root root 4.0K Jul 15 02:08 .. drwxrwxr-x 2 maltfield maltfield 4.0K Jul 11 18:13 microft drwxrwxr-x 2 maltfield maltfield 4.0K Jul 11 18:04 oseblog drwxrwxr-x 2 maltfield maltfield 4.0K Jul 6 23:38 osecivi drwxrwxr-x 2 maltfield maltfield 4.0K Jul 6 23:47 oseforum drwxrwxr-x 2 maltfield maltfield 4.0K Jul 11 17:48 osemain drwxr-xr-x 2 root root 4.0K Jul 16 18:38 osesurv ./microft: total 6.2G drwxrwxr-x 2 maltfield maltfield 4.0K Jul 11 18:13 . drwx------ 8 root root 4.0K Jul 16 18:29 .. -rw-r--r-- 1 maltfield maltfield 1.4G Jul 11 18:12 final_backup_before_hetzner1_deprecation_microft_20180706-234228_home.tar.bz2 -rw-r--r-- 1 maltfield maltfield 523K Jul 11 18:12 final_backup_before_hetzner1_deprecation_microft_20180706-234228_mysqldump-microft_db2.20180706-234228.sql.bz2 -rw-r--r-- 1 maltfield maltfield 1.3M Jul 11 18:12 final_backup_before_hetzner1_deprecation_microft_20180706-234228_mysqldump-microft_drupal1.20180706-234228.sql.bz2 -rw-r--r-- 1 maltfield maltfield 3.3G Jul 11 18:13 final_backup_before_hetzner1_deprecation_microft_20180706-234228_mysqldump-microft_wiki.20180706-234228.sql.bz2 -rw-r--r-- 1 maltfield maltfield 1.7G Jul 11 18:14 final_backup_before_hetzner1_deprecation_microft_20180706-234228_webroot.tar.bz2 ./oseblog: total 4.4G drwxrwxr-x 2 maltfield maltfield 4.0K Jul 11 18:04 . drwx------ 8 root root 4.0K Jul 16 18:29 .. -rw-r--r-- 1 maltfield maltfield 1.2G Jul 11 18:01 final_backup_before_hetzner1_deprecation_oseblog_20180706-234052_home.tar.bz2 -rw-r--r-- 1 maltfield maltfield 135M Jul 11 18:01 final_backup_before_hetzner1_deprecation_oseblog_20180706-234052_mysqldump-oseblog.20180706-234052.sql.bz2 -rw-r--r-- 1 maltfield maltfield 3.1G Jul 11 18:02 final_backup_before_hetzner1_deprecation_oseblog_20180706-234052_webroot.tar.bz2 ./osecivi: total 15M drwxrwxr-x 2 maltfield maltfield 4.0K Jul 6 23:38 . drwx------ 8 root root 4.0K Jul 16 18:29 .. -rw-r--r-- 1 maltfield maltfield 2.3M Jul 6 23:38 final_backup_before_hetzner1_deprecation_osecivi_20180706-233128_home.tar.bz2 -rw-r--r-- 1 maltfield maltfield 1.1M Jul 6 23:38 final_backup_before_hetzner1_deprecation_osecivi_20180706-233128_mysqldump-osecivi.20180706-233128.sql.bz2 -rw-r--r-- 1 maltfield maltfield 173K Jul 6 23:38 final_backup_before_hetzner1_deprecation_osecivi_20180706-233128_mysqldump-osedrupal.20180706-233128.sql.bz2 -rw-r--r-- 1 maltfield maltfield 12M Jul 6 23:38 final_backup_before_hetzner1_deprecation_osecivi_20180706-233128_webroot.tar.bz2 ./oseforum: total 955M drwxrwxr-x 2 maltfield maltfield 4.0K Jul 6 23:47 . drwx------ 8 root root 4.0K Jul 16 18:29 .. -rw-r--r-- 1 maltfield maltfield 853M Jul 6 23:47 final_backup_before_hetzner1_deprecation_oseforum_20180706-230007_home.tar.bz2 -rw-r--r-- 1 maltfield maltfield 46M Jul 6 23:47 final_backup_before_hetzner1_deprecation_oseforum_20180706-230007_mysqldump-oseforum.20180706-230007.sql.bz2 -rw-r--r-- 1 maltfield maltfield 57M Jul 6 23:47 final_backup_before_hetzner1_deprecation_oseforum_20180706-230007_webroot.tar.bz2 ./osemain: total 3.3G drwxrwxr-x 2 maltfield maltfield 4.0K Jul 11 17:48 . drwx------ 8 root root 4.0K Jul 16 18:29 .. -rw-r--r-- 1 maltfield maltfield 2.9G Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_home.tar.bz2 -rw-r--r-- 1 maltfield maltfield 1.2M Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_mysqldump-openswh.20180706-224656.sql.bz2 -rw-r--r-- 1 maltfield maltfield 187K Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_mysqldump-ose_fef.20180706-224656.sql.bz2 -rw-r--r-- 1 maltfield maltfield 157K Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_mysqldump-osesurv.20180706-224656.sql.bz2 -rw-r--r-- 1 maltfield maltfield 14 Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_mysqldump-ose_website.20180706-224656.sql.bz2 -rw-r--r-- 1 maltfield maltfield 203M Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_mysqldump-osewiki.20180706-224656.sql.bz2 -rw-r--r-- 1 maltfield maltfield 212M Jul 11 17:48 final_backup_before_hetzner1_deprecation_osemain_20180706-224656_webroot.tar.bz2 ./osesurv: total 260K drwxr-xr-x 2 root root 4.0K Jul 16 18:38 . drwx------ 8 root root 4.0K Jul 16 18:29 .. -rw-r--r-- 1 maltfield maltfield 252K Jul 16 18:37 final_backup_before_hetzner1_deprecation_osesurv_20180711-165315_home.tar.bz2 [root@hetzner2 deprecateHetzner1]# du -sh 15G .
For more information on the investigation & creation process of these archives, please see all log entries between:
I (Michael Altfield) opened a support ticket with Hetzner asking if there was a way that we could enable a firewall to drop all non-ssh traffic in & out of the server. Ultimately, they said it was not possible. Moreover, they said we didn't have the ability to toggle power on the server, but they could shut it down at our request.
I requested them to shut down hetzner1, but was very explicit that they should not cancel our contract or resuse the server; we needed the ability to boot the server again in-case we discovered any issues.
2018-07-05
Michael Altfield began initial investigation of all accounts on hetzner1 to determine what to backup to glaicer before shutting down this server.
Conclusion: ~34.87G of data (before compression) from files & databases spanning 5x distinct user accounts (osemain, osecivi, oseblog, oseforum, and microft). There was an additional user = 'osesurv' that I was unable to access.
Note that Hetzner1 is a "dedicated shared" server, so we don't have root on the box, making investigation much more difficult (the admin who setup this box has long-since left OSE).
Purpose
When I (Michael Altfield) joined OSE over a year ago, we had 1x website on Hetzner2 and 5x websites on Hetzner1.
Hetzner1 refers to a shared hosting plan with Hetzner that we began using in 2011. It does not give us root access, runs debian 8, has an AMD Athlon 64 X2 5600+ Processor, 4 GB RAM, 2x 400 GB Harddisks, and 1 Gbit/s Connection.
Hetzner2 refers to a dedicated server that we began using in 2016. It has root access, runs centos 7, and has an Intel Core i7-6700 Quad-Core Skylake4 CPU, 64G RAM, and 2x 250G SSDs in a software RAID1.
Note that Hetzner1 is less powerful *and* more expensive. Hetzner1 costs 74.79 eur/mo while Hetzner2 costs 39.00 eur/mo.
At the time of the creation of this CHG, all known production websites that existed on Hetzner1 have been fully migrated onto Hetzner2. However, there's a lot of projects and data still on Hetzner1, and it's unclear if any of it is important or not. Therefore, this CHG exists to backup all data we can find to some durable & cheap backup server (we plan to use aws glacier) prior to shutting down Hetzner1 for about a month. If a month has passed and nothing has broken, we'll terminate our contract with Hetzner1 & stop (over)paying 75 eur/mo.
Points of Contact
Change being performed by: Michael Altfield
Service owners: by Catarina Mota & Marcin Jakubowski for Open Building Institute and Open Source Ecology
