Maltfield Log/2024 Q3
My work log from the third quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
Contents
- 1 See Also
- 2 Mon Sep 30, 2024
- 3 Sun Sep 29, 2024
- 4 Sat Sep 28, 2024
- 5 Fri Sep 27, 2024
- 6 Thr Sep 26, 2024
- 7 Wed Sep 25, 2024
- 8 Tue Sep 24, 2024
- 9 Mon Sep 23, 2024
- 10 Sun Sep 22, 2024
- 11 Sat Sep 21, 2024
- 12 Fri Sep 20, 2024
- 13 Tue Sep 18, 2024
- 14 Mon Sep 17, 2024
- 15 Mon Sep 16, 2024
- 16 Sun Sep 15, 2024
- 17 Sat Sep 14, 2024
- 18 Fri Sep 13, 2024
- 19 Thr Sep 12, 2024
- 20 Wed Sep 11, 2024
- 21 Tue Sep 10, 2024
- 22 Wed Aug 07, 2024
- 23 Mon Aug 05, 2024
- 24 Sun Aug 04, 2024
- 25 Wed July 31, 2024
- 26 Tue July 30, 2024
- 27 Fri July 26, 2024
- 28 Wed July 24, 2024
See Also
Mon Sep 30, 2024
- here's TOFU 3/3 (ISP, exit in Ecuador) for our wordpress plugins & themes
Ecuador 2024-10-01 ... 2024-10-01 e39147da6f75b11e2bcd3a23ab6512dc0f1d1715cc6b25d4b0f876cd88f9851a akismet.5.3.3.zip 206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0 black-studio-tinymce-widget.2.7.3.zip 371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213 bouquet.1.2.5.zip 7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86 chartbeat.2.0.7.zip 3144da2236c6017ba1b7431363317d8db5bc541dd3c729c0aa1f597093723078 classic-editor.1.6.5.zip 67ccf342a4c7fe853e84b535bf3cf121c7a0110feaa96ae06f5eda202f8485ba coingate-for-woocommerce.2.1.1.zip 357b13dab0e2b9996444664c95bd7ee7ca49fb18aced65db4c40c747543afcf4 contact-form-7.5.9.8.zip fe9f4e60683b46eb409b9f782cbdee29e22c345430aafc898f78a1a00d88265c duplicate-page.4.5.zip a32d701644bc5ef0ac50047fcbb8b1c1ab9289d6531dfe986d6574f00d16b921 duplicate-post.4.5.zip 579488be50b99ea7401be9f8127efd7a243c8f213f8f90ede15b07577eb5e9d6 gk-portfolio.1.5.3.zip b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058 google-authenticator.0.54.zip 9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e google-authenticator-encourage-user-activation.0.2.zip 78a00cf800196b7a03599bfd8c49337a34916a09e570fd89b4c0ee82108d8f15 insert-headers-and-footers.2.2.2.zip 466b7a5a05140ce0d7ae2a2e475cdb22cd91ff7d10bc0d14068eaa8af71fb944 jetpack.13.8.1.zip 9bc91e2e6e2172f5284b24893352b8a233a24da31b34ce72a15ce584ec66d7b7 meta-box.5.10.2.zip 1f092fa51e2e3abd0c7a518db9d6ab5ee5dbc1abfa7551439350ddac9adb99fe ml-slider.3.91.0.zip c6123863c3472a40d52cba2ded434a37f6bd488f9e0c280470ccac68256f40c9 open-in-new-window-plugin.3.0.zip 33a58c7c61b1068b112b525556506e5f175988638db583883f7d2c196ef5e2d5 portfolio-press.2.8.0.zip c313fcee168a055005545c98ef665dcdc7b0693838851e5ba77a05c9b9a99fe6 post-types-order.2.2.6.zip d9740796220eb255937d81f8d2b3e7066a154bbce88f7a8d4555ad87a90b2583 revision-control.2.3.2.zip 80149aa1c6f83282d420b6c90ffd1d3c3b1ddfdcec2096c7411b568662fe0dd8 shareaholic.9.7.12.zip 24384df22558e87536aa848b21eee183ca6226fbfcea75af3821437325313ac0 share-on-diaspora.0.7.9.zip 254f144ad18558a91ed632dbb017e3a50a81872f8f9933d7ae77c85436175bd0 shariff.4.6.14.zip 1894750514b49301b3cb93ced16d48b60dee8654a5ce86f803b8454fffd9305e sketch.1.2.4.zip 122fa98f8b3674e4dfef0fa34a235db0cc6cff80c870d14d8159ba88d8796a1d ssl-insecure-content-fixer.2.7.2.zip 576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311 storefront.4.6.0.zip 02d33b647e5309a5817b58c3a51cb17ca801293a2a9653e33fd4ccd0fdb132b7 twentyeleven.4.7.zip 56309284b46e1953b855a21668cfba0f9fdd5e55acf9cdf2578f4cfd768048c7 twentyfifteen.3.8.zip 9af0e22d0a4e98535d852963e6bb23c7a2866167532df312edc27aa42257cb60 twentyfourteen.4.0.zip 8cecbed27f3af39c8964f09a2dbf2b58e50958fa7b7b00437b95fb5db891839a twentynineteen.2.9.zip 215aabae3e9cea494c012c419f31836a8ce2fb321b4210bccd05d0d263d6b727 twentyseventeen.3.7.zip 7df868e367ef21faf0ef71a0aab87dc2df8e35d53aa954165960f87d07b84f92 twentysixteen.3.3.zip 15b8b27950e68c655daed10d3ed0cd06ba55b814afed524374a68aff50856d5a twentyten.4.2.zip d1f82982f19e9aa75ae69db2eb3d76e2dd1f2cee737fbfac7390870970bbf67b twentythirteen.4.2.zip 2e699f0d0542d132aabc84f4813573aed246cac3624871299ed42d87152d4e10 twentytwelve.4.3.zip 832f4cf27722c448b44f5b7ff5583de3b5e8390ac881d37bcce4cdd5c5089be4 varnish-http-purge.5.2.2.zip 02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c vcaching.1.8.3.zip 1dbadde30b8e1e8b8785ec4ab319059e1ff158e224ce38c1df2a2688657294d9 w3-total-cache.2.7.6.zip 5c1e3005e5e5f3f614e50d0824ded56f822dd0a826fdad46a2e9c2dd76cf02be wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip a5f5ee58d239b9d170f6e6bfcbc29ae9c19d0298429ce73a8275f74e95524295 woocommerce.9.3.3.zip 9f0d2c2d82f6ec68f75307b65f723dda44ad7dce7931c307a8872d04ec9ca268 wordpress-importer.0.8.2.zip b7bda6589f00689021187d102912127fa7dc8e3289689f4d2ae2563c0a487a9c wordpress-seo.23.5.zip dfa3820f767ff202f942100aadd6e05acb144d5edaf842cfea26a6520aa71995 wpautop-control.1.6.zip fa9a177e7228bfdc6fddabb6be802a1fff32f8dddaba77cfea6c25c56788d8b0 wp-memory-usage.1.2.10.zip 79cacaea2b1f12c664e24347b2bc58f08db806bd25144f7e7332c494be115e66 wp-optimize.3.6.0.zip 17a675cb0262512d5247745bc0a1de0f72483c629b185dd43483071885d2e5eb wp-smushit.3.16.6.zip 2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d wp-super-cache.1.12.4.zip
- I confirmed that the hashes on all 3 TOFUs matched
- finally, I downloaded all the payloads on hetzner3, and I confirmed that matched all the TOFUs
root@hetzner3 /var/tmp/wordpress/plugins # sha256sum * e39147da6f75b11e2bcd3a23ab6512dc0f1d1715cc6b25d4b0f876cd88f9851a akismet.5.3.3.zip 206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0 black-studio-tinymce-widget.2.7.3.zip 371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213 bouquet.1.2.5.zip 7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86 chartbeat.2.0.7.zip 3144da2236c6017ba1b7431363317d8db5bc541dd3c729c0aa1f597093723078 classic-editor.1.6.5.zip 67ccf342a4c7fe853e84b535bf3cf121c7a0110feaa96ae06f5eda202f8485ba coingate-for-woocommerce.2.1.1.zip 357b13dab0e2b9996444664c95bd7ee7ca49fb18aced65db4c40c747543afcf4 contact-form-7.5.9.8.zip fe9f4e60683b46eb409b9f782cbdee29e22c345430aafc898f78a1a00d88265c duplicate-page.4.5.zip a32d701644bc5ef0ac50047fcbb8b1c1ab9289d6531dfe986d6574f00d16b921 duplicate-post.4.5.zip 579488be50b99ea7401be9f8127efd7a243c8f213f8f90ede15b07577eb5e9d6 gk-portfolio.1.5.3.zip b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058 google-authenticator.0.54.zip 9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e google-authenticator-encourage-user-activation.0.2.zip 78a00cf800196b7a03599bfd8c49337a34916a09e570fd89b4c0ee82108d8f15 insert-headers-and-footers.2.2.2.zip 466b7a5a05140ce0d7ae2a2e475cdb22cd91ff7d10bc0d14068eaa8af71fb944 jetpack.13.8.1.zip 9bc91e2e6e2172f5284b24893352b8a233a24da31b34ce72a15ce584ec66d7b7 meta-box.5.10.2.zip 1f092fa51e2e3abd0c7a518db9d6ab5ee5dbc1abfa7551439350ddac9adb99fe ml-slider.3.91.0.zip c6123863c3472a40d52cba2ded434a37f6bd488f9e0c280470ccac68256f40c9 open-in-new-window-plugin.3.0.zip 33a58c7c61b1068b112b525556506e5f175988638db583883f7d2c196ef5e2d5 portfolio-press.2.8.0.zip c313fcee168a055005545c98ef665dcdc7b0693838851e5ba77a05c9b9a99fe6 post-types-order.2.2.6.zip d9740796220eb255937d81f8d2b3e7066a154bbce88f7a8d4555ad87a90b2583 revision-control.2.3.2.zip 80149aa1c6f83282d420b6c90ffd1d3c3b1ddfdcec2096c7411b568662fe0dd8 shareaholic.9.7.12.zip 24384df22558e87536aa848b21eee183ca6226fbfcea75af3821437325313ac0 share-on-diaspora.0.7.9.zip 254f144ad18558a91ed632dbb017e3a50a81872f8f9933d7ae77c85436175bd0 shariff.4.6.14.zip 1894750514b49301b3cb93ced16d48b60dee8654a5ce86f803b8454fffd9305e sketch.1.2.4.zip 122fa98f8b3674e4dfef0fa34a235db0cc6cff80c870d14d8159ba88d8796a1d ssl-insecure-content-fixer.2.7.2.zip 576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311 storefront.4.6.0.zip 02d33b647e5309a5817b58c3a51cb17ca801293a2a9653e33fd4ccd0fdb132b7 twentyeleven.4.7.zip 56309284b46e1953b855a21668cfba0f9fdd5e55acf9cdf2578f4cfd768048c7 twentyfifteen.3.8.zip 9af0e22d0a4e98535d852963e6bb23c7a2866167532df312edc27aa42257cb60 twentyfourteen.4.0.zip 8cecbed27f3af39c8964f09a2dbf2b58e50958fa7b7b00437b95fb5db891839a twentynineteen.2.9.zip 215aabae3e9cea494c012c419f31836a8ce2fb321b4210bccd05d0d263d6b727 twentyseventeen.3.7.zip 7df868e367ef21faf0ef71a0aab87dc2df8e35d53aa954165960f87d07b84f92 twentysixteen.3.3.zip 15b8b27950e68c655daed10d3ed0cd06ba55b814afed524374a68aff50856d5a twentyten.4.2.zip d1f82982f19e9aa75ae69db2eb3d76e2dd1f2cee737fbfac7390870970bbf67b twentythirteen.4.2.zip 2e699f0d0542d132aabc84f4813573aed246cac3624871299ed42d87152d4e10 twentytwelve.4.3.zip 832f4cf27722c448b44f5b7ff5583de3b5e8390ac881d37bcce4cdd5c5089be4 varnish-http-purge.5.2.2.zip 02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c vcaching.1.8.3.zip 1dbadde30b8e1e8b8785ec4ab319059e1ff158e224ce38c1df2a2688657294d9 w3-total-cache.2.7.6.zip 5c1e3005e5e5f3f614e50d0824ded56f822dd0a826fdad46a2e9c2dd76cf02be wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip a5f5ee58d239b9d170f6e6bfcbc29ae9c19d0298429ce73a8275f74e95524295 woocommerce.9.3.3.zip 9f0d2c2d82f6ec68f75307b65f723dda44ad7dce7931c307a8872d04ec9ca268 wordpress-importer.0.8.2.zip b7bda6589f00689021187d102912127fa7dc8e3289689f4d2ae2563c0a487a9c wordpress-seo.23.5.zip dfa3820f767ff202f942100aadd6e05acb144d5edaf842cfea26a6520aa71995 wpautop-control.1.6.zip fa9a177e7228bfdc6fddabb6be802a1fff32f8dddaba77cfea6c25c56788d8b0 wp-memory-usage.1.2.10.zip 79cacaea2b1f12c664e24347b2bc58f08db806bd25144f7e7332c494be115e66 wp-optimize.3.6.0.zip 17a675cb0262512d5247745bc0a1de0f72483c629b185dd43483071885d2e5eb wp-smushit.3.16.6.zip 2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d wp-super-cache.1.12.4.zip root@hetzner3 /var/tmp/wordpress/plugins #
- and I separated the themes from the plugins
root@hetzner3 /var/tmp/wordpress # ls plugins/ akismet.5.3.3.zip jetpack.13.8.1.zip vcaching.1.8.3.zip black-studio-tinymce-widget.2.7.3.zip meta-box.5.10.2.zip w3-total-cache.2.7.6.zip chartbeat.2.0.7.zip ml-slider.3.91.0.zip wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip classic-editor.1.6.5.zip open-in-new-window-plugin.3.0.zip woocommerce.9.3.3.zip coingate-for-woocommerce.2.1.1.zip post-types-order.2.2.6.zip wordpress-importer.0.8.2.zip contact-form-7.5.9.8.zip revision-control.2.3.2.zip wordpress-seo.23.5.zip duplicate-page.4.5.zip shareaholic.9.7.12.zip wpautop-control.1.6.zip duplicate-post.4.5.zip share-on-diaspora.0.7.9.zip wp-memory-usage.1.2.10.zip google-authenticator.0.54.zip shariff.4.6.14.zip wp-optimize.3.6.0.zip google-authenticator-encourage-user-activation.0.2.zip ssl-insecure-content-fixer.2.7.2.zip wp-smushit.3.16.6.zip insert-headers-and-footers.2.2.2.zip varnish-http-purge.5.2.2.zip wp-super-cache.1.12.4.zip root@hetzner3 /var/tmp/wordpress # root@hetzner3 /var/tmp/wordpress # ls themes/ bouquet.1.2.5.zip sketch.1.2.4.zip twentyfifteen.3.8.zip twentyseventeen.3.7.zip twentythirteen.4.2.zip gk-portfolio.1.5.3.zip storefront.4.6.0.zip twentyfourteen.4.0.zip twentysixteen.3.3.zip twentytwelve.4.3.zip portfolio-press.2.8.0.zip twentyeleven.4.7.zip twentynineteen.2.9.zip twentyten.4.2.zip root@hetzner3 /var/tmp/wordpress #
- the last thing we're missing to be able to upgrade our wordpress sites is wordpress core!
- I downloaded wordpress-6.6.1, and I confirmed that its checksum matches what we confirmed during 3TOFU 3 weeks ago on sept 10
root@hetzner3 /var/tmp/wordpress/core # wget https://downloads.wordpress.org/release/wordpress-6.6.1.zip --2024-10-01 04:40:01-- https://downloads.wordpress.org/release/wordpress-6.6.1.zip Resolving downloads.wordpress.org (downloads.wordpress.org)... 198.143.164.250 Connecting to downloads.wordpress.org (downloads.wordpress.org)|198.143.164.250|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 26138467 (25M) [application/zip] Saving to: ‘wordpress-6.6.1.zip’ wordpress-6.6.1.zip 100%[=======================================================================>] 24,93M 13,4MB/s in 1,9s 2024-10-01 04:40:04 (13,4 MB/s) - ‘wordpress-6.6.1.zip’ saved [26138467/26138467] You have new mail in /var/mail/root root@hetzner3 /var/tmp/wordpress/core # root@hetzner3 /var/tmp/wordpress/core # sha256sum wordpress-6.6.1.zip 3757aa0f30e5e6f9952bcd08ca02c82f15b5fd25fb0cc6f9a8bc437af5a8f09f wordpress-6.6.1.zip root@hetzner3 /var/tmp/wordpress/core #
- alright, so now we have all the files we need verified & downloaded for all the wordpress sites
- for our first site to upgrade (store.opensourceecology.org) on hetzner3, here's what we need
root@hetzner3 /var/tmp/wordpress # ls /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins akismet contact-form-7 masterslider oshine-modules typehub be-gdpr force-strong-passwords meta-box redux-vendor-support vcaching be-portfolio-post google-authenticator meta-box-conditional-logic rename-wp-login woocommerce classic-editor google-authenticator-encourage-user-activation meta-box-show-hide revslider coingate-for-woocommerce hello.php meta-box-tabs ssl-insecure-content-fixer colorhub index.php oshine-core tatsu root@hetzner3 /var/tmp/wordpress # root@hetzner3 /var/tmp/wordpress # ls /var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/ index.php storefront twentyfifteen twentynineteen twentysixteen twentythirteen oshin twentyeleven twentyfourteen twentyseventeen twentyten twentytwelve root@hetzner3 /var/tmp/wordpress #
- oh cool, I just realized that wp-cli allows you to update wordpress by specifying a local .zip file of the release. That should be safe. The only thin we don't trust in wp-cli is its downloads, which it doesn't verify cryptographically https://developer.wordpress.org/cli/commands/core/update/
- I went ahead and granted the wp user access to our downladed payloads in /var/tmp/wordpress/
root@hetzner3 /var/tmp # chown -R root:wp /var/tmp/wordpress root@hetzner3 /var/tmp # root@hetzner3 /var/tmp # chmod 0770 /var/tmp/wordpress root@hetzner3 /var/tmp #
- well, fuck, I didn't think that one through. Of course it'll fail with the same PHP fatal error that is making wordpress itself broken now
maltfield@hetzner3:~$ sudo su - wp wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs core update /var/tmp/wordpress/core/wordpress-6.6.1.zip PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/script-loader.php on line 706 Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/script-loader.php on line 706 wp@hetzner3:~$
- that puts us in a catch-22; we just have to upgrade manually, it seems
- the docs describe a complicated process https://wordpress.org/documentation/article/updating-wordpress/#manual-update
- but SE says you can literally just overwrite the files. I'll try the latter https://wordpress.org/documentation/article/updating-wordpress/#manual-update
root@hetzner3 /var/tmp/wordpress/core # rsync -av --progress wordpress/ /var/www/html/store.opensourceecology.org/htdocs/
...
wp-includes/widgets/class-wp-widget-text.php
21.459 100% 46,47kB/s 0:00:00 (xfr#3008, to-chk=0/3317)
sent 71.113.087 bytes received 59.171 bytes 142.344.516,00 bytes/sec
total size is 70.887.875 speedup is 1,00
root@hetzner3 /var/tmp/wordpress/core #
- then I executed the permissions update https://wiki.opensourceecology.org/wiki/Wordpress#Proper_File.2FDirectory_Ownership_.26_Permissions
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
- and here's the result
root@hetzner3 /var/tmp/wordpress/core # ls -lah /var/www/html/store.opensourceecology.org/htdocs/ total 252K d---r-x--- 6 not-apache www-data 4,0K Jul 23 15:15 . d---r-x--- 4 not-apache www-data 4,0K Apr 9 2019 .. ----r----- 1 not-apache www-data 234 Apr 9 2019 .htaccess ----r----- 1 not-apache www-data 405 Feb 6 2020 index.php ----r----- 1 not-apache www-data 5 Sep 27 04:44 is_hetzner3 ----r----- 1 not-apache www-data 20K Jan 1 2024 license.txt ----r----- 1 not-apache www-data 7,3K Jun 18 11:59 readme.html d---r-x--- 4 not-apache www-data 4,0K Apr 9 2019 .svn ----r----- 1 not-apache www-data 7,3K Feb 13 2024 wp-activate.php d---r-x--- 9 not-apache www-data 4,0K Jul 23 15:15 wp-admin ----r----- 1 not-apache www-data 351 Feb 6 2020 wp-blog-header.php ----r----- 1 not-apache www-data 2,3K Jun 14 2023 wp-comments-post.php ----r----- 1 not-apache www-data 3,0K Mar 11 2024 wp-config-sample.php d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 wp-content ----r----- 1 not-apache www-data 5,6K May 30 2023 wp-cron.php d---r-x--- 31 not-apache www-data 12K Jul 23 15:15 wp-includes ----r----- 1 not-apache www-data 2,5K Nov 26 2022 wp-links-opml.php ----r----- 1 not-apache www-data 3,9K Mar 11 2024 wp-load.php ----r----- 1 not-apache www-data 51K May 28 11:13 wp-login.php ----r----- 1 not-apache www-data 8,4K Sep 16 2023 wp-mail.php ----r----- 1 not-apache www-data 29K Jul 9 15:43 wp-settings.php ----r----- 1 not-apache www-data 34K Jun 19 2023 wp-signup.php ----r----- 1 not-apache www-data 4,8K Jun 22 2023 wp-trackback.php ----r----- 1 not-apache www-data 3,2K Mar 2 2024 xmlrpc.php root@hetzner3 /var/tmp/wordpress/core #
- well, now wordpress has a different fatal error -- caused by a plugin (as opposed to core)
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
PHP Fatal error: Uncaught TypeError: round(): Argument #1 ($num) must be of type int|float, string given in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/slt-force-strong-passwords.php:46
Stack trace:
#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/slt-force-strong-passwords.php(46): round()
#1 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(522): include_once('...')
#2 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1375): require('...')
#3 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1294): WP_CLI\Runner->load_wordpress()
#4 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(28): WP_CLI\Runner->start()
#5 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/bootstrap.php(83): WP_CLI\Bootstrap\LaunchRunner->process()
#6 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/wp-cli.php(32): WP_CLI\bootstrap()
#7 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/php/boot-phar.php(20): include('...')
#8 /home/wp/.wp-cli/wp-cli-2.11.0.phar(4): include('...')
#9 {main}
thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/slt-force-strong-passwords.php on line 46
Fatal error: Uncaught TypeError: round(): Argument #1 ($num) must be of type int|float, string given in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/slt-force-strong-passwords.php:46
Stack trace:
#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/slt-force-strong-passwords.php(46): round()
#1 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(522): include_once('...')
#2 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1375): require('...')
#3 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1294): WP_CLI\Runner->load_wordpress()
#4 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(28): WP_CLI\Runner->start()
#5 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/bootstrap.php(83): WP_CLI\Bootstrap\LaunchRunner->process()
#6 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/wp-cli.php(32): WP_CLI\bootstrap()
#7 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/php/boot-phar.php(20): include('...')
#8 /home/wp/.wp-cli/wp-cli-2.11.0.phar(4): include('...')
#9 {main}
thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/slt-force-strong-passwords.php on line 46
Error: There has been a critical error on this website.Learn more about troubleshooting WordPress. There has been a critical error on this website.
wp@hetzner3:~$
- well, this was one of the plugins that we don't have updates for anymore, so I'll just delete it
root@hetzner3 /var/tmp # ls -lah /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords/ total 48K d---r-x--- 3 not-apache www-data 4,0K Apr 9 2019 . d---r-x--- 27 not-apache www-data 4,0K Jul 23 15:15 .. ----r----- 1 not-apache www-data 459 Apr 9 2019 force-zxcvbn.js ----r----- 1 not-apache www-data 227 Apr 9 2019 force-zxcvbn.min.js ----r----- 1 not-apache www-data 65 Apr 9 2019 .gitignore ----r----- 1 not-apache www-data 559 Apr 9 2019 js-admin.js ----r----- 1 not-apache www-data 432 Apr 9 2019 js-admin.min.js d---r-x--- 2 not-apache www-data 4,0K Apr 9 2019 languages ----r----- 1 not-apache www-data 5,6K Apr 9 2019 readme.txt ----r----- 1 not-apache www-data 7,2K Apr 9 2019 slt-force-strong-passwords.php root@hetzner3 /var/tmp # rm -rf /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/force-strong-passwords root@hetzner3 /var/tmp #
- now it's complaining about syntax of the woocommerce plugin
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/woocommerce/includes/wc-formatting-functions.php on line 776 Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/woocommerce/includes/wc-formatting-functions.php on line 776 Error: There has been a critical error on this website.Learn more about troubleshooting WordPress. There has been a critical error on this website. wp@hetzner3:~$
- I should probably write some quick script that loops through each of these dirs in 'plugins' and 'themes' and then [a] deletes the dir and [b] attempts to copy the dir from /var/tmp/wordpress/X/Y
Sun Sep 29, 2024
- well, the mdadm process never restarted since I killed it last night. I don't know if this is a problem or not
- I'm not worried about syncing being stopped, but I am worried about monitoring being stopped. But I guess somewhere in systemd it will kick it off again? I think it just needs to run once a week or month or something
root@hetzner3 ~ # /sbin/mdadm --monitor --scan --test
mdadm: Only one autorebuild process allowed in scan mode, aborting
root@hetzner3 ~ #
root@hetzner3 ~ # killall mdadm
root@hetzner3 ~ #
root@hetzner3 ~ # /sbin/mdadm --monitor --scan --test
^C
root@hetzner3 ~ #
root@hetzner3 ~ # ps -ef | grep mdadm
root 1916477 1543052 0 18:35 pts/107 00:00:00 grep mdadm
You have new mail in /var/mail/root
root@hetzner3 ~ #
root@hetzner3 ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
bitmap: 2/4 pages [8KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
root@hetzner3 ~ #
- alright, so this one says it kicks-off the first sunday of every month at 1 AM https://wiki.archlinux.org/title/Systemd/Timers
root@hetzner3 ~ # cat /lib/systemd/system/mdcheck_start.timer # This file is part of mdadm. # # mdadm is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=MD array scrubbing [Timer] OnCalendar=Sun *-*-1..7 1:00:00 [Install] WantedBy= mdmonitor.service Also= mdcheck_continue.timer root@hetzner3 ~ #
- some others
root@hetzner3 /lib/systemd/system # grep -irl 'mdadm' * mdadm-grow-continue@.service mdadm-last-resort@.service mdadm-shutdown.service mdcheck_continue.service mdcheck_continue.timer mdcheck_start.service mdcheck_start.timer mdmonitor-oneshot.service mdmonitor-oneshot.timer mdmonitor.service mdmon@.service root@hetzner3 /lib/systemd/system #
- there's this mdcheck unit
root@hetzner3 /lib/systemd/system # cat mdcheck_start.service
# This file is part of mdadm.
#
# mdadm is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
[Unit]
Description=MD array scrubbing
Wants=mdcheck_continue.timer
Documentation=man:mdadm(8)
[Service]
Type=oneshot
Environment="MDADM_CHECK_DURATION=6 hours"
EnvironmentFile=-/run/sysconfig/mdadm
ExecStartPre=-/usr/lib/mdadm/mdadm_env.sh
ExecStart=/usr/share/mdadm/mdcheck --duration ${MDADM_CHECK_DURATION}
root@hetzner3 /lib/systemd/system #
root@hetzner3 /lib/systemd/system # cat mdcheck_start.timer
# This file is part of mdadm.
#
# mdadm is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
[Unit]
Description=MD array scrubbing
[Timer]
OnCalendar=Sun *-*-1..7 1:00:00
[Install]
WantedBy= mdmonitor.service
Also= mdcheck_continue.timer
root@hetzner3 /lib/systemd/system #
- but note that "mdadm --check" is distinct from "mdcheck". And there's also "mdmon" :/
root@hetzner3 /lib/systemd/system # ls -lah /usr/share/mdadm total 24K drwxr-xr-x 2 root root 4,0K Jun 23 04:17 . drwxr-xr-x 115 root root 4,0K Sep 25 22:09 .. -rwxr-xr-x 1 root root 6,5K Feb 24 2023 checkarray -rwxr-xr-x 1 root root 3,8K Dec 30 2021 mdcheck -rwxr-xr-x 1 root root 2,8K Feb 24 2023 mkconf root@hetzner3 /lib/systemd/system # root@hetzner3 /lib/systemd/system # ls -lah /sbin/md* -rwxr-xr-x 1 root root 602K Feb 24 2023 /sbin/mdadm -rwxr-xr-x 1 root root 334K Feb 24 2023 /sbin/mdmon root@hetzner3 /lib/systemd/system #
- the description of the mdmonitor systemd unit (Reminder for degraded MD arrays) does sound like what we want, and it runs every day at 2 AM
root@hetzner3 /lib/systemd/system # cat mdmonitor-oneshot.service # This file is part of mdadm. # # mdadm is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=Reminder for degraded MD arrays Documentation=man:mdadm(8) [Service] Environment=MDADM_MONITOR_ARGS=--scan EnvironmentFile=-/run/sysconfig/mdadm ExecStartPre=-/usr/lib/mdadm/mdadm_env.sh ExecStart=/sbin/mdadm --monitor --oneshot $MDADM_MONITOR_ARGS root@hetzner3 /lib/systemd/system # cat mdmonitor-oneshot.timer # This file is part of mdadm. # # mdadm is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=Reminder for degraded MD arrays [Timer] OnCalendar= 2:00:00 [Install] WantedBy= mdmonitor.service root@hetzner3 /lib/systemd/system #
- but that's just "oneshot" -- whatever that means
- oh, that means it only checks once. But if it's wrapped with systemd and called once per day, that should be fine https://explainshell.com/explain/8/mdadm
- there's another one that's exactly what I was looking-for, but there's no cooresponding timer
root@hetzner3 /lib/systemd/system # cat mdmon@.service # This file is part of mdadm. # # mdadm is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=MD Metadata Monitor on /dev/%I DefaultDependencies=no Before=initrd-switch-root.target Documentation=man:mdmon(8) [Service] # mdmon should never complain due to lack of a platform, # that is mdadm's job if at all. Environment=IMSM_NO_PLATFORM=1 # The mdmon starting in the initramfs (with dracut at least) # cannot see sysfs after root is mounted, so we will have to # 'takeover'. As the '--offroot --takeover' don't hurt when # not necessary, are are useful with root-on-md in dracut, # have them always present. ExecStart=/sbin/mdmon --offroot --takeover %I Type=forking # Don't set the PIDFile. It isn't necessary (systemd can work # it out) and systemd will remove it when transitioning from # initramfs to rootfs. #PIDFile=/run/mdadm/%I.pid KillMode=none root@hetzner3 /lib/systemd/system #
- looks like it's disabled
root@hetzner3 /lib/systemd/system # systemctl status mdmon Unit mdmon.service could not be found. root@hetzner3 /lib/systemd/system # systemctl status mdmon@ Failed to get properties: Unit name mdmon@.service is neither a valid invocation ID nor unit name. root@hetzner3 /lib/systemd/system # systemctl status mdmon@.service Failed to get properties: Unit name mdmon@.service is neither a valid invocation ID nor unit name. root@hetzner3 /lib/systemd/system # root@hetzner3 /lib/systemd/system # systemctl list-units | grep mdmon root@hetzner3 /lib/systemd/system # systemctl list-units | grep mdadm root@hetzner3 /lib/systemd/system #
- anyway, the oneshot that runs once per day satisfies me enough
- I started building an ansible role for configuring mdadm, but I decided against it .. since we don't want to list email addresses on github. So I'll just edit this manually
root@hetzner3 ~ # cd /etc/mdadm/ root@hetzner3 /etc/mdadm # cp mdadm.conf mdadm.conf.20240929.orig root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # vim mdadm.conf root@hetzner3 /etc/mdadm #
- I updated it with MAILFROM and MAILADDR to my OSE email address, and it worked; I got three email messages in my inbox almost immediately
- I changed it from my own email to the shared ops email, and ran the test again
root@hetzner3 /etc/mdadm # vim mdadm.conf root@hetzner3 /etc/mdadm # oot@hetzner3 /etc/mdadm # git diff mdadm.conf.20240929.orig mdadm.conf diff --git a/mdadm.conf.20240929.orig b/mdadm.conf index 1312105..ae12f08 100644 --- a/mdadm.conf.20240929.orig +++ b/mdadm.conf @@ -15,7 +15,8 @@ HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts -MAILADDR root +MAILFROM REDACTED@hetzner3.opensourceecology.org +MAILADDR REDACTED@opensourceecology.org # definitions of existing MD arrays ARRAY /dev/md/0 metadata=1.2 UUID=01010ebe:30bb5ba3:c2fb6223:c8c1591d name=rescue:0 root@hetzner3 /etc/mdadm # root@hetzner3 ~ # /sbin/mdadm --monitor --scan --test ^C root@hetzner3 ~ #
- this time it was less immediate (it took maybe 30 seconds), but I got three emails again. Perfect.
- oh, actually, this test alert shows that it's currently in the middle of doing a check on nvme1n1p3, which is our 443G '/' main root partition. Great :)
This is an automatically generated mail message from mdadm
running on hetzner3
A TestMessage event had been detected on md device /dev/md/2.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
[==========>..........] check = 53.9% (250981760/465370432) finish=16.5min speed=216204K/sec
bitmap: 3/4 pages [12KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
resync=DELAYED
unused devices: <none>
--
You received this message because you are subscribed to the Google Groups "ops-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ops-list+unsubscribe@opensourceecology.org.
To view this discussion on the web visit https://groups.google.com/a/opensourceecology.org/d/msgid/ops-list/20240929191605.D986EB8838C%40hetzner3.opensourceecology.org.
For more options, visit https://groups.google.com/a/opensourceecology.org/d/optout.
- I don't even see the process, but it's definitely running (the percent increased)
root@hetzner3 ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
[============>........] check = 61.6% (286869888/465370432) finish=14.4min speed=206492K/sec
bitmap: 2/4 pages [8KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
resync=DELAYED
unused devices: <none>
root@hetzner3 ~ # ps -ef | grep md
root 1 0 0 Sep15 ? 00:01:29 /lib/systemd/systemd --system --deserialize=39
root 69 2 0 Sep15 ? 00:00:00 [ksmd]
root 273 2 0 Sep15 ? 00:00:00 [md]
root 291 2 0 Sep15 ? 00:00:15 [md0_raid1]
root 292 2 0 Sep15 ? 00:00:00 [md1_raid1]
root 294 2 0 Sep15 ? 00:04:45 [md2_raid1]
root 377 2 0 Sep15 ? 00:01:40 [jbd2/md2-8]
root 430 1 0 Sep15 ? 00:00:40 /lib/systemd/systemd-journald
root 447 1 0 Sep15 ? 00:00:01 /lib/systemd/systemd-udevd
root 517 2 0 Sep15 ? 00:00:00 [jbd2/md1-8]
systemd+ 541 1 0 Sep15 ? 00:00:09 /lib/systemd/systemd-timesyncd
message+ 619 1 0 Sep15 ? 00:03:09 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 624 1 0 Sep15 ? 00:01:23 /lib/systemd/systemd-logind
maltfie+ 1587 1 0 Sep15 ? 00:00:02 /lib/systemd/systemd --user
root 117518 2 0 Sep25 ? 00:00:00 [dio/md2]
root 1920464 2 0 18:52 ? 00:00:00 [md1_resync]
root 1920471 2 10 18:52 ? 00:02:46 [md2_resync]
root 1927022 1543052 0 19:19 pts/107 00:00:00 grep md
root@hetzner3 ~ # cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
[============>........] check = 62.7% (291853824/465370432) finish=13.2min speed=218240K/sec
bitmap: 2/4 pages [8KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
resync=DELAYED
unused devices: <none>
root@hetzner3 ~ #
- finally, I sent a support request to hetzner asking them if they are already monitoring for hardware signs of disk failure, and what happens if a drive does fail
Hi Hetzner, Can you please tell us more about the process of disk failure on our new dedicated server plan (Server Auction REDACTED)? Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk? Or do we have to pay a fee? If so, how much? And does Hetzner have some system in-place that monitors the hardware for disk failure? Or do we have to monitor this in software and alert Hetnzer that a disk is failing? If Hetzner does monitor for disk failure, how does it do it? Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
Sat Sep 28, 2024
- while I wait for the 3TOFUs of wordpress plugins & themes to finish, I'll take a look at awstats and munin
- first, I copied the hetzner2 .htpasswd file to /var/www/html/.htpasswd
root@hetzner3 /var/www/html # ls -lah total 20K d---r-x--- 4 root www-data 4,0K Sep 28 22:19 . drwxr-xr-x 3 root root 4,0K Sep 25 02:37 .. d---r-x--- 5 root www-data 4,0K Jul 11 2018 forum.opensourceecology.org -rw------- 1 root www-data 138 Mar 3 2018 .htpasswd d---r-x--- 4 root www-data 4,0K Apr 9 2019 store.opensourceecology.org root@hetzner3 /var/www/html #
- I updated my laptop's /etc/hosts to include munin
user@disp3202:~$ cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback disp3202 ff02::1 ip6-allnodes ff02::2 ip6-allrouters 144.76.164.201 forum.opensourceecology.org 144.76.164.201 store.opensourceecology.org 144.76.164.201 munin.opensourceecology.org 127.0.1.1 disp3202 user@disp3202:~$
- curl demands auth; that's a good sign
user@disp3202:~$ curl -i https://munin.opensourceecology.org:4443 HTTP/1.1 401 Unauthorized Server: nginx Date: Sat, 28 Sep 2024 22:24:36 GMT Content-Type: text/html Content-Length: 172 Connection: keep-alive WWW-Authenticate: Basic realm="auth required" <html> <head><title>401 Authorization Required</title></head> <body> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx</center> </body> </html> user@disp3202:~$
- I tried in the web browser. I typed my username/password, and then I got a 500 Internal Server Error from nginx :(
- nginx logs say it doesn't like the permissions on the htpasswd file
==> nginx/munin.opensourceecology.org/error.log <== 2024/09/28 22:27:24 [crit] 1033409#1033409: *2487 open() "/var/www/html/.htpasswd" failed (13: Permission denied), client: 66.115.189.137, server: munin.opensourceecology.org, request: "GET / HTTP/1.1", host: "munin.opensourceecology.org:4443" ==> nginx/munin.opensourceecology.org/access.log <== 66.115.189.137 - admin [28/Sep/2024:22:27:24 +0000] "GET / HTTP/1.1" 500 170 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "-"
- oh, I set it (above) to root:www-data 0600. That's dumb. It should be 0640
root@hetzner3 /var/www/html # chmod 0640 .htpasswd root@hetzner3 /var/www/html # ls -lah .htpasswd -rw-r----- 1 root www-data 138 Mar 3 2018 .htpasswd root@hetzner3 /var/www/html #
- cool, I refresh and now the 501 goes away
- ...but the page is pretty broken. for some reason lots of the charts are missing, replaced with just a hyperlink and a big white gap https://munin.opensourceecology.org:4443/localdomain/localhost.localdomain/index.html
- oh, looks like nginx is too aggressively rate-limiting, including the css -- which is why the layout is broken
==> nginx/munin.opensourceecology.org/error.log <== 2024/09/28 22:31:22 [error] 1033411#1033411: *2493 limiting requests, excess: 0.380 by zone "one", client: 66.115.189.137, server: munin.opensourceecology.org, request: "GET /static/style-new.css HTTP/1.1", host: "munin.opensourceecology.org:4443", referrer: "https://munin.opensourceecology.org:4443/localdomain/localhost.localdomain/index.html" ==> nginx/munin.opensourceecology.org/access.log <== 66.115.189.137 - admin [28/Sep/2024:22:31:22 +0000] "GET /static/style-new.css HTTP/1.1" 503 190 "https://munin.opensourceecology.org:4443/localdomain/localhost.localdomain/index.html" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0" "-"
- looks like the rate limit is set in the main nginx.conf file, so this should affect all our sites
root@hetzner3 /etc/nginx # grep -ir 'limit' * | grep -i zone | grep -i one nginx.conf: limit_conn_zone $binary_remote_addr zone=addr:10m; nginx.conf: limit_req_zone $binary_remote_addr zone=one:10m rate=120r/m; nginx.conf: limit_req zone=one; root@hetzner3 /etc/nginx #
- am I reading this right? The limit is 120 requests per minute? That's 2 requests per second.
- oh, after reading the nginx guide, I understand what's wrong https://blog.nginx.org/blog/rate-limiting-nginx
- nginx tracks requests at a millisecond granularity, so even if you say "120 requests per minute", it converts it down to X requests per millisecond
- we also need to take account for bursts. normally we'd expect a user to load a page, download a bunch of things, then have a pause while they read and navigate, then they click something and we get another burst of requests
- what we want to limit is a single IP that's just constantly downloading without pauses
- munin is a good example; it actually downloads 122 requests on first page load!
root@hetzner3 /var/log # grep '22:55' nginx/munin.opensourceecology.org/access.log | wc -l 122 root@hetzner3 /var/log #
- I decided to use a "300" burst with "100" delay. That means that nginx will quickly process 200 requests in excess of the 120 r/s rate-limit, and it will slowly process the next 100 requests. After 300 requests, it will throw 503 errors until the leaky bucket is drained.
- I updated this in ansible and pushed it out
diff --git a/hetzner3/roles/maltfield.nginx/templates/nginx.conf.j2 b/hetzner3/roles/maltfie
ld.nginx/templates/nginx.conf.j2
index 4dfab0b..3826ccf 100644
--- a/hetzner3/roles/maltfield.nginx/templates/nginx.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/nginx.conf.j2
@@ -66,7 +66,7 @@ http {
# only permit a max of 30 requests per min from each client ip address
# prevents DOS
limit_req_zone $binary_remote_addr zone=one:10m rate=120r/m;
- limit_req zone=one;
+ limit_req zone=one burst=300 delay=100;
- cool, so I can load munin now, but there's some noticible things missing:
- apache
- nginx
- mysql
- varnish
- memory rss (multips_memory)
- proc
- ps
- there's also a new one that I haven't seen before called "CPU frequency scaling"
- anyway, it does look like ansible installed the plugins
root@hetzner3 /etc/munin # ls -lah plugins total 8,0K drwxr-xr-x 2 root root 4,0K Sep 25 01:47 . drwxr-xr-x 7 root root 4,0K Sep 25 01:47 .. lrwxrwxrwx 1 root root 29 Sep 25 01:47 acpi -> /usr/share/munin/plugins/acpi lrwxrwxrwx 1 root root 40 Sep 25 01:47 apache_accesses -> /usr/share/munin/plugins/apache_accesses lrwxrwxrwx 1 root root 41 Sep 25 01:47 apache_processes -> /usr/share/munin/plugins/apache_processes lrwxrwxrwx 1 root root 38 Sep 25 01:47 apache_volume -> /usr/share/munin/plugins/apache_volume lrwxrwxrwx 1 root root 28 Sep 25 01:47 cpu -> /usr/share/munin/plugins/cpu lrwxrwxrwx 1 root root 33 Sep 25 01:47 cpuspeed -> /usr/share/munin/plugins/cpuspeed lrwxrwxrwx 1 root root 27 Sep 25 01:47 df -> /usr/share/munin/plugins/df lrwxrwxrwx 1 root root 33 Sep 25 01:47 df_inode -> /usr/share/munin/plugins/df_inode lrwxrwxrwx 1 root root 34 Sep 25 01:47 diskstats -> /usr/share/munin/plugins/diskstats lrwxrwxrwx 1 root root 32 Sep 25 01:47 entropy -> /usr/share/munin/plugins/entropy lrwxrwxrwx 1 root root 30 Sep 25 01:47 forks -> /usr/share/munin/plugins/forks lrwxrwxrwx 1 root root 37 Sep 25 01:47 fw_conntrack -> /usr/share/munin/plugins/fw_conntrack lrwxrwxrwx 1 root root 43 Sep 25 01:47 fw_forwarded_local -> /usr/share/munin/plugins/fw_forwarded_local lrwxrwxrwx 1 root root 35 Sep 25 01:47 fw_packets -> /usr/share/munin/plugins/fw_packets lrwxrwxrwx 1 root root 38 Sep 25 01:47 http_loadtime -> /usr/share/munin/plugins/http_loadtime lrwxrwxrwx 1 root root 28 Sep 25 01:47 if_enp0s31f6 -> /usr/share/munin/plugins/if_ lrwxrwxrwx 1 root root 32 Sep 25 01:47 if_err_enp0s31f6 -> /usr/share/munin/plugins/if_err_ lrwxrwxrwx 1 root root 35 Sep 25 01:47 interrupts -> /usr/share/munin/plugins/interrupts lrwxrwxrwx 1 root root 33 Sep 25 01:47 irqstats -> /usr/share/munin/plugins/irqstats lrwxrwxrwx 1 root root 29 Sep 25 01:47 load -> /usr/share/munin/plugins/load lrwxrwxrwx 1 root root 31 Sep 25 01:47 memory -> /usr/share/munin/plugins/memory lrwxrwxrwx 1 root root 39 Sep 25 01:47 multips_memory -> /usr/share/munin/plugins/multips_memory lrwxrwxrwx 1 root root 36 Sep 25 01:47 munin_stats -> /usr/share/munin/plugins/munin_stats lrwxrwxrwx 1 root root 31 Sep 25 01:47 mysql_ -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 36 Sep 25 01:47 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes lrwxrwxrwx 1 root root 37 Sep 25 01:47 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb lrwxrwxrwx 1 root root 42 Sep 25 01:47 mysql_isam_space_ -> /usr/share/munin/plugins/mysql_isam_space_ lrwxrwxrwx 1 root root 38 Sep 25 01:47 mysql_queries -> /usr/share/munin/plugins/mysql_queries lrwxrwxrwx 1 root root 42 Sep 25 01:47 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries lrwxrwxrwx 1 root root 38 Sep 25 01:47 mysql_threads -> /usr/share/munin/plugins/mysql_threads lrwxrwxrwx 1 root root 32 Sep 25 01:47 netstat -> /usr/share/munin/plugins/netstat lrwxrwxrwx 1 root root 38 Sep 25 01:47 nginx_request -> /usr/share/munin/plugins/nginx_request lrwxrwxrwx 1 root root 37 Sep 25 01:47 nginx_status -> /usr/share/munin/plugins/nginx_status lrwxrwxrwx 1 root root 35 Sep 25 01:47 open_files -> /usr/share/munin/plugins/open_files lrwxrwxrwx 1 root root 36 Sep 25 01:47 open_inodes -> /usr/share/munin/plugins/open_inodes lrwxrwxrwx 1 root root 42 Sep 25 01:47 postfix_mailqueue -> /usr/share/munin/plugins/postfix_mailqueue lrwxrwxrwx 1 root root 29 Sep 25 01:47 proc -> /usr/share/munin/plugins/proc lrwxrwxrwx 1 root root 34 Sep 25 01:47 processes -> /usr/share/munin/plugins/processes lrwxrwxrwx 1 root root 33 Sep 25 01:47 proc_pri -> /usr/share/munin/plugins/proc_pri lrwxrwxrwx 1 root root 28 Sep 25 01:47 ps_ -> /usr/share/munin/plugins/ps_ lrwxrwxrwx 1 root root 29 Sep 25 01:47 swap -> /usr/share/munin/plugins/swap lrwxrwxrwx 1 root root 32 Sep 25 01:47 threads -> /usr/share/munin/plugins/threads lrwxrwxrwx 1 root root 31 Sep 25 01:47 uptime -> /usr/share/munin/plugins/uptime lrwxrwxrwx 1 root root 30 Sep 25 01:47 users -> /usr/share/munin/plugins/users lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_backend_traffic -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_bad -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_expunge -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_hit_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_memory_usage -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_objects -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_request_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_threads -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_transfer_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_uptime -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 31 Sep 25 01:47 vmstat -> /usr/share/munin/plugins/vmstat root@hetzner3 /etc/munin #
- I gave it a restart
root@hetzner3 /etc/munin # systemctl restart munin root@hetzner3 /etc/munin # root@hetzner3 /etc/munin # service munin-node restart root@hetzner3 /etc/munin # root@hetzner3 /etc/munin # sudo -u munin /usr/bin/munin-cron You have new mail in /var/mail/root root@hetzner3 /etc/munin #
- I refreshed munin in the web browser, and now I have a lot more charts. cool.
- now everything appears except for varnish
- this is a known bug that I reported back in 2020 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959812
- the solution is to just delete munin's varnish plugin, and replace it with this one
- I've already been through this on my personal server; here's the hash of the file I have there
root@mail:~# sha256sum /usr/share/munin/plugins/varnish5_ 3009548b40e04b9709384089ec4b23af377cabb191a4ea9e20c64b667b20b51a /usr/share/munin/plugins/varnish5_ root@mail:~#
- I copied and pasted the file from the two websites above, and -- fuck -- all three files are unique
root@hetzner3 /var/tmp/munin/varnish # sha256sum */varnish5_ 863dc8555114cfe9f625d0cc586f3a79ffb9203562495cb6eb6bf0c5bf812c17 github.com/varnish5_ 75eb64ad903562548fe85e7e66cb4924c2fed8ab3491fd31fe96a96394731a20 munin-monitoring.org/varnish5_ root@hetzner3 /var/tmp/munin/varnish #
- good news: they all have different numbers of newlines at the end
- my server has only 1, so I edited the ones on hetzner3 to also only have 1 newline at the end of their files
root@mail:~# tail /usr/share/munin/plugins/varnish5_
if ($DEBUG) {
print STDERR "Error: $value not part of "
. "varnishstat.\n";
}
next;
}
print "$pvalue.value ";
print "$data{$value}{'value'}\n";
}
}
root@mail:~#
- cool, now they're all identical
root@hetzner3 /var/tmp/munin/varnish # sha256sum */varnish5_ 3009548b40e04b9709384089ec4b23af377cabb191a4ea9e20c64b667b20b51a github.com/varnish5_ 3009548b40e04b9709384089ec4b23af377cabb191a4ea9e20c64b667b20b51a munin-monitoring.org/varnish5_ root@hetzner3 /var/tmp/munin/varnish #
- I replaced the debian-provided one with the one we just downloaded & confirmed
root@hetzner3 /var/tmp/munin/varnish # cd /usr/share/munin/plugins/ root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # mv varnish5_ varnish5_.20240928.orig root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # cp /var/tmp/munin/varnish/github.com/varnish5_ . root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # ls -lah | head total 1,6M drwxr-xr-x 2 root root 12K Sep 28 23:48 . drwxr-xr-x 3 root root 4,0K Sep 25 01:47 .. -rwxr-xr-x 1 root root 2,0K Mar 21 2023 acpi -rwxr-xr-x 1 root root 3,2K Mar 21 2023 amavis -rwxr-xr-x 1 root root 5,1K Mar 21 2023 apache_accesses -rwxr-xr-x 1 root root 6,5K Mar 21 2023 apache_processes -rwxr-xr-x 1 root root 5,1K Mar 21 2023 apache_volume -rwxr-xr-x 1 root root 2,3K Mar 21 2023 apc_envunit_ -rwxr-xr-x 1 root root 3,7K Mar 21 2023 apc_nis root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # ls -lah varnish5_ -rw-r--r-- 1 root root 28K Sep 28 23:48 varnish5_ root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # chmod 0755 varnish5_ root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # ls -lah varnish5_ -rwxr-xr-x 1 root root 28K Sep 28 23:48 varnish5_ root@hetzner3 /usr/share/munin/plugins #
- now when I refresh munin, a new section titled "webserver" appears -- that should be our data
- these new charts are mostly still empty, and I won't be able to verify them until they get some data (or don't), so I'll have to revisit this tomorrow
...
- what about awstats?
- first, I added it to my laptop /etc/hosts
user@disp3202:~$ cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback disp3202 ff02::1 ip6-allnodes ff02::2 ip6-allrouters 144.76.164.201 forum.opensourceecology.org 144.76.164.201 store.opensourceecology.org 144.76.164.201 munin.opensourceecology.org 144.76.164.201 awstats.opensourceecology.org 127.0.1.1 disp3202 user@disp3202:~$
- curl gives me an auth challenge, good
user@disp3202:~$ curl -i https://awstats.opensourceecology.org:4443 HTTP/1.1 401 Unauthorized Server: nginx Date: Sat, 28 Sep 2024 23:53:24 GMT Content-Type: text/html Content-Length: 172 Connection: keep-alive WWW-Authenticate: Basic realm="auth required" <html> <head><title>401 Authorization Required</title></head> <body> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx</center> </body> </html> user@disp3202:~$
- I auth'd in the browser, and I get a directory of files
- I can browse to the forum's Sept 2024 index, but I get a 404 when I click on it :/ https://awstats.opensourceecology.org:4443/forum.opensourceecology.org/2024/09/index.en.html
- nginx logs say it is, in fact, unable to find the file
==> awstats.opensourceecology.org/error.log <== 2024/09/28 23:56:09 [error] 1283460#1283460: *139 open() "/var/cache/awstats/forum.opensourceecology.org/2024/09/index.en.html" failed (2: No such file or directory), client: 66.115.189.137, server: awstats.opensourceecology.org, request: "GET /forum.opensourceecology.org/2024/09/index.en.html HTTP/1.1", host: "awstats.opensourceecology.org:4443", referrer: "https://awstats.opensourceecology.org:4443/forum.opensourceecology.org/2024/09/" ==> awstats.opensourceecology.org/access.log <== 66.115.189.137 - admin [28/Sep/2024:23:56:09 +0000] "GET /forum.opensourceecology.org/2024/09/index.en.html HTTP/1.1" 404 118 "https://awstats.opensourceecology.org:4443/forum.opensourceecology.org/2024/09/" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0" "-"
- looks like the cron has two steps; I gave the first a manual run, and I got an error
root@hetzner3 ~ # cd /etc/cron.d
root@hetzner3 /etc/cron.d #
root@hetzner3 /etc/cron.d # ls
awstats backup_to_backblaze certbot e2scrub_all mdadm munin munin-node php phplist_cron xhprof_cron
root@hetzner3 /etc/cron.d #
root@hetzner3 /etc/cron.d # cat awstats
MAILTO=root
*/10 * * * * www-data [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh
# Generate static reports:
10 03 * * * www-data [ -x /usr/share/awstats/tools/buildstatic.sh ] && /usr/share/awstats/tools/buildstatic.sh
root@hetzner3 /etc/cron.d #
root@hetzner3 /etc/cron.d # sudo -u www-data /usr/share/awstats/tools/update.sh
Error while processing /etc/awstats/awstats.forum.opensourceecology.org.conf
Error: Plugin load for plugin 'ipv6' failed with return code: Error: Can't locate Net/IP.pm in @INC (you may need to install the Net::IP module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl /usr/share/awstats/lib /usr/share/awstats/plugins) at (eval 5) line 1.
Setup ('/etc/awstats/awstats.forum.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.opensourceecology.org.conf
Error: Plugin load for plugin 'ipv6' failed with return code: Error: Can't locate Net/IP.pm in @INC (you may need to install the Net::IP module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.36.0 /usr/local/share/perl/5.36.0 /usr/lib/x86_64-linux-gnu/perl5/5.36 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.36 /usr/share/perl/5.36 /usr/local/lib/site_perl /usr/share/awstats/lib /usr/share/awstats/plugins) at (eval 5) line 1.
Setup ('/etc/awstats/awstats.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.conf
Error: SiteDomain parameter not defined in your config/domain file. You must edit it for using this version of AWStats.
Setup ('/etc/awstats/awstats.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
root@hetzner3 /etc/cron.d #
- it seems unhappy about the permissions on our config files, which are all root:root 644
root@hetzner3 /etc/cron.d # ls -lah /etc/awstats/ total 88K drwxr-xr-x 2 root root 4,0K Sep 25 22:09 . drwxr-xr-x 93 root root 4,0K Sep 28 08:20 .. -rw-r--r-- 1 root root 64K Oct 10 2023 awstats.conf -rw-r--r-- 1 root root 240 Jul 7 2023 awstats.conf.local -rw-r--r-- 1 root root 554 Sep 25 22:08 awstats.forum.opensourceecology.org.conf -rw-r--r-- 1 root root 585 Sep 25 22:08 awstats.opensourceecology.org.conf -rw-r--r-- 1 root root 1,9K Sep 25 22:08 common.conf root@hetzner3 /etc/cron.d #
- I checked this on the old server and, yep, it's the same :/
[root@opensourceecology ~]# ls -lah /etc/awstats/ total 300K drwxr-xr-x 2 root root 4.0K Jan 18 2023 . drwxr-xr-x. 104 root root 12K Sep 11 22:02 .. -rw-r--r-- 1 root root 175 Feb 12 2018 awstats.fef.opensourceecology.org.conf -rw-r--r-- 1 root root 64K Jan 9 2023 awstats.localhost.localdomain.conf -rw-r--r-- 1 root root 193 Aug 27 2018 awstats.microfactory.opensourceecology.org.conf -rw-r--r-- 1 root root 64K Jan 9 2023 awstats.model.conf -rw-r--r-- 1 root root 141 Dec 24 2017 awstats.seedhome.openbuildinginstitute.org.conf -rw-r--r-- 1 root root 179 Apr 9 2019 awstats.store.opensourceecology.org.conf -rw-r--r-- 1 root root 182 Feb 20 2018 awstats.wiki.opensourceecology.org.conf -rw-r--r-- 1 root root 187 Jan 18 2018 awstats.www.openbuildinginstitute.org.conf -rw-r--r-- 1 root root 175 Feb 12 2018 awstats.www.opensourceecology.org.conf -rw-r--r-- 1 root root 63K Dec 24 2017 common.conf -rw-r--r-- 1 root root 63K Dec 24 2017 openbuildinginstitute.org.conf.20171223.bak [root@opensourceecology ~]#
- I checked my personal server (also running Debian 12), and it's the same
- so I guess it's not permissions. Maybe it's just talking about the error above it, regarding IPv6?
- my guess is that this broke because we're no longer installing geoip-extra, since it's no longer available in Debian 12
- SE says to install 'libgeo-ipfree-perl' https://stackoverflow.com/a/65911/1174102
- but hetzner3 says we already got it
root@hetzner3 /etc/cron.d # sudo apt-get install libgeo-ipfree-perl Reading package lists... Done Building dependency tree... Done Reading state information... Done libgeo-ipfree-perl is already the newest version (1.160000-2). 0 upgraded, 0 newly installed, 0 to remove and 21 not upgraded. You have new mail in /var/mail/root root@hetzner3 /etc/cron.d #
- maybe one of these will work?
root@hetzner3 /etc/cron.d # apt-cache search perl | grep -i ipv6 python3-ipy - Python3 module for handling IPv4 and IPv6 addresses and networks libio-socket-ip-perl - module for using IPv4 and IPv6 sockets in a protocol-independent way libnet-cidr-lite-perl - module for merging IPv4 or IPv6 CIDR address ranges libnet-cidr-perl - module to manipulate IPv4/IPv6 netblocks in CIDR notation libnet-frame-layer-ipv6-perl - module for encoding and decoding of the IPv6 layer libnet-inet6glue-perl - glue module to make perl modules IPv6 ready libnet-ip-perl - Perl extension for manipulating IPv4/IPv6 addresses libnet-ip-xs-perl - Perl extension for manipulating IPv4/IPv6 addresses (XS) libnet-iptrie-perl - Perl module for building IPv4 and IPv6 address space hierarchies libnet-ipv6addr-perl - module to validate/manipulate IPv6 addresses libnet-libdnet6-perl - module to add IPv6 support to Net::Libdnet libnet-subnet-perl - Fast IP-in-subnet matcher module for IPv4 and IPv6 libregexp-ipv6-perl - Regular expression for IPv6 addresses libsocket-multicast6-perl - base module for IPv4 and IPv6 multicast socket operations libsocket6-perl - Perl extensions for IPv6 python3-radix - radix tree implementation for storage of IPv4 and IPv6 networks (Python 3) You have new mail in /var/mail/root root@hetzner3 /etc/cron.d #
- I tried this
root@hetzner3 /etc/cron.d # apt-get install libnet-ip-perl Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: libnet-ip-perl 0 upgraded, 1 newly installed, 0 to remove and 21 not upgraded. Need to get 28,8 kB of archives. After this operation, 108 kB of additional disk space will be used. Get:1 http://mirror.hetzner.com/debian/packages bookworm/main amd64 libnet-ip-perl all 1.26-3 [28,8 kB] Fetched 28,8 kB in 0s (84,1 kB/s) Selecting previously unselected package libnet-ip-perl. (Reading database ... 66437 files and directories currently installed.) Preparing to unpack .../libnet-ip-perl_1.26-3_all.deb ... Unpacking libnet-ip-perl (1.26-3) ... Setting up libnet-ip-perl (1.26-3) ... Processing triggers for man-db (2.11.2-2) ... root@hetzner3 /etc/cron.d #
- well this looks like progress, actually
root@hetzner3 /etc/cron.d # sudo -u www-data /usr/share/awstats/tools/update.sh
Error while processing /etc/awstats/awstats.forum.opensourceecology.org.conf
Error: Plugin init for plugin 'geoip_city_maxmind' failed with return code: Error: Failed to create gi object for datafile=/usr/share/GeoIP/GeoIPCity.dat
Setup ('/etc/awstats/awstats.forum.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.opensourceecology.org.conf
Error: Plugin init for plugin 'geoip_city_maxmind' failed with return code: Error: Failed to create gi object for datafile=/usr/share/GeoIP/GeoIPCity.dat
Setup ('/etc/awstats/awstats.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.conf
Error: SiteDomain parameter not defined in your config/domain file. You must edit it for using this version of AWStats.
Setup ('/etc/awstats/awstats.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
You have new mail in /var/mail/root
root@hetzner3 /etc/cron.d #
- looks like these .dat files are already present, so it just wants to create city-specific data?
root@hetzner3 /etc/cron.d # ls -lah /usr/share/GeoIP/ total 7,8M drwxr-xr-x 2 root root 4,0K Sep 25 22:09 . drwxr-xr-x 115 root root 4,0K Sep 25 22:09 .. -rw-r--r-- 1 root root 2,4M Jan 31 2023 GeoIP.dat -rw-r--r-- 1 root root 5,4M Jan 31 2023 GeoIPv6.dat root@hetzner3 /etc/cron.d #
- fwiw, here's what we have on the old server
[root@opensourceecology ~]# ls -lah /usr/share/GeoIP/ total 3.5M drwxr-xr-x 2 root root 4.0K May 4 2020 . drwxr-xr-x. 128 root root 4.0K Jul 26 17:02 .. lrwxrwxrwx 1 root root 17 May 4 2020 GeoIP.dat -> GeoIP-initial.dat -rw-r--r-- 1 root root 1.2M Aug 8 2019 GeoIP-initial.dat lrwxrwxrwx 1 root root 19 May 4 2020 GeoIPv6.dat -> GeoIPv6-initial.dat -rw-r--r-- 1 root root 2.3M Aug 8 2019 GeoIPv6-initial.dat [root@opensourceecology ~]#
- google doesn't turn-up much info about this
- this bug from 7 years old has some info https://debian-bugs-dist.debian.narkive.com/bZirDUYL/bug-860142-libgeo-ip-perl-incompatible-with-geoip-database-cannot-load
- looks like there's no more geoip-database-extras package
root@hetzner3 /etc/awstats # apt-cache search geoip-database geoip-database - IP lookup command line tools that use the GeoIP library (country database) root@hetzner3 /etc/awstats #
- if city data is now missing, I wonder if we can just comment something out that doesn't try to get us city info?
root@hetzner3 /etc/awstats # grep -ir 'city' * awstats.conf:# PLUGIN: GeoIP_City_Maxmind awstats.conf:# PARAMETERS: [GEOIP_STANDARD | GEOIP_MEMORY_CACHE] [/pathto/GeoIPCity.dat[+/pathto/override.txt]] awstats.conf:# and hits by city including regions. awstats.conf:#LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP/GeoIPCity.dat" awstats.conf:# PLUGIN: GeoIP2_City awstats.conf:# PARAMETERS: [/pathto/GeoLite2-City.mmdb[+/pathto/override.txt]] awstats.conf:# and hits by city including regions. awstats.conf:#LoadPlugin="geoip2_city /pathto/GeoLite2-City.mmdb" common.conf:LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP/GeoIPCity.dat" root@hetzner3 /etc/awstats #
- I tried commenting it out in ansible & pushed it
diff --git a/hetzner3/roles/maltfield.awstats/templates/common.conf.j2 b/hetzner3/roles/maltfield.awstats/templates/common.conf.j2 index 084d154..7a20f17 100644 --- a/hetzner3/roles/maltfield.awstats/templates/common.conf.j2 +++ b/hetzner3/roles/maltfield.awstats/templates/common.conf.j2 @@ -2,11 +2,11 @@ ################################################################################ # File: common.conf -# Version: 0.1 +# Version: 0.2 # Purpose: Common config options to be included in other awstats config files # Author: Michael Altfield <michael@michaelaltfield.net> # Created: 2024-09-14 -# Updated: 2024-09-14 +# Updated: 2024-09-28 ################################################################################ LogFormat=1 @@ -61,5 +61,6 @@ LoadPlugin="hashfiles" # Note that geoip6 also does ipv4 addresses.. #LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat" LoadPlugin="geoip6 GEOIP_STANDARD /usr/share/GeoIP/GeoIPv6.dat" -LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP/GeoIPCity.dat" +# 2024-09: city data is no longer available in Debian 12 +#LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP/GeoIPCity.dat"
- that error went away; it's only complaining about a site that doesn't yet exist
root@hetzner3 /etc/awstats # sudo -u www-data /usr/share/awstats/tools/update.sh
Error while processing /etc/awstats/awstats.opensourceecology.org.conf
Create/Update database for config "/etc/awstats/awstats.opensourceecology.org.conf" by AWStats version 7.8 (build 20200416)
From data in log file "/var/log/nginx/www.opensourceecology.org/access.log"...
Error: Couldn't open server log file "/var/log/nginx/www.opensourceecology.org/access.log" : No such file or directory
Setup ('/etc/awstats/awstats.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.conf
Error: SiteDomain parameter not defined in your config/domain file. You must edit it for using this version of AWStats.
Setup ('/etc/awstats/awstats.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
root@hetzner3 /etc/awstats #
- and I gave it a run
root@hetzner3 /etc/awstats # sudo -u www-data /usr/share/awstats/tools/update.sh
Error while processing /etc/awstats/awstats.opensourceecology.org.conf
Create/Update database for config "/etc/awstats/awstats.opensourceecology.org.conf" by AWStats version 7.8 (build 20200416)
From data in log file "/var/log/nginx/www.opensourceecology.org/access.log"...
Error: Couldn't open server log file "/var/log/nginx/www.opensourceecology.org/access.log" : No such file or directory
Setup ('/etc/awstats/awstats.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.conf
Error: SiteDomain parameter not defined in your config/domain file. You must edit it for using this version of AWStats.
Setup ('/etc/awstats/awstats.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
root@hetzner3 /etc/awstats #
root@hetzner3 /etc/awstats # sudo -u www-data /usr/share/awstats/tools/buildstatic.sh
root@hetzner3 /etc/awstats #
- this last run took a bit longer, that's good
- I refreshed the web browser, and now this loads :D https://awstats.opensourceecology.org:4443/forum.opensourceecology.org/2024/09/
- actually, all the numbers are 0 and "Last Update" says this error
Never updated (See 'Build/Update' on awstats_setup.html page)
- ...but that's better than a 404, at least
- thinking that maybe it'll only collect recent data (since the cron runs every 10 minutes) I tried to click around a bit in the forums and re-run, but there was no change to the awstats page
- I think the issue might be caused by there being no nginx log data for www.opensourceecology.org
- there's a special awstats config for that domian. I think it's so that we can define the ip addresses as aliases, so that all traffic hitting our server over the ip address will get logged here too (since it's teh default vhost)
- but that's one of the last sites that we're going to prepare, so let's just revisit awstats after we finish setting up vhost (yet before we do the migration of the first prod site to this server)
- anyway, it's finally Sep 29 in UTC, so let's do TOFU 2/3 for our wordpress files
- here's TOFU 2/3 (VPN, exit in Sweden)
Sweden 2024-09-29 ... 2024-09-29 e39147da6f75b11e2bcd3a23ab6512dc0f1d1715cc6b25d4b0f876cd88f9851a akismet.5.3.3.zip 206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0 black-studio-tinymce-widget.2.7.3.zip 371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213 bouquet.1.2.5.zip 7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86 chartbeat.2.0.7.zip 3144da2236c6017ba1b7431363317d8db5bc541dd3c729c0aa1f597093723078 classic-editor.1.6.5.zip 67ccf342a4c7fe853e84b535bf3cf121c7a0110feaa96ae06f5eda202f8485ba coingate-for-woocommerce.2.1.1.zip 357b13dab0e2b9996444664c95bd7ee7ca49fb18aced65db4c40c747543afcf4 contact-form-7.5.9.8.zip fe9f4e60683b46eb409b9f782cbdee29e22c345430aafc898f78a1a00d88265c duplicate-page.4.5.zip a32d701644bc5ef0ac50047fcbb8b1c1ab9289d6531dfe986d6574f00d16b921 duplicate-post.4.5.zip 579488be50b99ea7401be9f8127efd7a243c8f213f8f90ede15b07577eb5e9d6 gk-portfolio.1.5.3.zip b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058 google-authenticator.0.54.zip 9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e google-authenticator-encourage-user-activation.0.2.zip 78a00cf800196b7a03599bfd8c49337a34916a09e570fd89b4c0ee82108d8f15 insert-headers-and-footers.2.2.2.zip 466b7a5a05140ce0d7ae2a2e475cdb22cd91ff7d10bc0d14068eaa8af71fb944 jetpack.13.8.1.zip 9bc91e2e6e2172f5284b24893352b8a233a24da31b34ce72a15ce584ec66d7b7 meta-box.5.10.2.zip 1f092fa51e2e3abd0c7a518db9d6ab5ee5dbc1abfa7551439350ddac9adb99fe ml-slider.3.91.0.zip c6123863c3472a40d52cba2ded434a37f6bd488f9e0c280470ccac68256f40c9 open-in-new-window-plugin.3.0.zip 33a58c7c61b1068b112b525556506e5f175988638db583883f7d2c196ef5e2d5 portfolio-press.2.8.0.zip c313fcee168a055005545c98ef665dcdc7b0693838851e5ba77a05c9b9a99fe6 post-types-order.2.2.6.zip d9740796220eb255937d81f8d2b3e7066a154bbce88f7a8d4555ad87a90b2583 revision-control.2.3.2.zip 80149aa1c6f83282d420b6c90ffd1d3c3b1ddfdcec2096c7411b568662fe0dd8 shareaholic.9.7.12.zip 24384df22558e87536aa848b21eee183ca6226fbfcea75af3821437325313ac0 share-on-diaspora.0.7.9.zip 254f144ad18558a91ed632dbb017e3a50a81872f8f9933d7ae77c85436175bd0 shariff.4.6.14.zip 1894750514b49301b3cb93ced16d48b60dee8654a5ce86f803b8454fffd9305e sketch.1.2.4.zip 122fa98f8b3674e4dfef0fa34a235db0cc6cff80c870d14d8159ba88d8796a1d ssl-insecure-content-fixer.2.7.2.zip 576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311 storefront.4.6.0.zip 02d33b647e5309a5817b58c3a51cb17ca801293a2a9653e33fd4ccd0fdb132b7 twentyeleven.4.7.zip 56309284b46e1953b855a21668cfba0f9fdd5e55acf9cdf2578f4cfd768048c7 twentyfifteen.3.8.zip 9af0e22d0a4e98535d852963e6bb23c7a2866167532df312edc27aa42257cb60 twentyfourteen.4.0.zip 8cecbed27f3af39c8964f09a2dbf2b58e50958fa7b7b00437b95fb5db891839a twentynineteen.2.9.zip 215aabae3e9cea494c012c419f31836a8ce2fb321b4210bccd05d0d263d6b727 twentyseventeen.3.7.zip 7df868e367ef21faf0ef71a0aab87dc2df8e35d53aa954165960f87d07b84f92 twentysixteen.3.3.zip 15b8b27950e68c655daed10d3ed0cd06ba55b814afed524374a68aff50856d5a twentyten.4.2.zip d1f82982f19e9aa75ae69db2eb3d76e2dd1f2cee737fbfac7390870970bbf67b twentythirteen.4.2.zip 2e699f0d0542d132aabc84f4813573aed246cac3624871299ed42d87152d4e10 twentytwelve.4.3.zip 832f4cf27722c448b44f5b7ff5583de3b5e8390ac881d37bcce4cdd5c5089be4 varnish-http-purge.5.2.2.zip 02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c vcaching.1.8.3.zip 1dbadde30b8e1e8b8785ec4ab319059e1ff158e224ce38c1df2a2688657294d9 w3-total-cache.2.7.6.zip 5c1e3005e5e5f3f614e50d0824ded56f822dd0a826fdad46a2e9c2dd76cf02be wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip a5f5ee58d239b9d170f6e6bfcbc29ae9c19d0298429ce73a8275f74e95524295 woocommerce.9.3.3.zip 9f0d2c2d82f6ec68f75307b65f723dda44ad7dce7931c307a8872d04ec9ca268 wordpress-importer.0.8.2.zip b7bda6589f00689021187d102912127fa7dc8e3289689f4d2ae2563c0a487a9c wordpress-seo.23.5.zip dfa3820f767ff202f942100aadd6e05acb144d5edaf842cfea26a6520aa71995 wpautop-control.1.6.zip fa9a177e7228bfdc6fddabb6be802a1fff32f8dddaba77cfea6c25c56788d8b0 wp-memory-usage.1.2.10.zip 79cacaea2b1f12c664e24347b2bc58f08db806bd25144f7e7332c494be115e66 wp-optimize.3.6.0.zip 17a675cb0262512d5247745bc0a1de0f72483c629b185dd43483071885d2e5eb wp-smushit.3.16.6.zip 2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d wp-super-cache.1.12.4.zip
- I decided to listen to my former self and create a 'not-apache' user who will own the vhost files (perhaps to avoid SUID; currently it's root which seems like a bad idea)
root@hetzner3 /etc/awstats # adduser not-apache --disabled-password --gecos '' --home /dev/null --shell /usr/sbin/nologin'' Adding user `not-apache' ... Adding new group `not-apache' (1006) ... Adding new user `not-apache' (1006) with group `not-apache (1006)' ... adduser: The home directory `/dev/null' already exists. Not touching this directory. adduser: Warning: The home directory `/dev/null' does not belong to the user you are currently creating. Adding new user `not-apache' to supplemental / extra groups `users' ... Adding user `not-apache' to group `users' ... root@hetzner3 /etc/awstats # root@hetzner3 /etc/awstats # grep not-apache /etc/passwd not-apache:x:1006:1006:,,,:/dev/null:/usr/sbin/nologin root@hetzner3 /etc/awstats # root@hetzner3 /etc/awstats # grep not-apache /etc/group users:x:100:maltfield,wp,not-apache not-apache:x:1006: root@hetzner3 /etc/awstats #
- here's our new commands to fix permissions on /var/www/html
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
...
- I also need to create use accounts for other humans
- on hetzner2, here's all the users we had in the 'sshaccess' group
[maltfield@opensourceecology ~]$ grep sshaccess /etc/group sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas [maltfield@opensourceecology ~]$
- some of these users should also have root
[maltfield@opensourceecology ~]$ grep wheel /etc/group wheel:x:10:maltfield,crupp,tgriffing,root,jthomas [maltfield@opensourceecology ~]$
- my next question is: do all of these users have authorized keys? because if they don't have a key setup on hetzner2, then they never had access as long as I've been at OSE, and we shouldn't bother migrating htem
- looks like they all have keys
[root@opensourceecology ~]# users='cmota marcin tgriffing maltfield lberezhny crupp jthomas' [root@opensourceecology ~]# for user in $users; do > echo $user > cat /home/$user/.ssh/authorized_keys > echo '-----------' > echo > done cmota #ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDOhf/CGM3t3sklCPAiwaAl7NQPiyDwy5WwV2oxO6rCsDC3jmwn2JCJ6LZ4jXI9NvQQXwg+1kOqsSv2ZFS6RUq1Rs/EW8jR5lFxy9+FgRFpAIlzglAWIBbjpamX0duCMAwpOH2bK1oEDokNM37TtBynFyNytJqLgIGI48IR45KIU4Ib/IN8KzcQRdfPMPFW3EcO1jHJCMUl5SIKt36rgFvhc6JmHF9sl2fCPZ6Jmbi21DOz8fVQDdxQNAc2ubkAS2Sn27LZS7ww6I1VPK43IKBZlZXm3/FQ5HyHrXsGelIDCwdrwM/GTffClDc9EX1ujEHXZIILBDDVLVbwzI+3QHsHvaU9pY+pXBC1dw4LN8InmVazGeV9MoH6SeSeEx4g4KS+2qMiF6fjViPzpLeDW7hIRiTc2+8pjoivNKcgO0xCIBJ/dvTpTExCOPDBVJisnU+3Psq6Wn0Nb7ts7A0tyCtGOG2rxQEYCRmIy8AvQ6WlKztGFvqrfdHaRO9v7WY0hMtyD7kh0tHhHgxSFNcWSIVAIf8JSfhm3f/Orcsg2JDliT4wJzMz7OAYGOQmpqDI+qYa/ULgp9qlf/XBQWeMrQwAU0YeepRJjx3rz9xMxOLj2ddV3JSS8wHg+BGLpPwvqaoFrVIXIgHkW0AbBoXHmtSIZSZ5t5+odWlT1aLb5GmlXw== catarinamota@catarinas-air.pk5001z ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer ----------- marcin ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500 ----------- tgriffing ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0 ----------- maltfield ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGNYjR7UKiJSAG/AbP+vlCBqNfQZ2yuSXfsEDuM7cEU8PQNJyuJnS7m0VcA48JRnpUpPYYCCB0fqtIEhpP+szpMg2LByfTtbU0vDBjzQD9mEfwZ0mzJsfzh1Nxe86l/d6h6FhxAqK+eG7ljYBElDhF4l2lgcMAl9TiSba0pcqqYBRsvJgQoAjlZOIeVEvM1lyfWfrmDaFK37jdUCBWq8QeJ98qpNDX4A76f9T5Y3q5EuSFkY0fcU+zwFxM71bGGlgmo5YsMMdSsW+89fSG0652/U4sjf4NTHCpuD0UaSPB876NJ7QzeDWtOgyBC4nhPpS8pgjsnl48QZuVm6FNDqbXr9bVk5BdntpBgps+gXdSL2j0/yRRayLXzps1LCdasMCBxCzK+lJYWGalw5dNaIDHBsEZiK55iwPp0W3lU9vXFO4oKNJGFgbhNmn+KAaW82NBwlTHo/tOlj2/VQD9uaK5YLhQqAJzIq0JuWZWFLUC2FJIIG0pJBIonNabANcN+vq+YJqjd+JXNZyTZ0mzuj3OAB/Z5zS6lT9azPfnEjpcOngFs46P7S/1hRIrSWCvZ8kfECpa8W+cTMus4rpCd40d1tVKzJA/n0MGJjEs2q4cK6lC08pXxq9zAyt7PMl94PHse2uzDFhrhh7d0ManxNZE+I5/IPWOnG1PJsDlOe4Yqw== guttersnipe@guttersnipe ----------- lberezhny ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIS+nkIqdJVcPQ8X2eVyvCOCOISwmxn0muYMUwvsKQms lex@hortus ----------- crupp ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDXW69msrd6s4Gs5cs/L4zdLJGgFp8xGtsEwXrxSUcyM1TnHu3lpz++ZOCGBssCQcuo5BhPP1i4fwcBrz1Zm3mK2a4NoxzoRwJtcX961YPqecg4aTbTmT+HlUqIOF5tQ3ZsKo3x7G8is948cx2SfBMPqQKPpOaNkCv8U2Uaq9hVZs5WbODXeCve6oPwkhFNhRtZ4frSlpVOBVX+2WMhwo48KrtsAw3cLqJs7fiHAo8hCeXHgBcpfWmMnOeLUiLrtIvoqXueu5SWPC0y48Qgy8bBJFUk6nGyg83Z5wDZi0NopZKE04CAqcatcxgzWwOtqcoB9tYYXLJsT2VH4myq8UZAt1WeyP/haVprrVCRqC6BRxIUAmmCl/5FbNEtZUSaUsxH7f2DRxGYQnrcsB1pm05bI8wBtJLZwkxep9MYayP2q13E0mRIKmzexgR93JGcw3QyxvqUixbdYBrVl/lLPDO5VBVOWT6wZbGGg4M2RKDwOlF94OPCPDn28LcfxF0U694H82T7NHadA411HMLiBQ7X/0JGZd5Pi7IN2svlScJ1GRc4V5QEbxGjsTwbvKqd6uCKxas1xz2muC+IKeZYDs7LngnVwCyC+3BFo/N2YM/HpJw3+QMSLYQwehp+5NnkiVBMua32ycKsQzhXD7sN/s/F9quaWtR2txWBNbAB3r97wQ== root@WebPi ----------- jthomas ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDYgPQsHpClJaLSKj0NOSwimdGGch4YFYetV6Uv6dW2i1eKyd0ncpNWRJvZ4RV2jVCwVYpI13SZv4f+c6EL/FfrQHBCDNv/ENRxbrvOYG1c0c5qc9VwLvab5ar1hWtn/mLbazGH2OaOEhPuO/U3snli3a/MhSKXCSXUb8jjxjKBk9SXW0EzoycR8FjXfwvSGnGeMHyc6ZPoJdiMeO3bTLu8xVJMm8OMzUKkEWRXMBFlGH0Y7wapYziTFov3BrNSw34mWqYx7wUwGGhVb9jQmfAuQgJ/pABT57+EIKLl58GCznHzyO12BpKPMXAikYX1tb3D6RtHBE9l1vqv+cXsrMvlRADAWoBD6RMU936mAC+iTVuxRAmsrlOmzp7jQFhynSKhOpwwQ23KikXW+cFoOZsLUKA+MFr6PCmznHbrgBGJlD3CW76DsTeaBNqxg6W8lmdP2ZjUV+9vakXI+tVf9hImwrTL2qmVeFIlm8Zk2Tz0BqNRtplOszVCU2Dc8jjW2jX7P52ZGoaedRKGvQvF92OkZrUU1FNO9nkAr8MgxCs1QK1XuQgqySZ8cheEGzpqQ0/6lbOBQh/LkroE9a+R2uuN5m5C9bv6d5xMKge6Fz0178the8vdLGYer3EBvPrGhM1nPKd+v94wAS0sdTUF9l6tvDLGr4aU76b4wNPkVLMnbQ== ----------- [root@opensourceecology ~]#
- I sent an email to Marcin asking me if I should migrate all the users, or just a subset
Hey Marcin, Can you tell me if I should migrate all of these users to Hetzner3, or only a subset? I need to create user accounts on the hetzner3 server. All of these users have ssh access to hetzner2, but I imagine some of them might be inactive and you might want to intentionally not want me to grant them access to the hetzner3 server. Here's the user list cmota marcin tgriffing maltfield lberezhny crupp jthomas Please let me know if you want me to create accounts for all of these users on Hetzner3, or if you only want me to create accounts for some of them. Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
...
- I realized that we're missing the raid-check cron on hetzner3
- hetzner2
[maltfield@opensourceecology ~]$ ls /etc/cron.d 0hourly backup_to_backblaze letsencrypt raid-check awstats_generate_static_files cacti phplist sysstat [maltfield@opensourceecology ~]$ cat /etc/cron.d/raid-check # Run system wide raid-check once a week on Sunday at 1am by default 0 1 * * Sun root /usr/sbin/raid-check [maltfield@opensourceecology ~]$
- hetzner3
maltfield@hetzner3:~$ ls /etc/cron.d awstats backup_to_backblaze certbot e2scrub_all mdadm munin munin-node php phplist_cron xhprof_cron maltfield@hetzner3:~$
- hmm..doesn't seem to exist on hetzner3
root@hetzner3 ~ # raid-check -bash: raid-check: command not found root@hetzner3 ~ # /usr/sbin/raid-check -bash: /usr/sbin/raid-check: No such file or directory root@hetzner3 ~ # /sbin/raid-check -bash: /sbin/raid-check: No such file or directory root@hetzner3 ~ #
- I spent some time looking into how to monitor and get notifications about drive failures in the software raid.
- first, it looks like Debain 12 moved the mdadm check from cron to systemd https://unix.stackexchange.com/a/228518/419003
root@hetzner3 ~ # ls /etc/cron.d awstats backup_to_backblaze certbot e2scrub_all mdadm munin munin-node php phplist_cron xhprof_cron root@hetzner3 ~ # root@hetzner3 ~ # ls /lib/systemd/system/md* /lib/systemd/system/mdadm-grow-continue@.service /lib/systemd/system/mdadm-waitidle.service /lib/systemd/system/mdmonitor-oneshot.service /lib/systemd/system/mdadm-last-resort@.service /lib/systemd/system/mdcheck_continue.service /lib/systemd/system/mdmonitor-oneshot.timer /lib/systemd/system/mdadm-last-resort@.timer /lib/systemd/system/mdcheck_continue.timer /lib/systemd/system/mdmonitor.service /lib/systemd/system/mdadm.service /lib/systemd/system/mdcheck_start.service /lib/systemd/system/mdmon@.service /lib/systemd/system/mdadm-shutdown.service /lib/systemd/system/mdcheck_start.timer root@hetzner3 ~ # root@hetzner3 ~ # cat /lib/systemd/system/mdcheck_start.timer # This file is part of mdadm. # # mdadm is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. [Unit] Description=MD array scrubbing [Timer] OnCalendar=Sun *-*-1..7 1:00:00 [Install] WantedBy= mdmonitor.service Also= mdcheck_continue.timer root@hetzner3 ~ #
- ok, it looks like CentOS's 'raid-check' is 'mdcheck' in Debian -- and it's located outside path and has no man page :/
root@hetzner3 ~ # cat /lib/systemd/system/mdcheck_start.service
# This file is part of mdadm.
#
# mdadm is free software; you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
[Unit]
Description=MD array scrubbing
Wants=mdcheck_continue.timer
Documentation=man:mdadm(8)
[Service]
Type=oneshot
Environment="MDADM_CHECK_DURATION=6 hours"
EnvironmentFile=-/run/sysconfig/mdadm
ExecStartPre=-/usr/lib/mdadm/mdadm_env.sh
ExecStart=/usr/share/mdadm/mdcheck --duration ${MDADM_CHECK_DURATION}
root@hetzner3 ~ #
root@hetzner3 ~ # /usr/share/mdadm/mdcheck --help
Usage: mdcheck [--continue] [--duration time-offset]
time-offset must be understood by "date --date"
root@hetzner3 ~ #
- anyway, this says that it will automatically send an email to `MAILADDR` if it detects a disk is degraded https://unix.stackexchange.com/a/28642/419003
- so it looks like alerts will go to root@localhost, which currently won't get forwarded off-machine anywhere
root@hetzner3 ~ # cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=01010ebe:30bb5ba3:c2fb6223:c8c1591d name=rescue:0
ARRAY /dev/md/1 metadata=1.2 UUID=7a94e7cb:8379474a:aad8e37d:624c7561 name=rescue:1
ARRAY /dev/md/2 metadata=1.2 UUID=b67ca22a:8ac0e283:fcc092c4:3c976042 name=rescue:2
# This configuration was auto-generated on Wed, 31 Jul 2024 23:44:14 +0200 by mkconf
root@hetzner3 ~ #
# I checked the 'root' user's mail for the first time; it's currently being spammed by awstats and phplist crons
<pre>
From www-data@hetzner3.opensourceecology.org Sun Sep 29 04:30:02 2024
Return-Path: <www-data@hetzner3.opensourceecology.org>
X-Original-To: root
Received: by hetzner3.opensourceecology.org (Postfix, from userid 33)
id 7FE5DB883A3; Sun, 29 Sep 2024 04:30:02 +0000 (UTC)
From: root@hetzner3.opensourceecology.org (Cron Daemon)
To: root@hetzner3.opensourceecology.org
Subject: Cron <www-data@hetzner3> [ -x /usr/share/awstats/tools/update.sh ] && /usr/share/awstats/tools/update.sh
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <MAILTO=root>
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/www>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=www-data>
Message-Id: <20240929043002.7FE5DB883A3@hetzner3.opensourceecology.org>
Date: Sun, 29 Sep 2024 04:30:02 +0000 (UTC)
Status: RO
Content-Length: 909
Lines: 10
Error while processing /etc/awstats/awstats.opensourceecology.org.conf
Create/Update database for config "/etc/awstats/awstats.opensourceecology.org.conf" by AWStats version 7.8 (build 20200416)
>From data in log file "/var/log/nginx/www.opensourceecology.org/access.log"...
Error: Couldn't open server log file "/var/log/nginx/www.opensourceecology.org/access.log" : No such file or directory
Setup ('/etc/awstats/awstats.opensourceecology.org.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
Error while processing /etc/awstats/awstats.conf
Error: SiteDomain parameter not defined in your config/domain file. You must edit it for using this version of AWStats.
Setup ('/etc/awstats/awstats.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).
From root@hetzner3.opensourceecology.org Sun Sep 29 04:30:37 2024
Return-Path: <root@hetzner3.opensourceecology.org>
X-Original-To: root
Received: by hetzner3.opensourceecology.org (Postfix, from userid 0)
id D69F9B883A3; Sun, 29 Sep 2024 04:30:37 +0000 (UTC)
From: root@hetzner3.opensourceecology.org (Cron Daemon)
To: root@hetzner3.opensourceecology.org
Subject: Cron <root@hetzner3> sleep $(( RANDOM % 60 )) && su -s /bin/sh www-data -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessqueue &>> /var/log/phplist/processQueue.log"
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/bash>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=root>
Message-Id: <20240929043037.D69F9B883A3@hetzner3.opensourceecology.org>
Date: Sun, 29 Sep 2024 04:30:37 +0000 (UTC)
Status: RO
Content-Length: 101
Lines: 2
sh: 1: cannot create /var/log/phplist/processQueue.log: Directory nonexistent
sh: 1: time: not found
- I'm not opposed to setting this to email marcin and myself, but it's going to be useless if we don't check it because it's buried in spam
- I went ahead and updated the ansible 'maltfield.cron' role to create this dir and switch from 'sh' to 'bash' so it had 'time'
- I don't think that's going to suppress these emails though; now it's just sending the the stdout of the `time`
root@hetzner3 ~ # tail -n20 /var/mail/root To: root@hetzner3.opensourceecology.org Subject: Cron <root@hetzner3> sleep $(( RANDOM % 60 )) && su -s /bin/bash www-data -c "time /usr/bin/php /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php -c /var/www/html/phplist.opensourceecology.org/config.php -pprocessqueue &>> /var/log/phplist/processQueue.log" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Cron-Env: <SHELL=/bin/bash> X-Cron-Env: <HOME=/root> X-Cron-Env: <PATH=/usr/bin:/bin> X-Cron-Env: <LOGNAME=root> Message-Id: <20240929045524.5392DB88372@hetzner3.opensourceecology.org> Date: Sun, 29 Sep 2024 04:55:24 +0000 (UTC) Status: RO Content-Length: 42 Lines: 4 real 0m0,024s user 0m0,021s sys 0m0,003s root@hetzner3 ~ #
- alternatively, we could just enter the actual opslist email address in our mdadm config
- the only reason I wouldn't want to do this is if the email will contain sensitive info
- I need to produce a test email, and this says how https://ioflood.com/blog/mdadm-status-notifications-how-to-configure-mdadm-email-alerts-for-software-raid-drive-failures/
root@hetzner3 ~ # killall mdadm root@hetzner3 ~ # /sbin/mdadm --monitor --scan --test ^C root@hetzner3 ~ #
- after about 30 seconds or so, I killed the process with ctrl-c
- indeed, root got an email. 3 actually.
From root@hetzner3.opensourceecology.org Sun Sep 29 05:11:12 2024
Return-Path: <root@hetzner3.opensourceecology.org>
X-Original-To: root
Received: by hetzner3.opensourceecology.org (Postfix, from userid 0)
id 01331B88372; Sun, 29 Sep 2024 05:11:11 +0000 (UTC)
From: mdadm monitoring <root@hetzner3.opensourceecology.org>
To: root@hetzner3.opensourceecology.org
Subject: TestMessage event on /dev/md/2:hetzner3
Message-Id: <20240929051112.01331B88372@hetzner3.opensourceecology.org>
Date: Sun, 29 Sep 2024 05:11:11 +0000 (UTC)
Status: RO
Content-Length: 663
Lines: 21
This is an automatically generated mail message from mdadm
running on hetzner3
A TestMessage event had been detected on md device /dev/md/2.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
bitmap: 2/4 pages [8KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
From root@hetzner3.opensourceecology.org Sun Sep 29 05:11:12 2024
Return-Path: <root@hetzner3.opensourceecology.org>
X-Original-To: root
Received: by hetzner3.opensourceecology.org (Postfix, from userid 0)
id 05875B88372; Sun, 29 Sep 2024 05:11:12 +0000 (UTC)
From: mdadm monitoring <root@hetzner3.opensourceecology.org>
To: root@hetzner3.opensourceecology.org
Subject: TestMessage event on /dev/md/1:hetzner3
Message-Id: <20240929051112.05875B88372@hetzner3.opensourceecology.org>
Date: Sun, 29 Sep 2024 05:11:12 +0000 (UTC)
Status: O
Content-Length: 663
Lines: 21
This is an automatically generated mail message from mdadm
running on hetzner3
A TestMessage event had been detected on md device /dev/md/1.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
bitmap: 2/4 pages [8KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
From root@hetzner3.opensourceecology.org Sun Sep 29 05:11:12 2024
Return-Path: <root@hetzner3.opensourceecology.org>
X-Original-To: root
Received: by hetzner3.opensourceecology.org (Postfix, from userid 0)
id 09709B88372; Sun, 29 Sep 2024 05:11:12 +0000 (UTC)
From: mdadm monitoring <root@hetzner3.opensourceecology.org>
To: root@hetzner3.opensourceecology.org
Subject: TestMessage event on /dev/md/0:hetzner3
Message-Id: <20240929051112.09709B88372@hetzner3.opensourceecology.org>
Date: Sun, 29 Sep 2024 05:11:12 +0000 (UTC)
Status: O
Content-Length: 663
Lines: 21
This is an automatically generated mail message from mdadm
running on hetzner3
A TestMessage event had been detected on md device /dev/md/0.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md2 : active raid1 nvme0n1p3[1] nvme1n1p3[0]
465370432 blocks super 1.2 [2/2] [UU]
bitmap: 2/4 pages [8KB], 65536KB chunk
md0 : active raid1 nvme0n1p1[1] nvme1n1p1[0]
33520640 blocks super 1.2 [2/2] [UU]
md1 : active raid1 nvme0n1p2[1] nvme1n1p2[0]
1046528 blocks super 1.2 [2/2] [UU]
unused devices: <none>
- oh, it was one alert for each of the three /dev/md/X devices
- looks like this doesn't contain anything sensitive, should be ok to send unencrypted.
- but we should also change the "from" address
Fri Sep 27, 2024
- I decided to install wp-cli on hetzner3
- wp-cli is *not* safe to download things, but it should be safe for local internet gathering
- Because wp-cli is itself signed, it should be safe to install. We should just prevent it from having Internet access. which should be the default for the 'wp' user.
- first, I exported the wp-cli release signing pgp key from my personal keychain and uploaded it to hetzner3
user@personal:~/tmp/wp-cli/wp-cli-2.11.0$ gpg --list-keys wp-cli pub rsa2048/0xA3A2E8F226F0BC06 2018-05-31 [SC] Key fingerprint = 63AF 7AA1 5067 C056 16FD DD88 A3A2 E8F2 26F0 BC06 uid [ unknown] WP-CLI Releases <releases@wp-cli.org> sub rsa2048/0xF6B7309F0E6A9622 2018-05-31 [E] user@personal:~/tmp/wp-cli/wp-cli-2.11.0$ gpg -a --export wp-cli > wp-cli.asc user@personal:~/tmp/wp-cli/wp-cli-2.11.0$ rsync -av --progress wp-cli.asc hetzner3: sending incremental file list wp-cli.asc 1,749 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1) sent 1,858 bytes received 35 bytes 420.67 bytes/sec total size is 1,749 speedup is 0.92 user@personal:~/tmp/wp-cli/wp-cli-2.11.0$
- and I imported it into both my users and root's gpg key on hetzner3
maltfield@hetzner3:~$ ls bin ossec.conf.31557.2024-09-15@04:36:48~ wp-cli.asc maltfield@hetzner3:~$ maltfield@hetzner3:~$ gpg --import wp-cli.asc maltfield@hetzner3:~$ gpg: directory '/home/maltfield/.gnupg' created gpg: keybox '/home/maltfield/.gnupg/pubring.kbx' created gpg: /home/maltfield/.gnupg/trustdb.gpg: trustdb created gpg: key A3A2E8F226F0BC06: public key "WP-CLI Releases <releases@wp-cli.org>" imported gpg: Total number processed: 1 gpg: imported: 1 maltfield@hetzner3:~$ sudo su - root@hetzner3 ~ # root@hetzner3 ~ # gpg --import /home/maltfield/wp-cli.asc gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key A3A2E8F226F0BC06: public key "WP-CLI Releases <releases@wp-cli.org>" imported gpg: Total number processed: 1 gpg: imported: 1 root@hetzner3 ~ #
- I downloaded the latest version of wp-cli and confirmed its authenticity using the gpg signature
root@hetzner3 ~ # mkdir /var/tmp/wp-cli-2.11.0 root@hetzner3 ~ # chown root:root /var/tmp/wp-cli-2.11.0 root@hetzner3 ~ # chmod 0700 /var/tmp/wp-cli-2.11.0/ root@hetzner3 ~ # cd /var/tmp/wp-cli-2.11.0/ root@hetzner3 /var/tmp/wp-cli-2.11.0 # root@hetzner3 /var/tmp/wp-cli-2.11.0 # wget https://github.com/wp-cli/wp-cli/releases/download/v2.11.0/wp-cli-2.11.0.phar.asc --2024-09-28 03:31:13-- https://github.com/wp-cli/wp-cli/releases/download/v2.11.0/wp-cli-2.11.0.phar.asc Resolving github.com (github.com)... 140.82.121.3 Connecting to github.com (github.com)|140.82.121.3|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/2360755/169456ad-1244-47ee-aeb9-519ca6355884?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240928%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240928T033004Z&X-Amz-Expires=300&X-Amz-Signature=5e78356f2fecc58902253c90e99518b9d11f823a218d6e020e942f00d437a040&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B filename%3Dwp-cli-2.11.0.phar.asc&response-content-type=application%2Foctet-stream [following] --2024-09-28 03:31:14-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/2360755/169456ad-1244-47ee-aeb9-519ca6355884?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240928%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240928T033004Z&X-Amz-Expires=300&X-Amz-Signature=5e78356f2fecc58902253c90e99518b9d11f823a218d6e020e942f00d437a040&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B filename%3Dwp-cli-2.11.0.phar.asc&response-content-type=application%2Foctet-stream Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ... Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 516 [application/octet-stream] Saving to: ‘wp-cli-2.11.0.phar.asc’ wp-cli-2.11.0.phar.asc 100%[=======================================================================>] 516 --.-KB/s in 0s 2024-09-28 03:31:15 (5,80 MB/s) - ‘wp-cli-2.11.0.phar.asc’ saved [516/516] root@hetzner3 /var/tmp/wp-cli-2.11.0 # wget https://github.com/wp-cli/wp-cli/releases/download/v2.11.0/wp-cli-2.11.0.phar --2024-09-28 03:31:18-- https://github.com/wp-cli/wp-cli/releases/download/v2.11.0/wp-cli-2.11.0.phar Resolving github.com (github.com)... 140.82.121.3 Connecting to github.com (github.com)|140.82.121.3|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/2360755/9dedae94-2f58-4f81-a823-dd7c03820d2f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240928%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240928T033008Z&X-Amz-Expires=300&X-Amz-Signature=6dbf79a5bd953e19a8887cb7803746192c727c544e2d1fdd2a325173856a94ab&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B filename%3Dwp-cli-2.11.0.phar&response-content-type=application%2Foctet-stream [following] --2024-09-28 03:31:18-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/2360755/9dedae94-2f58-4f81-a823-dd7c03820d2f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240928%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240928T033008Z&X-Amz-Expires=300&X-Amz-Signature=6dbf79a5bd953e19a8887cb7803746192c727c544e2d1fdd2a325173856a94ab&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B filename%3Dwp-cli-2.11.0.phar&response-content-type=application%2Foctet-stream Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ... Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7048108 (6,7M) [application/octet-stream] Saving to: ‘wp-cli-2.11.0.phar’ wp-cli-2.11.0.phar 100%[=======================================================================>] 6,72M --.-KB/s in 0,08s 2024-09-28 03:31:19 (86,6 MB/s) - ‘wp-cli-2.11.0.phar’ saved [7048108/7048108] root@hetzner3 /var/tmp/wp-cli-2.11.0 # root@hetzner3 /var/tmp/wp-cli-2.11.0 # gpg --verify wp-cli-2.11.0.phar.asc gpg: assuming signed data in 'wp-cli-2.11.0.phar' gpg: Signature made 2024-08-08T03:46:14 UTC gpg: using RSA key 63AF7AA15067C05616FDDD88A3A2E8F226F0BC06 gpg: issuer "releases@wp-cli.org" gpg: Good signature from "WP-CLI Releases <releases@wp-cli.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 63AF 7AA1 5067 C056 16FD DD88 A3A2 E8F2 26F0 BC06 root@hetzner3 /var/tmp/wp-cli-2.11.0 #
- I had to update the documentation a bit for how to install it on Debian https://wiki.opensourceecology.org/wiki/Wordpress#WP-CLI
root@hetzner3 /var/tmp/wp-cli-2.11.0 # adduser wp --disabled-password --gecos ''
Adding user `wp' ...
Adding new group `wp' (1005) ...
Adding new user `wp' (1005) with group `wp (1005)' ...
Creating home directory `/home/wp' ...
Copying files from `/etc/skel' ...
Adding new user `wp' to supplemental / extra groups `users' ...
Adding user `wp' to group `users' ...
root@hetzner3 /var/tmp/wp-cli-2.11.0 #
root@hetzner3 /var/tmp/wp-cli-2.11.0 # gpasswd -a wp www-data
Adding user wp to group www-data
root@hetzner3 /var/tmp/wp-cli-2.11.0 #
root@hetzner3 /var/tmp/wp-cli-2.11.0 # gpasswd -a wp apache-admins
Adding user wp to group apache-admins
root@hetzner3 /var/tmp/wp-cli-2.11.0 #
root@hetzner3 /var/tmp/wp-cli-2.11.0 # mkdir /home/wp/.wp-cli
root@hetzner3 /var/tmp/wp-cli-2.11.0 #
root@hetzner3 /var/tmp/wp-cli-2.11.0 # cp wp-cli-2.11.0.phar /home/wp/.wp-cli/
root@hetzner3 /var/tmp/wp-cli-2.11.0 #
root@hetzner3 /var/tmp/wp-cli-2.11.0 # chown -R wp:wp /home/wp/.wp-cli
root@hetzner3 /var/tmp/wp-cli-2.11.0 #
root@hetzner3 /var/tmp/wp-cli-2.11.0 # su - wp
wp@hetzner3:~$
wp@hetzner3:~$ mkdir -p $HOME/bin
wp@hetzner3:~$
wp@hetzner3:~$ ln -s $HOME/.wp-cli/wp-cli-2.11.0.phar $HOME/bin/wp
wp@hetzner3:~$
wp@hetzner3:~$ chown wp:wp -R /$HOME/.wp-cli
wp@hetzner3:~$
wp@hetzner3:~$ find $HOME/.wp-cli -type d -exec chmod 0700 {} \;
wp@hetzner3:~$
wp@hetzner3:~$ find $HOME/.wp-cli -type f -exec chmod 0600 {} \;
wp@hetzner3:~$
wp@hetzner3:~$ chmod 0700 $HOME/.wp-cli/wp-cli-2.11.0.phar
wp@hetzner3:~$
wp@hetzner3:~$ rm -f $HOME/bin/wp
wp@hetzner3:~$
- finally, I tried to execute it but, well, it fails with the same error that wordpress failed with; we're gonna have to update wordpress first
wp@hetzner3:~$ bin/wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/script-loader.php on line 706 Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/script-loader.php on line 706 wp@hetzner3:~$
- anyway, let's go ahead and prepare the 3TOFU commands for verifying these updated versions of wordpress plugins and themes. Yesterday we got these, but they need to be cleaned-up a bit
[root@opensourceecology hetzner3]# ls wordpress_plugins.csv wordpress_plugins_table.txt wordpress_plugins_full.txt wordpress_themes.csv [root@opensourceecology hetzner3]#
- I copied these over to hetnzer3
[root@opensourceecology hetzner3]# ls wordpress_plugins.csv wordpress_plugins_table.txt wordpress_plugins_full.txt wordpress_themes.csv [root@opensourceecology hetzner3]# [root@opensourceecology hetzner3]# cp *.csv /home/maltfield/ [root@opensourceecology hetzner3]# [root@opensourceecology hetzner3]# chown maltfield /home/maltfield/wordpress*.csv [root@opensourceecology hetzner3]# [root@opensourceecology hetzner3]# logout [maltfield@opensourceecology ~]$ [maltfield@opensourceecology ~]$ rsync -e 'ssh -p 32415' -av --progress wordpress*csv 144.76.164.201:: rsync error: syntax or usage error (code 1) at clientserver.c(1296) [Receiver=3.2.7] rsync: did not see server greeting rsync error: error starting client-server protocol (code 5) at main.c(1656) [sender=3.1.2] [maltfield@opensourceecology ~]$ ^C [maltfield@opensourceecology ~]$ rsync -e 'ssh -p 32415' -av --progress wordpress*csv 144.76.164.201: sending incremental file list wordpress_plugins.csv 14,858 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/2) wordpress_themes.csv 3,289 100% 3.14MB/s 0:00:00 (xfr#2, to-chk=0/2) sent 18,330 bytes received 54 bytes 12,256.00 bytes/sec total size is 18,147 speedup is 0.99 [maltfield@opensourceecology ~]$
- I created a secure staging area for all the wordpress downloads that we can use as a trustworthy source for all future upgrades on all our vhost sites
root@hetzner3 /var/tmp # mkdir wordpress
root@hetzner3 /var/tmp # mkdir -p wordpress/{themes,plugins}
root@hetzner3 /var/tmp # mv wp-cli-2.11.0/ wordpress
root@hetzner3 /var/tmp # chown root:root wordpress
root@hetzner3 /var/tmp # chown root:root wordpress/themes
root@hetzner3 /var/tmp # chown root:root wordpress/plugins
root@hetzner3 /var/tmp # chmod 0700 wordpress/
root@hetzner3 /var/tmp # chmod 0700 wordpress/themes/
root@hetzner3 /var/tmp # chmod 0700 wordpress/plugins
root@hetzner3 /var/tmp #
root@hetzner3 /var/tmp # mv /home/maltfield/wordpress_themes.csv wordpress/themes/
root@hetzner3 /var/tmp #
root@hetzner3 /var/tmp # mv /home/maltfield/wordpress_plugins.csv wordpress/plugins/
root@hetzner3 /var/tmp #
- ok, let's start with plugins
root@hetzner3 /var/tmp # cd wordpress/ root@hetzner3 /var/tmp/wordpress # ls plugins themes wp-cli-2.11.0 root@hetzner3 /var/tmp/wordpress # cd plugins/ root@hetzner3 /var/tmp/wordpress/plugins # ls wordpress_plugins.csv root@hetzner3 /var/tmp/wordpress/plugins #
- first let's remove the column
root@hetzner3 /var/tmp/wordpress/plugins # cp wordpress_plugins.csv wordpress_plugins.csv.orig root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # vim wordpress_plugins.csv root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # diff wordpress_plugins.csv.orig wordpress_plugins.csv 1d0 < name,status,update,version root@hetzner3 /var/tmp/wordpress/plugins #
- oh shit, apparently there's many of these (one for each site). I'll have to 'uniq' and then delete
root@hetzner3 /var/tmp/wordpress/plugins # cat wordpress_plugins.csv | cut -d, -f1 | sort | uniq > wordpress_plugins1.txt root@hetzner3 /var/tmp/wordpress/plugins # vim wordpress_plugins1.txt root@hetzner3 /var/tmp/wordpress/plugins #
- I threw together a quick script to get the latest version & download URLs for each of these plugins
for plugin in $(cat wordpress_plugins1.txt); do
echo $plugin;
curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
echo -e "\t$url"
done
- unfortunately an awful lot of them are "null"
root@hetzner3 /var/tmp/wordpress/plugins # for plugin in $(cat wordpress_plugins1.txt); do echo $plugin; curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json; latest_version=$(cat plugin.json | jq -r .version); url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\""); echo -e "\t$url"; done
...
wpautop-control
https://downloads.wordpress.org/plugin/wpautop-control.1.6.zip
wp-facebook-open-graph-protocol
null
wp-memory-usage
https://downloads.wordpress.org/plugin/wp-memory-usage.1.2.10.zip
wpmudev-updates
null
wp-optimize
https://downloads.wordpress.org/plugin/wp-optimize.3.6.0.zip
wp-simple-galleries
null
wp-smushit
https://downloads.wordpress.org/plugin/wp-smushit.3.16.6.zip
wp-smush-pro
null
wp-super-cache
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
root@hetzner3 /var/tmp/wordpress/plugins #
- I did a spot-check on just one of these, and I found that the website says that the plugin was removed from wordpress due to a "Security Issue" — ah crap, is that what all these are? https://wordpress.org/plugins/wp-simple-galleries/
This plugin has been closed as of October 27, 2023 and is not available for download. Reason: Security Issue.
- alright, well, at least the API returns the reason
root@hetzner3 /var/tmp/wordpress/plugins # curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/wp-simple-galleries.json
root@hetzner3 /var/tmp/wordpress/plugins # cat plugin.json
{"error":"closed","name":"WP Simple Galleries","slug":"wp-simple-galleries","description":"This plugin has been closed as of October 27, 2023 and is not available for download. Reason: Security Issue.","closed":true,"closed_date":"2023-10-27 21:14:19","reason":"security-issue","reason_text":"Security Issue"}root@hetzner3 /var/tmp/wordpress/plugins #
- let's update the script to output some error handling for each
for plugin in $(cat wordpress_plugins1.txt); do
echo $plugin;
curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
echo -e "\t$url"
if [ "${url}" = "null" ]; then
cat plugin.json
echo
fi
echo
done
- here's the output
root@hetzner3 /var/tmp/wordpress/plugins # for plugin in $(cat wordpress_plugins1.txt); do
echo $plugin;
curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
echo -e "\t$url"
if [ "${url}" = "null" ]; then
cat plugin.json
echo
fi
echo
done > urls.txt
root@hetzner3 /var/tmp/wordpress/plugins #
- and here's the output
akismet
https://downloads.wordpress.org/plugin/akismet.5.3.3.zip
amr-shortcode-any-widget
null
{"error":"closed","name":"amr shortcode any widget","slug":"amr-shortcode-any-widget","description":"This plugin has been closed as of January 19, 2023 and is not available for download. Reason: Security Issue.","closed":true,"closed_date":"2023-01-19 19:04:59","reason":"security-issue","reason_text":"Security Issue"}
be-gdpr
null
{"error":"Plugin not found."}
be-page-builder
null
{"error":"Plugin not found."}
be-portfolio-post
null
{"error":"Plugin not found."}
be-themes-one-click-import
null
{"error":"Plugin not found."}
black-studio-tinymce-widget
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
brankic-photostream-widget
null
{"error":"closed","name":"Brankic Photostream Widget","slug":"brankic-photostream-widget","description":"This plugin has been closed as of October 19, 2018 and is not available for download. Reason: Guideline Violation.","closed":true,"closed_date":"2018-10-19 19:17:29","reason":"guideline-violation","reason_text":"Guideline Violation"}
chartbeat
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
classic-editor
https://downloads.wordpress.org/plugin/classic-editor.1.6.5.zip
coingate-for-woocommerce
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.1.1.zip
colorhub
null
{"error":"Plugin not found."}
contact-form-7
https://downloads.wordpress.org/plugin/contact-form-7.5.9.8.zip
cyclone-slider-2
null
{"error":"closed","name":"Cyclone Slider","slug":"cyclone-slider-2","description":"This plugin has been closed as of August 2, 2018 and is not available for download. Reason: Guideline Violation.","closed":true,"closed_date":"2018-08-02 23:18:54","reason":"guideline-violation","reason_text":"Guideline Violation"}
duplicate-page
null
{"name":"Duplicate Page","slug":"duplicate-page","version":"4.5.4","author":"<a href=\"https:\/\/profiles.wordpress.org\/mndpsingh287\/\">mndpsingh287<\/a>","author_profile":"https:\/\/profiles.wordpress.org\/mndpsingh287\/","requires":"3.4","tested":"6.6.2","requires_php":false,"requires_plugins":[],"compatibility":[],"rating":96,"ratings":{"5":358,"4":5,"3":4,"2":4,"1":12},"num_ratings":383,"support_threads":4,"support_threads_resolved":1,"downloaded":28887807,"last_updated":"2024-07-29 10:47am GMT","added":"2016-05-04","homepage":"https:\/\/wordpress.org\/plugins\/duplicate-page\/","sections":{"description":"<p>Duplicate Posts, Pages and Custom Posts easily using single click. You can duplicate your pages, posts and custom post by just one click and it will save as your selected options (draft, private, public, pending).<\/p>\n<h4>Key Features in Duplicate Page Pro Editions<\/h4>\n<ul>\n<li><strong>User Roles:<\/strong> Allow User Roles To access Duplicate Page.<\/li>\n<li><strong>Post Types:<\/strong> Filter to show Duplicate Page link in post types.<\/li>\n<li><strong>Clone Link Location:<\/strong> Option where to show clone link.<\/li>\n<li><strong>Status:<\/strong> Option to select Duplicate Posts Status.<\/li>\n<li><strong>Redirection:<\/strong> Option to Redirect after click on clone link..<\/li>\n<li><strong>Clone Link Title:<\/strong> Option to change Duplicate Post Link Title.<\/li>\n<li><strong>Post Prefix:<\/strong> Option to add Post Prefix.<\/li>\n<li><strong>Post Suffix:<\/strong> Option to add Post Suffix.<\/li>\n<li><strong>Editor<\/strong>: And Many More Filters and Features.<\/li>\n<\/ul>\n<blockquote>\n<p><strong><a href=\"https:\/\/duplicatepro.com\/pro\/?utm_source=Wordpress.org&utm_medium=Website&utm_campaign=Duplicate%20Page%20Pro\" rel=\"nofollow ugc\">Buy Pro Version<\/a><\/strong> with various features & support.<\/p>\n<p><strong><a href=\"https:\/\/duplicatepro.com\/contact\/?utm_source=Wordpress.org&utm_medium=Website&utm_campaign=Duplicate%20Page%20Pro\" rel=\"nofollow ugc\">Contact us<\/a><\/strong> for Support Only Pro Version Users.<\/p>\n<\/blockquote>\n<p><strong><a href=\"https:\/\/duplicatepro.com\/pro\/?utm_source=Wordpress.org&utm_medium=Website&utm_campaign=Duplicate%20Page%20Pro\" rel=\"nofollow ugc\">Upgrade to Pro Version<\/a><\/strong><\/p>\n<p><iframe loading=\"lazy\" title=\"Duplicate page WordPress Plugin | Webdesi9\" width=\"750\" height=\"422\" src=\"https:\/\/www.youtube.com\/embed\/Fj8BHxvebXs?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p>\n<h3>How to use<\/h3>\n<ol>\n<li>First Activate Plugin.<\/li>\n<li>Go Select to Duplicate Page settings Menu from Settings Tab and savings settings. <\/li>\n<li>Then Create New Post\/Page or Use old.<\/li>\n<li>After click on duplicate this link, then duplicate post\/ page will be created and saved as draft,publish,pending,private depending upon settings.<\/li>\n<\/ol>\n<h3>Minimum requirements for Duplicate Page<\/h3>\n<ul>\n<li>WordPress 3.3+<\/li>\n<li>PHP 5.x<\/li>\n<li>MySQL 5.x<\/li>\n<\/ul>\n","installation":"<ol>\n<li>Upload the <code>duplicate-page<\/code> folder to the directory <code>\/wp-content\/plugins\/<\/code>.<\/li>\n<li>Activate the plugin using the ‘Plugins’ menu in WordPress.<\/li>\n<\/ol>\n","changelog":"<h4>4.5.4 (29th July, 2024)<\/h4>\n<ul>\n<li>Compatible with WordPress 6.6.1<\/li>\n<\/ul>\n<p>= 4.5.3(11th Sept, 2023)<\/p>\n<ul>\n<li>Compatible with wordpress 6.3.1<\/li>\n<\/ul>\n<p>= 4.5.2(4th May, 2023)<\/p>\n<ul>\n<li>Compatible with wordpress 6.2<\/li>\n<\/ul>\n<p>= 4.5.1(8th Feb, 2023)<\/p>\n<ul>\n<li>Fixes compatibility issues with event calendar plugin.<\/li>\n<li>Minor bug fixes<\/li>\n<\/ul>\n<p>= 4.5(14th Dec, 2022)<\/p>\n<ul>\n<li>Checked compatibility with wordpress v6.1.1 <\/li>\n<li>Fixes elementor sections compatibility issue<\/li>\n<\/ul>\n<p>= 4.4.9(15th Jul, 2022)<\/p>\n<ul>\n<li>Checked compatibility with wordpress v6.0.1<\/li>\n<\/ul>\n<p>= 4.4.8(27th Jan, 2022)<\/p>\n<ul>\n<li>Checked compatibility with wordpress v5.9<\/li>\n<\/ul>\n<p>= 4.4.7(16th Dec, 2021)<\/p>\n<ul>\n<li>Added editor role check<\/li>\n<\/ul>\n<p>= 4.4.6(7th Dec, 2021)<\/p>\n<ul>\n<li>Fixed security issues.<\/li>\n<\/ul>\n<p>= 4.4.5(29th Sep, 2021)<\/p>\n<ul>\n<li>Fixed the issue for content HTML displaying in the sidebar.<\/li>\n<\/ul>\n<p>= 4.4.4(16th Sep, 2021)<\/p>\n<ul>\n<li>Fixed sanitization Issues as per wordpress standards.<\/li>\n<\/ul>\n<p>= 4.4.3(8th Sep, 2021)<\/p>\n<ul>\n<li>Updated code sanitization process for secure data submission<\/li>\n<\/ul>\n<p>= 4.4.2(6th Sep, 2021)<\/p>\n<ul>\n<li>Added sanitization as per wordpress standards.<\/li>\n<\/ul>\n<h4>4.4.1(27th July, 2021)<\/h4>\n<ul>\n<li>Checked compatibility with wordpress v5.8<\/li>\n<li>Fixed php warnings<\/li>\n<\/ul>\n<h4>4.4(18th Mar, 2021)<\/h4>\n<ul>\n<li>Tested upto wordpress v5.7<\/li>\n<\/ul>\n<h4>4.3(28th Aug, 2020)<\/h4>\n<ul>\n<li>Tested with wordpress v5.5<\/li>\n<\/ul>\n<h4>4.2(26th March, 2020)<\/h4>\n<ul>\n<li>Fixed Translations string issue<\/li>\n<\/ul>\n<h4>4.1(20th Feb, 2020)<\/h4>\n<ul>\n<li>Fixed Translations issue<\/li>\n<\/ul>\n<h4>4.0(20th Sep, 2019)<\/h4>\n<ul>\n<li>Fixed slug duplication issue<\/li>\n<\/ul>\n<h4>3.9 (6th Aug, 2019)<\/h4>\n<ul>\n<li>Fixed some compatibility issues<\/li>\n<\/ul>\n<h4>3.8 (18th July, 2019)<\/h4>\n<ul>\n<li>Fixed compatibility issues with GT3 editor<\/li>\n<\/ul>\n<h4>3.7 (18th July, 2019)<\/h4>\n<ul>\n<li>Fixed compatibility issues with other editors<\/li>\n<\/ul>\n<h4>3.6 (16th July, 2019)<\/h4>\n<ul>\n<li>Security fixes addressed by wordpress<\/li>\n<\/ul>\n<h4>3.5 (2nd Apr, 2019)<\/h4>\n<ul>\n<li>Gutenberg and advanced custom fields compatibility <\/li>\n<\/ul>\n<h4>3.4 (1st Apr, 2019)<\/h4>\n<ul>\n<li>Security issues fixes addressed by Securi<\/li>\n<\/ul>\n<h4>3.3 (14th March, 2019)<\/h4>\n<ul>\n<li>Compatible with WordPress 5.1.1<\/li>\n<\/ul>\n<h4>3.2 (25th Dec, 2018)<\/h4>\n<ul>\n<li>Gutenberg Compatible<\/li>\n<\/ul>\n<h4>3.1 (5th Dec, 2018)<\/h4>\n<ul>\n<li>removed nag<\/li>\n<\/ul>\n<h4>3.0 (3rd Dec, 2018)<\/h4>\n<ul>\n<li>German Formal lang added<\/li>\n<\/ul>\n<h4>2.9 (16th Nov, 2018)<\/h4>\n<ul>\n<li>Minor bug fixes<\/li>\n<\/ul>\n<h4>2.8 (20th Oct, 2018)<\/h4>\n<ul>\n<li>Compatible with php 7.3 and wordpress 5.0<\/li>\n<\/ul>\n<h4>2.7 (28th July, 2018)<\/h4>\n<ul>\n<li>removed useless adsense code<\/li>\n<\/ul>\n<h4>2.6 (24th March, 2018)<\/h4>\n<ul>\n<li>Added signup popup and minor fixes.<\/li>\n<\/ul>\n<h4>2.5 (5th Feb, 2018)<\/h4>\n<ul>\n<li>Translations issue resolve.<\/li>\n<\/ul>\n<h4>2.4 (29th Nov, 2017)<\/h4>\n<ul>\n<li>Duplicate Page appends original post name issue fixed.<\/li>\n<\/ul>\n<h4>2.3 (27st april, 2017)<\/h4>\n<ul>\n<li>Minor Fixes<\/li>\n<\/ul>\n<h4>2.2 (28th Jan, 2017)<\/h4>\n<ul>\n<li>Strings Translations<\/li>\n<\/ul>\n<h4>2.1 (25th Aug, 2016)<\/h4>\n<ul>\n<li>New Text field added in settings page for duplicate post title suffix to remove confusion of currently duplicate page.<\/li>\n<\/ul>\n<h4>1.4 (18th Jun, 2016)<\/h4>\n<ul>\n<li>New Features added<\/li>\n<\/ul>\n<h4>1.3 (23th May, 2016)<\/h4>\n<ul>\n<li>New Features added<\/li>\n<\/ul>\n<h4>1.2 (05th May, 2016)<\/h4>\n<ul>\n<li>Duplicate Page Settings Menu Added.<\/li>\n<\/ul>\n<h4>1.1 (04th May ,2016)<\/h4>\n<ul>\n<li>fix some Bug in 1.0<\/li>\n<\/ul>\n","screenshots":"<ol><li><a href=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-1.png?rev=1410933\"><img src=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-1.png?rev=1410933\" alt=\"Activate Screen\"><\/a><p>Activate Screen<\/p><\/li><li><a href=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-2.png?rev=1779481\"><img src=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-2.png?rev=1779481\" alt=\"Duplicate Page Settings Screen\"><\/a><p>Duplicate Page Settings Screen<\/p><\/li><li><a href=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-3.png?rev=1410933\"><img src=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-3.png?rev=1410933\" alt=\"Select Option from Settings Page.\"><\/a><p>Select Option from Settings Page.<\/p><\/li><li><a href=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-4.png?rev=1410933\"><img src=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-4.png?rev=1410933\" alt=\"Click on Duplicate This.\"><\/a><p>Click on Duplicate This.<\/p><\/li><li><a href=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-5.png?rev=1410933\"><img src=\"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-5.png?rev=1410933\" alt=\"Duplicate Post \/ Page will Appear.\"><\/a><p>Duplicate Post \/ Page will Appear.<\/p><\/li><\/ol>"},"download_link":"https:\/\/downloads.wordpress.org\/plugin\/duplicate-page.zip","screenshots":{"1":{"src":"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-1.png?rev=1410933","caption":"Activate Screen"},"2":{"src":"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-2.png?rev=1779481","caption":"Duplicate Page Settings Screen"},"3":{"src":"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-3.png?rev=1410933","caption":"Select Option from Settings Page."},"4":{"src":"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-4.png?rev=1410933","caption":"Click on Duplicate This."},"5":{"src":"https:\/\/ps.w.org\/duplicate-page\/assets\/screenshot-5.png?rev=1410933","caption":"Duplicate Post \/ Page will Appear."}},"tags":{"duplicate-custom-posts":"duplicate custom posts","duplicate-page":"duplicate page","duplicate-post":"duplicate post","page-duplicate":"Page Duplicate","post-duplicate":"Post duplicate"},"versions":{"1.1":"https:\/\/downloads.wordpress.org\/plugin\/duplicate-page.1.1.zip","2.5":"https:\/\/downloads.wordpress.org\/plugin\/duplicate-page.2.5.zip","4.4.9":"https:\/\/downloads.wordpress.org\/plugin\/duplicate-page.4.4.9.zip","4.5":"https:\/\/downloads.wordpress.org\/plugin\/duplicate-page.4.5.zip"},"donate_link":"","contributors":{"mndpsingh287":"https:\/\/profiles.wordpress.org\/mndpsingh287\/"}}
duplicate-post
https://downloads.wordpress.org/plugin/duplicate-post.4.5.zip
flattr
null
{"error":"closed","name":"Flattr","slug":"flattr","description":"This plugin has been closed as of April 28, 2024 and is not available for download. Reason: Security Issue.","closed":true,"closed_date":"2024-04-28 19:13:13","reason":"security-issue","reason_text":"Security Issue"}
force-strong-passwords
null
{"error":"closed","name":"Force Strong Passwords","slug":"force-strong-passwords","description":"This plugin has been closed as of November 23, 2020 and is not available for download. This closure is permanent. Reason: Author Request.","closed":true,"closed_date":"2020-11-23 22:35:26","reason":"author-request","reason_text":"Author Request"}
fundraising
null
{"error":"Plugin not found."}
google-authenticator
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
google-authenticator-encourage-user-activation
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
hello
null
{"error":"Plugin not found."}
insert-headers-and-footers
https://downloads.wordpress.org/plugin/insert-headers-and-footers.2.2.2.zip
jetpack
https://downloads.wordpress.org/plugin/jetpack.13.8.1.zip
masterslider
null
{"error":"Plugin not found."}
media-element-html5-video-and-audio-player
null
{"error":"closed","name":"MediaElement.js – HTML5 Video & Audio Player","slug":"media-element-html5-video-and-audio-player","description":"This plugin has been closed as of December 29, 2022 and is not available for download. Reason: Security Issue.","closed":true,"closed_date":"2022-12-29 17:26:05","reason":"security-issue","reason_text":"Security Issue"}
meta-box
https://downloads.wordpress.org/plugin/meta-box.5.10.2.zip
meta-box-conditional-logic
null
{"error":"Plugin not found."}
meta-box-show-hide
null
{"error":"Plugin not found."}
meta-box-tabs
null
{"error":"Plugin not found."}
ml-slider
https://downloads.wordpress.org/plugin/ml-slider.3.91.0.zip
ml-slider-pro
null
{"error":"Plugin not found."}
newsletter-sign-up
null
{"error":"closed","name":"Newsletter Sign-Up","slug":"newsletter-sign-up","description":"This plugin has been closed as of November 18, 2022 and is not available for download. This closure is permanent. Reason: Author Request.","closed":true,"closed_date":"2022-11-18 12:00:14","reason":"author-request","reason_text":"Author Request"}
oa-open-graph-for-fb
null
{"error":"closed","name":"Facebook Open Graph by OAlves","slug":"oa-open-graph-for-fb","description":"This plugin has been closed as of September 7, 2019 and is not available for download. Reason: Licensing\/Trademark Violation.","closed":true,"closed_date":"2019-09-07 01:03:59","reason":"licensing-trademark-violation","reason_text":"Licensing\/Trademark Violation"}
open-in-new-window-plugin
https://downloads.wordpress.org/plugin/open-in-new-window-plugin.3.0.zip
oshine-core
null
{"error":"Plugin not found."}
oshine-modules
null
{"error":"Plugin not found."}
patch-for-revolution-slider
null
{"error":"closed","name":"Patch for Revolution Slider","slug":"patch-for-revolution-slider","description":"This plugin has been closed as of March 16, 2020 and is not available for download. This closure is permanent. Reason: Author Request.","closed":true,"closed_date":"2020-03-16 16:42:05","reason":"author-request","reason_text":"Author Request"}
post-types-order
https://downloads.wordpress.org/plugin/post-types-order.2.2.6.zip
really-simple-facebook-twitter-share-buttons
null
{"error":"closed","name":"Really Simple Share","slug":"really-simple-facebook-twitter-share-buttons","description":"This plugin has been closed and is no longer available for download.","closed":true,"closed_date":false,"reason":"","reason_text":"Unknown"}
recent-tweets-widget
null
{"error":"closed","name":"Recent Tweets Widget","slug":"recent-tweets-widget","description":"This plugin has been closed as of March 20, 2024 and is not available for download. Reason: Guideline Violation.","closed":true,"closed_date":"2024-03-20 07:26:02","reason":"guideline-violation","reason_text":"Guideline Violation"}
redux-vendor-support
null
{"error":"Plugin not found."}
rename-wp-login
null
{"error":"closed","name":"Rename wp-login.php","slug":"rename-wp-login","description":"This plugin has been closed as of June 13, 2022 and is not available for download. Reason: Security Issue.","closed":true,"closed_date":"2022-06-13 16:28:54","reason":"security-issue","reason_text":"Security Issue"}
revision-control
https://downloads.wordpress.org/plugin/revision-control.2.3.2.zip
revslider
null
{"error":"Plugin not found."}
shareaholic
https://downloads.wordpress.org/plugin/shareaholic.9.7.12.zip
share-on-diaspora
https://downloads.wordpress.org/plugin/share-on-diaspora.0.7.9.zip
shariff
https://downloads.wordpress.org/plugin/shariff.4.6.14.zip
social-media-buttons-toolbar
null
{"error":"closed","name":"Social Media Follow Buttons Bar","slug":"social-media-buttons-toolbar","description":"This plugin has been closed as of November 10, 2022 and is not available for download. Reason: Guideline Violation.","closed":true,"closed_date":"2022-11-10 16:34:23","reason":"guideline-violation","reason_text":"Guideline Violation"}
ssl-insecure-content-fixer
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
tatsu
null
{"error":"Plugin not found."}
typehub
null
{"error":"Plugin not found."}
varnish-http-purge
https://downloads.wordpress.org/plugin/varnish-http-purge.5.2.2.zip
vcaching
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
vcaching.bak
w3-total-cache
https://downloads.wordpress.org/plugin/w3-total-cache.2.7.6.zip
wonderm00ns-simple-facebook-open-graph-tags
https://downloads.wordpress.org/plugin/wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip
woocommerce
https://downloads.wordpress.org/plugin/woocommerce.9.3.3.zip
wordpress-importer
https://downloads.wordpress.org/plugin/wordpress-importer.0.8.2.zip
wordpress-seo
https://downloads.wordpress.org/plugin/wordpress-seo.23.5.zip
wpautop-control
https://downloads.wordpress.org/plugin/wpautop-control.1.6.zip
wp-facebook-open-graph-protocol
null
{"error":"closed","name":"WP Facebook Open Graph protocol","slug":"wp-facebook-open-graph-protocol","description":"This plugin has been closed as of September 6, 2019 and is not available for download. Reason: Licensing\/Trademark Violation.","closed":true,"closed_date":"2019-09-06 02:57:41","reason":"licensing-trademark-violation","reason_text":"Licensing\/Trademark Violation"}
wp-memory-usage
https://downloads.wordpress.org/plugin/wp-memory-usage.1.2.10.zip
wpmudev-updates
null
{"error":"Plugin not found."}
wp-optimize
https://downloads.wordpress.org/plugin/wp-optimize.3.6.0.zip
wp-simple-galleries
null
{"error":"closed","name":"WP Simple Galleries","slug":"wp-simple-galleries","description":"This plugin has been closed as of October 27, 2023 and is not available for download. Reason: Security Issue.","closed":true,"closed_date":"2023-10-27 21:14:19","reason":"security-issue","reason_text":"Security Issue"}
wp-smushit
https://downloads.wordpress.org/plugin/wp-smushit.3.16.6.zip
wp-smush-pro
null
{"error":"Plugin not found."}
wp-super-cache
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
- so I see the following reasons for failure
- plugin was closed
- for "Security Issue"
- for "Guideline Violation"
- for "Author Request"
- for "Licensing/Trademark Violation"
- for "Unknown"
- plugin not found
- the "version" doesn't actually have a cooresponding link for that version (but we can probably just manually parse it to grab the latest version)
- it was just a backup that I made (eg 'vcaching.bak')
- plugin was closed
- let's just investigate the ones that fail
for plugin in $(cat wordpress_plugins1.txt); do
#echo $plugin;
curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
#echo -e "\t$url"
if [ "${url}" = "null" ]; then
echo $plugin
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
echo -e "\t $error"
echo -e "\t $description"
fi
done
- here's the execution
root@hetzner3 /var/tmp/wordpress/plugins # for plugin in $(cat wordpress_plugins1.txt); do
#echo $plugin;
curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
#echo -e "\t$url"
if [ "${url}" = "null" ]; then
echo $plugin
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
echo -e "\t $error"
echo -e "\t $description"
fi
done > fail.txt
root@hetzner3 /var/tmp/wordpress/plugins #
- looks like there's 36 of them (out of 68)
root@hetzner3 /var/tmp/wordpress/plugins # cat fail.txt | grep -viE '^\s' amr-shortcode-any-widget be-gdpr be-page-builder be-portfolio-post be-themes-one-click-import brankic-photostream-widget colorhub cyclone-slider-2 duplicate-page flattr force-strong-passwords fundraising hello masterslider media-element-html5-video-and-audio-player meta-box-conditional-logic meta-box-show-hide meta-box-tabs ml-slider-pro newsletter-sign-up oa-open-graph-for-fb oshine-core oshine-modules patch-for-revolution-slider really-simple-facebook-twitter-share-buttons recent-tweets-widget redux-vendor-support rename-wp-login revslider social-media-buttons-toolbar tatsu typehub wp-facebook-open-graph-protocol wpmudev-updates wp-simple-galleries wp-smush-pro root@hetzner3 /var/tmp/wordpress/plugins # cat fail.txt | grep -viE '^\s' | wc -l 36 root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # wc -l wordpress_plugins2.txt 68 wordpress_plugins2.txt root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins #
- here's the full output
amr-shortcode-any-widget closed This plugin has been closed as of January 19, 2023 and is not available for download. Reason: Security Issue. be-gdpr Plugin not found. null be-page-builder Plugin not found. null be-portfolio-post Plugin not found. null be-themes-one-click-import Plugin not found. null brankic-photostream-widget closed This plugin has been closed as of October 19, 2018 and is not available for download. Reason: Guideline Violation. colorhub Plugin not found. null cyclone-slider-2 closed This plugin has been closed as of August 2, 2018 and is not available for download. Reason: Guideline Violation. duplicate-page null null flattr closed This plugin has been closed as of April 28, 2024 and is not available for download. Reason: Security Issue. force-strong-passwords closed This plugin has been closed as of November 23, 2020 and is not available for download. This closure is permanent. Reason: Author Request. fundraising Plugin not found. null hello Plugin not found. null masterslider Plugin not found. null media-element-html5-video-and-audio-player closed This plugin has been closed as of December 29, 2022 and is not available for download. Reason: Security Issue. meta-box-conditional-logic Plugin not found. null meta-box-show-hide Plugin not found. null meta-box-tabs Plugin not found. null ml-slider-pro Plugin not found. null newsletter-sign-up closed This plugin has been closed as of November 18, 2022 and is not available for download. This closure is permanent. Reason: Author Request. oa-open-graph-for-fb closed This plugin has been closed as of September 7, 2019 and is not available for download. Reason: Licensing/Trademark Violation. oshine-core Plugin not found. null oshine-modules Plugin not found. null patch-for-revolution-slider closed This plugin has been closed as of March 16, 2020 and is not available for download. This closure is permanent. Reason: Author Request. really-simple-facebook-twitter-share-buttons closed This plugin has been closed and is no longer available for download. recent-tweets-widget closed This plugin has been closed as of March 20, 2024 and is not available for download. Reason: Guideline Violation. redux-vendor-support Plugin not found. null rename-wp-login closed This plugin has been closed as of June 13, 2022 and is not available for download. Reason: Security Issue. revslider Plugin not found. null social-media-buttons-toolbar closed This plugin has been closed as of November 10, 2022 and is not available for download. Reason: Guideline Violation. tatsu Plugin not found. null typehub Plugin not found. null wp-facebook-open-graph-protocol closed This plugin has been closed as of September 6, 2019 and is not available for download. Reason: Licensing/Trademark Violation. wpmudev-updates Plugin not found. null wp-simple-galleries closed This plugin has been closed as of October 27, 2023 and is not available for download. Reason: Security Issue. wp-smush-pro Plugin not found. null
- I sent an email to Marcin & Catarina to warn them about this
Hey Marcin, Hey Catarina, I wanted to let you know that there are a large number of wordpress plugins that you used to use that are not going to be available on your new server. Across all of your wordpress sites, I counted that you currently use 68 wordpress plugins. Of those, I couldn't get the download link for the latest version of 36 wordpress plugins. The reasons are varied, and include: 1. Plugin was closed 2. Plugin not found 3. Version manifest issue 4. Plugin was removed for "Security Issue" 5. Plugin was removed for "Guideline Violation" 6. Plugin was removed for "Author Request" 7. Plugin was removed for "Licensing/Trademark Violation" 8. Plugin was removed for "Unknown" Some of these might be private plugins that just get installed with the theme, and those won't be an issue. But most of them we'll just have to live without -- which may break some functionality on the website(s) after we migrate. Anyway, please consider the following list of plugins, and do let me know if there's any that you recognize as particularly important. amr-shortcode-any-widget be-gdpr be-page-builder be-portfolio-post be-themes-one-click-import brankic-photostream-widget colorhub cyclone-slider-2 duplicate-page flattr force-strong-passwords fundraising hello masterslider media-element-html5-video-and-audio-player meta-box-conditional-logic meta-box-show-hide meta-box-tabs ml-slider-pro newsletter-sign-up oa-open-graph-for-fb oshine-core oshine-modules patch-for-revolution-slider really-simple-facebook-twitter-share-buttons recent-tweets-widget redux-vendor-support rename-wp-login revslider social-media-buttons-toolbar tatsu typehub wp-facebook-open-graph-protocol wpmudev-updates wp-simple-galleries wp-smush-pro Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- anyway, I repeated this process for themes
- fortunately, the list here is much shorter
root@hetzner3 /var/tmp/wordpress/themes # cat wordpress_themes2.txt ArtWorksResponsive bouquet enigmatic Eventor gk-portfolio gridsby gridthemeresponsive oshin portfolio-press simplephotoRes sketch storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve root@hetzner3 /var/tmp/wordpress/themes #
- all right, here's the 3TOFU script
REMOTE_FILES="https://downloads.wordpress.org/plugin/akismet.5.3.3.zip https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip https://downloads.wordpress.org/plugin/classic-editor.1.6.5.zip https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.1.1.zip https://downloads.wordpress.org/plugin/contact-form-7.5.9.8.zip https://downloads.wordpress.org/plugin/duplicate-post.4.5.zip https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip https://downloads.wordpress.org/plugin/insert-headers-and-footers.2.2.2.zip https://downloads.wordpress.org/plugin/jetpack.13.8.1.zip https://downloads.wordpress.org/plugin/meta-box.5.10.2.zip https://downloads.wordpress.org/plugin/ml-slider.3.91.0.zip https://downloads.wordpress.org/plugin/open-in-new-window-plugin.3.0.zip https://downloads.wordpress.org/plugin/post-types-order.2.2.6.zip https://downloads.wordpress.org/plugin/revision-control.2.3.2.zip https://downloads.wordpress.org/plugin/shareaholic.9.7.12.zip https://downloads.wordpress.org/plugin/share-on-diaspora.0.7.9.zip https://downloads.wordpress.org/plugin/shariff.4.6.14.zip https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip https://downloads.wordpress.org/plugin/varnish-http-purge.5.2.2.zip https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip https://downloads.wordpress.org/plugin/w3-total-cache.2.7.6.zip https://downloads.wordpress.org/plugin/wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip https://downloads.wordpress.org/plugin/woocommerce.9.3.3.zip https://downloads.wordpress.org/plugin/wordpress-importer.0.8.2.zip https://downloads.wordpress.org/plugin/wordpress-seo.23.5.zip https://downloads.wordpress.org/plugin/wpautop-control.1.6.zip https://downloads.wordpress.org/plugin/wp-memory-usage.1.2.10.zip https://downloads.wordpress.org/plugin/wp-optimize.3.6.0.zip https://downloads.wordpress.org/plugin/wp-smushit.3.16.6.zip https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip https://downloads.wordpress.org/plugin/duplicate-page.4.5.zip https://downloads.wordpress.org/theme/bouquet.1.2.5.zip https://downloads.wordpress.org/theme/gk-portfolio.1.5.3.zip https://downloads.wordpress.org/theme/portfolio-press.2.8.0.zip https://downloads.wordpress.org/theme/sketch.1.2.4.zip https://downloads.wordpress.org/theme/storefront.4.6.0.zip https://downloads.wordpress.org/theme/twentyeleven.4.7.zip https://downloads.wordpress.org/theme/twentyfifteen.3.8.zip https://downloads.wordpress.org/theme/twentyfourteen.4.0.zip https://downloads.wordpress.org/theme/twentynineteen.2.9.zip https://downloads.wordpress.org/theme/twentyseventeen.3.7.zip https://downloads.wordpress.org/theme/twentysixteen.3.3.zip https://downloads.wordpress.org/theme/twentyten.4.2.zip https://downloads.wordpress.org/theme/twentythirteen.4.2.zip https://downloads.wordpress.org/theme/twentytwelve.4.3.zip"
CURL="/usr/bin/curl"
WGET="/usr/bin/wget --retry-on-host-error --retry-connrefused"
PYTHON="/usr/bin/python3"
# in tails, we must torify
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
WGET="/usr/bin/torify ${WGET}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
# get the file
for file in ${REMOTE_FILES}; do
wget ${file}
done
# checksum
date -u +"%Y-%m-%d"
sha256sum *
- and here's TOFU 1/3 (Tor, exit in US)
... Congratulations. This browser is configured to use Tor. 2024-09-28 ... 2024-09-28 e39147da6f75b11e2bcd3a23ab6512dc0f1d1715cc6b25d4b0f876cd88f9851a akismet.5.3.3.zip 206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0 black-studio-tinymce-widget.2.7.3.zip 371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213 bouquet.1.2.5.zip 7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86 chartbeat.2.0.7.zip 3144da2236c6017ba1b7431363317d8db5bc541dd3c729c0aa1f597093723078 classic-editor.1.6.5.zip 67ccf342a4c7fe853e84b535bf3cf121c7a0110feaa96ae06f5eda202f8485ba coingate-for-woocommerce.2.1.1.zip 357b13dab0e2b9996444664c95bd7ee7ca49fb18aced65db4c40c747543afcf4 contact-form-7.5.9.8.zip fe9f4e60683b46eb409b9f782cbdee29e22c345430aafc898f78a1a00d88265c duplicate-page.4.5.zip a32d701644bc5ef0ac50047fcbb8b1c1ab9289d6531dfe986d6574f00d16b921 duplicate-post.4.5.zip 579488be50b99ea7401be9f8127efd7a243c8f213f8f90ede15b07577eb5e9d6 gk-portfolio.1.5.3.zip b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058 google-authenticator.0.54.zip 9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e google-authenticator-encourage-user-activation.0.2.zip 78a00cf800196b7a03599bfd8c49337a34916a09e570fd89b4c0ee82108d8f15 insert-headers-and-footers.2.2.2.zip 466b7a5a05140ce0d7ae2a2e475cdb22cd91ff7d10bc0d14068eaa8af71fb944 jetpack.13.8.1.zip 9bc91e2e6e2172f5284b24893352b8a233a24da31b34ce72a15ce584ec66d7b7 meta-box.5.10.2.zip 1f092fa51e2e3abd0c7a518db9d6ab5ee5dbc1abfa7551439350ddac9adb99fe ml-slider.3.91.0.zip c6123863c3472a40d52cba2ded434a37f6bd488f9e0c280470ccac68256f40c9 open-in-new-window-plugin.3.0.zip 33a58c7c61b1068b112b525556506e5f175988638db583883f7d2c196ef5e2d5 portfolio-press.2.8.0.zip c313fcee168a055005545c98ef665dcdc7b0693838851e5ba77a05c9b9a99fe6 post-types-order.2.2.6.zip d9740796220eb255937d81f8d2b3e7066a154bbce88f7a8d4555ad87a90b2583 revision-control.2.3.2.zip 80149aa1c6f83282d420b6c90ffd1d3c3b1ddfdcec2096c7411b568662fe0dd8 shareaholic.9.7.12.zip 24384df22558e87536aa848b21eee183ca6226fbfcea75af3821437325313ac0 share-on-diaspora.0.7.9.zip 254f144ad18558a91ed632dbb017e3a50a81872f8f9933d7ae77c85436175bd0 shariff.4.6.14.zip 1894750514b49301b3cb93ced16d48b60dee8654a5ce86f803b8454fffd9305e sketch.1.2.4.zip 122fa98f8b3674e4dfef0fa34a235db0cc6cff80c870d14d8159ba88d8796a1d ssl-insecure-content-fixer.2.7.2.zip 576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311 storefront.4.6.0.zip 02d33b647e5309a5817b58c3a51cb17ca801293a2a9653e33fd4ccd0fdb132b7 twentyeleven.4.7.zip 56309284b46e1953b855a21668cfba0f9fdd5e55acf9cdf2578f4cfd768048c7 twentyfifteen.3.8.zip 9af0e22d0a4e98535d852963e6bb23c7a2866167532df312edc27aa42257cb60 twentyfourteen.4.0.zip 8cecbed27f3af39c8964f09a2dbf2b58e50958fa7b7b00437b95fb5db891839a twentynineteen.2.9.zip 215aabae3e9cea494c012c419f31836a8ce2fb321b4210bccd05d0d263d6b727 twentyseventeen.3.7.zip 7df868e367ef21faf0ef71a0aab87dc2df8e35d53aa954165960f87d07b84f92 twentysixteen.3.3.zip 15b8b27950e68c655daed10d3ed0cd06ba55b814afed524374a68aff50856d5a twentyten.4.2.zip d1f82982f19e9aa75ae69db2eb3d76e2dd1f2cee737fbfac7390870970bbf67b twentythirteen.4.2.zip 2e699f0d0542d132aabc84f4813573aed246cac3624871299ed42d87152d4e10 twentytwelve.4.3.zip 832f4cf27722c448b44f5b7ff5583de3b5e8390ac881d37bcce4cdd5c5089be4 varnish-http-purge.5.2.2.zip 02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c vcaching.1.8.3.zip 1dbadde30b8e1e8b8785ec4ab319059e1ff158e224ce38c1df2a2688657294d9 w3-total-cache.2.7.6.zip 5c1e3005e5e5f3f614e50d0824ded56f822dd0a826fdad46a2e9c2dd76cf02be wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip a5f5ee58d239b9d170f6e6bfcbc29ae9c19d0298429ce73a8275f74e95524295 woocommerce.9.3.3.zip 9f0d2c2d82f6ec68f75307b65f723dda44ad7dce7931c307a8872d04ec9ca268 wordpress-importer.0.8.2.zip b7bda6589f00689021187d102912127fa7dc8e3289689f4d2ae2563c0a487a9c wordpress-seo.23.5.zip dfa3820f767ff202f942100aadd6e05acb144d5edaf842cfea26a6520aa71995 wpautop-control.1.6.zip fa9a177e7228bfdc6fddabb6be802a1fff32f8dddaba77cfea6c25c56788d8b0 wp-memory-usage.1.2.10.zip 79cacaea2b1f12c664e24347b2bc58f08db806bd25144f7e7332c494be115e66 wp-optimize.3.6.0.zip 17a675cb0262512d5247745bc0a1de0f72483c629b185dd43483071885d2e5eb wp-smushit.3.16.6.zip 2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d wp-super-cache.1.12.4.zip user@host:/tmp/user/1000/tmp.o4xa3t3gHj$
- back to the list above, there's two plugins that were missing that I personally recognize, and I'm going to want to find replacements for
- rename-wp-login
- force-strong-passwords
- rename-wp-login was a nice, simple plugin that just changed the path of 'wp-login.php', which cut the head off of a ton of dumb brute force scanners
- looks like a popular replacement is wps-hide-login https://wordpress.org/plugins/wps-hide-login/
- it advertises itself as extremely basic, which is what we want
- wp-hive confirms that it's super lightweight, great! https://wphive.com/plugins/wps-hide-login/
- actually, when I went looking for a simple plugin that replaces force-strong-passwords, the best one I found also includes moving the login page; so this may work for both
- in general, I'm trying to avoid all the top "secure all the things" plugins, because I just want something simple -- not something complex that will slow down the site
- here's the plugin https://wordpress.org/plugins/melapress-login-security/
- wphive confirms it's lightweight and has little impact on the site; I say we take it https://wphive.com/plugins/melapress-login-security/
- this will replace both those plugins, perfect!
Thr Sep 26, 2024
- it's time to copy a snapshot of hetzner2's www files from to hetzner3, so I can test these vhosts
root@mail /var/tmp # mkdir hetzner2-www-20240926 root@mail /var/tmp # chown root:root hetzner2-www-20240926 root@mail /var/tmp # chmod 0700 hetzner2-www-20240926/ root@mail /var/tmp #
- looks like I can just use rclone to download the last backup hetzner2 put on backblaze directly from hetzner3. Cool
time rclone -vvv --progress copy b2:ose-server-backups/daily_hetzner2_20240926_072001.tar.gpg hetzner2-www-20240926/
- it took less than 10 minutes to download our latest 21 GB backup file. Nice.
root@mail /var/tmp # time rclone -vvv --progress copy b2:ose-server-backups/daily_hetzner2_20240926_072001.tar.gpg hetzner2-www-20240926/ 2024/09/26 22:55:05 DEBUG : rclone: Version "v1.60.1-DEV" starting with parameters ["rclone" "-vvv" "--progress" "copy" "b2:ose-server-backups/daily_hetzner2_20240926_072001.tar.gpg" "hetzner2-www-20240926/"] 2024/09/26 22:55:05 DEBUG : Creating backend with remote "b2:ose-server-backups/daily_hetzner2_20240926_072001.tar.gpg" 2024/09/26 22:55:05 DEBUG : Using config file from "/root/.config/rclone/rclone.conf" 2024/09/26 22:55:06 DEBUG : fs cache: adding new entry for parent of "b2:ose-server-backups/daily_hetzner2_20240926_072001.tar.gpg", "b2:ose-server-backups" 2024/09/26 22:55:06 DEBUG : Creating backend with remote "hetzner2-www-20240926/" 2024/09/26 22:55:06 DEBUG : fs cache: renaming cache item "hetzner2-www-20240926/" to be canonical "/var/tmp/hetzner2-www-20240926" 2024-09-26 22:55:07 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: Sizes differ (src 21743146163 vs dst 16389643686) 2024-09-26 22:55:07 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: Starting multi-thread copy with 4 parts of size 5.062Gi 2024-09-26 22:55:07 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 4/4 (16307453952-21743146163) size 5.062Gi starting 2024-09-26 22:55:07 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 1/4 (0-5435817984) size 5.062Gi starting 2024-09-26 22:55:07 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 2/4 (5435817984-10871635968) size 5.062Gi starting 2024-09-26 22:55:07 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 3/4 (10871635968-16307453952) size 5.062Gi starting 2024-09-26 23:00:25 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 1/4 (0-5435817984) size 5.062Gi finished 2024-09-26 23:00:31 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 3/4 (10871635968-16307453952) size 5.062Gi finished 2024-09-26 23:00:45 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 4/4 (16307453952-21743146163) size 5.062Gi finished 2024-09-26 23:00:50 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: multi-thread copy: stream 2/4 (5435817984-10871635968) size 5.062Gi finished 2024-09-26 23:00:50 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: Finished multi-thread copy with 4 parts of size 5.062Gi 2024-09-26 23:01:14 DEBUG : daily_hetzner2_20240926_072001.tar.gpg: sha1 = 4372d11b20710920d93c74cb73455514e0c76a8c OK 2024-09-26 23:01:14 INFO : daily_hetzner2_20240926_072001.tar.gpg: Multi-thread Copied (replaced existing) Transferred: 20.250 GiB / 20.250 GiB, 100%, 7.451 MiB/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 6m8.3s 2024/09/26 23:01:14 INFO : Transferred: 20.250 GiB / 20.250 GiB, 100%, 7.451 MiB/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 6m8.3s 2024/09/26 23:01:14 DEBUG : 4 go routines active real 6m8.362s user 1m30.296s sys 1m39.245s You have new mail in /var/mail/root root@mail /var/tmp # root@mail /var/tmp # du -sh hetzner2-www-20240926/* 21G hetzner2-www-20240926/daily_hetzner2_20240926_072001.tar.gpg root@mail /var/tmp # root@mail /var/tmp # df -h Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 916K 6.3G 1% /run /dev/md2 436G 34G 381G 9% / tmpfs 32G 80K 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/md1 989M 66M 872M 8% /boot tmpfs 6.3G 0 6.3G 0% /run/user/1000 root@mail /var/tmp #
- it took another 3 minutes to decrypt
root@mail /var/tmp/hetzner2-www-20240926 # time gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --output daily_hetzner2_20240926_072001.tar daily_hetzner2_20240926_072001.tar.gpg gpg: WARNING: no command supplied. Trying to guess what you mean ... gpg: AES256.CFB encrypted data gpg: encrypted with 1 passphrase real 2m7.766s user 1m41.110s sys 0m19.485s You have new mail in /var/mail/root root@mail /var/tmp/hetzner2-www-20240926 # root@mail /var/tmp/hetzner2-www-20240926 # ls -lah total 41G drwx------ 2 root root 4.0K Sep 26 23:04 . drwxrwxrwt 10 root root 4.0K Sep 26 22:45 .. -rw-r--r-- 1 root root 21G Sep 26 23:06 daily_hetzner2_20240926_072001.tar -rw-r--r-- 1 root root 21G Sep 26 07:52 daily_hetzner2_20240926_072001.tar.gpg root@mail /var/tmp/hetzner2-www-20240926 #
- and 20 seconds to extract
root@mail /var/tmp/hetzner2-www-20240926 # time tar -xf daily_hetzner2_20240926_072001.tar real 0m18.223s user 0m2.217s sys 0m15.582s root@mail /var/tmp/hetzner2-www-20240926 #
- extracting the compress archive for /var/www/ took another 3 minutes
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www # time tar -xf www.20240926_072001.tar.gz real 2m48.249s user 2m45.639s sys 0m44.612s You have new mail in /var/mail/root root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www # root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www # ls -lah total 19G drwxr-xr-x 3 root root 4.0K Sep 26 23:09 . drwxr-xr-x 8 root root 4.0K Sep 26 23:07 .. drwxr-xr-x 3 root root 4.0K Sep 26 23:09 var -rw-r--r-- 1 root root 19G Sep 26 07:38 www.20240926_072001.tar.gz root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www # root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www # cd var/www/ root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www # ls cgi-bin html html.old phpinfo.php root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www # root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www # cd html root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html # ls 3dp.opensourceecology.org fef.opensourceecology.org oswh.opensourceecology.org store.opensourceecology.org awstats.openbuildinginstitute.org forum.opensourceecology.org phplist.opensourceecology.org varnishTest awstats.opensourceecology.org microfactory.opensourceecology.org seedhome.openbuildinginstitute.org wiki.opensourceecology.org cacti.opensourceecology.org.old munin SITE_DOWN www.openbuildinginstitute.org certbot munin.opensourceecology.org staging.openbuildinginstitute.org www.opensourceecology.org d3d.opensourceecology.org openbuildinginstitute.org staging.opensourceecology.org root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html #
- at this point the disk is 26% full
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html # df -h Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 916K 6.3G 1% /run /dev/md2 436G 106G 309G 26% / tmpfs 32G 80K 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/md1 989M 66M 872M 8% /boot tmpfs 6.3G 0 6.3G 0% /run/user/1000 root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html #
- right, so the first vhost on our list is the forums, which is just one big, simple static site. Let's replace what we have with this
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html # find /var/www/html/forum.opensourceecology.org/
/var/www/html/forum.opensourceecology.org/
/var/www/html/forum.opensourceecology.org/htdocs
/var/www/html/forum.opensourceecology.org/htdocs/index.html
/var/www/html/forum.opensourceecology.org/htdocs/index.php
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html #
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html # rm -rf /var/www/html/forum.opensourceecology.org/
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html #
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html # rsync -av --progress forum.opensourceecology.org /var/www/html/
...
forum.opensourceecology.org/vanilla_docroot_backup.20180113/uploads/userpics/944/pIO15T2FR3HFS.jpg
3,258 100% 15.83kB/s 0:00:00 (xfr#107816, to-chk=0/116865)
sent 3,738,215,467 bytes received 2,103,360 bytes 257,953,022.55 bytes/sec
total size is 3,727,355,888 speedup is 1.00
root@mail /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html #
- I spent a bit of time updating the command to update the permissions on /var/www/
- a few things changed: for one, the group is 'www-data' on Debian, whereas it was 'apache' on CentOS
- also, I decided to make the owner 'root' (as opposed to 'not-apache' on hetzner2) — I don't know why I did that, maybe to avoid SUID?
- I made it a bit more robust; with find commands that could discover all the wordpress & phplist sites
# first pass, whole site
chown -R root:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R root:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown root:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R root:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R root:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R root:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R root:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
- that took a while to run, but here's what I got when it finished
root@hetzner3 ~ # ls -lah /var/www/html/forum.opensourceecology.org/ total 24K d---r-x--- 5 root www-data 4,0K Jul 11 2018 . d---r-x--- 3 root www-data 4,0K Sep 26 23:37 .. d---r-x--- 2 root www-data 4,0K Jul 11 2018 final_backup_before_hetzner1_deprecation_oseforum_20180706-230007 d---r-x--- 16 root www-data 4,0K Feb 4 2018 htdocs ----r----- 1 root www-data 1,1K Jul 11 2018 readme.txt d---r-x--- 13 root www-data 4,0K Jan 25 2018 vanilla_docroot_backup.20180113 root@hetzner3 ~ # root@hetzner3 ~ # ls -lah /var/www/html/forum.opensourceecology.org/htdocs/ total 8,1M d---r-x--- 16 root www-data 4,0K Feb 4 2018 . d---r-x--- 5 root www-data 4,0K Jul 11 2018 .. d---r-x--- 3 root www-data 4,0K Feb 1 2018 activity ----r----- 1 root www-data 47K Feb 1 2018 activity.html d---r-x--- 3 root www-data 4,0K Feb 1 2018 applications d---r-x--- 3 root www-data 4,0K Feb 1 2018 cache d---r-x--- 55 root www-data 4,0K Feb 1 2018 categories ----r----- 1 root www-data 14K Feb 1 2018 categories.1.html ----r----- 1 root www-data 14K Feb 1 2018 categories.html d---r-x--- 6 root www-data 4,0K Feb 1 2018 dashboard d---r-x--- 1007 root www-data 20K Feb 1 2018 discussion d---r-x--- 4 root www-data 4,0K Feb 1 2018 discussions ----r----- 1 root www-data 47K Feb 1 2018 discussions.html d---r-x--- 2 root www-data 7,1M Feb 1 2018 entry ----r----- 1 root www-data 148K Feb 1 2018 index.html d---r-x--- 11 root www-data 4,0K Feb 1 2018 plugins d---r-x--- 948 root www-data 60K Feb 1 2018 profile ----r----- 1 root www-data 12K Feb 1 2018 search.html ----r----- 1 root www-data 26K Feb 1 2018 'search?Search=#000000&Mode=like.html' ----r----- 1 root www-data 29K Feb 1 2018 'search?Search=#0&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#13Lifetime&Mode=like.html' ----r----- 1 root www-data 17K Feb 1 2018 'search?Search=#197187&Mode=like.html' ----r----- 1 root www-data 29K Feb 1 2018 'search?Search=#1&Mode=like.html' ----r----- 1 root www-data 28K Feb 1 2018 'search?Search=#2&Mode=like.html' ----r----- 1 root www-data 15K Feb 1 2018 'search?Search=#363636&Mode=like.html' ----r----- 1 root www-data 29K Feb 1 2018 'search?Search=#3&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#458&Mode=like.html' ----r----- 1 root www-data 19K Feb 1 2018 'search?Search=#4&Mode=like.html' ----r----- 1 root www-data 29K Feb 1 2018 'search?Search=#5&Mode=like.html' ----r----- 1 root www-data 18K Feb 1 2018 'search?Search=#6&Mode=like.html' ----r----- 1 root www-data 14K Feb 1 2018 'search?Search=#7&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#8-High&Mode=like.html' ----r----- 1 root www-data 27K Feb 1 2018 'search?Search=#8&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#9-Industrial&Mode=like.html' ----r----- 1 root www-data 16K Feb 1 2018 'search?Search=#9&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#Another&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#apollo&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#Cloud&Mode=like.html' ----r----- 1 root www-data 14K Feb 1 2018 'search?Search=#Edit&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#Fabiofranca&Mode=like.html' ----r----- 1 root www-data 15K Feb 1 2018 'search?Search=#freecad&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#HUBCAMP&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#openhardware&Mode=like.html' ----r----- 1 root www-data 15K Feb 1 2018 'search?Search=#opensourceecology&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#qi-hardware&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#qihardware&Mode=like.html' ----r----- 1 root www-data 13K Feb 1 2018 'search?Search=#REDIRECT&Mode=like.html' ----r----- 1 root www-data 15K Feb 1 2018 'search?Search=#toc&Mode=like.html' ----r----- 1 root www-data 15K Feb 1 2018 'search?Search=#toctitle&Mode=like.html' d---r-x--- 3 root www-data 4,0K Feb 1 2018 themes d---r-x--- 4 root www-data 4,0K Feb 1 2018 uploads d---r-x--- 3 root www-data 4,0K Feb 1 2018 vanilla d---r-x--- 2 root www-data 4,0K Sep 25 18:02 .well-known root@hetzner3 ~ #
- cool, looks like the site is live (on my system with overridden /etc/hosts)
user@disp3202:~$ curl -si https://forum.opensourceecology.org/ | tail <input type="hidden" id="ConfirmText" value="Are you sure you want to do that?" /> <input type="hidden" id="Okay" value="Okay" /> <input type="hidden" id="Cancel" value="Cancel" /> <input type="hidden" id="Search" value="Search" /> </div><li><a href="http://vanillaforums.org">Powered by Vanilla</a></li> </ul> </div> </div> </body> </html> user@disp3202:~$
- also looks fine in my web browser
- since these sites are identical, I realized we need some way to figure out if we're *actually* looking at hetzner2 or hetnzer3
- we had this same issue on prod vs staging, and the solution was to create a file '/is_staging' in the docroot of every vhost on the staging server
</pre> for docroot in $(sudo find /var/www/html/* -maxdepth 1 -regextype awk -regex ".*(htdocs|public_html)" -type d); do echo "true" | sudo tee "$docroot/is_staging"; done
## I'll steal that code and call it '/_is_hetzner3' instead <pre> for docroot in $(sudo find /var/www/html/* -maxdepth 1 -regextype awk -regex ".*(htdocs|public_html)" -type d); do echo "true" | sudo tee "$docroot/is_hetzner3"; done
- ok, that worked
root@hetzner3 ~ # for docroot in $(sudo find /var/www/html/* -maxdepth 1 -regextype awk -regex ".*(htdocs|public_html)" -type d); do echo "true" | sudo tee "$docroot/is_hetzner3"; done true root@hetzner3 ~ # find /var/www/html -name is_hetzner3 /var/www/html/forum.opensourceecology.org/htdocs/is_hetzner3 root@hetzner3 ~ #
- aaand, confirmed. awesome
user@disp3202:~$ curl -si https://forum.opensourceecology.org/is_hetzner3 | tail X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 65563 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3202:~$
- ok, so that's the easiest site. before I start actually migrating things, I want to go through each site and make sure the process works
- next-up: store.opensourceecology.org
- before I can restore any wordpress sites, I first have to restore the db
- first, I confirmed the db config is working and mariadb is running
root@hetzner3 ~ # systemctl status mariadb
● mariadb.service - MariaDB 10.11.6 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; preset: enabled)
Active: active (running) since Fri 2024-09-27 00:44:31 UTC; 13s ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 854644 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 854645 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 854647 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0>
Process: 854720 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 854722 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 854709 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 12 (limit: 76834)
Memory: 162.3M
CPU: 552ms
CGroup: /system.slice/mariadb.service
└─854709 /usr/sbin/mariadbd
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Note] InnoDB: log sequence number 46980; transaction id 14
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Note] Plugin 'FEEDBACK' is disabled.
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-lo>
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Note] InnoDB: Buffer pool(s) load completed at 240927 0:44:31
Sep 27 00:44:31 hetzner3 mariadbd[854709]: 2024-09-27 0:44:31 0 [Note] /usr/sbin/mariadbd: ready for connections.
Sep 27 00:44:31 hetzner3 mariadbd[854709]: Version: '10.11.6-MariaDB-0+deb12u1' socket: '/run/mysqld/mysqld.sock' port: 0 Debian 12
Sep 27 00:44:31 hetzner3 systemd[1]: Started mariadb.service - MariaDB 10.11.6 database server.
Sep 27 00:44:31 hetzner3 /etc/mysql/debian-start[854724]: Upgrading MySQL tables if necessary.
root@hetzner3 ~ #
root@hetzner3 ~ # systemctl restart mariadb
root@hetzner3 ~ #
root@hetzner3 ~ # systemctl status mariadb
● mariadb.service - MariaDB 10.11.6 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; preset: enabled)
Active: active (running) since Fri 2024-09-27 00:44:48 UTC; 2s ago
Docs: man:mariadbd(8)
https://mariadb.com/kb/en/library/systemd/
Process: 854776 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Process: 854777 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 854779 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0>
Process: 854853 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 854855 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
Main PID: 854842 (mariadbd)
Status: "Taking your SQL requests now..."
Tasks: 12 (limit: 76834)
Memory: 207.4M
CPU: 448ms
CGroup: /system.slice/mariadb.service
└─854842 /usr/sbin/mariadbd
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the fil>
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] InnoDB: log sequence number 46980; transaction id 14
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] Plugin 'FEEDBACK' is disabled.
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-lo>
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] InnoDB: Buffer pool(s) load completed at 240927 0:44:48
Sep 27 00:44:48 hetzner3 mariadbd[854842]: 2024-09-27 0:44:48 0 [Note] /usr/sbin/mariadbd: ready for connections.
Sep 27 00:44:48 hetzner3 mariadbd[854842]: Version: '10.11.6-MariaDB-0+deb12u1' socket: '/run/mysqld/mysqld.sock' port: 0 Debian 12
Sep 27 00:44:48 hetzner3 systemd[1]: Started mariadb.service - MariaDB 10.11.6 database server.
root@hetzner3 ~ #
- I ran the mysql secure script
root@hetzner3 /etc/mysql # mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.
You already have your root account protected, so you can safely answer 'n'.
Switch to unix_socket authentication [Y/n] n
... skipping.
You already have your root account protected, so you can safely answer 'n'.
Change the root password? [Y/n] n
... skipping.
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] y
... Success!
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
You have new mail in /var/mail/root
root@hetzner3 /etc/mysql #
- I actually was expecting that to prompt us for a new root password, but it didn't :(
- yeah, looks like there's no password
- I wonder if we restore the whole db, will it restore the root password too? Let's try
root@hetzner3 /etc/mysql # mysql -u root Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 33 Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
- I extracted and tried to restore the db, but it failed :(
root@hetzner3 ~ # cd /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump/ root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # ls mysqldump.20240926_072001.sql.gz root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # gunzip mysqldump.20240926_072001.sql.gz root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # du -sh * 4,8G mysqldump.20240926_072001.sql root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysql < mysqldump.20240926_072001.sql ERROR 1050 (42S01) at line 6103: Table 'user' already exists You have new mail in /var/mail/root root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- oh, apparently this is a consequence of mysql jumping versions. It was changed to a "view" in 10.4 https://dba.stackexchange.com/questions/266480/mariadb-mysql-all-db-import-table-user-already-exists
- we ran mariadb 5.5.68 on Cent7
[root@opensourceecology ~]# rpm -qa | grep -i mariadb mariadb-5.5.68-1.el7.x86_64 mariadb-server-5.5.68-1.el7.x86_64 mariadb-libs-5.5.68-1.el7.x86_64 [root@opensourceecology ~]#
- and we're running mariadb 10.11.6 on Debian 12
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # dpkg -l | grep -i mariadb ii libdbd-mysql-perl:amd64 4.050-5+b1 amd64 Perl5 database interface to the MariaDB/MySQL database ii libmariadb3:amd64 1:10.11.6-0+deb12u1 amd64 MariaDB database client library ii mariadb-client 1:10.11.6-0+deb12u1 amd64 MariaDB database client binaries ii mariadb-client-core 1:10.11.6-0+deb12u1 amd64 MariaDB database core client binaries ii mariadb-common 1:10.11.6-0+deb12u1 all MariaDB common configuration files ii mariadb-server 1:10.11.6-0+deb12u1 amd64 MariaDB database server binaries ii mariadb-server-core 1:10.11.6-0+deb12u1 amd64 MariaDB database core server files root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- the answer says you just have to add these two lines to the top of the .sql file
DROP TABLE IF EXISTS `mysql`.`global_priv`; DROP VIEW IF EXISTS `mysql`.`user`;
- so I made a backup and added them
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # cp mysqldump.20240926_072001.sql mysqldump.20240926_072001b.sql root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # vim mysqldump.20240926_072001b.sql root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # diff mysqldump.20240926_072001.sql mysqldump.20240926_072001b.sql 0a1,2 > DROP TABLE IF EXISTS `mysql`.`global_priv`; > DROP VIEW IF EXISTS `mysql`.`user`; root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- ok, that worked. the db was restored in less than 5 minutes runtime
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # time mysql -u root < mysqldump.20240926_072001b.sql real 3m13,352s user 0m36,301s sys 0m2,031s You have new mail in /var/mail/root root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- unfortunately, that did *not* restore the root password
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysql -u root Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 39 Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
- I tried to set it (to the root mysql password store in our shared OSE keepass), but it complained about the DB state
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysqladmin -u root password New password: Confirm new password: mysqladmin: unable to change password; error: 'Column count of mysql.user is wrong. Expected 3, found 42. Created with MariaDB 101106, now running 101106. Please use mariadb-upgrade to fix this error' root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- I ran the command like it asked, but it refused
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mariadb-upgrade This installation of MariaDB is already upgraded to 10.11.6-MariaDB. There is no need to run mysql_upgrade again for 10.11.6-MariaDB. You can use --force if you still want to run mysql_upgrade You have new mail in /var/mail/root root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- I forced the run, and it gave a ton of output
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mariadb-upgrade --force ... store_db.wp_woocommerce_tax_rates OK sys sys.sys_config OK Phase 7/8: uninstalling plugins Phase 8/8: Running 'FLUSH PRIVILEGES' OK root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- when I tried to set the password again, it wouldn't let me. Does that mean the upgrade converted the user table to a view? Do we have the old password now??
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysqladmin -u root password mysqladmin: connect to server at 'localhost' failed error: 'Access denied for user 'root'@'localhost' (using password: NO)' root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump #
- yep, that worked. It wouldn't let me in with no password or an empty password. But when I entered the hetzner2 root password, it let me in. Cool
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysql -u root ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysql -u root -p Enter password: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # mysql -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 60 Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>
- with the DB in-place, I copied the vhost files for store.opensourceecology.org, created 'is_hetzner3', and reset the permissions again
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html # rsync -av --progress store.opensourceecology.org /var/www/html/
...
store.opensourceecology.org/oshineTheme.20190409/__MACOSX/Oshine Buyers Package 6.6.4.4/plugins/._revslider.zip
1.317 100% 1,37kB/s 0:00:00 (xfr#11772, to-chk=1/13258)
store.opensourceecology.org/oshineTheme.20190409/__MACOSX/Oshine Buyers Package 6.6.4.4/plugins/._typehub.zip
176 100% 0,18kB/s 0:00:00 (xfr#11773, to-chk=0/13258)
sent 398.589.915 bytes received 234.353 bytes 159.529.707,20 bytes/sec
total size is 397.617.059 speedup is 1,00
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html #
- while that ran, I uncommented the 'store.opensourceecology.org' vhost from the ansible provisioning playbook, and uncommented the nginx, varnish, and apache roles. And gave it a run.
- I had to fix the varnish template with the same changes I had to make for the forums (purge key path and missing semicolon)
- ok, that worked
root@hetzner3 /etc/varnish # systemctl restart varnish root@hetzner3 /etc/varnish # varnishadm backend.list Backend name Admin Probe Health Last change boot.forum_opensourceecology_org healthy 0/0 healthy Fri, 27 Sep 2024 04:53:46 GMT boot.store_opensourceecology_org healthy 0/0 healthy Fri, 27 Sep 2024 04:53:46 GMT root@hetzner3 /etc/varnish #
- I updated my /etc/hosts to point to hetnzer3, and gave curl a go
user@disp3202:~$ cat /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback disp3202 ff02::1 ip6-allnodes ff02::2 ip6-allrouters 144.76.164.201 forum.opensourceecology.org 144.76.164.201 store.opensourceecology.org 127.0.1.1 disp3202 user@disp3202:~$
- I had to manually restart nginx (I guess due to that ansible bug), but that just changed it from a 404 to a 500 error
user@disp3202:~$ curl -si https://store.opensourceecology.org/ HTTP/1.1 500 Internal Server Error Server: nginx Date: Fri, 27 Sep 2024 04:57:05 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 32773 Age: 0 Via: 1.1 varnish (Varnish/7.1) user@disp3202:~$
- curiously, is_hetzner3 works tho
user@disp3202:~$ curl -si https://store.opensourceecology.org/is_hetzner3 | tail X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 32782 12 Age: 64 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3202:~$
- ok, the error is coming from php
==> store.opensourceecology.org/error.log <== [Fri Sep 27 05:00:19.531330 2024] [proxy_fcgi:error] [pid 1032904:tid 1032938] [client 185.213.155.162:0] AH01071: Got error 'PHP message: PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/script-loader.php on line 706' ==> modsec_audit.log <== --f8a0792d-A-- [27/Sep/2024:05:00:19.531459 +0000] ZvY745fESsiaVmdbnJCmfAAAAEw 127.0.0.1 47370 127.0.0.1 8000 --f8a0792d-B-- GET / HTTP/1.1 X-Real-IP: 185.213.155.162 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: store.opensourceecology.org User-Agent: curl/7.88.1 Accept: */* hash: #store.opensourceecology.org Accept-Encoding: gzip X-Varnish: 32791 --f8a0792d-F-- HTTP/1.1 500 Internal Server Error X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Content-Length: 0 Connection: close Content-Type: text/html; charset=UTF-8 --f8a0792d-E-- --f8a0792d-H-- Apache-Error: [file "mod_proxy_fcgi.c"] [line 911] [level 3] AH01071: Got error 'PHP message: PHP Fatal error: Array and string offset access syntax with curly braces is no longer supported in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/script-loader.php on line 706' Apache-Handler: proxy:unix:/run/php/php8.2-fpm.sock|fcgi://localhost Stopwatch: 1727413219521532 10041 (- - -) Stopwatch2: 1727413219521532 10041; combined=37, p1=23, p2=12, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --f8a0792d-Z-- ==> store.opensourceecology.org/access.log <== 185.213.155.162 - - [27/Sep/2024:05:00:19 +0000] "GET / HTTP/1.1" 500 332 "-" "curl/7.88.1"
- and the fun begins. this fails because PHP 7.4 deprecated the syntax. Basically, we need to upgrade wordpress. And probably all its plugins and themes too https://stackoverflow.com/questions/59158548/array-and-string-offset-access-syntax-with-curly-braces-is-deprecated
- in the past, I used to subversion to update wordpress. But at some point I switched to wp-cli so that it could manage not just the core, but also plugins and themes
- In 2022, I opened three tickets related to cryptographic signing of the wp-cli project. As of now, they're all still open; it looks like no progress has been made
- it's also come to my attention that wordpress doesn't verify signatures on their releases. I opened tickets on this, but I was told it's been a WIP for something like 10 years? They're letting perfect be the enemy of the good.
- anyway, it's clear that these are pretty easy to solve problems, but wordpress and wp-cli don't care enough about security to fix it
- either we accept the risk or we try to mitigate it; let's look at the effort to mitigate it
- using wp-cli on the old server, we can enumerate all the plugins that we'll have to download with 3TOFU
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
sudo -u wp -i wp --path="${wp_docroot}" plugin list
done
- the output is a bit nasty with errors that I can't suppress, so I stored it to a file and cleaned it up manually
- note that we got a couple errors about some sites
- d3d.opensourceecology.org
- 3dp.opensourceecology.org
PHP Warning: realpath(): open_basedir restriction in effect. File(/var/www/html/d3d.opensourceecology.org/htdocs) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/staging.opensourceecology.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/var/www/html/store.opensourceecology.org:/var/www/html/microfactory.opensourceecology.org:/var/www/html/phplist.opensourceecology.org) in phar:///home/wp/.wp-cli/wp-cli.phar/php/WP_CLI/Runner.php on line 209
- ok, actually, if you just redirect to a file, then all the errors are gone
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
sudo -u wp -i wp --path="${wp_docroot}" plugin list
done
- but that default table view doesn't wrap lines, so we need a CSV or something easier to parse
for wordpress_site in $wordpress_sites; do wp_docroot="$(dirname "${wordpress_site}")"; sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list >> /var/tmp/hetzner3/wordpress_plugins.csv; done
- allright, we're looking at 70 plugins
[root@opensourceecology hetzner3]# cat wordpress_plugins.csv | cut -d, -f1 | sort | uniq akismet amr-shortcode-any-widget be-gdpr be-page-builder be-portfolio-post be-themes-one-click-import black-studio-tinymce-widget brankic-photostream-widget chartbeat classic-editor coingate-for-woocommerce colorhub contact-form-7 cyclone-slider-2 duplicate-page duplicate-post flattr force-strong-passwords fundraising google-authenticator google-authenticator-encourage-user-activation hello insert-headers-and-footers jetpack masterslider media-element-html5-video-and-audio-player meta-box meta-box-conditional-logic meta-box-show-hide meta-box-tabs ml-slider ml-slider-pro name newsletter-sign-up oa-open-graph-for-fb open-in-new-window-plugin oshine-core oshine-modules patch-for-revolution-slider post-types-order really-simple-facebook-twitter-share-buttons recent-tweets-widget redux-vendor-support rename-wp-login revision-control revslider shareaholic share-on-diaspora shariff social-media-buttons-toolbar ssl-insecure-content-fixer tatsu typehub varnish-http-purge vcaching vcaching.bak w3-total-cache wonderm00ns-simple-facebook-open-graph-tags woocommerce wordpress-importer wordpress-seo wpautop-control wp-facebook-open-graph-protocol wp-memory-usage wpmudev-updates wp-optimize wp-simple-galleries wp-smushit wp-smush-pro wp-super-cache [root@opensourceecology hetzner3]# [root@opensourceecology hetzner3]# cat wordpress_plugins.csv | cut -d, -f1 | sort | uniq | wc -l 70 [root@opensourceecology hetzner3]#
- well, it's less than 70. We quickly see irrelevant ones like 'vcaching.bak'
- anyway, let's look at themes
for wordpress_site in $wordpress_sites; do wp_docroot="$(dirname "${wordpress_site}")"; sudo -u wp -i wp --path="${wp_docroot}" --format=csv theme list >> /var/tmp/hetzner3/wordpress_themes.csv; done
- we're looking at 24 themes
[root@opensourceecology hetzner3]# cat wordpress_themes.csv | cut -d, -f1 | sort | uniq ArtWorksResponsive bouquet enigmatic enigmatic.20170714.bak Eventor Eventor.old gk-portfolio gridsby gridthemeresponsive name oshin portfolio-press simplephotoRes sketch storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve [root@opensourceecology hetzner3]#
- oh, for each of these the entry named 'name' is just the column header. disregard that too.
- it looks like there are some API endpoints to determine the latest version of a given plugin slug https://codex.wordpress.org/WordPress.org_API#Plugins
- that official page links to this unofficial guide https://wplake.org/blog/wordpress-org-api/
- oh, ugh, that's using php. I want to do this in bash with curl.
- ok, this works https://api.wordpress.org/plugins/info/1.0/google-authenticator.json
- and we can begin to parse it
user@disp3202:~$ curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/google-authenticator.json
user@disp3202:~$ cat plugin.json | jq .version
"0.54"
user@disp3202:~$
user@disp3202:~$ latest_version=$(cat plugin.json | jq -r .version)
user@disp3202:~$
user@disp3202:~$ cat plugin.json | jq -r ".versions.\"${latest_version}\""
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
user@disp3202:~$
- alright, the api had to be modified a bit, but here's the script to get the download links for these themes
for theme in $(cat wordpress_themes2.txt); do
echo $theme;
curl -so theme.json "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}"
latest_version=$(cat theme.json | jq -r .version)
url=$(cat theme.json | jq -r ".download_link")
echo -e "\t$url"
if [ "${url}" = "null" ]; then
error=$(cat theme.json | jq -r .error);
description=$(cat theme.json | jq -r .description);
echo -e "\t $error"
echo -e "\t $description"
fi
done
- ..and the execution
root@hetzner3 /var/tmp/wordpress/themes # for theme in $(cat wordpress_themes2.txt); do
echo $theme;
curl -so theme.json "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}"
latest_version=$(cat theme.json | jq -r .version)
url=$(cat theme.json | jq -r ".download_link")
echo -e "\t$url"
if [ "${url}" = "null" ]; then
error=$(cat theme.json | jq -r .error);
description=$(cat theme.json | jq -r .description);
echo -e "\t $error"
echo -e "\t $description"
fi
done
ArtWorksResponsive
null
Invalid slug provided
null
bouquet
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
enigmatic
null
Theme not found
null
Eventor
null
Invalid slug provided
null
gk-portfolio
https://downloads.wordpress.org/theme/gk-portfolio.1.5.3.zip
gridsby
null
Theme not found
null
gridthemeresponsive
null
Theme not found
null
oshin
null
Theme not found
null
portfolio-press
https://downloads.wordpress.org/theme/portfolio-press.2.8.0.zip
simplephotoRes
null
Invalid slug provided
null
sketch
https://downloads.wordpress.org/theme/sketch.1.2.4.zip
storefront
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
twentyeleven
https://downloads.wordpress.org/theme/twentyeleven.4.7.zip
twentyfifteen
https://downloads.wordpress.org/theme/twentyfifteen.3.8.zip
twentyfourteen
https://downloads.wordpress.org/theme/twentyfourteen.4.0.zip
twentynineteen
https://downloads.wordpress.org/theme/twentynineteen.2.9.zip
twentyseventeen
https://downloads.wordpress.org/theme/twentyseventeen.3.7.zip
twentysixteen
https://downloads.wordpress.org/theme/twentysixteen.3.3.zip
twentyten
https://downloads.wordpress.org/theme/twentyten.4.2.zip
twentythirteen
https://downloads.wordpress.org/theme/twentythirteen.4.2.zip
twentytwelve
https://downloads.wordpress.org/theme/twentytwelve.4.3.zip
You have new mail in /var/mail/root
root@hetzner3 /var/tmp/wordpress/themes #
- alright, for the 3TOFU, we have these plugins and themes
root@hetzner3 /var/tmp/wordpress # cat plugins/urls1.txt https://downloads.wordpress.org/plugin/akismet.5.3.3.zip https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip https://downloads.wordpress.org/plugin/classic-editor.1.6.5.zip https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.1.1.zip https://downloads.wordpress.org/plugin/contact-form-7.5.9.8.zip https://downloads.wordpress.org/plugin/duplicate-post.4.5.zip https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip https://downloads.wordpress.org/plugin/insert-headers-and-footers.2.2.2.zip https://downloads.wordpress.org/plugin/jetpack.13.8.1.zip https://downloads.wordpress.org/plugin/meta-box.5.10.2.zip https://downloads.wordpress.org/plugin/ml-slider.3.91.0.zip https://downloads.wordpress.org/plugin/open-in-new-window-plugin.3.0.zip https://downloads.wordpress.org/plugin/post-types-order.2.2.6.zip https://downloads.wordpress.org/plugin/revision-control.2.3.2.zip https://downloads.wordpress.org/plugin/shareaholic.9.7.12.zip https://downloads.wordpress.org/plugin/share-on-diaspora.0.7.9.zip https://downloads.wordpress.org/plugin/shariff.4.6.14.zip https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip https://downloads.wordpress.org/plugin/varnish-http-purge.5.2.2.zip https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip https://downloads.wordpress.org/plugin/w3-total-cache.2.7.6.zip https://downloads.wordpress.org/plugin/wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip https://downloads.wordpress.org/plugin/woocommerce.9.3.3.zip https://downloads.wordpress.org/plugin/wordpress-importer.0.8.2.zip https://downloads.wordpress.org/plugin/wordpress-seo.23.5.zip https://downloads.wordpress.org/plugin/wpautop-control.1.6.zip https://downloads.wordpress.org/plugin/wp-memory-usage.1.2.10.zip https://downloads.wordpress.org/plugin/wp-optimize.3.6.0.zip https://downloads.wordpress.org/plugin/wp-smushit.3.16.6.zip https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip https://downloads.wordpress.org/plugin/duplicate-page.4.5.zip root@hetzner3 /var/tmp/wordpress # root@hetzner3 /var/tmp/wordpress # cat themes/urls1.txt https://downloads.wordpress.org/theme/bouquet.1.2.5.zip https://downloads.wordpress.org/theme/gk-portfolio.1.5.3.zip https://downloads.wordpress.org/theme/portfolio-press.2.8.0.zip https://downloads.wordpress.org/theme/sketch.1.2.4.zip https://downloads.wordpress.org/theme/storefront.4.6.0.zip https://downloads.wordpress.org/theme/twentyeleven.4.7.zip https://downloads.wordpress.org/theme/twentyfifteen.3.8.zip https://downloads.wordpress.org/theme/twentyfourteen.4.0.zip https://downloads.wordpress.org/theme/twentynineteen.2.9.zip https://downloads.wordpress.org/theme/twentyseventeen.3.7.zip https://downloads.wordpress.org/theme/twentysixteen.3.3.zip https://downloads.wordpress.org/theme/twentyten.4.2.zip https://downloads.wordpress.org/theme/twentythirteen.4.2.zip https://downloads.wordpress.org/theme/twentytwelve.4.3.zip root@hetzner3 /var/tmp/wordpress #
Wed Sep 25, 2024
- yesterday I found that blockuseragents.rules was missing, causing nginx to fail to start
root@hetzner3 /etc/nginx # systemctl restart nginx Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details. root@hetzner3 /etc/nginx # root@hetzner3 /etc/nginx # journalctl -u nginx -f ... Sep 25 03:02:39 hetzner3 systemd[1]: Stopping nginx.service - A high performance web server and a reverse proxy server... Sep 25 03:02:39 hetzner3 systemd[1]: nginx.service: Deactivated successfully. Sep 25 03:02:39 hetzner3 systemd[1]: Stopped nginx.service - A high performance web server and a reverse proxy server. Sep 25 03:02:39 hetzner3 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server... Sep 25 03:02:39 hetzner3 nginx[171224]: 2024/09/25 03:02:39 [emerg] 171224#171224: open() "/etc/nginx/blockuseragents.rules" failed (2: No such file or directory) in /etc/nginx/nginx.conf:61 Sep 25 03:02:39 hetzner3 nginx[171224]: nginx: configuration file /etc/nginx/nginx.conf test failed Sep 25 03:02:39 hetzner3 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Sep 25 03:02:39 hetzner3 systemd[1]: nginx.service: Failed with result 'exit-code'. Sep 25 03:02:39 hetzner3 systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.
- I added a new task to the 'maltifeld.nginx' ansible role, which provisions this file
- now it's failing on the Let's Encrypt TLS cert missing
Sep 25 17:09:15 hetzner3 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server... Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [warn] 245515#245515: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [warn] 245515#245515: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [warn] 245515#245515: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [warn] 245515#245515: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [warn] 245515#245515: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [warn] 245515#245515: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 Sep 25 17:09:15 hetzner3 nginx[245515]: 2024/09/25 17:09:15 [emerg] 245515#245515: cannot load certificate "/etc/letsencrypt/live/opensourceecology.org/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/opensourceecology.org/fullchain.pem, r) error:10000080:BIO routines::no such file) Sep 25 17:09:15 hetzner3 nginx[245515]: nginx: configuration file /etc/nginx/nginx.conf test failed Sep 25 17:09:15 hetzner3 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Sep 25 17:09:15 hetzner3 systemd[1]: nginx.service: Failed with result 'exit-code'. Sep 25 17:09:15 hetzner3 systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.
- this is a chicken-and-egg problem:
- We need DNS to point to the new server in order for the new server to generate valid certificates
- We can't point DNS to the new server until we've verified the new server
- Verifying the new server requires us to have the server with certificates
- I think probably the best option is to manually copy the LE files (including the config and the certs) from hetzner2 to hetnzer3
- looks like certbot isn't eve installed on hetnzer3; ansible failed before it made it to this role
root@hetzner3 /etc/nginx # dpkg -l | grep -i cert ii ca-certificates 20230311 all Common CA certificates ii dirmngr 2.2.40-1.1 amd64 GNU privacy guard - network certificate management service ii python3-certifi 2022.9.24-1 all root certificates for validating SSL certs and verifying TLS hosts (python3) ii ssl-cert 1.1.2 all simple debconf wrapper for OpenSSL root@hetzner3 /etc/nginx #
- I commented-out all the other roles and pushed ansible just the certbot role with ansible
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml PLAY [hetzner3] **************************************************************** TASK [Gathering Facts] ********************************************************* ok: [hetzner3] TASK [maltfield.certbot : install certbot for Let's Encrypt autorenew] ********* changed: [hetzner3] TASK [install basic essential packages] **************************************** ok: [hetzner3] PLAY RECAP ********************************************************************* hetzner3 : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 user@ose:~/sandbox_local/ansible/hetzner3$
- that looks better
root@hetzner3 /etc/nginx # dpkg -l | grep -i cert ii ca-certificates 20230311 all Common CA certificates ii certbot 2.1.0-4 all automatically configure HTTPS using Let's Encrypt ii dirmngr 2.2.40-1.1 amd64 GNU privacy guard - network certificate management service ii python3-certbot 2.1.0-4 all main library for certbot ii python3-certifi 2022.9.24-1 all root certificates for validating SSL certs and verifying TLS hosts (python3) ii ssl-cert 1.1.2 all simple debconf wrapper for OpenSSL root@hetzner3 /etc/nginx # root@hetzner3 /etc/nginx # ls -lah /etc/cron.d total 40K drwxr-xr-x 2 root root 4,0K Sep 25 17:58 . drwxr-xr-x 92 root root 4,0K Sep 25 17:58 .. -rw-r--r-- 1 root root 248 Sep 16 16:45 backup_to_backblaze -rw-r--r-- 1 root root 802 Apr 16 2023 certbot -rw-r--r-- 1 root root 201 Mar 5 2023 e2scrub_all -rw-r--r-- 1 root root 563 Jul 31 21:44 mdadm -rw-r--r-- 1 root root 1,1K Mar 21 2023 munin -rw-r--r-- 1 root root 506 Mar 21 2023 munin-node -rw-r--r-- 1 root root 712 Jul 13 2022 php -rw-r--r-- 1 root root 102 Mar 2 2023 .placeholder root@hetzner3 /etc/nginx # root@hetzner3 /etc/nginx # ls -lah /etc/letsencrypt/ total 12K drwxr-xr-x 2 root root 4,0K Sep 25 17:58 . drwxr-xr-x 92 root root 4,0K Sep 25 17:58 .. -rw-r--r-- 1 root root 207 Nov 12 2021 cli.ini root@hetzner3 /etc/nginx #
- Let's Encrypt has a max cert validity of 90-days
- back on hetzner2, we can see we have two certs (with many, many SANs) that are valid for 46 days
- so, before I copy them, I'll force renew them so they're valid for longer on hetzner3 before they break (in-case we don't finish the migration before 90 days)
[root@opensourceecology ~]# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/openbuildinginstitute.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/opensourceecology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not yet due for renewal
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certificates are not due for renewal yet:
/etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem expires on 2024-11-11 (skipped)
/etc/letsencrypt/live/opensourceecology.org/fullchain.pem expires on 2024-11-11 (skipped)
No renewals were attempted.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[root@opensourceecology ~]#
[root@opensourceecology ~]# certbot --force-renew renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/openbuildinginstitute.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate for www.openbuildinginstitute.org and 3 more domains
Performing the following challenges:
http-01 challenge for awstats.openbuildinginstitute.org
http-01 challenge for openbuildinginstitute.org
http-01 challenge for seedhome.openbuildinginstitute.org
http-01 challenge for www.openbuildinginstitute.org
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/opensourceecology.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate for fef.opensourceecology.org and 11 more domains
Performing the following challenges:
http-01 challenge for awstats.opensourceecology.org
http-01 challenge for fef.opensourceecology.org
http-01 challenge for forum.opensourceecology.org
http-01 challenge for microfactory.opensourceecology.org
http-01 challenge for munin.opensourceecology.org
http-01 challenge for opensourceecology.org
http-01 challenge for oswh.opensourceecology.org
http-01 challenge for phplist.opensourceecology.org
http-01 challenge for staging.opensourceecology.org
http-01 challenge for store.opensourceecology.org
http-01 challenge for wiki.opensourceecology.org
http-01 challenge for www.opensourceecology.org
Using the webroot path /var/www/html/staging.opensourceecology.org/htdocs for all unmatched domains.
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/opensourceecology.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded:
/etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (success)
/etc/letsencrypt/live/opensourceecology.org/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[root@opensourceecology ~]#
[root@opensourceecology ~]# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: openbuildinginstitute.org
Serial Number: 34b09858ae1ae039474513ab579189bd996
Key Type: RSA
Domains: www.openbuildinginstitute.org awstats.openbuildinginstitute.org openbuildinginstitute.org seedhome.openbuildinginstitute.org
Expiry Date: 2024-12-24 17:03:55+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem
Certificate Name: opensourceecology.org
Serial Number: 4e34880100277209c4f459ea08cb84546da
Key Type: RSA
Domains: fef.opensourceecology.org awstats.opensourceecology.org forum.opensourceecology.org microfactory.opensourceecology.org munin.opensourceecology.org opensourceecology.org oswh.opensourceecology.org phplist.opensourceecology.org staging.opensourceecology.org store.opensourceecology.org wiki.opensourceecology.org www.opensourceecology.org
Expiry Date: 2024-12-24 17:04:08+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/opensourceecology.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[root@opensourceecology ~]#
- well, I guess we only get 89 days. Good enough.
- I created a tarball of /etc/letsencrypt on hetnzer2
[root@opensourceecology ~]# tar -czf /home/maltfield/letsencrypt.tar.gz /etc/letsencrypt/* tar: Removing leading `/' from member names [root@opensourceecology ~]# [root@opensourceecology ~]# chown maltfield /home/maltfield/letsencrypt.tar.gz [root@opensourceecology ~]#
- I created a backup of the existing dir on hetzner3, and I moved it out of the way
root@hetzner3 /etc/nginx # tar -czf /etc/letsencrypt.20240925.tar.gz /etc/letsencrypt/* tar: Removing leading `/' from member names root@hetzner3 /etc/nginx # root@hetzner3 /etc/nginx # tar -tf /etc/letsencrypt.20240925.tar.gz etc/letsencrypt/cli.ini root@hetzner3 /etc/nginx # root@hetzner3 /etc/nginx # mv /etc/letsencrypt /var/tmp/letsencrypt.20240925 root@hetzner3 /etc/nginx #
- I securely copied the tarball to hetzner3, and I shredded the less-privileged source file in my user's $HOME on hetzner2
[maltfield@opensourceecology ~]$ rsync -e 'ssh -p 32415' -av --progress letsencrypt.tar.gz hetzner3.opensourceecology.org:
[maltfield@opensourceecology ~]$ rsync -e 'ssh -p 32415' -av --progress letsencrypt.tar.gz 144.76.164.201:
sending incremental file list
letsencrypt.tar.gz
624,634 100% 94.07MB/s 0:00:00 (xfr#1, to-chk=0/1)
sent 624,897 bytes received 35 bytes 249,972.80 bytes/sec
total size is 624,634 speedup is 1.00
[maltfield@opensourceecology ~]$
[maltfield@opensourceecology ~]$ shred -u letsencrypt.tar.gz
[maltfield@opensourceecology ~]$
- I copied the files in-place on hetzner3, then cleaned-up the files in my $HOME
root@hetzner3 /etc # cd /home/maltfield/
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # ls
bin letsencrypt.tar.gz ossec.conf.31557.2024-09-15@04:36:48~
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # mkdir letsencrypt
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # mv letsencrypt.tar.gz letsencrypt
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # cd letsencrypt/
root@hetzner3 /home/maltfield/letsencrypt #
root@hetzner3 /home/maltfield/letsencrypt # tar -xzf letsencrypt.tar.gz
root@hetzner3 /home/maltfield/letsencrypt #
root@hetzner3 /home/maltfield/letsencrypt # ls
etc letsencrypt.tar.gz
root@hetzner3 /home/maltfield/letsencrypt #
root@hetzner3 /home/maltfield/letsencrypt # ls etc/
letsencrypt
root@hetzner3 /home/maltfield/letsencrypt #
root@hetzner3 /home/maltfield/letsencrypt # rsync -a --progress etc/letsencrypt /etc/
...
letsencrypt/renewal/
letsencrypt/renewal/openbuildinginstitute.org.conf
966 100% 17,15kB/s 0:00:00 (xfr#684, to-chk=2/718)
letsencrypt/renewal/opensourceecology.org.conf
2.047 100% 36,35kB/s 0:00:00 (xfr#685, to-chk=1/718)
letsencrypt/renewal/old/
root@hetzner3 /home/maltfield/letsencrypt #
- looks like a couple files on hetzner2 are missing from hetzner3
- hetzner2
[root@opensourceecology ~]# ls -lah /etc/letsencrypt total 68K drwxr-xr-x 9 root root 4.0K Sep 25 18:03 . drwxr-xr-x. 104 root root 12K Sep 11 22:02 .. drwx------ 5 root root 4.0K Aug 1 2018 accounts drwx------ 5 root root 4.0K Nov 27 2017 archive drwxr-xr-x 2 root root 4.0K Sep 25 18:02 csr drwx------ 2 root root 4.0K Sep 25 18:02 keys drwx------ 4 root root 4.0K Nov 27 2017 live -rw-r--r-- 1 root root 1.6K Nov 24 2017 options-ssl-apache.conf -rw-r--r-- 1 root root 1.2K Nov 24 2017 options-ssl-nginx.conf drwxr-xr-x 3 root root 4.0K Sep 25 18:02 renewal drwxr-xr-x 5 root root 4.0K Nov 24 2017 renewal-hooks -rw-r--r-- 1 root root 424 Nov 24 2017 ssl-dhparams.pem -rw-r--r-- 1 root root 64 Nov 24 2017 .updated-options-ssl-apache-conf-digest.txt -rw-r--r-- 1 root root 64 Nov 24 2017 .updated-options-ssl-nginx-conf-digest.txt -rw-r--r-- 1 root root 64 Nov 24 2017 .updated-ssl-dhparams-pem-digest.txt [root@opensourceecology ~]#
- hetzner3
root@hetzner3 /home/maltfield/letsencrypt # ls -lah /etc/letsencrypt total 48K drwxr-xr-x 9 root root 4,0K Sep 25 19:02 . drwxr-xr-x 92 root root 4,0K Sep 25 18:27 .. drwx------ 5 root root 4,0K Aug 1 2018 accounts drwx------ 5 root root 4,0K Nov 27 2017 archive drwxr-xr-x 2 root root 4,0K Sep 25 18:02 csr drwx------ 2 root root 4,0K Sep 25 18:02 keys drwx------ 4 root root 4,0K Nov 27 2017 live -rw-r--r-- 1 root root 1,6K Nov 24 2017 options-ssl-apache.conf -rw-r--r-- 1 root root 1,2K Nov 24 2017 options-ssl-nginx.conf drwxr-xr-x 3 root root 4,0K Sep 25 18:02 renewal drwxr-xr-x 5 root root 4,0K Nov 24 2017 renewal-hooks -rw-r--r-- 1 root root 424 Nov 24 2017 ssl-dhparams.pem root@hetzner3 /home/maltfield/letsencrypt #
- but it does appears to be working
root@hetzner3 /home/maltfield/letsencrypt # certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: openbuildinginstitute.org
Serial Number: 34b09858ae1ae039474513ab579189bd996
Key Type: RSA
Domains: www.openbuildinginstitute.org awstats.openbuildinginstitute.org openbuildinginstitute.org seedhome.openbuildinginstitute.org
Expiry Date: 2024-12-24 17:03:55+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem
Certificate Name: opensourceecology.org
Serial Number: 4e34880100277209c4f459ea08cb84546da
Key Type: RSA
Domains: fef.opensourceecology.org awstats.opensourceecology.org forum.opensourceecology.org microfactory.opensourceecology.org munin.opensourceecology.org opensourceecology.org oswh.opensourceecology.org phplist.opensourceecology.org staging.opensourceecology.org store.opensourceecology.org wiki.opensourceecology.org www.opensourceecology.org
Expiry Date: 2024-12-24 17:04:08+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/opensourceecology.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@hetzner3 /home/maltfield/letsencrypt #
- I shredded the LE files from my $HOME
root@hetzner3 /home/maltfield # find letsencrypt -type f -exec shred -u '{}' \;
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # find letsencrypt -type f
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # ls -lah letsencrypt/
total 12K
drwxr-xr-x 3 root root 4,0K Sep 25 19:06 .
drwx------ 7 maltfield maltfield 4,0K Sep 25 19:02 ..
drwxr-xr-x 3 root root 4,0K Sep 25 19:02 etc
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # rm -rf letsencrypt/
root@hetzner3 /home/maltfield #
root@hetzner3 /home/maltfield # ls
bin ossec.conf.31557.2024-09-15@04:36:48~
root@hetzner3 /home/maltfield #
- I gave nginx a restart; no errors this time
root@hetzner3 ~ # systemctl restart nginx root@hetzner3 ~ #
- now when I attempt to load forums.opensourceecology.org (with the overridden /etc/hosts pointing it to the new server), it gives me a varnish error. Progress.
user@disp3202:~$ curl -iL http://forum.opensourceecology.org
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 25 Sep 2024 19:09:37 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://forum.opensourceecology.org/
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=1;includeSubDomains
HTTP/1.1 503 Backend fetch failed
Server: nginx
Date: Wed, 25 Sep 2024 19:09:39 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 282
Connection: keep-alive
Retry-After: 5
X-Varnish: 32773
Age: 0
Via: 1.1 varnish (Varnish/7.1)
<!DOCTYPE html>
<html>
<head>
<title>503 Backend fetch failed</title>
</head>
<body>
<h1>Error 503 Backend fetch failed</h1>
<p>Backend fetch failed</p>
<h3>Guru Meditation:</h3>
<p>XID: 32774</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
user@disp3202:~$
- If I monitor varnish during the curl, I get a log entry just complaining about the apache backend
root@hetzner3 ~ # varnishlog * << BeReq >> 18 - Begin bereq 17 fetch - VCL_use boot - Timestamp Start: 1727299105.021436 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.0 - BereqHeader X-Real-IP: 146.70.201.205 - BereqHeader X-Forwarded-Proto: https - BereqHeader X-Forwarded-Port: 443 - BereqHeader Host: forum.opensourceecology.org - BereqHeader User-Agent: curl/7.88.1 - BereqHeader Accept: */* - BereqHeader X-Forwarded-For: 146.70.201.205, 127.0.0.1 - BereqHeader Accept-Encoding: gzip - BereqProtocol HTTP/1.1 - BereqHeader X-Varnish: 18 - VCL_call BACKEND_FETCH - VCL_return fetch - Timestamp Fetch: 1727299105.021468 0.000031 0.000031 - FetchError backend default: fail errno 111 (Connection refused) - Timestamp Beresp: 1727299105.021692 0.000256 0.000224 - Timestamp Error: 1727299105.021695 0.000259 0.000002 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Backend fetch failed - BerespHeader Date: Wed, 25 Sep 2024 21:18:25 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=utf-8 - BerespHeader Retry-After: 5 - VCL_return deliver - Storage malloc Transient - Length 279 - BereqAcct 0 0 0 0 0 0 - End
- I confirmed that apache is listening on 127.0.0.1:8000 and 127.0.0.1:8010
- on port 8000, it's listening for requests to Host = 'forum.opensourceecology.org'
root@hetzner3 /etc/varnish # netstat -plan | grep -i apache tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 169319/apache2 tcp 0 0 127.0.0.1:8010 0.0.0.0:* LISTEN 169319/apache2 unix 2 [ ACC ] STREAM LISTENING 3810492 146294/SCREEN /run/screen/S-maltfield/146294.apache root@hetzner3 /etc/varnish # root@hetzner3 /etc/varnish # apachectl -S VirtualHost configuration: 127.0.0.1:8000 forum.opensourceecology.org (/etc/apache2/sites-enabled/00-forum.opensourceecology.org.conf:13) ServerRoot: "/etc/apache2" Main DocumentRoot: "/var/www/html" Main ErrorLog: "/var/log/apache2/error.log" Mutex default: dir="/var/run/apache2/" mechanism=default Mutex watchdog-callback: using_defaults Mutex rewrite-map: using_defaults PidFile: "/var/run/apache2/apache2.pid" Define: DUMP_VHOSTS Define: DUMP_RUN_CFG Define: MODSEC_2.5 Define: MODSEC_2.9 User: name="www-data" id=33 Group: name="www-data" id=33 root@hetzner3 /etc/varnish #
- curiously, if I try to curl it locally, it works fine. So maybe varnish has the backend info wrong?
root@hetzner3 /etc/varnish # curl -H 'Host: forum.opensourceecology.org' 127.0.0.1:8000 it works! root@hetzner3 /etc/varnish #
- If I try the curl on my local machine again, I see no entries in the apache access logs, which tells me varnish isn't even reaching apache
- oh, uhh, looks like our backend is missing
root@hetzner3 /var/log/varnish # varnishadm backend.list Backend name Admin Probe Health Last change boot.default healthy 0/0 healthy Tue, 24 Sep 2024 04:17:38 GMT root@hetzner3 /var/log/varnish #
- yeah, config is stale because restart fails
root@hetzner3 /var/log/varnish # systemctl restart varnish Job for varnish.service failed because of unavailable resources or another system error. See "systemctl status varnish.service" and "journalctl -xeu varnish.service" for details. root@hetzner3 /var/log/varnish #
- looks like systemd is complaining about the "environment file"
Sep 25 21:23:18 hetzner3 varnishd[92059]: CLI telnet 127.0.0.1 60942 127.0.0.1 6082 Rd auth REDACTED
Sep 25 21:23:18 hetzner3 varnishd[92059]: CLI telnet 127.0.0.1 60942 127.0.0.1 6082 Wr 200 -----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,6.1.0-21-amd64,x86_64,-junix,-smalloc,-sdefault,-hcritbit
varnish-7.1.1 revision 7cee1c581bead20e88d101ab3d72afb29f14d87a
Type 'help' for command list.
Type 'quit' to close CLI session.
Sep 25 21:23:18 hetzner3 varnishd[92059]: CLI telnet 127.0.0.1 60942 127.0.0.1 6082 Rd ping
Sep 25 21:23:18 hetzner3 varnishd[92059]: CLI telnet 127.0.0.1 60942 127.0.0.1 6082 Wr 200 PONG 1727299398 1.0
Sep 25 21:23:18 hetzner3 varnishd[92059]: CLI telnet 127.0.0.1 60942 127.0.0.1 6082 Rd backend.list
Sep 25 21:23:18 hetzner3 varnishd[92059]: CLI telnet 127.0.0.1 60942 127.0.0.1 6082 Wr 200 Backend name Admin Probe Health Last change
boot.default healthy 0/0 healthy Tue, 24 Sep 2024 04:17:38 GMT
Sep 25 21:23:58 hetzner3 varnishd[92059]: Error: Manager got SIGTERM
Sep 25 21:23:58 hetzner3 varnishd[92059]: Debug: Stopping Child
Sep 25 21:23:58 hetzner3 varnishd[92059]: Manager got SIGTERM
Sep 25 21:23:58 hetzner3 systemd[1]: Stopping varnish.service - Varnish HTTP accelerator...
Sep 25 21:23:58 hetzner3 varnishd[92059]: Stopping Child
Sep 25 21:23:58 hetzner3 varnishd[92059]: Error: Child (92074) died signal=15
Sep 25 21:23:58 hetzner3 varnishd[92059]: Child (92074) died signal=15
Sep 25 21:23:58 hetzner3 varnishd[92059]: Debug: Child cleanup complete
Sep 25 21:23:58 hetzner3 varnishd[92059]: Child cleanup complete
Sep 25 21:23:58 hetzner3 varnishd[92059]: Info: manager stopping child
Sep 25 21:23:58 hetzner3 varnishd[92059]: Info: manager dies
Sep 25 21:23:58 hetzner3 varnishd[92059]: manager stopping child
Sep 25 21:23:58 hetzner3 varnishd[92059]: manager dies
Sep 25 21:23:58 hetzner3 systemd[1]: varnish.service: Main process exited, code=exited, status=64/USAGE
Sep 25 21:23:58 hetzner3 systemd[1]: varnish.service: Failed with result 'exit-code'.
Sep 25 21:23:58 hetzner3 systemd[1]: Stopped varnish.service - Varnish HTTP accelerator.
Sep 25 21:23:58 hetzner3 systemd[1]: varnish.service: Consumed 1min 43.887s CPU time.
Sep 25 21:23:58 hetzner3 systemd[1]: varnish.service: Failed to load environment files: No such file or directory
Sep 25 21:23:58 hetzner3 systemd[1]: varnish.service: Failed to run 'start' task: No such file or directory
Sep 25 21:23:58 hetzner3 systemd[1]: varnish.service: Failed with result 'resources'.
Sep 25 21:23:58 hetzner3 systemd[1]: Failed to start varnish.service - Varnish HTTP accelerator.
- looks like the provisioning of the file /etc/varnish/varnish.params' was missing in the ansible role
- I added a new task to the maltfield.varnish ansible role to push this file
- now it won't restart because my new include line for the purge keys is missing a semicolon at the end; fixed in ansible role and re-pushed
- now it won't restart because the purge key file doesn't exist
- hmm...looks like it *does* exist
root@hetzner3 /etc/varnish # ls -lah conf total 16K drwxr-xr-x 2 root root 4,0K Sep 25 01:37 . drwxr-xr-x 5 root root 4,0K Sep 25 21:30 .. -rw-r--r-- 1 root root 515 Sep 24 04:16 acl.vcl -rw------- 1 root root 56 Sep 25 01:37 purge-key_forum.opensourceecology.org.vcl root@hetzner3 /etc/varnish #
- oh, I hard-coded the wrong domain :(
Sep 25 21:32:32 hetzner3 systemd[1]: Started varnish.service - Varnish HTTP accelerator.
Sep 25 21:32:32 hetzner3 varnishd[276621]: Could not delete 'vcl_boot.1727299952.335280/vgc.sym': No such file or directory
Sep 25 21:32:32 hetzner3 varnishd[276621]: Error:
Sep 25 21:32:32 hetzner3 varnishd[276621]: Message from VCC-compiler:
Sep 25 21:32:32 hetzner3 varnishd[276621]: Cannot read file '../conf/purge-key_forum.opensourceecology.org.vcl' (No such file or directory)
Sep 25 21:32:32 hetzner3 varnishd[276621]: ('/etc/varnish/sites-enabled/forum.opensourceecology.org.vcl' Line 82 Pos 25)
Sep 25 21:32:32 hetzner3 varnishd[276621]: include "../conf/purge-key_forum.opensourceecology.org.vcl";
Sep 25 21:32:32 hetzner3 varnishd[276621]: ------------------------###################################################-
Sep 25 21:32:32 hetzner3 varnishd[276621]: ('/etc/varnish/all-vhosts.vcl' Line 14 Pos 9)
Sep 25 21:32:32 hetzner3 varnishd[276621]: include "sites-enabled/forum.opensourceecology.org.vcl";
Sep 25 21:32:32 hetzner3 varnishd[276621]: --------###############################################-
Sep 25 21:32:32 hetzner3 varnishd[276621]: ('/etc/varnish/default.vcl' Line 26 Pos 9)
Sep 25 21:32:32 hetzner3 varnishd[276621]: include "all-vhosts.vcl";
Sep 25 21:32:32 hetzner3 varnishd[276621]: --------################-
Sep 25 21:32:32 hetzner3 varnishd[276621]: Running VCC-compiler failed, exited with 2
Sep 25 21:32:32 hetzner3 varnishd[276621]: VCL compilation failed
Sep 25 21:32:32 hetzner3 systemd[1]: varnish.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 25 21:32:32 hetzner3 systemd[1]: varnish.service: Failed with result 'exit-code'.
- I also had to change the path (without '../') and permissions of the file (owned by vcache)
- and I also had to add a semicolon to the purge file too
- finally I got it to start; the backends look good
root@hetzner3 /etc/varnish # varnishadm backend.list Backend name Admin Probe Health Last change boot.forum_opensourceecology_org healthy 0/0 healthy Wed, 25 Sep 2024 21:50:09 GMT root@hetzner3 /etc/varnish #
- and the curl works now; yay!
user@disp3202:~$ curl -iL http://forum.opensourceecology.org HTTP/1.1 301 Moved Permanently Server: nginx Date: Wed, 25 Sep 2024 21:51:37 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://forum.opensourceecology.org/ X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=1;includeSubDomains HTTP/1.1 200 OK Server: nginx Date: Wed, 25 Sep 2024 21:51:38 GMT Content-Type: text/html Content-Length: 10 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 25 Sep 2024 02:38:18 GMT ETag: "a-622e883c35942" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 5 3 Age: 79 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" it works! user@disp3202:~$
- finally, the varnishlog shows a hit; cache is working!
* << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1727301126.990628 0.000000 0.000000 - Timestamp Req: 1727301126.990628 0.000000 0.000000 - VCL_use boot - ReqStart 127.0.0.1 59986 a0 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 146.70.201.205 - ReqHeader X-Forwarded-For: 146.70.201.205 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: forum.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.88.1 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 146.70.201.205 - ReqHeader X-Forwarded-For: 146.70.201.205, 127.0.0.1 - VCL_call RECV - ReqHeader PS-CapabilityList: fully general optimizations only - ReqUnset X-Forwarded-For: 146.70.201.205, 127.0.0.1 - ReqHeader X-Forwarded-For: 146.70.201.205, 127.0.0.1, 127.0.0.1 - ReqHeader X-VC-My-Purge-Key: OBFUSCATED - ReqUnset X-VC-My-Purge-Key: OBFUSCATED - ReqURL / - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqHeader Cookie: - ReqUnset Cookie: - ReqURL / - VCL_return hash - VCL_call HASH - ReqHeader hash: #forum.opensourceecology.org - VCL_return lookup - Hit 3 191.982091 120.000000 0.000000 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Wed, 25 Sep 2024 21:50:18 GMT - RespHeader Server: Apache - RespHeader X-Frame-Options: SAMEORIGIN - RespHeader Last-Modified: Wed, 25 Sep 2024 02:38:18 GMT - RespHeader ETag: "a-622e883c35942" - RespHeader Content-Length: 10 - RespHeader X-Content-Type-Options: nosniff - RespHeader X-XSS-Protection: 1; mode=block - RespHeader X-Frame-Options: deny - RespHeader Referrer-Policy: no-referrer-when-downgrade - RespHeader Pragma: public - RespHeader Cache-Control: public, max-age=300 - RespHeader Content-Type: text/html - RespHeader X-VC-Req-Host: forum.opensourceecology.org - RespHeader X-VC-Req-URL: / - RespHeader X-VC-Req-URL-Base: / - RespHeader X-Varnish: 32770 3 - RespHeader Age: 108 - RespHeader Via: 1.1 varnish (Varnish/7.1) - RespHeader Accept-Ranges: bytes - VCL_call DELIVER - RespUnset X-VC-Req-Host: forum.opensourceecology.org - RespUnset X-VC-Req-URL: / - RespUnset X-VC-Req-URL-Base: / - RespHeader X-VC-Cache: HIT - RespUnset X-VC-Cache: HIT - VCL_return deliver - Timestamp Process: 1727301126.990872 0.000243 0.000243 - Filters - RespHeader Connection: close - Timestamp Resp: 1727301126.990976 0.000348 0.000104 - ReqAcct 219 0 219 506 10 516 - End
- I went ahead and commented-out all the roles and tried to run ansible against them all again
- ansible failed with an issue at awstats
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
...
TASK [maltfield.awstats : install awstats and plugin depends] ******************
fatal: [hetzner3]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "msg": "No package matching 'geoip-database-extra' is available"}
RUNNING HANDLER [maltfield.apache : restart apache] ****************************
PLAY RECAP *********************************************************************
hetzner3 : ok=162 changed=5 unreachable=0 failed=1 skipped=18 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- ok, looks like this package was available in buster https://packages.debian.org/buster/geoip-database-extra
- ...but it's not available in bullseye https://packages.debian.org/bullseye/geoip-database-extra
- I just commented-out that package. I'm not 100% sure what "extras" it provides -- maybe IPv6? I couldn't find much info about it on the 'net
- next I hit an issue where a logrorate bugfix was breaking. But according to the bug report, it was fixed in awstats 7.8-1 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=890414
- we're running awstats 7.8-3, so it shouldn't be necessary
root@hetzner3 ~ # dpkg -l | grep -i awstats ii awstats 7.8-3+deb12u1 all powerful and featureful web server log analyzer You have new mail in /var/mail/root root@hetzner3 ~ #
- with that, ansible finished awstats without errors
- I re-ran ansible with all of the roles uncommented
- we made it further; this time logrotate failed
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
...
TASK [maltfield.logrotate : logrotate.conf] ************************************
changed: [hetzner3]
TASK [maltfield.logrotate : ose custom logrotate configs] **********************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
fatal: [hetzner3]: FAILED! => {"changed": false, "msg": "Could not find or access 'ose.j2'\nSearched in:\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.logrotate/templates/ose.j2\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.logrotate/ose.j2\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.logrotate/tasks/templates/ose.j2\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.logrotate/tasks/ose.j2\n\t/home/user/sandbox_local/ansible/hetzner3/templates/ose.j2\n\t/home/user/sandbox_local/ansible/hetzner3/ose.j2 on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
RUNNING HANDLER [maltfield.apache : restart apache] ****************************
PLAY RECAP *********************************************************************
hetzner3 : ok=170 changed=7 unreachable=0 failed=1 skipped=18 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- that was fixed by a mixmatch of the task and template file 'ose.j2' vs 'ose_logrotate.j2'
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml PLAY [hetzner3] **************************************************************** TASK [Gathering Facts] ********************************************************* ok: [hetzner3] TASK [maltfield.logrotate : logrotate.conf] ************************************ ok: [hetzner3] TASK [maltfield.logrotate : ose custom logrotate configs] ********************** changed: [hetzner3] TASK [maltfield.logrotate : rsyslog logrotate configs] ************************* changed: [hetzner3] TASK [install basic essential packages] **************************************** ok: [hetzner3] PLAY RECAP ********************************************************************* hetzner3 : ok=5 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 user@ose:~/sandbox_local/ansible/hetzner3$
- I re-ran ansible with all of the roles uncommented
- it made it to the end, but somehow it had an unclear error on ansible restarting apache
- I tried the restart manually on the box itself, and it didn't have any errors; I don't see the issue
- unfortunately, there appears to be a few non-idempotent tasks in the apache role
TASK [maltfield.apache : apache2_maint.conf (apache2.conf but maintenance mode)] ***
changed: [hetzner3]
TASK [maltfield.apache : create apache2_maint.conf from apache2_prod.conf, except only one vhost is included] ***
changed: [hetzner3]
...
TASK [maltfield.apache : sites-enabled/{{ item }}.conf] ************************
changed: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.apache : replace default vhost symlink] ************************
changed: [hetzner3]
- those shouldn't break anything, but it will try to restart apache on every run.
- I did a double-tap, but I got the same error
- I tried again, this time with the '-vvv' flag, which gave us this traceback
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 158, in run
res = self._execute()
File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 663, in _execute
result = self._handler.run(task_vars=variables)
File "/usr/lib/python3/dist-packages/ansible_mitogen/mixins.py", line 142, in run
return super(ActionModuleMixin, self).run(tmp, task_vars)
File "/usr/lib/python3/dist-packages/ansible/plugins/action/service.py", line 92, in run
result.update(self._execute_module(module_name=module, module_args=new_module_args, task_vars=task_vars, wrap_async=self._task.async_val))
File "/usr/lib/python3/dist-packages/ansible_mitogen/mixins.py", line 382, in _execute_module
result = ansible_mitogen.planner.invoke(
File "/usr/lib/python3/dist-packages/ansible_mitogen/planner.py", line 609, in invoke
response = invocation.connection.get_chain().call(
File "/usr/lib/python3/dist-packages/ansible_mitogen/connection.py", line 451, in call
return self._rethrow(recv)
File "/usr/lib/python3/dist-packages/ansible_mitogen/connection.py", line 437, in _rethrow
return recv.get().unpickle()
File "/usr/lib/python3/dist-packages/mitogen/core.py", line 963, in unpickle
raise obj
mitogen.core.CallError: mitogen.core.StreamError: cannot unpickle 'ansible.utils.unsafe_proxy'/'AnsibleUnsafeText'
File "<stdin>", line 3668, in _dispatch_one
File "<stdin>", line 3651, in _parse_request
File "<stdin>", line 952, in unpickle
File "<stdin>", line 743, in find_class
File "<stdin>", line 853, in _find_global
fatal: [hetzner3]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
- looks like this is a bug in the latest stable version of mitogen in the debian repos, which was fixed in mitogen 0.3.6. We're using 0.3.0-rc1.4 https://github.com/mitogen-hq/mitogen/issues/1034
user@ose:~/sandbox_local/ansible/hetzner3$ dpkg -l | grep -i ansible ii ansible 2.10.7+merged+base+2.10.17+dfsg-0+deb11u1 all Configuration management, deployment, and task execution system ii ansible-mitogen 0.3.0~rc1-4 all Fast connection strategy for Ansible user@ose:~/sandbox_local/ansible/hetzner3$
- I tried editing the files per the patch outlined in the ticket, but I couldn't get the error to go away. Anyway, the server *is* working. This appears to just be a cosmetic error reporting error
- anyway, here's the full run
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.postfix/tasks/main.yml, line 8,
column 3, found a duplicate dict key (command). Using last defined value only.
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.nginx/tasks/main.yml, line 8,
column 3, found a duplicate dict key (command). Using last defined value only.
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.varnish/tasks/main.yml, line 107,
column 3, found a duplicate dict key (command). Using last defined value only.
PLAY [hetzner3] ****************************************************************************
TASK [Gathering Facts] *********************************************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include_tasks] ***********************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/hardening.yml for hetzner3
TASK [dev-sec.ssh-hardening : Set OS dependent variables] **********************************
ok: [hetzner3] => (item=/home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/vars/Debian.yml)
TASK [dev-sec.ssh-hardening : get openssh-version] *****************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to create crypo-vars] **************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/crypto.yml for hetzner3
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] *******************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] *******************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] *******************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 7.6] *****
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 6.6] *****
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] ***********************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] ***********************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version] ********************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version if openssh >= 6.6] ******
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version] ************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : create revoked_keys and set permissions to root/600] *********
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : create sshd_config and set permissions to root/600] **********
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : create ssh_config and set permissions to root/644] ***********
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : Check if /etc/ssh/moduli contains weak DH parameters] ********
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : remove all small primes] *************************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup ca keys and principals] ***************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup 2FA] **********************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include selinux specific tasks] ******************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] **********************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/rules.yml for hetzner3
TASK [mikegleasonjr.firewall : Generate v4 rules] ******************************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Load v4 rules] **********************************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : Generate v6 rules] ******************************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Load v6 rules] **********************************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] **********************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/persist-debian.yml for hetzner3
TASK [mikegleasonjr.firewall : Remove any obsolete scripts used by an old version of the role] ***
ok: [hetzner3] => (item=/etc/network/if-post-down.d/iptables-v4)
ok: [hetzner3] => (item=/etc/network/if-pre-up.d/iptables-v4)
ok: [hetzner3] => (item=/etc/iptables.v4.saved)
TASK [mikegleasonjr.firewall : Install iptables-persistent] ********************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Install ipset-persistent] ***********************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Check if netfilter-persistent is present] *******************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (netfilter-persistent)] **************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (iptables-persistent)] ***************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] **********************************************
skipping: [hetzner3]
TASK [maltfield.unattended-upgrades : install unattended-upgrades] *************************
ok: [hetzner3]
TASK [maltfield.unattended-upgrades : 20auto-upgrades] *************************************
ok: [hetzner3]
TASK [maltfield.unattended-upgrades : 50unattended-upgrades] *******************************
ok: [hetzner3]
TASK [maltfield.wazuh : install wazuh prereqs] *********************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh gpg key] *****************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh repo] ********************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : install wazuh manager] *********************************************
ok: [hetzner3]
TASK [maltfield.wazuh : ossec.conf] ********************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : local_rules.xml] ***************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : email encryption .forward file] ************************************
ok: [hetzner3]
TASK [maltfield.wazuh : email encryption script] *******************************************
ok: [hetzner3]
TASK [maltfield.locale : Set timezone to UTC] **********************************************
ok: [hetzner3]
TASK [maltfield.locale : Ensure localisation files for 'en_DK.UTF-8' are available] ********
ok: [hetzner3]
TASK [maltfield.locale : Ensure localisation files for 'en_US.UTF-8' are available] ********
ok: [hetzner3]
TASK [maltfield.locale : Get current locale and language configuration] ********************
ok: [hetzner3]
TASK [maltfield.locale : Parse 'LANG' from current locale and language configuration] ******
ok: [hetzner3]
TASK [maltfield.locale : Parse 'LANGUAGE' from current locale and language configuration] ***
ok: [hetzner3]
TASK [maltfield.locale : Configure locale to 'en_DK.UTF-8' and language to 'en_US.UTF-8'] ***
ok: [hetzner3]
TASK [maltfield.locale : set EDITOR to vim] ************************************************
ok: [hetzner3]
TASK [maltfield.kernel : /etc/sysctl.d/local.conf] *****************************************
ok: [hetzner3]
TASK [maltfield.dns : install packages for encrypted DNS] **********************************
ok: [hetzner3]
TASK [maltfield.dns : Add stubby user] *****************************************************
ok: [hetzner3]
TASK [maltfield.dns : /etc/systemd/system/stubby.service] **********************************
ok: [hetzner3]
TASK [maltfield.dns : /etc/stubby/stubby.yml] **********************************************
ok: [hetzner3]
TASK [maltfield.dns : /etc/unbound/unbound.conf.d/ose.conf] ********************************
ok: [hetzner3]
TASK [maltfield.dns : /etc/resolvconf.conf] ************************************************
ok: [hetzner3]
TASK [maltfield.postfix : install postfix] *************************************************
ok: [hetzner3]
TASK [maltfield.postfix : generate dh parameters file] *************************************
[WARNING]: Consider using the file module with mode rather than running 'chmod'. If you
need to use command because file is insufficient you can add 'warn: false' to this command
task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
ok: [hetzner3]
TASK [maltfield.postfix : Whitelisted Domains] *********************************************
ok: [hetzner3]
TASK [maltfield.postfix : Postfox main.cf] *************************************************
ok: [hetzner3]
TASK [maltfield.postfix : Postfox master.cf] ***********************************************
ok: [hetzner3]
TASK [maltfield.backups : install backups prereqs] *****************************************
ok: [hetzner3]
TASK [maltfield.backups : Make root's hardened backups dir] ********************************
ok: [hetzner3]
TASK [maltfield.backups : Add b2user user] *************************************************
ok: [hetzner3]
TASK [maltfield.backups : Make b2user's hardened backups] **********************************
ok: [hetzner3]
TASK [maltfield.backups : Make b2user's sync dir] ******************************************
ok: [hetzner3]
TASK [maltfield.backups : Backup logs dir] *************************************************
ok: [hetzner3]
TASK [maltfield.backups : Backup script] ***************************************************
ok: [hetzner3]
TASK [maltfield.backups : Backup Report script] ********************************************
ok: [hetzner3]
TASK [maltfield.backups : Backup README.txt] ***********************************************
ok: [hetzner3]
TASK [maltfield.backups : Backup cron] *****************************************************
ok: [hetzner3]
TASK [maltfield.certbot : install certbot for Let's Encrypt autorenew] *********************
ok: [hetzner3]
TASK [maltfield.nginx : install nginx] *****************************************************
ok: [hetzner3]
TASK [maltfield.nginx : generate dh parameters file] ***************************************
ok: [hetzner3]
TASK [maltfield.nginx : remove default sites] **********************************************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/nginx.conf] *********************************************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/blockuseragents.rules] **********************************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/conf.d/secure.include] **********************************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/conf.d/https.opensourceecology.org.include] *************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/conf.d/https.openbuildinginstitute.org.include] *********
ok: [hetzner3]
TASK [maltfield.nginx : Create log dir for munin.opensourceecology.org] ********************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/sites-enabled/munin.opensourceecology.org.conf] *********
ok: [hetzner3]
TASK [maltfield.nginx : Create log dir for awstats.opensourceecology.org] ******************
ok: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/sites-enabled/awstats.opensourceecology.org.conf] *******
ok: [hetzner3]
TASK [maltfield.nginx : Create /var/log/nginx/{{ item }}] **********************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.nginx : sites-available/{{ item }}.conf] ***********************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.nginx : sites-enabled/{{ item }}.conf] *************************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.nginx : /etc/nginx/sites-enabled/00-default.conf] **************************
ok: [hetzner3]
TASK [maltfield.varnish : install varnish] *************************************************
ok: [hetzner3]
TASK [maltfield.varnish : Create /etc/vanish/conf] *****************************************
ok: [hetzner3]
TASK [maltfield.varnish : Create /etc/vanish/lib] ******************************************
ok: [hetzner3]
TASK [maltfield.varnish : Create /etc/vanish/sites-enabled] ********************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish systemd params file] *************************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish systemd config] ******************************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish main config file (default.vlc)] **************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish conf/acl.vcl] ********************************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish lib/purge.vcl] *******************************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish all-vhosts.vcl] ******************************************
ok: [hetzner3]
TASK [maltfield.varnish : varnish catch-all.vcl] *******************************************
ok: [hetzner3]
TASK [maltfield.varnish : generate random varnish /etc/varnish/secret] *********************
ok: [hetzner3]
TASK [maltfield.varnish : varnish {{ item }}.vcl] ******************************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.varnish : varnish conf/purge-key_{{ item }}.vcl] ***************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.php : install php apache module] *******************************************
ok: [hetzner3]
TASK [maltfield.php : install php extensions/modules] **************************************
ok: [hetzner3]
TASK [maltfield.php : install pear Text_CAPTCHA module] ************************************
ok: [hetzner3]
TASK [maltfield.php : install pear Image_Text module] **************************************
ok: [hetzner3]
TASK [maltfield.php : php tmp dir] *********************************************************
ok: [hetzner3]
TASK [maltfield.php : php wsdl soap cache dir] *********************************************
ok: [hetzner3]
TASK [maltfield.php : install php apache module] *******************************************
ok: [hetzner3]
TASK [maltfield.mariadb : install mariadb-server] ******************************************
ok: [hetzner3]
TASK [maltfield.mariadb : main mariadb-server config file] *********************************
ok: [hetzner3]
TASK [maltfield.apache : install apache] ***************************************************
ok: [hetzner3]
TASK [maltfield.apache : Add apache-admins group] ******************************************
ok: [hetzner3]
TASK [maltfield.apache : Add www-data to apache-admins] ************************************
ok: [hetzner3]
TASK [maltfield.apache : install modsecurity] **********************************************
ok: [hetzner3]
TASK [maltfield.apache : modsec tmp dir] ***************************************************
ok: [hetzner3]
TASK [maltfield.apache : modsec data dir] **************************************************
ok: [hetzner3]
TASK [maltfield.apache : modsecurity conf] *************************************************
ok: [hetzner3]
TASK [maltfield.apache : remove default apache site] ***************************************
ok: [hetzner3]
TASK [maltfield.apache : remove default apache ssl site] ***********************************
ok: [hetzner3]
TASK [maltfield.apache : enable module = rewrite] ******************************************
ok: [hetzner3]
TASK [maltfield.apache : enable module = headers] ******************************************
ok: [hetzner3]
TASK [maltfield.apache : status.conf] ******************************************************
ok: [hetzner3]
TASK [maltfield.apache : enable module = status] *******************************************
ok: [hetzner3]
TASK [maltfield.apache : enable module = remoteip] *****************************************
ok: [hetzner3]
TASK [maltfield.apache : mods-enabled/pagespeed.conf] **************************************
ok: [hetzner3]
TASK [maltfield.apache : security.conf] ****************************************************
ok: [hetzner3]
TASK [maltfield.apache : conf-enabled/security.conf] ***************************************
ok: [hetzner3]
TASK [maltfield.apache : main apache4_prod.conf] *******************************************
ok: [hetzner3]
TASK [maltfield.apache : apache2_maint.conf (apache2.conf but maintenance mode)] ***********
changed: [hetzner3]
TASK [maltfield.apache : create apache2_maint.conf from apache2_prod.conf, except only one vhost is included] ***
changed: [hetzner3]
TASK [maltfield.apache : And the actual vhost for maintenance mode] ************************
ok: [hetzner3]
TASK [maltfield.apache : symlink "apache2.conf" to our "_prod" config] *********************
ok: [hetzner3]
TASK [maltfield.apache : apache ports.conf] ************************************************
ok: [hetzner3]
TASK [maltfield.apache : security.directory.include - generic hardened contents for <Directory ...> blocks] ***
ok: [hetzner3]
TASK [maltfield.apache : security.virtualhost.include - generic hardened contents for <VirtualHost ...> blocks] ***
ok: [hetzner3]
TASK [maltfield.apache : wordpress.virtualhost.include - common wordpress config for <VirtualHost ...> blocks] ***
ok: [hetzner3]
TASK [maltfield.apache : wordpress.directory.include - common wordpress config for <Directory> blocks] ***
ok: [hetzner3]
TASK [maltfield.apache : phplist.virtualhost.include - common phplist config for <VirtualHost ...> blocks] ***
ok: [hetzner3]
TASK [maltfield.apache : Create /var/log/apache2/{{ item }}] *******************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.apache : sites-available/{{ item }}.conf] **********************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.apache : sites-enabled/{{ item }}.conf] ************************************
changed: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.apache : replace default vhost symlink] ************************************
changed: [hetzner3]
TASK [maltfield.apache : default vhost symlink prepended with '00-'] ***********************
ok: [hetzner3]
TASK [maltfield.munin : install munin and plugins] *****************************************
ok: [hetzner3]
TASK [maltfield.munin : install munin plugin depends] **************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_ plugin] ***********************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_bytes plugin] ******************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_innodb plugin] *****************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_isam_space_ plugin] ************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_queries plugin] ****************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_slowqueries plugin] ************************************
ok: [hetzner3]
TASK [maltfield.munin : munin mysql_threads plugin] ****************************************
ok: [hetzner3]
TASK [maltfield.munin : munin nginx_request plugin] ****************************************
ok: [hetzner3]
TASK [maltfield.munin : munin nginx_status plugin] *****************************************
ok: [hetzner3]
TASK [maltfield.munin : munin multips_memory plugin] ***************************************
ok: [hetzner3]
TASK [maltfield.munin : munin ps_ plugin] **************************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish5_ plugin] ********************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_objects plugin] **************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_hit_rate plugin] *************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_transfer_rate plugin] ********************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_uptime plugin] ***************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_threads plugin] **************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_bad plugin] ******************************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_backend_traffic plugin] ******************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_request_rate plugin] *********************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_memory_usage plugin] *********************************
ok: [hetzner3]
TASK [maltfield.munin : munin varnish_expunge plugin] **************************************
ok: [hetzner3]
TASK [maltfield.munin : munin apache_accesses plugin] **************************************
ok: [hetzner3]
TASK [maltfield.munin : munin apache_processes plugin] *************************************
ok: [hetzner3]
TASK [maltfield.munin : munin apache_volume plugin] ****************************************
ok: [hetzner3]
TASK [maltfield.munin : munin proc plugin] *************************************************
ok: [hetzner3]
TASK [maltfield.munin : mariadb plugins config] ********************************************
ok: [hetzner3]
TASK [maltfield.awstats : install awstats and plugin depends] ******************************
ok: [hetzner3]
TASK [maltfield.awstats : /etc/awstats/common.conf] ****************************************
ok: [hetzner3]
TASK [maltfield.awstats : /etc/awstats/awstats.{{ item }}.conf] ****************************
ok: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.awstats : replace vhost for opensourceecology.org] *************************
ok: [hetzner3]
TASK [maltfield.awstats : /etc/awstats/awstats.opensourceecology.org.conf] *****************
ok: [hetzner3]
TASK [maltfield.cron : phplist cron] *******************************************************
ok: [hetzner3]
TASK [maltfield.cron : xhprof cron] ********************************************************
ok: [hetzner3]
TASK [maltfield.logrotate : logrotate.conf] ************************************************
ok: [hetzner3]
TASK [maltfield.logrotate : ose custom logrotate configs] **********************************
ok: [hetzner3]
TASK [maltfield.logrotate : rsyslog logrotate configs] *************************************
ok: [hetzner3]
TASK [install basic essential packages] ****************************************************
ok: [hetzner3]
RUNNING HANDLER [maltfield.apache : restart apache] ****************************************
ERROR! [mux 118614] 18:27:32.403928 E mitogen.[ssh.144.76.164.201:32415.sudo.root]: raw pickle was: b'\x80\x02(X\'\x00\x00\x00ose-119373-7af604291740-58c43726c4q\x00X\x16\x00\x00\x00ansible_mitogen.targetq\x01NX\n\x00\x00\x00run_moduleq\x02)cmitogen.core\nKwargs\nq\x03}q\x04X\x06\x00\x00\x00kwargsq\x05}q\x06(X\x0b\x00\x00\x00runner_nameq\x07X\x0e\x00\x00\x00NewStyleRunnerq\x08X\x06\x00\x00\x00moduleq\tcansible.utils.unsafe_proxy\nAnsibleUnsafeText\nq\nX\x16\x00\x00\x00ansible.legacy.systemdq\x0b\x85q\x0c\x81q\rX\x04\x00\x00\x00pathq\x0eX9\x00\x00\x00/usr/lib/python3/dist-packages/ansible/modules/systemd.pyq\x0fX\t\x00\x00\x00json_argsq\x10XR\x02\x00\x00{"name": "apache2.service", "state": "restarted", "_ansible_check_mode": false, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 0, "_ansible_version": "2.10.17", "_ansible_module_name": "ansible.legacy.systemd", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}q\x11X\x03\x00\x00\x00envq\x12}q\x13X\x14\x00\x00\x00interpreter_fragmentq\x14NX\t\x00\x00\x00is_pythonq\x15NX\n\x00\x00\x00module_mapq\x16}q\x17(X\x07\x00\x00\x00builtinq\x18]q\x19(X\x1a\x00\x00\x00ansible.module_utils._textq\x1aX\x1a\x00\x00\x00ansible.module_utils.basicq\x1bX\x1b\x00\x00\x00ansible.module_utils.commonq\x1cX/\x00\x00\x00ansible.module_utils.common._collections_compatq\x1dX(\x00\x00\x00ansible.module_utils.common._json_compatq\x1eX"\x00\x00\x00ansible.module_utils.common._utilsq\x1fX\'\x00\x00\x00ansible.module_utils.common.collectionsq X \x00\x00\x00ansible.module_utils.common.fileq!X&\x00\x00\x00ansible.module_utils.common.parametersq"X#\x00\x00\x00ansible.module_utils.common.processq#X$\x00\x00\x00ansible.module_utils.common.sys_infoq$X \x00\x00\x00ansible.module_utils.common.textq%X+\x00\x00\x00ansible.module_utils.common.text.convertersq&X+\x00\x00\x00ansible.module_utils.common.text.formattersq\'X&\x00\x00\x00ansible.module_utils.common.validationq(X$\x00\x00\x00ansible.module_utils.common.warningsq)X\x1b\x00\x00\x00ansible.module_utils.compatq*X\'\x00\x00\x00ansible.module_utils.compat._selectors2q+X%\x00\x00\x00ansible.module_utils.compat.selectorsq,X\x1b\x00\x00\x00ansible.module_utils.distroq-X#\x00\x00\x00ansible.module_utils.distro._distroq.X\x1a\x00\x00\x00ansible.module_utils.factsq/X,\x00\x00\x00ansible.module_utils.facts.ansible_collectorq0X$\x00\x00\x00ansible.module_utils.facts.collectorq1X!\x00\x00\x00ansible.module_utils.facts.compatq2X-\x00\x00\x00ansible.module_utils.facts.default_collectorsq3X#\x00\x00\x00ansible.module_utils.facts.hardwareq4X\'\x00\x00\x00ansible.module_utils.facts.hardware.aixq5X(\x00\x00\x00ansible.module_utils.facts.hardware.baseq6X*\x00\x00\x00ansible.module_utils.facts.hardware.darwinq7X-\x00\x00\x00ansible.module_utils.facts.hardware.dragonflyq8X+\x00\x00\x00ansible.module_utils.facts.hardware.freebsdq9X(\x00\x00\x00ansible.module_utils.facts.hardware.hpuxq:X(\x00\x00\x00ansible.module_utils.facts.hardware.hurdq;X)\x00\x00\x00ansible.module_utils.facts.hardware.linuxq<X*\x00\x00\x00ansible.module_utils.facts.hardware.netbsdq=X+\x00\x00\x00ansible.module_utils.facts.hardware.openbsdq>X)\x00\x00\x00ansible.module_utils.facts.hardware.sunosq?X$\x00\x00\x00ansible.module_utils.facts.namespaceq@X"\x00\x00\x00ansible.module_utils.facts.networkqAX&\x00\x00\x00ansible.module_utils.facts.network.aixqBX\'\x00\x00\x00ansible.module_utils.facts.network.baseqCX)\x00\x00\x00ansible.module_utils.facts.network.darwinqDX,\x00\x00\x00ansible.module_utils.facts.network.dragonflyqEX)\x00\x00\x00ansible.module_utils.facts.network.fc_wwnqFX*\x00\x00\x00ansible.module_utils.facts.network.freebsdqGX.\x00\x00\x00ansible.module_utils.facts.network.generic_bsdqHX\'\x00\x00\x00ansible.module_utils.facts.network.hpuxqIX\'\x00\x00\x00ansible.module_utils.facts.network.hurdqJX(\x00\x00\x00ansible.module_utils.facts.network.iscsiqKX(\x00\x00\x00ansible.module_utils.facts.network.linuxqLX)\x00\x00\x00ansible.module_utils.facts.network.netbsdqMX\'\x00\x00\x00ansible.module_utils.facts.network.nvmeqNX*\x00\x00\x00ansible.module_utils.facts.network.openbsdqOX(\x00\x00\x00ansible.module_utils.facts.network.sunosqPX \x00\x00\x00ansible.module_utils.facts.otherqQX\'\x00\x00\x00ansible.module_utils.facts.other.facterqRX%\x00\x00\x00ansible.module_utils.facts.other.ohaiqSX!\x00\x00\x00ansible.module_utils.facts.sysctlqTX!\x00\x00\x00ansible.module_utils.facts.systemqUX*\x00\x00\x00ansible.module_utils.facts.system.apparmorqVX&\x00\x00\x00ansible.module_utils.facts.system.capsqWX(\x00\x00\x00ansible.module_utils.facts.system.chrootqXX)\x00\x00\x00ansible.module_utils.facts.system.cmdlineqYX+\x00\x00\x00ansible.module_utils.facts.system.date_timeqZX.\x00\x00\x00ansible.module_utils.facts.system.distributionq[X%\x00\x00\x00ansible.module_utils.facts.system.dnsq\\X%\x00\x00\x00ansible.module_utils.facts.system.envq]X&\x00\x00\x00ansible.module_utils.facts.system.fipsq^X\'\x00\x00\x00ansible.module_utils.facts.system.localq_X%\x00\x00\x00ansible.module_utils.facts.system.lsbq`X)\x00\x00\x00ansible.module_utils.facts.system.pkg_mgrqaX*\x00\x00\x00ansible.module_utils.facts.system.platformqbX(\x00\x00\x00ansible.module_utils.facts.system.pythonqcX)\x00\x00\x00ansible.module_utils.facts.system.selinuxqdX-\x00\x00\x00ansible.module_utils.facts.system.service_mgrqeX.\x00\x00\x00ansible.module_utils.facts.system.ssh_pub_keysqfX&\x00\x00\x00ansible.module_utils.facts.system.userqgX"\x00\x00\x00ansible.module_utils.facts.timeoutqhX \x00\x00\x00ansible.module_utils.facts.utilsqiX"\x00\x00\x00ansible.module_utils.facts.virtualqjX\'\x00\x00\x00ansible.module_utils.facts.virtual.baseqkX,\x00\x00\x00ansible.module_utils.facts.virtual.dragonflyqlX*\x00\x00\x00ansible.module_utils.facts.virtual.freebsdqmX\'\x00\x00\x00ansible.module_utils.facts.virtual.hpuxqnX(\x00\x00\x00ansible.module_utils.facts.virtual.linuxqoX)\x00\x00\x00ansible.module_utils.facts.virtual.netbsdqpX*\x00\x00\x00ansible.module_utils.facts.virtual.openbsdqqX(\x00\x00\x00ansible.module_utils.facts.virtual.sunosqrX)\x00\x00\x00ansible.module_utils.facts.virtual.sysctlqsX\x1c\x00\x00\x00ansible.module_utils.parsingqtX)\x00\x00\x00ansible.module_utils.parsing.convert_boolquX\x1f\x00\x00\x00ansible.module_utils.pycompat24qvX\x1c\x00\x00\x00ansible.module_utils.serviceqwX\x18\x00\x00\x00ansible.module_utils.sixqxeX\x06\x00\x00\x00customqy]qzuX\x0e\x00\x00\x00py_module_nameq{X\x17\x00\x00\x00ansible.modules.systemdq|X\r\x00\x00\x00good_temp_dirq}X\x12\x00\x00\x00/root/.ansible/tmpq~X\x03\x00\x00\x00cwdq\x7fNX\t\x00\x00\x00extra_envq\x80NX\x0b\x00\x00\x00emulate_ttyq\x81\x88X\x0f\x00\x00\x00service_contextq\x82cmitogen.core\n_unpickle_context\nq\x83K\x00N\x86q\x84Rq\x85us\x85q\x86Rq\x87tq\x88.'
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: File "<stdin>", line 869, in _find_global
fatal: [hetzner3]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
NO MORE HOSTS LEFT *************************************************************************
PLAY RECAP *********************************************************************************
hetzner3 : ok=173 changed=4 unreachable=0 failed=1 skipped=18 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
...
- now, before I copy-over the /var/www from hetzner2 to hetzner3, I want to make sure my changes for php are working. Because if they're not, I could easily make the config files (with passwords) publicly accessible. That would be bad.
- I created a file on the forum site
root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs # echo '<?php echo "it works!"; ?>' > index.php root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs #
- sure enough, yep, it's leaking the code
user@disp3202:~$ curl -iL http://forum.opensourceecology.org/index.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Wed, 25 Sep 2024 23:41:36 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://forum.opensourceecology.org/index.php X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=1;includeSubDomains HTTP/1.1 200 OK Server: nginx Date: Wed, 25 Sep 2024 23:41:37 GMT Content-Length: 27 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 25 Sep 2024 23:39:25 GMT X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 10 32773 Age: 126 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" <?php echo "it works!"; ?> user@disp3202:~$
- ok, I fixed it. I had to enable two apache modules, which are dependencies to the 'php8.2-fpm.conf' config that ships with Debian
- proxy (/etc/apache2/mods-enabled/proxy.load -> /etc/apache2/mods-available/proxy.load), and
- proxy_fcgi.load (/etc/apache2/mods-enabled/proxy_fcgi.load -> /etc/apache2/mods-available/proxy_fcgi.load)
user@ose:~/sandbox_local/ansible/hetzner3$ git diff
...
diff --git a/hetzner3/roles/maltfield.php/tasks/main.yml b/hetzner3/roles/maltfield.php/tasks/main.yml
index f325af3..518de2f 100644
--- a/hetzner3/roles/maltfield.php/tasks/main.yml
+++ b/hetzner3/roles/maltfield.php/tasks/main.yml
@@ -49,7 +49,34 @@
state: directory
mode: 0700
-- name: install php apache module
+- name: enable apache proxy (depend)
+ file:
+ src: /etc/apache2/mods-available/proxy.load
+ dest: /etc/apache2/mods-enabled/proxy.load
+ state: link
+ notify:
+ - restart php-fpm
+ - restart apache
+
+- name: enable apache proxy_fcgi (depend)
+ file:
+ src: /etc/apache2/mods-available/proxy_fcgi.load
+ dest: /etc/apache2/mods-enabled/proxy_fcgi.load
+ state: link
+ notify:
+ - restart php-fpm
+ - restart apache
+
+- name: enable apache php-fpm
+ file:
+ src: /etc/apache2/conf-available/php8.2-fpm.conf
+ dest: /etc/apache2/conf-enabled/php8.2-fpm.conf
+ state: link
+ notify:
+ - restart php-fpm
+ - restart apache
+
+- name: configure php.ini
template:
src: php_8.2.ini.j2
dest: /etc/php/8.2/fpm/php.ini
user@ose:~/sandbox_local/ansible/hetzner3$
- and now it processes the php, as desired
root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs # curl -H 'Host: forum.opensourceecology.org' 127.0.0.1:8000/index.php it works!root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs #
- and from my laptop
user@disp3202:~$ curl -iL http://forum.opensourceecology.org/index.php?nocache=1 HTTP/1.1 301 Moved Permanently Server: nginx Date: Thu, 26 Sep 2024 00:12:18 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://forum.opensourceecology.org/index.php?nocache=1 X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=1;includeSubDomains HTTP/1.1 200 OK Server: nginx Date: Thu, 26 Sep 2024 00:12:20 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 9 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 32781 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" it works!user@disp3202:~$
- I guess it wasn't an enormous risk, since we do have layered security that blocks access to these dangerous files
user@disp3202:~$ curl -i https://forum.opensourceecology.org/wp-config.php HTTP/1.1 403 Forbidden Server: nginx Date: Thu, 26 Sep 2024 00:19:13 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3202:~$ curl -i https://forum.opensourceecology.org/.git HTTP/1.1 403 Forbidden Server: nginx Date: Thu, 26 Sep 2024 00:19:19 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3202:~$
- the above response was from nginx; here's testing apache too
root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs # curl -H 'Host: forum.opensourceecology.org' 127.0.0.1:8000/wp-config.php <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs # curl -H 'Host: forum.opensourceecology.org' 127.0.0.1:8000/.git <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> root@hetzner3 /var/www/html/forum.opensourceecology.org/htdocs #
- I went ahead and pushed ansible to GitHub https://github.com/OpenSourceEcology/ansible
Tue Sep 24, 2024
- I re-ran the ansible provisioning
- I had to fix the site files to include the '.vcl.j2' extension
- I also had to update the varnish site-specific-purge-key file to be a "shell" task (instead of a "copy" task) so that it supported the "creates" arg (see notes from yesterday)
- the apache ansible role left apache in a broken state
Sep 25 02:04:42 hetzner3 systemd[1]: Starting apache2.service - The Apache HTTP Server... Sep 25 02:04:42 hetzner3 apachectl[147257]: AH00534: apache2: Configuration error: More than one MPM loaded. Sep 25 02:04:42 hetzner3 apachectl[147254]: Action 'start' failed. Sep 25 02:04:42 hetzner3 apachectl[147254]: The Apache error log may have more information. Sep 25 02:04:42 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Sep 25 02:04:42 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'. Sep 25 02:04:42 hetzner3 systemd[1]: Failed to start apache2.service - The Apache HTTP Server.
- it appears that our custom MPM config is conflicting with the default MPM config
root@hetzner3 /etc/apache2 # ls -lah mods-enabled/ | grep mpm lrwxrwxrwx 1 root root 32 Sep 25 01:24 mpm_event.conf -> ../mods-available/mpm_event.conf lrwxrwxrwx 1 root root 32 Sep 25 01:24 mpm_event.load -> ../mods-available/mpm_event.load lrwxrwxrwx 1 root root 44 Sep 25 01:24 mpm_prefork.conf -> /etc/apache2/mods-available/mpm_prefork.conf lrwxrwxrwx 1 root root 44 Sep 25 01:24 mpm_prefork.load -> /etc/apache2/mods-available/mpm_prefork.load root@hetzner3 /etc/apache2 #
- looks like I'm setting up mpm_prefork, but the default is mpm_event
- according to this, mpm_prefork works with more things, but it's less performant than mpm_event https://serverfault.com/questions/383526/how-do-i-select-which-apache-mpm-to-use
- looks like mpm_event didn't become stable until apache 2.4, which is what we're using on hetzner3
root@hetzner3 /etc/apache2 # dpkg -l | grep -i apache ii apache2 2.4.62-1~deb12u1 amd64 Apache HTTP Server ii apache2-bin 2.4.62-1~deb12u1 amd64 Apache HTTP Server (modules and other binary files) ii apache2-data 2.4.62-1~deb12u1 all Apache HTTP Server (common files) ii apache2-utils 2.4.62-1~deb12u1 amd64 Apache HTTP Server (utility programs for web servers) ii libapache2-mod-php 2:8.2+93 all server-side, HTML-embedded scripting language (Apache 2 module) (default) ii libapache2-mod-php8.2 8.2.20-1~deb12u1 amd64 server-side, HTML-embedded scripting language (Apache 2 module) ii libapache2-mod-security2 2.9.7-1+b1 amd64 Tighten web applications security for Apache ii libapr1:amd64 1.7.2-3 amd64 Apache Portable Runtime Library ii libaprutil1:amd64 1.6.3-1 amd64 Apache Portable Runtime Utility Library ii libaprutil1-dbd-sqlite3:amd64 1.6.3-1 amd64 Apache Portable Runtime Utility Library - SQLite3 Driver ii libaprutil1-ldap:amd64 1.6.3-1 amd64 Apache Portable Runtime Utility Library - LDAP Driver ii libsvn1:amd64 1.14.2-4+b2 amd64 Shared libraries used by Apache Subversion root@hetzner3 /etc/apache2 #
- one of the things that lead us to setup mpm was an issue on hetzner2 where apache was spawning runaway children until the server died and Marcin needed to reboot it
- we never figured out why it was doing this, but setting-up MaxServers in an mpm config patched the issue
- in that research, I also learned that php-fpm was generally more preformant than mod_php
- currently we're using mod_php on hetzner3. I think now might be a good time to try-out php-fpm and mpm_event, as those seem to be the best recommendations
- I finally got php and apache setup (at least to the point it could start), but I'm still getting a default debian page :(
- looks like I need to create the document root manually, at least until I copy the files over
root@hetzner3 /etc/apache2 # ls -lah /var/www/html total 24K drwxr-xr-x 2 root root 4,0K Sep 25 01:24 . drwxr-xr-x 3 root root 4,0K Sep 24 04:17 .. -rw-r--r-- 1 root root 11K Sep 25 01:24 index.html -rw-r--r-- 1 root root 615 Sep 24 04:17 index.nginx-debian.html root@hetzner3 /etc/apache2 #
- I made a backup of what's there, deleted it, and replaced it
root@hetzner3 /var/www # tar -czvf html.20240924.tar.gz html/* html/index.html html/index.nginx-debian.html root@hetzner3 /var/www # root@hetzner3 /var/www # rm html/* root@hetzner3 /var/www # root@hetzner3 /var/www # ls html root@hetzner3 /var/www # root@hetzner3 /var/www # mkdir -p html/forum.opensourceecology.org/htdocs/ root@hetzner3 /var/www # root@hetzner3 /var/www # echo 'it works!' > html/forum.opensourceecology.org/htdocs/index.html root@hetzner3 /var/www # root@hetzner3 /var/www # chown root:www-data -R html root@hetzner3 /var/www # root@hetzner3 /var/www # systemctl restart apache2 root@hetzner3 /var/www #
- after that, I now get a 403 from nginx trying to load the server over port 80 http://hetzner3.opensourceecology.org/
- ok, looks like that's failing because the Host doesn't match, and the default (www.opensourceecology.org) doesn't exist (yet)
- if I override /etc/hosts on my local machine to point to the new server, I still get a 403
144.76.164.201 forum.opensourceecology.org
- here's a simple curl
</pre> user@disp3202:~$ curl -i forum.opensourceecology.org HTTP/1.1 403 Forbidden Server: nginx/1.22.1 Date: Wed, 25 Sep 2024 02:53:23 GMT Content-Type: text/html Content-Length: 153 Connection: keep-alive
403 Forbidden
- that request triggers this error message in the logs
==> access.log <== 185.204.1.184 - - [25/Sep/2024:02:53:23 +0000] "GET / HTTP/1.1" 403 153 "-" "curl/7.88.1" ==> error.log <== 2024/09/25 02:53:23 [error] 89740#89740: *214 directory index of "/var/www/html/" is forbidden, client: 185.204.1.184, server: _, request: "GET / HTTP/1.1", host: "forum.opensourceecology.org"
- ok, it's saying we can't query the directory. That's fair, I guess?
- but this is actually http over port 80; it's supposed to just redirect the requested URL to https
- ah, crap, there's a default site there
root@hetzner3 /etc/nginx # grep -ir 'listen 80' * nginx.conf: listen 80; sites-available/default: listen 80 default_server; sites-available/default:# listen 80; root@hetzner3 /etc/nginx #
- wait, no, that's just "available" not "enabled"
- oh, ugh, the current configs are just stale
root@hetzner3 /etc/nginx # systemctl restart nginx Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details. root@hetzner3 /etc/nginx #
Mon Sep 23, 2024
- I checked the backup logs, and it looks like the cron backup kicked-off this morning was successful
- I also checked for it in the rclone listing
root@mail ~/backups # rclone --b2-versions ls b2:ose-server-backups
21812118477 daily_hetzner2_20240921_072001.tar-v2024-09-21-075216-801.gpg
21757200532 daily_hetzner2_20240922_072001.tar.gpg
41605227 daily_hetzner3_20240922_224521.tar.gpg
21349753244 monthly_hetzner2_20231001_072001.tar.gpg
21360808568 monthly_hetzner2_20231101_072001.tar.gpg
21360301269 monthly_hetzner2_20231201_072001.tar.gpg
21820017340 monthly_hetzner2_20240201_072001.tar.gpg
21683700909 monthly_hetzner2_20240301_072001.tar.gpg
21660296728 monthly_hetzner2_20240401_072001.tar.gpg
21790035424 monthly_hetzner2_20240501_072001.tar.gpg
21603737883 monthly_hetzner2_20240601_072001.tar.gpg
21663769333 monthly_hetzner2_20240701_072001.tar.gpg
21991147307 monthly_hetzner2_20240801_072001.tar.gpg
21896377523 monthly_hetzner2_20240901_072001.tar.gpg
21942660432 weekly_hetzner2_20240826_072001.tar.gpg
21902006508 weekly_hetzner2_20240902_072001.tar.gpg
21873908566 weekly_hetzner2_20240909_072001.tar.gpg
21830987241 weekly_hetzner2_20240916_072001.tar.gpg
21738236693 weekly_hetzner2_20240923_072001.tar.gpg
10346 weekly_hetzner3_20240922_220636.tar.gpg
41717868 weekly_hetzner3_20240923_080135.tar.gpg
17516124812 yearly_hetzner2_20190101_111520.tar.gpg
18872422001 yearly_hetzner2_20200101_072001.tar.gpg
19827971632 yearly_hetzner2_20210101_072001.tar.gpg
21079942509 yearly_hetzner2_20230101_072001.tar.gpg
21541199047 yearly_hetzner2_20240101_072001.tar.gpg
root@mail ~/backups #
- And it looks like the next backup is set to run in about 4 hours
root@hetzner3 ~ # date -u 2024-09-24T04:09:32 UTC root@hetzner3 ~ #
...
- finally, now that the server is hardened and backups are confirmed to be working: let's do the rest!
- I uncommented every role in the ansible playbook and gave it a push; probably this is going to have errors, but let's start fixing them!
- Here's the full run; it failed at varnish
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from /home/user/sandbox_local/ansible/h
etzner3/roles/maltfield.postfix/tasks/main.yml, line 8, column 3, found a
duplicate dict key (command). Using last defined value only.
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.nginx/tasks/main.yml,
line 8, column 3, found a duplicate dict key (command). Using last defined
value only.
PLAY [hetzner3] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include_tasks] ***********************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/hardening.yml for hetzner3
TASK [dev-sec.ssh-hardening : Set OS dependent variables] **********************
ok: [hetzner3] => (item=/home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/vars/Debian.yml)
TASK [dev-sec.ssh-hardening : get openssh-version] *****************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to create crypo-vars] **************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/crypto.yml for hetzner3
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] *******
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] *******
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] *******
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 7.6] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] ***********
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] ***********
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version] ********
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version if openssh >= 6.6] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version] ************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : create revoked_keys and set permissions to root/600] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : create sshd_config and set permissions to root/600] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : create ssh_config and set permissions to root/644] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : Check if /etc/ssh/moduli contains weak DH parameters] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : remove all small primes] *************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup ca keys and principals] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup 2FA] **********************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include selinux specific tasks] ******************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] **********************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/rules.yml for hetzner3
TASK [mikegleasonjr.firewall : Generate v4 rules] ******************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Load v4 rules] **********************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Generate v6 rules] ******************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Load v6 rules] **********************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] **********************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/persist-debian.yml for hetzner3
TASK [mikegleasonjr.firewall : Remove any obsolete scripts used by an old version of the role] ***
ok: [hetzner3] => (item=/etc/network/if-post-down.d/iptables-v4)
ok: [hetzner3] => (item=/etc/network/if-pre-up.d/iptables-v4)
ok: [hetzner3] => (item=/etc/iptables.v4.saved)
TASK [mikegleasonjr.firewall : Install iptables-persistent] ********************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Install ipset-persistent] ***********************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Check if netfilter-persistent is present] *******
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (netfilter-persistent)] **************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (iptables-persistent)] ***************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] **********************************
skipping: [hetzner3]
TASK [maltfield.postfix : install postfix] *************************************
ok: [hetzner3]
TASK [maltfield.postfix : generate dh parameters file] *************************
[WARNING]: Consider using the file module with mode rather than running
'chmod'. If you need to use command because file is insufficient you can add
'warn: false' to this command task or set 'command_warnings=False' in
ansible.cfg to get rid of this message.
ok: [hetzner3]
TASK [maltfield.postfix : Whitelisted Domains] *********************************
ok: [hetzner3]
TASK [maltfield.postfix : Postfox main.cf] *************************************
ok: [hetzner3]
TASK [maltfield.postfix : Postfox master.cf] ***********************************
ok: [hetzner3]
TASK [maltfield.wazuh : install wazuh prereqs] *********************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh gpg key] *****************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh repo] ********************************************
ok: [hetzner3]
TASK [maltfield.wazuh : install wazuh manager] *********************************
ok: [hetzner3]
TASK [maltfield.wazuh : ossec.conf] ********************************************
ok: [hetzner3]
TASK [maltfield.wazuh : local_rules.xml] ***************************************
ok: [hetzner3]
TASK [maltfield.wazuh : email encryption .forward file] ************************
ok: [hetzner3]
TASK [maltfield.wazuh : email encryption script] *******************************
ok: [hetzner3]
TASK [maltfield.unattended-upgrades : install unattended-upgrades] *************
ok: [hetzner3]
TASK [maltfield.unattended-upgrades : 20auto-upgrades] *************************
ok: [hetzner3]
TASK [maltfield.unattended-upgrades : 50unattended-upgrades] *******************
ok: [hetzner3]
TASK [maltfield.kernel : /etc/sysctl.d/local.conf] *****************************
changed: [hetzner3]
TASK [maltfield.dns : install packages for encrypted DNS] **********************
ok: [hetzner3]
TASK [maltfield.dns : Add stubby user] *****************************************
ok: [hetzner3]
TASK [maltfield.dns : /etc/systemd/system/stubby.service] **********************
ok: [hetzner3]
TASK [maltfield.dns : /etc/stubby/stubby.yml] **********************************
ok: [hetzner3]
TASK [maltfield.dns : /etc/unbound/unbound.conf.d/ose.conf] ********************
ok: [hetzner3]
TASK [maltfield.dns : /etc/resolvconf.conf] ************************************
ok: [hetzner3]
TASK [maltfield.locale : Set timezone to UTC] **********************************
ok: [hetzner3]
TASK [maltfield.locale : Ensure localisation files for 'en_DK.UTF-8' are available] ***
ok: [hetzner3]
TASK [maltfield.locale : Ensure localisation files for 'en_US.UTF-8' are available] ***
ok: [hetzner3]
TASK [maltfield.locale : Get current locale and language configuration] ********
ok: [hetzner3]
TASK [maltfield.locale : Parse 'LANG' from current locale and language configuration] ***
ok: [hetzner3]
TASK [maltfield.locale : Parse 'LANGUAGE' from current locale and language configuration] ***
ok: [hetzner3]
TASK [maltfield.locale : Configure locale to 'en_DK.UTF-8' and language to 'en_US.UTF-8'] ***
ok: [hetzner3]
TASK [maltfield.locale : set EDITOR to vim] ************************************
ok: [hetzner3]
TASK [maltfield.nginx : install nginx] *****************************************
changed: [hetzner3]
TASK [maltfield.nginx : generate dh parameters file] ***************************
ok: [hetzner3]
TASK [maltfield.nginx : remove default sites] **********************************
changed: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/nginx.conf] *********************************
changed: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/conf.d/secure.include] **********************
changed: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/conf.d/https.opensourceecology.org.include] ***
changed: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/conf.d/https.openbuildinginstitute.org.include] ***
changed: [hetzner3]
TASK [maltfield.nginx : Create log dir for munin.opensourceecology.org] ********
changed: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/sites-enabled/munin.opensourceecology.org.conf] ***
changed: [hetzner3]
TASK [maltfield.nginx : Create log dir for awstats.opensourceecology.org] ******
changed: [hetzner3]
TASK [maltfield.nginx : /etc/nginx/sites-enabled/awstats.opensourceecology.org.conf] ***
changed: [hetzner3]
TASK [maltfield.nginx : Create /var/log/nginx/{{ item }}] **********************
changed: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.nginx : sites-available/{{ item }}.conf] ***********************
changed: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.nginx : sites-enabled/{{ item }}.conf] *************************
changed: [hetzner3] => (item=forum.opensourceecology.org)
TASK [maltfield.nginx : /etc/nginx/sites-enabled/00-default.conf] **************
changed: [hetzner3]
TASK [maltfield.varnish : install varnish] *************************************
changed: [hetzner3]
TASK [maltfield.varnish : Create /etc/vanish/conf] *****************************
changed: [hetzner3]
TASK [maltfield.varnish : Create /etc/vanish/lib] ******************************
changed: [hetzner3]
TASK [maltfield.varnish : Create /etc/vanish/sites-enabled] ********************
changed: [hetzner3]
TASK [maltfield.varnish : varnish systemd config] ******************************
changed: [hetzner3]
TASK [maltfield.varnish : varnish main config file (default.vlc)] **************
changed: [hetzner3]
TASK [maltfield.varnish : varnish conf/acl.vcl] ********************************
changed: [hetzner3]
TASK [maltfield.varnish : varnish lib/purge.vcl] *******************************
changed: [hetzner3]
TASK [maltfield.varnish : varnish all-vhosts.vcl] ******************************
changed: [hetzner3]
TASK [maltfield.varnish : varnish catch-all.vcl] *******************************
changed: [hetzner3]
TASK [maltfield.varnish : create random varnish /etc/varnish/secret] ***********
fatal: [hetzner3]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "checksum": "dc202b5391d52091ef270b7e0d5a69b531470825", "msg": "Unsupported parameters for (ansible.legacy.copy) module: creates Supported parameters include: _original_basename, attributes, backup, checksum, content, dest, directory_mode, follow, force, group, local_follow, mode, owner, remote_src, selevel, serole, setype, seuser, src, unsafe_writes, validate"}
RUNNING HANDLER [maltfield.varnish : restart varnish] **************************
RUNNING HANDLER [maltfield.varnish : reload systemd] ***************************
RUNNING HANDLER [maltfield.certbot : restart nginx] ****************************
PLAY RECAP *********************************************************************
hetzner3 : ok=79 changed=30 unreachable=0 failed=1 skipped=14 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- looks like "creates" isn't supported in "copy"? Well that's shit; if we don't include that, it's not idempotent
- I tested "copy" without "creates" -- yep, it just overwrites the file with a new secret on every run
- I tried changing from "copy" to "template" -- it has the same issue that it doesn't support "copy"
- I ran it twice with the "template" without "copy" -- again it just overwrites the file every time
- I changed it to a command. Sigh. This works
-- name: create random varnish /etc/varnish/secret
- copy:
- dest: /etc/varnish/secret
- owner: root
- group: root
- mode: 0644
- backup: yes
- content: {{ lookup('ansible.builtin.password', '/dev/null', chars=['ascii_lowercase', 'digits'], length=32) }}
+- name: generate random varnish /etc/varnish/secret
+ command: echo "{{ lookup('ansible.builtin.password', '/dev/null', chars=['ascii_lowercase', 'digits'], length=32) }}" > /etc/varnish/secret
+ command: chown root:root /etc/ssl/certs/dhparam.pem
+ command: chmod 0600 /etc/ssl/certs/dhparam.pem
args:
- creates: /etc/varnish/secret
+ creates: "/etc/varnish/secret"
Sun Sep 22, 2024
- I loggged-back into Backblaze this morning and started investigating each of the keys. They each have names, buckets, and capabilities
- dev, ose-dev-server-backups, deleteFiles, listBuckets, listFiles, readFiles, shareFiles, writeFiles
- master2, -, deleteBuckets, deleteFiles, deleteKeys, listBuckets, listFiles, listKeys, readFiles, shareFiles, writeBuckets, writeFiles, writeKeys
- prod-append-only, ose-server-backups, listBuckets, writeFiles
- prod-append-only-2022-10, ose-server-backups, readFiles, writeFiles
- prod-list-and-append-only-2022-10, ose-server-backups, listFiles, readFiles, writeFiles
- it's a bit concerning that the dev one clearly isn't append-only, but that data isn't so important. And I don't think it's a good use of my time to fight with the dev server (which is probably broken atm and would be a time sink just to investigate it), so I'll just leave that one as-is. It can only access the dev bucket, anyway
- I don't recognize the "master2" key, and it definitely shouldn't exist on any server, so I deleted it
- I confirmed from my logs yesterday that the hetzner2 prod server uses the last two keys (
prod-append-only-2022-10for b2 user andprod-list-and-append-only-2022-10for the root user), so I deleted the old keyprod-append-only - I created a new (temporary) master key named
master-2024-09in the Backblaze B2 WUI - I installed the backblaze-b2 package on my local debian 12 laptop
sudo apt-get install backblaze-b2
- I configured my local
backblaze-b2CLI tool with the new master key that I created in the Backblaze B2 WUI (above)
user@disp3202:~$ backblaze-b2 authorize-account Using https://api.backblazeb2.com Backblaze account ID: REDACTED Backblaze application key: user@disp3202:~$
- I created two new keys for the hetzner3 server
user@disp3202:~$ backblaze-b2 create-key --bucket 'ose-server-backups' 'hetzner3-append-only-2024-09' 'readFiles, writeFiles' OBFUSCATED OBFUSCATED user@disp3202:~$ user@disp3202:~$ backblaze-b2 create-key --bucket 'ose-server-backups' 'hetzner3-list-and-append-only-2024-09' 'listFiles, readFiles, writeFiles' OBFUSCATED OBFUSCATED user@disp3202:~$
- I configured rclone on hetzne3 for the root user with the key that includes the
listFilescapability
root@mail ~/backups # rclone config 2024/09/22 20:31:36 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n Enter name for new remote. name> b2 Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. 1 / 1Fichier \ (fichier) 2 / Akamai NetStorage \ (netstorage) 3 / Alias for an existing remote \ (alias) 4 / Amazon Drive \ (amazon cloud drive) 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi \ (s3) 6 / Backblaze B2 \ (b2) 7 / Better checksums for other remotes \ (hasher) 8 / Box \ (box) 9 / Cache a remote \ (cache) 10 / Citrix Sharefile \ (sharefile) 11 / Combine several remotes into one \ (combine) 12 / Compress a remote \ (compress) 13 / Dropbox \ (dropbox) 14 / Encrypt/Decrypt a remote \ (crypt) 15 / Enterprise File Fabric \ (filefabric) 16 / FTP \ (ftp) 17 / Google Cloud Storage (this is not Google Drive) \ (google cloud storage) 18 / Google Drive \ (drive) 19 / Google Photos \ (google photos) 20 / HTTP \ (http) 21 / Hadoop distributed file system \ (hdfs) 22 / HiDrive \ (hidrive) 23 / In memory object storage system. \ (memory) 24 / Internet Archive \ (internetarchive) 25 / Jottacloud \ (jottacloud) 26 / Koofr, Digi Storage and other Koofr-compatible storage providers \ (koofr) 27 / Local Disk \ (local) 28 / Mail.ru Cloud \ (mailru) 29 / Microsoft Azure Blob Storage \ (azureblob) 30 / Microsoft OneDrive \ (onedrive) 31 / OpenDrive \ (opendrive) 32 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH) \ (swift) 33 / Pcloud \ (pcloud) 34 / Put.io \ (putio) 35 / SMB / CIFS \ (smb) 36 / SSH/SFTP \ (sftp) 37 / Sia Decentralized Cloud \ (sia) 38 / Sugarsync \ (sugarsync) 39 / Transparently chunk/split large files \ (chunker) 40 / Union merges the contents of several upstream fs \ (union) 41 / Uptobox \ (uptobox) 42 / WebDAV \ (webdav) 43 / Yandex Disk \ (yandex) 44 / Zoho \ (zoho) 45 / premiumize.me \ (premiumizeme) 46 / seafile \ (seafile) Storage> b2 Option account. Account ID or Application Key ID. Enter a value. account> OBFUSCATED Option key. Application Key. Enter a value. key> OBFUSCATED Option hard_delete. Permanently delete files on remote removal, otherwise hide files. Enter a boolean value (true or false). Press Enter for the default (false). hard_delete> true Edit advanced config? y) Yes n) No (default) y/n> n Configuration complete. Options: - type: b2 - account: OBFUSCATED - key: OBFUSCATED - hard_delete: true Keep this "b2" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes: Name Type = b2 b2 e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> q root@mail ~/backups #
- I confirmed that it's now working
root@mail ~/backups # rclone --b2-versions ls b2:ose-server-backups 21812118477 daily_hetzner2_20240921_072001.tar.gpg 21757200532 daily_hetzner2_20240922_072001.tar.gpg 21349753244 monthly_hetzner2_20231001_072001.tar.gpg 21360808568 monthly_hetzner2_20231101_072001.tar.gpg 21360301269 monthly_hetzner2_20231201_072001.tar.gpg 21820017340 monthly_hetzner2_20240201_072001.tar.gpg 21683700909 monthly_hetzner2_20240301_072001.tar.gpg 21660296728 monthly_hetzner2_20240401_072001.tar.gpg 21790035424 monthly_hetzner2_20240501_072001.tar.gpg 21603737883 monthly_hetzner2_20240601_072001.tar.gpg 21663769333 monthly_hetzner2_20240701_072001.tar.gpg 21991147307 monthly_hetzner2_20240801_072001.tar.gpg 21896377523 monthly_hetzner2_20240901_072001.tar.gpg 21942660432 weekly_hetzner2_20240826_072001.tar.gpg 21902006508 weekly_hetzner2_20240902_072001.tar.gpg 21873908566 weekly_hetzner2_20240909_072001.tar.gpg 21830987241 weekly_hetzner2_20240916_072001.tar.gpg 17516124812 yearly_hetzner2_20190101_111520.tar.gpg 18872422001 yearly_hetzner2_20200101_072001.tar.gpg 19827971632 yearly_hetzner2_20210101_072001.tar.gpg 21079942509 yearly_hetzner2_20230101_072001.tar.gpg 21541199047 yearly_hetzner2_20240101_072001.tar.gpg root@mail ~/backups #
- I switched to the
b2useruser, and I did the same process for the other key
sudo su - b2user rclone config
- I spent some time documenting this whole key creation and rclone config process on Backblaze
- I also spent some time updating the Hetzner3 article with a summary of the steps to configure the server until now
- I checked the backups log. I see two problems
- as expected, rclone failed to upload because it didn't have the b2 endpoint setup yet
- but it's also failing to copy the backups to the b2 user
+ /bin/mv /root/backups/sync/daily_hetzner3_20240922_061931.tar.gpg /home/b2user/backups/sync/daily_hetzner3_20240922_061931.tar.gpg /bin/mv: cannot stat '/root/backups/sync/daily_hetzner3_20240922_061931.tar.gpg': No such file or directory
- oh, the issue is higher-up; it fails to create the gpg file because gpg is not installed!
+ echo -e '\tINFO: Encrypting the single-file tarball' INFO: Encrypting the single-file tarball + /bin/nice /bin/gpg2 --output /root/backups/sync/daily_hetzner3_20240922_061931.tar.gpg --batch --symmetric --cipher-algo aes256 --compress-algo none --passphrase-file /root/backups/ose-backups-cron.2.key /root/backups/sync/daily_hetzner3_20240922_061931.tar /bin/nice: ‘/bin/gpg2’: No such file or directory real 0m0.002s user 0m0.002s sys 0m0.000s
- huh, gpg *is* installed. I guess it just doesn't use `gpg2` anymore
root@mail ~/backups # which gpg2 root@mail ~/backups # which gpg /usr/bin/gpg root@mail ~/backups #
- I updated the
backup.settingsfile to use/usr/bin/gpg, kicked-off the backup script, and took lunch
...
- the update failed after almost exactly 10 minutes, due to a timeout
root@mail ~/backups # time /root/backups/backup.sh ... + /bin/sudo -u b2user /bin/rclone -v --bwlimit 3M copy /home/b2user/backups/sync/daily_hetzner3_20240922_190109.tar.gpg b2:ose-server-backups 2024/09/22 21:01:10 INFO : Starting bandwidth limiter at 3Mi Byte/s 2024/09/22 21:11:10 Failed to create file system for "b2:ose-server-backups": failed to authorize account: failed to authenticate: Get "https://api.backblazeb2.com/b2api/v1/b2_authorize_account": dial tcp 104.153.233.180:443: i/o timeout real 10m0.067s user 0m0.001s sys 0m0.006s + echo ================================================================================ ================================================================================ ++ date -u +%Y%m%d_%H%M%S + echo 'INFO: Finished Backup Run at 20240922_191110' INFO: Finished Backup Run at 20240922_191110 + echo ================================================================================ ================================================================================ + exit 0 real 10m1.345s user 0m0.242s sys 0m0.043s root@mail ~/backups #
- this smells like I didn't update the firewall rules for this new
b2useruser (our firewall rules block traffic for users by default, unless they're explicitly allowed - I updated the ansible playbook to poke a hole in the firewall for the
b2user, ran ansible, and kicked it off again. - ok, that worked. The backup is stupid small, currently only 11 KB
root@mail ~/backups # time /root/backups/backup.sh ... + /bin/sudo -u b2user /bin/rclone -v --bwlimit 3M copy /home/b2user/backups/sync/weekly_hetzner3_20240922_220636.tar.gpg b2:ose-server-backups 2024/09/23 00:06:36 INFO : Starting bandwidth limiter at 3Mi Byte/s 2024/09/23 00:06:39 INFO : weekly_hetzner3_20240922_220636.tar.gpg: Copied (new) 2024/09/23 00:06:39 INFO : Transferred: 10.104 KiB / 10.104 KiB, 100%, 5.046 KiB/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 3.2s real 0m3.231s user 0m0.003s sys 0m0.003s + echo ================================================================================ ================================================================================ ++ date -u +%Y%m%d_%H%M%S + echo 'INFO: Finished Backup Run at 20240922_220639' INFO: Finished Backup Run at 20240922_220639 + echo ================================================================================ ================================================================================ + exit 0 real 0m3.506s user 0m0.232s sys 0m0.042s root@mail ~/backups # ls -lah /home/b2user/backups total 16K drwx------ 4 b2user b2user 4.0K Sep 23 00:06 . drwxr-xr-x 4 b2user b2user 4.0K Sep 17 07:29 .. drwxr-xr-x 2 root root 4.0K Sep 23 00:06 sync drwx------ 2 b2user b2user 4.0K Sep 23 00:04 sync.old root@mail ~/backups # ls -lah /home/b2user/backups/sync total 20K drwxr-xr-x 2 root root 4.0K Sep 23 00:06 . drwx------ 4 b2user b2user 4.0K Sep 23 00:06 .. -rw-r--r-- 1 b2user root 11K Sep 23 00:06 weekly_hetzner3_20240922_220636.tar.gpg root@mail ~/backups #
- now, the final test: can we download the backup from the Backblaze B2 WUI, decrypt it, and view the files?
- I logged into the BackBlaze B2 WUI, and clicked "Browse Files" on the left-hand navigation
- I clicked on the
ose-server-backupsbucket - I saw one file with "hetzner3" in its name =
weekly_hetzner3_20240922_220636.tar.gpg, and I clicked on it - In the popup modal that appeared, I clicked the
Downloadbutton
- I was successfully able to decrypt and extract its contents, but everything was missing except for mysql, which is funny because that's one that I expected to be missing because we don't even have mysql installed!
user@ose:~/tmp/hetzner3/backup-restore-test$ gpg --batch --passphrase-file ose-backups-cron.2.key --output weekly_hetzner3_20240922_220636.tar weekly_hetzner3_20240922_220636.tar.gpg gpg: WARNING: no command supplied. Trying to guess what you mean ... gpg: AES256.CFB encrypted data gpg: encrypted with 1 passphrase user@ose:~/tmp/hetzner3/backup-restore-test$ user@ose:~/tmp/hetzner3/backup-restore-test$ ls ose-backups-cron.2.key weekly_hetzner3_20240922_220636.tar.gpg weekly_hetzner3_20240922_220636.tar user@ose:~/tmp/hetzner3/backup-restore-test$ user@ose:~/tmp/hetzner3/backup-restore-test$ tar -xf weekly_hetzner3_20240922_220636.tar user@ose:~/tmp/hetzner3/backup-restore-test$ user@ose:~/tmp/hetzner3/backup-restore-test$ du -sh * 4.0K ose-backups-cron.2.key 40K root 12K weekly_hetzner3_20240922_220636.tar 12K weekly_hetzner3_20240922_220636.tar.gpg user@ose:~/tmp/hetzner3/backup-restore-test$ user@ose:~/tmp/hetzner3/backup-restore-test$ ls root/backups/sync/weekly_hetzner3_20240922_220636/ etc home log mysqldump root www user@ose:~/tmp/hetzner3/backup-restore-test$ user@ose:~/tmp/hetzner3/backup-restore-test$ cd root/backups/sync/weekly_hetzner3_20240922_220636/ user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/weekly_hetzner3_20240922_220636/$ user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/weekly_hetzner3_20240922_220636$ find . ./log ./www ./root ./home ./etc ./mysqldump ./mysqldump/mysqldump.20240922_220636.sql.gz user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/weekly_hetzner3_20240922_220636$ cd mysqldump/ user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/weekly_hetzner3_20240922_220636/mysqldump$ user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/weekly_hetzner3_20240922_220636/mysqldump$ du -sh mysqldump.20240922_220636.sql.gz 0 mysqldump.20240922_220636.sql.gz user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/weekly_hetzner3_20240922_220636/mysqldump$
- ok, so the tarball has literally just one file that's 0 bytes large. Fail. This is why we test restores ;)
- hmm, I also realized that today is Sunday, but the backup is marked as a 'weekly'. Weeklys are supposed to happen on the first of the week, which is supposed to be Monday
- ah, hmm, it looks like the locale never got setup properly
root@mail ~/backups # date +%u 1 root@mail ~/backups # date Mon Sep 23 12:36:58 AM CEST 2024 root@mail ~/backups # date -u Sun Sep 22 10:38:33 PM UTC 2024 root@mail ~/backups # <pre> # I uncommented the 'maltfield.locale' role and gave ansible another run # after that, the timezone was UTC (as desired) <pre> root@mail ~/backups # date Sun Sep 22 10:40:21 PM UTC 2024 root@mail ~/backups # root@mail ~/backups # date +%u 7 root@mail ~/backups #
- I reviewed the output of the
backup.shrun. Looks like all thetarcommands are failing because of the change I made to use pgiz
+ /bin/nice /bin/tar --use-compress-program=pigz --exclude '/home/b2user/backups/sync*' -czf /root/backups/sync/weekly_hetzner3_20240922_220636/home/home.20240922_220636.tar.gz /home/b2user /home/maltfield /bin/tar: Conflicting compression options Try '/bin/tar --help' or '/bin/tar --usage' for more information. real 0m0.003s user 0m0.000s sys 0m0.003s
- I triggered another backup run. This time it was 40 MB, and the whole backup & upload process finished in less than 20 seconds
root@mail ~/backups # time /root/backups/backup.sh ... INFO: Beginning upload to backblaze b2 + /bin/sudo -u b2user /bin/rclone -v --bwlimit 3M copy /home/b2user/backups/sync/daily_hetzner3_20240922_224521.tar.gpg b2:ose-server-backups 2024/09/22 22:45:23 INFO : Starting bandwidth limiter at 3Mi Byte/s 2024/09/22 22:45:38 INFO : daily_hetzner3_20240922_224521.tar.gpg: Copied (new) 2024/09/22 22:45:38 INFO : Transferred: 39.678 MiB / 39.678 MiB, 100%, 2.834 MiB/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 15.4s real 0m15.442s user 0m0.000s sys 0m0.006s + echo ================================================================================ ================================================================================ ++ date -u +%Y%m%d_%H%M%S + echo 'INFO: Finished Backup Run at 20240922_224538' INFO: Finished Backup Run at 20240922_224538 + echo ================================================================================ ================================================================================ + exit 0 real 0m16.788s user 0m6.406s sys 0m0.371s root@mail ~/backups #
- back in the web browser, I downloaded this latest backup file from the Backblaze B2 WUI
- and on my computer, I decrypted it, extracted it, and confirmed that I can restore files from log, www, root, home, and etc. Backups are working :)
user@ose:~/tmp/hetzner3/backup-restore-test$ gpg --batch --passphrase-file ose-backups-cron.2.key --output daily_hetzner3_20240922_224521.tar daily_hetzner3_20240922_224521.tar.gpg
gpg: WARNING: no command supplied. Trying to guess what you mean ...
gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
user@ose:~/tmp/hetzner3/backup-restore-test$
user@ose:~/tmp/hetzner3/backup-restore-test$ tar -xf daily_hetzner3_20240922_224521.tar
user@ose:~/tmp/hetzner3/backup-restore-test$
user@ose:~/tmp/hetzner3/backup-restore-test$ du -sh *
40M daily_hetzner3_20240922_224521.tar
40M daily_hetzner3_20240922_224521.tar.gpg
4.0K ose-backups-cron.2.key
40M root
user@ose:~/tmp/hetzner3/backup-restore-test$
user@ose:~/tmp/hetzner3/backup-restore-test$ cd root/backups/sync/daily_hetzner3_20240922_224521/
user@ose:~/tmp/hetzner3/backup-restore-test$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521$ find
.
./log
./log/log.20240922_224521.tar.gz
./www
./www/www.20240922_224521.tar.gz
./root
./root/root.20240922_224521.tar.gz
./home
./home/home.20240922_224521.tar.gz
./etc
./etc/etc.20240922_224521.tar.gz
./mysqldump
./mysqldump/mysqldump.20240922_224521.sql.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521$ cd log
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$ tar -xf log.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$ ls
log.20240922_224521.tar.gz var/
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$ ls var/log/
alternatives.log audit btmp.1 faillog private unattended-upgrades
alternatives.log.1 backups dpkg.log journal README wtmp
apt btmp dpkg.log.1 lastlog runit
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$ tail var/log/apt/history.log
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install stubby unbound resolvconf
Requested-By: maltfield (1000)
Install: stubby:amd64 (1.6.0-3+b1), resolvconf:amd64 (1.91+nmu1), libyaml-0-2:amd64 (0.2.5-1, automatic), libevent-2.1-7:amd64 (2.1.12-stable-8, automatic), libunbound8:amd64 (1.17.1-2+deb12u2, automatic), libgetdns10:amd64 (1.6.0-3+b1, automatic), libevent-core-2.1-7:amd64 (2.1.12-stable-8, automatic), libev4:amd64 (1:4.33-1, automatic), unbound:amd64 (1.17.1-2+deb12u2), dns-root-data:amd64 (2024041801~deb12u1, automatic)
End-Date: 2024-09-15 21:50:47
Start-Date: 2024-09-16 18:46:10
Commandline: /usr/bin/apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install pigz
Requested-By: maltfield (1000)
Install: pigz:amd64 (2.6-1)
End-Date: 2024-09-16 18:46:10
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/log$ cd ../www
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$ tar -xf www.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$ ls
www.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$ du www.20240922_224521.tar.gz
4 www.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$ tar -tf www.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/www$ cd ../root
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$ tar -xf root.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$ ls root
backups dead.letter Mail
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$ ls root/backups/
backupReport.sh backup.settings backup.settings.20240916 backup.sh backup.sh.74511.2024-09-22@22:45:04~ ose-backups-cron.2.key ose-backups-cron.key README.txt
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$ tail root/backups/README.txt
gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --decrypt daily_hetzner2_20240726_160837.tar.gpg > daily_hetzner2_20240726_160837.tar
Then you can untar the wrapper tarball and the compressed tarball inside of that. For example:
tar -xf daily_hetzner2_20240726_160837.tar
cd root/backups/sync/daily_hetzner2_20240726_160837/www/
tar -xf www.20240726_160837.tar.gz
head var/www/html/www.opensourceecology.org/htdocs/index.php
--Michael Altfield <https://michaelaltfield.net.>
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/root$ cd ../home
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$ tar -xf home.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$ ls home
b2user maltfield
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$ ls home/maltfield/
bin ossec.conf.31557.2024-09-15@04:36:48~
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$ tail home/maltfield/ossec.conf.31557.2024-09-15@04\:36\:48~
<log_format>syslog</log_format>
<location>/var/ossec/logs/active-responses.log</location>
</localfile>
<localfile>
<log_format>syslog</log_format>
<location>/var/log/dpkg.log</location>
</localfile>
</ossec_config>
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/home$ cd ../etc
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/etc$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/etc$ tar -xf etc.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/etc$
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/etc$ ls
etc etc.20240922_224521.tar.gz
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/etc$ tail etc/hostname
hetzner3
user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_202user@ose:~/tmp/hetzner3/backup-restore-test/root/backups/sync/daily_hetzner3_20240922_224521/etc$
- note that the contents of www was emtpy, because that dir doesn't exist on the server (yet)
root@mail ~/backups # ls -lah /var/www ls: cannot access '/var/www': No such file or directory root@mail ~/backups #
- I spent some time updating Wazuh with the changes from hetzner2 to hetzner3
Sat Sep 21, 2024
- basically, from my research yesterday, I discovered two ways to be able to login to our backblaze gmail account again:
- enable 2FA for this (shared) user account
- temporarily turn-off "two step authentication" (which is a distinct Google concept from "two factor authentication")
- I did some reading about how one "should" setup a shared mailbox in Google Groups. There's a few ways, the it appears the most recommended is to create a Google Group https://old.reddit.com/r/gsuite/comments/rwx0wb/set_up_an_info_general_email_address_for_all_users/
- I checked OSE's Google Groups, and I see that, apparently, I created a group TODO
- I went ahead and created a new group with the following description:
This is a Google Group. People in this Google Group will receive emails from services related to OSE's internet presence and Internet infrastructure. Please note that "reset password" functionality usually works by sending a link to a user's email address, so we should assume that anyone on this list can login to these services, even if they don't have the account password. So please only ever put trusted users on this list.
- I made it so that anyone can't join this group if they'd like, but anyone with an @opensourceecology.org email account can ask to join
- I made it so that only group members can read & reply to emails in this list
- I made it so anyone with an @opensourceecology.org email account can see who is a member of this list
- I invited Marcin to join this group, with the following message
Hey Marcin, I'm creating this "Google Group" to forward messages about OSE's online presence and Internet infrastructure (eg OSE Server) to anyone who is a member. The reason I'm setting this up is so that an event like what just happened with backblaze (we failed to pay ~$10 for a couple months, and they deleted all our backups!) won't happen again, because we can setup service-specific accounts (eg for backblaze) to forward all of its mail to this email list, which in-turn will make these alerts appear in multiple people's inboxes. Please note that "reset password" functionality usually works by sending a link to a user's email address, so we should assume that anyone on this list can login to these services, even if they don't have the account password. So please only ever put trusted users on this list. Cheers, Michael Altfield
- I set the "subscription" to "Each Email" -- as opposed to the others, which I guess group-together all the emails into a summary email
- To re-gain access to our backblaze email account, I logged into the admin.google.com panel
- Clicked Directory -> Users
- Clicked on the backblaze username
- Clicked on the "Security" tab
- Scrolled-down to "Login challenge" and clicked the "TURN OFF FOR 10 MINS" button
- finally, this time when I attempted to login to the backblaze gmail account, I was let-in!
- I saw some security alerts and marked them all as "yeah it as me"
- I setup email forwarding of all incoming mail to this new address
- Additionally, from the backblaze gmail UI, I went to Settings -> Accounts -> "Grant access to your account" and I added marcin & myself to have "access" to the backblaze account. This is not a replacement for the Google Groups list, but I'm hoping its another way for our users to be able to login as the backblaze user (using our own creds) to be able to read and send mail on behalf of this user
- Surprisingly, I don't think we had *any* documentation on the wiki about Google Workspace, so I created Google Workspace
- I documented what I did today, and how we should use Google Groups for service-specific accounts
- I logged-into our backblaze account, and I see that the $6.12 account balance is marked as "Paid"
- I sent an email reply to Backblaze support asking to confirm that our account is now in good-standing, and that we don't need to do anything further to prevent data loss
- I checked our "buckets" page, and they're both there (thank god)
- ose-server-backups @ 446.4 GB
- ose-dev-server-backups @ 303.1 MB
- I also went ahead and set the following service-specific accounts to forward to our new Google Groups list
- Let's Encrypt
- Cloudflare
- Status Cake
- I sent Marcin an email asking him to click the "accept" link in the emails that he received
- I replied to the Backblaze support ticket again, this time asking for them to forward some feature requests to their product team which, if implemented, would greatly reduce the risk of us loosing our data due to an event like this in the future
Hi Joshua, > I'd be happy to forward over your interest in these alternate options to our teams for consideration. Yes, we do have some feature requests that we'd like you to forward to your product team. 1. We would like the ability to decrease the amount of days in the Grace Period from 30 days to 0 days, such that it increases the No Service period from 14 days to 44 days. 2. We would like the ability to add additional users to our account, such that individual users can login (with their own personal username/credentials), and so that each user will receive "Billing Failed" email alerts separately in their own inboxes. 3. We would like to be able to transfer a sum of funds into our Backblaze account in excess of a given bill, such that we can maintain a positive balance from which Backblaze will deduct monthly fees-from, in the event that credit card payments failed 4. We would like the ability to pay with cryptocurrency, as a backup payment method in the event that we have issues with our bank blocking our credit card payments. Can you please forward these requests to your product team, and let us know if any are currently planned?
- within an hour, the backblaze support rep confirmed that our account is now in good-standing and that none of the features we requested are currently planned
I can confirm that the $6.12 charge has been successfully paid for and your account is now in good standing. No further action will be needed on that end. As for those feature requests, thank you for providing your input! I'll be sure to pass these over to our product team. At the moment, none of these requests are currently planned but I'll pass it forward for consideration. Let me know if you have any questions or any other feature requests.
..
- ok -- now -- with all that in-place, it's time to actually setup the new backblaze api keys, so we can get hetzner3 uploading its backups to backblaze
- currently we have 5x application keys (besides the master keys, which we shouldn't be using anywhere)
- this is odd; I would expect us to only have 1 or 2 keys in-use?
- I re-visited my notes from Maltfield_Log/2022#Fri_October_28.2C_2022, when I last setup rclone. It was when some python version on the OS broke b2 and our backups, so I switched from b2 (which was not available in yum on CentOS) to rclone
- Per my notes, however, it looks like I can't actually create an append-only (protected from ransomware) key with the Backblaze B2 WUI; I *have* to use the b2 CLI app directly
- The good news is: the b2 cli is directly available in the official Debian repos. So now that we've switched to Debian, we can simply install the backblaze b2 CLI
- ah, yeah, my notes from 2022-10-28 say that I created two keys:
- One for the b2 user with only
readFiles, writeFiles(namedprod-append-only-2022-10) - And one for the root user with
listFiles, readFiles, writeFiles(namedprod-list-and-append-only-2022-10)
- One for the b2 user with only
- I spent some time starting to document this at Backblaze
Fri Sep 20, 2024
- Marcin paid the backblaze bill last night
- he said that after payment, all our buckets (and therefore all of our backup data) was gone
- I submitted a support request asking [a] if we can restore the data, [b] how we can build better reporting tools to notify us of failed payments and [c] how we can setup better payment methods to avoid this issue in the future
Hi, We just discovered that our bank was blocking payments to backblaze "for our protection" As soon as we discovered the issue, we settled the balance. Unfortunately, now when we login, we don't see any buckets. Is it possible to restore the data? Also important: how can we prevent this from happening again? I see two major areas for improvement: 1. Better Alerting 2. Better Payment Methods === Better Alerting === We have a script that runs once per month (on the second of every month) that checks (with rclone) to see if our monthly, weekly, and yearly backups can be found in our B2 bucket. This works great when we have a technical issue. If a backup is missing, our server generates a high-priority email alert that gets sent out to everyone in our small non-profit organization. Unfortunately, we just learned that it, apparently, *doesn't work for payment issues*. Here's the problem: On 2024-09-02, our Backblaze Report indicated no issues – even though payment wasn't made for 17 days at that point. On 2024-10-02, we would have received our first report indicating an error. We only discovered the payment issue early because I happened to login to the Backblaze B2 WUI to create a new API key. We only do this every few years, so it was incredible that I noticed there was an issue with the account, at all. How can we update our reporting bash script (which uses rclone, not the backblaze CLI tool) to throw an error when the account is not in good-standing? How long is the grace period? Is it possible for us to make all our API calls fail during the grace period, yet still retain our data during the grace period? This would make it so that our reporting tools alert us about the lack-of-payment, without the risk of data loss of our very, very important backups. === Better Payment Methods === We have had numerous issues (with a couple different banks at this point) where our CNP transactions fail due to false-positives. We have to call the bank and tell them that, yes, that recurring transaction for Backblaze B2 is still valid. We've also had this issue with other service providers. Usually, to prevent the risk of service shutdown due to bank-blocked auto-payments, we hedge this risk by transferring a large sum of money to our service provider's accounts yearly -- such that we maintain a balance with them. We keep the balance on the account sufficient to cover the costs of 1 year of service, so if we miss 12 monthly payments in-a-row, they can be deducted from the balance without service disruptions. You get paid. Our service doesn't shutdown. Everyone wins. Some of these service providers accept cryptocurrency, which – while not a good option for recurring payments – it works great to top-up the account balance, because there's no faulty "fraud detection" system that can block the payment; payments always goes through, unlike with less-secure credit card systems utilizing faulty "fraud detection" systems. Is it possible for us to transfer a positive balance to our Backblaze account of, say, $100 – so that you can deduct payments from this account balance in the event that there's a failed payment due to issues with the bank? And what alternative payment methods exist? Is cryptocurrency an option? Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org -- *Full Disclosure:* OSE works openly. All conversations in this email are intended to be transparent and subject to sharing, with due respect. OSE does not sign NDAs in order to promote collaboration. All of our work is libre or open source. If you are discussing potential hardware development collaboration, your work must also be open source pursuant to the Open Source Hardware Association definition <http://www.oshwa.org/definition/>.
- I tried to login to our backblaze-specific gapps account, but I again got an error demanding a phone number to "recover" my account (false-positives, even though I entered the correct password on the first attempt!)
- The error said to contact the gapps admin. I am the admin!
- I tried to login to my gapps admin account. Last time I tried this (a couple days ago), I got the same error, but today it let me in
- I immediately setup 2FA via TOTP. There's no guarantee, but I hope that this additional protection will prevent Google from locking me out of my own account, just because they can't see any history in their panoptic datasets from tracking my browsers fingerprint across the Internet..
- anyway, I went to the admin console for the OSE gapps https://admin.google.com/
- I don't see any alerts about failed logins for the backblaze user
- if I open the backblaze user, I still see nothing. If I click on the "Investigate" tab and click "View logs" next to "Failed sign-ins", then I do see an entry for a failed login attempt 2 days ago
- curiously that entry says "Is suspicious" = "False"
- In another ephemeral/sandboxed browser, I tried to login ask the backblaze user; same issue.
- After I correctly enter the username and password on the first try, I'm told to "Enter a phone number to get a text message with a verification code."
- For security (and practicality; this is a shared account so it should definitely not be tied to anyone's phone number), I ignore the request and click "Try another way" -- but really I just want to click "skip"; I already entered the correct password on the first try -- just let me in!
- It then brings me to a page that says "Choose how you want to sign in". I have two options: "Get a verification code sent to your phone" or "Get help"
- of course, this is absurd; I've already signed-in! Why can't I just skip this?
- The only option is "Get Help", which then just displays "Contact your domain admin for help. Learn more"
- If I click "Learn more", I'm brought to this page https://support.google.com/accounts/answer/181627?hl=en
Sign in to your work, school, or other group account You can't reset or recover the password to a work, school, or other group account on this page because your organization has restricted password changes. You'll need to contact your administrator. For students who use Classroom, go to Troubleshooting for students.
- But ^ that page is ridiculous and not relevant. I'm not trying to reset or change my password; I just want to login! I already auth'd! Let me in!!
- Meanwhile, with my admin hat on in the other browser -- wtf am I supposed to do? Where's the button to say: "Google, stop being an asshole, just let that user in if they entered the correct password. Why are you being so stupid, Google?"
- If I refresh the "Investigate" tab, I don't even see the failed login from just now
- I did find the docs on the "failed login" entry, at least https://developers.google.com/admin-sdk/reports/v1/appendix/activity/login#login_failure
- If I click on the entry for my failed login attempt to my own admin account on 2024-09-18 (two days ago), I don't even see 'login_failure_type' -- like it's saying I failed to login, but there is no reason it failed. Yeah, that tracks.
- One of the options for 'login_failure_type' is 'login_failure_invalid_password' -- so, yeah, I guess this is saying that I had the right password, but it failed anyway. Why tho!?!
- Unfortunately, if I can't I still can't login to the account login to this account, then I won't be able to see the response from backblaze support
- in the meantime, I opened this question on SE asking why the fuck Google is blocking me from logging-in https://webapps.stackexchange.com/questions/177032/why-is-google-blocking-a-user-login-with-failed-login-when-they-entered-the-co
- I found this, but there is no answer saying how to disable this faulty bug https://webapps.stackexchange.com/questions/115156/how-to-stop-googles-google-prevented-suspicious-attempt
- I found this thread on reddit from another equally frustrated gapps admin https://old.reddit.com/r/gsuite/comments/14psu8s/how_do_i_disable_login_challenges_permanently/
- A common suggestion is to enable 2FA. That's not a great solution for a shared account. Anyway, a 100-char random passphrase should be good enough. And storing a 2FA secret key in the same keepass DB won't add any additional security.
- Ok, google may internally refer to this as a "login challenge security method" -- not very clear because a password is a "login challenge". But, anyway, here's a KB article by Google titled "How to disable login challenge security method permanently" https://knowledge.workspace.google.com/kb/how-to-disable-login-challenge-security-method-permanently-000007696
- the answer is totally unsatisfying. It's not a toggle button. It says "This is working to product specification. Login challenges cannot be disabled permanently. They say to enable 2FA.
- damn, it's a straight-up "we don't care about your business needs or that our tech is faulty and locking you out of your own accounts; fuck you and wontfix".
Tue Sep 18, 2024
- I need to create a new set of backblaze b2 api keys for hetzner3 (and then configure rclone to use it for backups)
- unfortunatly, after I logged into the backblaze wui, I got an error
- I clicked on "application keys" in the left-hand navigation menu, and I got a similar error
We were unable to retrieve your keys. Error: failed to advance iterator: B2 Storage API(s) not Enabled Please contact support if this continues.
- oh man, that was replaced with this
Your access to B2 has been suspended because your account has not been in good standing and your grace period has now ended. Please review your account and update your payment method at payment history, or contact tech support for assistance.
- shit, I checked the payments page and I see a lot of red
- last successful payment was July 2024
- there's 4x failed payments since then
- I manually ran the backup script; it alerted us immediately that our key was bad
[root@opensourceecology backups]# ./backupReport.sh INFO: email body below ATTENTION: BACKUPS MISSING! 2024/09/19 01:40:43 Failed to create file system for "b2:ose-server-backups": failed to authorize account: failed to authenticate: B2 Storage API(s) not Enabled (403 access_denied) WARNING: First of this month's backup (20240901) is missing! WARNING: First of last month's backup (20240801) is missing! WARNING: Yesterday's backup (20240918) is missing! WARNING: The day before yesterday's backup (20240917) is missing! See below for the contents of the backblaze b2 bucket = ose-server-backups 2024/09/19 01:40:43 Failed to create file system for "b2:ose-server-backups": failed to authorize account: failed to authenticate: B2 Storage API(s) not Enabled (403 access_denied) --- Note: This report was generated on 20240919_014042 UTC by script '/root/backups/backupReport.sh' This script was triggered by '/etc/cron.d/backup_to_backblaze' For more information about OSE backups, please see the relevant documentation pages on the wiki: * https://wiki.opensourceecology.org/wiki/Backblaze * https://wiki.opensourceecology.org/wiki/OSE_Server#Backups [root@opensourceecology backups]#
- Our last backups report came on Sep 02, and it was fine. So I guess our "grace period" was active then, and it ended sometime in the past ~2 weeks
No missing backups detected See below for the contents of the backblaze b2 bucket = ose-server-backups 21349753244 monthly_hetzner2_20231001_072001.tar.gpg 21360808568 monthly_hetzner2_20231101_072001.tar.gpg 21360301269 monthly_hetzner2_20231201_072001.tar.gpg 21820017340 monthly_hetzner2_20240201_072001.tar.gpg 21683700909 monthly_hetzner2_20240301_072001.tar.gpg 21660296728 monthly_hetzner2_20240401_072001.tar.gpg 21790035424 monthly_hetzner2_20240501_072001.tar.gpg 21603737883 monthly_hetzner2_20240601_072001.tar.gpg 21663769333 monthly_hetzner2_20240701_072001.tar.gpg 21991147307 monthly_hetzner2_20240801_072001.tar.gpg 21896377523 monthly_hetzner2_20240901_072001.tar.gpg 22031705783 weekly_hetzner2_20240812_072001.tar.gpg 21980940370 weekly_hetzner2_20240819_072001.tar.gpg 21942660432 weekly_hetzner2_20240826_072001.tar.gpg 21902006508 weekly_hetzner2_20240902_072001.tar.gpg 17516124812 yearly_hetzner2_20190101_111520.tar.gpg 18872422001 yearly_hetzner2_20200101_072001.tar.gpg 19827971632 yearly_hetzner2_20210101_072001.tar.gpg 21079942509 yearly_hetzner2_20230101_072001.tar.gpg 21541199047 yearly_hetzner2_20240101_072001.tar.gpg --- Note: This report was generated on 20240903_042001 UTC by script '/root/backups/backupReport.sh' This script was triggered by '/etc/cron.d/backup_to_backblaze' For more information about OSE backups, please see the relevant documentation pages on the wiki: * https://wiki.opensourceecology.org/wiki/Backblaze * https://wiki.opensourceecology.org/wiki/OSE_Server#Backups
- I tried to login to our OSE-specific backblaze gmail account for this (which I would hope was setup to forward emails to Marcin & myself), but I got an error on-login demanding a phone number. When I said I didn't have a phone number (it's definitely not a good idea to link phones numbers to google accounts), it said to contact the gapps admin. That's me!!
- I tried to login as my gapps admin account, and it had the same issue: enter phone number. Say no, and it says "enter the last password that you remember" — as if I didn't enter the creds correctly & perfectly the first time. Then it said to ask another admin to recover my account
- I did *not* forget my password. This happens to me ~50% of the time I use Google, and it's why I never recommend orgs use Google. They lock you out of your own account, even if you enter the correct credentials on the very first time. Google's faulty "fraud protection" isn't FOSS, but it's most likely using some super-shitty machine-learning anonmoly detection that false-positives on me because they can't track my activity on the internet. I'm "fresh" to them, so they raise a flag and ban me from my own account. There's numerous discussions on the Internet of people loosing access to their google accounts because of this.
- anyway, if I can't login go google, then I can't investigate email alerts
Mon Sep 17, 2024
- Marcin emailed me today saying that he can't email to gmail (but apparently OSE-to-OSE emails work)
- I replied-all, and I got an error back from Catarina's email
Message blocked Your message to OBFUSCATED@gmail.com has been blocked. See technical details below for more information. Learn more here: https://support.google.com/mail/answer/81126#authentication The response was: 550 5.7.26 Your email has been blocked because the sender is unauthenticated. Gmail requires all senders to authenticate with either SPF or DKIM. Authentication results: DKIM = did not pass SPF [opensourceecology.org] with ip: [209.85.220.41] = did not pass For instructions on setting up authentication, go to https://support.google.com/mail/answer/81126#authentication d75a77b69052e-459aad0da83sor46218481cf.5 - gsmtp
- I checked our DNS, and I got this
user@disp3433:~$ while true; do date; dig -t TXT opensourceecology.org; sleep 300; echo; done Tue Sep 17 11:58:03 PM -05 2024 ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t TXT opensourceecology.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21111 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;opensourceecology.org. IN TXT ;; ANSWER SECTION: opensourceecology.org. 120 IN TXT "spf1=v a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ip4:144.76.164.201 ip6:2a01:4f8:200:40d7::2 ~all" ;; Query time: 113 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Tue Sep 17 23:58:05 -05 2024 ;; MSG SIZE rcvd: 182 ^C user@disp3433:~$
- mxtoolbox.com also complains about the SPF record https://mxtoolbox.com/emailhealth/opensourceecology.org/
- oh, yikes, looks like it says "spf1=v" and it should be "v=spf1"
- I quickly changed it in cloudflare, and now it looks good
user@disp3433:~$ while true; do date; dig -t TXT opensourceecology.org; sleep 300; echo; done Wed Sep 18 12:02:36 AM -05 2024 ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t TXT opensourceecology.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10396 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;opensourceecology.org. IN TXT ;; ANSWER SECTION: opensourceecology.org. 120 IN TXT "v=spf1 a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ip4:144.76.164.201 ip6:2a01:4f8:200:40d7::2 ~all" ;; Query time: 106 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Wed Sep 18 00:02:36 -05 2024 ;; MSG SIZE rcvd: 182 ^C user@disp3433:~$
- I also refreshed mxtoolbox.com, and it no longer complains about the SPF record
- I sent marcin another reply, and it worked.
Mon Sep 16, 2024
- since last night, I have 4 PGP-encrypted emails in my inbox from wazuh@hetzner3.opensourceecology.org; it's working!
- it's warning me about a trojaned version of /bin/diff, but I've seen that error on other systems; seems to be a false-positive on Debian
- there's already a bunch of open/duplicate reports of this; wazuh team seems sloppy in managing their tickets https://github.com/wazuh/wazuh/issues/13278#issuecomment-2333707523
- but a PR was submitted last week https://github.com/wazuh/wazuh/issues/13278#issuecomment-2335327334
- anyway, sad truth is that OSE has 0 full-time sysadmins, so nobody is probably going to be reading wazuh alert emails, anyway. It's useful because;
- it creates an off-system audit log that can be reviewed post-incident
- it's a solid foundation for the future, if OSE scales and actually gets a full-time sysadmin team
- the other contents of the wazuh alerts overnight were netstat diff reports
- looks like some sshd connection disappeared, probably my laptop going to sleep
- oh, and I got another alert where the sshd connection popped-up again at 09:30-ish, so -- yeah -- my laptop reconnecting again
tcp 127.0.0.1:47362 0.0.0.0:* 1612/sshd
...
- I also went ahead and updated the hostname, so my shell clearly says which system I'm working-on
root@mail ~ # hostnamectl
Static hostname: mail
Icon name: computer-desktop
Chassis: desktop 🖥️
Machine ID: 6f64c84dc1094f43a1acbb37ba58b697
Boot ID: 4fb17ee3a5884a4f8f3a219ffd329cbc
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-21-amd64
Architecture: x86-64
Hardware Vendor: FUJITSU
Hardware Model: D3401-H1
Firmware Version: V5.0.0.11 R1.29.0 for D3401-H1x
root@mail ~ #
root@mail ~ # hostnamectl set-hostname hetzner3
root@mail ~ #
root@mail ~ # hostnamectl
Static hostname: hetzner3
Icon name: computer-desktop
Chassis: desktop 🖥️
Machine ID: 6f64c84dc1094f43a1acbb37ba58b697
Boot ID: 4fb17ee3a5884a4f8f3a219ffd329cbc
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.1.0-21-amd64
Architecture: x86-64
Hardware Vendor: FUJITSU
Hardware Model: D3401-H1
Firmware Version: V5.0.0.11 R1.29.0 for D3401-H1x
root@mail ~ #
root@mail ~ # cat /etc/hosts
### Hetzner Online GmbH installimage
127.0.0.1 localhost.localdomain localhost
144.76.164.201 mail.opensourceecology.org mail
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
2a01:4f8:200:40d7::2 mail.opensourceecology.org mail
root@mail ~ #
root@mail ~ # vim /etc/hosts
...
root@mail ~ #
root@mail ~ # cat /etc/hosts
### Hetzner Online GmbH installimage
127.0.0.1 localhost.localdomain localhost
144.76.164.201 hetzner3.opensourceecology.org hetzner3
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
2a01:4f8:200:40d7::2 hetzner3.opensourceecology.org hetzner3
root@mail ~ #
...
- now that we've used ansible to harden and configure our security-sensitive base packages, I want to setup backups before provisioning anything else (like databases, web servers, etc)
- so I commented-out all the roles except 'maltfield.backups' in ansible, and gave it a run; looks successful
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml PLAY [hetzner3] ************************************************************************ TASK [Gathering Facts] ***************************************************************** ok: [hetzner3] TASK [maltfield.backups : install backups prereqs] ************************************* changed: [hetzner3] TASK [maltfield.backups : Make root's hardened backups dir] **************************** changed: [hetzner3] TASK [maltfield.backups : Add b2user user] ********************************************* changed: [hetzner3] TASK [maltfield.backups : Make b2user's hardened backups] ****************************** changed: [hetzner3] TASK [maltfield.backups : Make b2user's sync dir] ************************************** changed: [hetzner3] TASK [maltfield.backups : Backup logs dir] ********************************************* changed: [hetzner3] TASK [maltfield.backups : Backup script] *********************************************** changed: [hetzner3] TASK [maltfield.backups : Backup Report script] **************************************** changed: [hetzner3] TASK [maltfield.backups : Backup README.txt] ******************************************* changed: [hetzner3] TASK [maltfield.backups : Backup cron] ************************************************* changed: [hetzner3] TASK [install basic essential packages] ************************************************ ok: [hetzner3] PLAY RECAP ***************************************************************************** hetzner3 : ok=12 changed=10 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 user@ose:~/sandbox_local/ansible/hetzner3$
- here's what it put on the server
root@mail ~ # ls -lah /root/backups/ total 24K drwx------ 2 root root 4.0K Sep 16 18:46 . drwx------ 8 root root 4.0K Sep 16 18:46 .. -rwx------ 1 root root 3.2K Sep 16 18:45 backupReport.sh -rwx------ 1 root root 6.4K Sep 16 18:45 backup.sh -rwx------ 1 root root 1.1K Sep 16 18:45 README.txt root@mail ~ # root@mail ~ # ls -lah /etc/cron.d total 24K drwxr-xr-x 2 root root 4.0K Sep 16 18:46 . drwxr-xr-x 84 root root 4.0K Sep 16 18:46 .. -rw-r--r-- 1 root root 248 Sep 16 18:45 backup_to_backblaze -rw-r--r-- 1 root root 201 Mar 5 2023 e2scrub_all -rw-r--r-- 1 root root 563 Jul 31 23:44 mdadm -rw-r--r-- 1 root root 102 Mar 2 2023 .placeholder root@mail ~ # root@mail ~ # cat /etc/cron.d/backup_to_backblaze # Ansible managed SHELL=/bin/bash 20 07 * * * root sleep $(( RANDOM \% 3600 )) && time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log 20 04 03 * * root sleep $(( RANDOM \% 3600 )) && time /bin/nice /root/backups/backupReport.sh root@mail ~ # root@mail ~ # dpkg -l | grep -i rclone ii rclone 1.60.1+dfsg-2+b5 amd64 rsync for commercial cloud storage root@mail ~ #
- oh, I also got an email from wazuh informing me that pigz was installed (this is new), nice :)
- it also says that /etc/passwd (and some other important system files) were updated
- unfortunately, the message was cut-off
- there's some things (intentionally) missing:
- the rclone config (with secret keys not to be stored in ansible)
- /root/backups/backups.settings (with passwords not to be stored in ansible)
- /root/backups/*.key (the private key used to encrypt/decrypt backups, not to be stored in ansible)
- I figured that I would generate a new key (it's a good idea to rotate keys at least once a year, and the last key was generated 2018-04-02 (over 6 years ago))
- I actually did generate a new key, but when I went to add it to our OSE shared keepass, I realized that I had already pre-generated an additional 2 unused keys
- I left a note in the OSE Keepass entry saying that I wanted to go ahead and pregenerate keys for future use, so that I was sure they'd be in Marcin's offline copy that we made when I was at FeF. Smart thinking, Michael!
2018-05-21: At the time of writing, we're currently using 'ose-backups-cron.key' for encrypting our backups. The following two key files are unused. They've just been pre-generated & stored here in keepass to ensure that our backups (namely, Marcin's Veracrypt'd usb drive) includes the current backup encryption key + future ones if we need to rotate keys.
- So I deleted my newly-generated key, and instead uploaded `ose-backups-cron.2.key` to hetzner3
- I also uploaded `ose-backups-cron.key` to hetzner3, of course, so that it can still decrypt backups made by hetzner2
root@mail ~/backups # ls -lah total 32K drwx------ 2 root root 4.0K Sep 16 19:36 . drwx------ 8 root root 4.0K Sep 16 19:12 .. -rwx------ 1 root root 3.2K Sep 16 18:45 backupReport.sh -rwx------ 1 root root 6.4K Sep 16 18:45 backup.sh -r-------- 1 root root 4.0K Sep 16 19:31 ose-backups-cron.2.key -r-------- 1 root root 4.0K Sep 16 19:34 ose-backups-cron.key -rwx------ 1 root root 1.1K Sep 16 18:45 README.txt root@mail ~/backups #
- next, I rsync'd over the old backup.settings file and made a backup of it
root@mail ~/backups # ls -lah total 36K drwx------ 2 root root 4.0K Sep 16 20:05 . drwx------ 8 root root 4.0K Sep 16 20:00 .. -rwx------ 1 root root 3.2K Sep 16 18:45 backupReport.sh -rw------- 1 root root 966 Oct 29 2022 backup.settings -rwx------ 1 root root 6.4K Sep 16 18:45 backup.sh -r-------- 1 root root 4.0K Sep 16 19:31 ose-backups-cron.2.key -r-------- 1 root root 4.0K Sep 16 19:34 ose-backups-cron.key -rwx------ 1 root root 1.1K Sep 16 18:45 README.txt root@mail ~/backups # cp backup.settings backup.settings.20240916 root@mail ~/backups # root@mail ~/backups # ls -lah total 40K drwx------ 2 root root 4.0K Sep 16 20:06 . drwx------ 8 root root 4.0K Sep 16 20:00 .. -rwx------ 1 root root 3.2K Sep 16 18:45 backupReport.sh -rw------- 1 root root 966 Oct 29 2022 backup.settings -rw------- 1 root root 966 Sep 16 20:06 backup.settings.20240916 -rwx------ 1 root root 6.4K Sep 16 18:45 backup.sh -r-------- 1 root root 4.0K Sep 16 19:31 ose-backups-cron.2.key -r-------- 1 root root 4.0K Sep 16 19:34 ose-backups-cron.key -rwx------ 1 root root 1.1K Sep 16 18:45 README.txt root@mail ~/backups #
- I editing the settings file, making a few changes
- I added the 'EMAIL_LIST=' variable, which was previously in the backupReports.sh file (but now that this is provisioned by ansible and we're uploading ansible roles to GitHub, I removed it because I didn't want our email addresses exposed on GitHub)
- I added a 'PIGZ=' variable, with the path to the `pigz` binary -- this will let us compress files across multiple cores, which was previously pegged to 1 CPU using just `gzip`
- I removed the amazon glacier creds; we don't use this anymore because min retention reqs for glacier made it insanely expensive
- I removed b2 binary stuff; this broke on hetzner2 due to python changes, and we ended-up switching to rclone, which is actually in the official debian repos
- I changed the path to the 'encryptionKeyFilePath=' from '/root/backups/ose-backups-cron.key' to '/root/backups/ose-backups-cron.2.key'
- I changed the path to the 'b2StagingDir' from '/home/b2user/sync' to '/home/b2user/backups/sync'
root@mail ~/backups # vim backup.settings ...
Sun Sep 15, 2024
- it looks like our ssh server keys aren't ideal (eg 3072-bit RSA key should be 4096-bit)
root@mail /etc/ssh # ssh-keygen -lf ssh_host_rsa_key.pub 3072 SHA256:sS0Nbe0cHagMevsRQxZ7BkRw2UlrnwLxPSOIl1JUWF0 root@mail.opensourceecology.org (RSA) root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -lf ssh_host_ecdsa_key.pub 256 SHA256:eTYBZQgqpbmUpMD1XKPFEkUiHHMwAfY4rB60osHI5jA root@mail.opensourceecology.org (ECDSA) root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -lf ssh_host_ed25519_key.pub 256 SHA256:xmAIk3DB/dXxxm2PunsITI8fRDwBXjH2W5ULXKsysBQ root@mail.opensourceecology.org (ED25519) root@mail /etc/ssh #
- so I rotated the current ssh host keys, and I regenerated new ones
revoked_keys root@mail /etc/ssh # mkdir original_host_keys.20240915 root@mail /etc/ssh # root@mail /etc/ssh # mv ssh_host* original_host_keys.20240915/ root@mail /etc/ssh # root@mail /etc/ssh # du -sh original_host_keys.20240915 28K original_host_keys.20240915 root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -b 4096 -o -a 100 Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /etc/ssh/ssh_host_rsa_key Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub The key fingerprint is: SHA256:1VDOqdL3X0cNt/2qDWHyqxLMd8Tz8hb2t89+kkEejPE root@mail The key's randomart image is: +---[RSA 4096]----+ | ... | | =.. | | ..==. .| | o .= Eo+| | oS +.+= o+| | +..=oo*..| | o .o+.=+| | . +=+*| | ...ooo+X| +----[SHA256]-----+ root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -t ecdsa -b 521 -o -a 100 Generating public/private ecdsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub The key fingerprint is: SHA256:hxzCaH018Wn8tEn7sZ22P/WrNMmZ29w5tK/w0KHq8eE root@mail The key's randomart image is: +---[ECDSA 521]---+ | +. | | + . + . | | o + o = o | | . + o . + + | | S . *. | | . .o++*| | . =Bo+*| | =.=*+=| | .o Eo+BX| +----[SHA256]-----+ root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -t ed25519 -a 100 Generating public/private ed25519 key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub The key fingerprint is: SHA256:5SGqAP2v//HV5aJ0VyhI7AcG3p5MKTzgXyfvzpuIP0Q root@mail The key's randomart image is: +--[ED25519 256]--+ | . . | | . . + + . | |. . . * & . | | . . o #EO . | | . . . S.B + . o| | . o .o o o.| | . . .. + + o| | . +.* + o | | ....o.+.*. | +----[SHA256]-----+ root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -lf ssh_host_rsa_key.pub 4096 SHA256:1VDOqdL3X0cNt/2qDWHyqxLMd8Tz8hb2t89+kkEejPE root@mail (RSA) root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -lf ssh_host_ecdsa_key.pub 521 SHA256:hxzCaH018Wn8tEn7sZ22P/WrNMmZ29w5tK/w0KHq8eE root@mail (ECDSA) root@mail /etc/ssh # root@mail /etc/ssh # ssh-keygen -lf ssh_host_ed25519_key.pub 256 SHA256:5SGqAP2v//HV5aJ0VyhI7AcG3p5MKTzgXyfvzpuIP0Q root@mail (ED25519) root@mail /etc/ssh #
- I was able to fix wazuh
- first I just took the default ossec.conf file that was put in-place at /var/ossec/etc/ossec.conf after wazuh 4.9.0-1 was installed fresh on Debian 12
- then I manually merged-in the important options from our old ossec.conf file (which was used on CentOS7 with, I think, ossec v2.9.0)
- one obvious change appears to be that <rules> no longer exists. Now it's <ruleset> -- where (instead of defining individual .xml rules files) we define a directory where the .xml rules files live
- anyway, after re-provisioning wazuh with this merged file, it started
- most importantly, wazuh should be managing active responses
- I tested this, by throwing a failed ssh "attack" in a loop
- $ while true; do date; ssh -p 32415 144.76.164.201; sleep 1; echo; done
Sun 15 Sep 2024 12:53:02 PM -05 user@144.76.164.201: Permission denied (publickey).
Sun 15 Sep 2024 12:53:06 PM -05 user@144.76.164.201: Permission denied (publickey).
Sun 15 Sep 2024 12:53:11 PM -05 user@144.76.164.201: Permission denied (publickey).
Sun 15 Sep 2024 12:53:14 PM -05 user@144.76.164.201: Permission denied (publickey).
Sun 15 Sep 2024 12:53:18 PM -05 user@144.76.164.201: Permission denied (publickey).
Sun 15 Sep 2024 12:53:23 PM -05 user@144.76.164.201: Permission denied (publickey).
Sun 15 Sep 2024 12:53:26 PM -05 user@144.76.164.201: Permission denied (publickey).
- after this try, my existing ssh session got stuck
- I changed IPs, reconnected, and saw this in the logs triggering both 'hosts-dey' and 'firewall-drop
2024/09/15 19:53:47 active-response/bin/host-deny: Starting
2024/09/15 19:53:47 active-response/bin/host-deny: {"version":1,"origin":{"name":"node01","module":"wazuh-execd"},"command":"add","parameters":{"extra_args":[],"alert":{"timestamp":"2024-09-15T19:53:47.144+0200","rule":{"level":10,"description":"sshd: brute force trying to get access to the system. Non existent user.","id":"5712","mitre":{"id":["T1110"],"tactic":["Credential Access"],"technique":["Brute Force"]},"frequency":8,"firedtimes":1,"mail":true,"groups":["syslog","sshd","authentication_failures"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["SI.4","AU.14","AC.7"],"pci_dss":["11.4","10.2.4","10.2.5"],"tsc":["CC6.1","CC6.8","CC7.2","CC7.3"]},"agent":{"id":"000","name":"mail"},"manager":{"name":"mail"},"id":"1726422827.1936013","previous_output":"Sep 15 17:53:42 mail sshd[100734]: Invalid user user from 146.70.188.50 port 38336\nSep 15 17:53:38 mail sshd[100722]: Invalid user user from 146.70.188.50 port 47542\nSep 15 17:53:34 mail sshd[100720]: Invalid user user from 146.70.188.50 port 47532\nSep 15 17:53:30 mail sshd[100718]: Invalid user user from 146.70.188.50 port 38758\nSep 15 17:53:26 mail sshd[100716]: Invalid user user from 146.70.188.50 port 38746\nSep 15 17:53:22 mail sshd[100714]: Invalid user user from 146.70.188.50 port 38732\nSep 15 17:52:52 mail sshd[100711]: Invalid user user from 146.70.188.50 port 51720","full_log":"Sep 15 17:53:46 mail sshd[100736]: Invalid user user from 146.70.188.50 port 38352","predecoder":{"program_name":"sshd","timestamp":"Sep 15 17:53:46","hostname":"mail"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"146.70.188.50","srcport":"38352","srcuser":"user"},"location":"journald"},"program":"active-response/bin/host-deny"}}
2024/09/15 19:53:47 active-response/bin/host-deny: {"version":1,"origin":{"name":"host-deny","module":"active-response"},"command":"check_keys","parameters":{"keys":["146.70.188.50"]}}
2024/09/15 19:53:47 active-response/bin/host-deny: {"version":1,"origin":{"name":"node01","module":"wazuh-execd"},"command":"continue","parameters":{"extra_args":[],"alert":{"timestamp":"2024-09-15T19:53:47.144+0200","rule":{"level":10,"description":"sshd: brute force trying to get access to the system. Non existent user.","id":"5712","mitre":{"id":["T1110"],"tactic":["Credential Access"],"technique":["Brute Force"]},"frequency":8,"firedtimes":1,"mail":true,"groups":["syslog","sshd","authentication_failures"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["SI.4","AU.14","AC.7"],"pci_dss":["11.4","10.2.4","10.2.5"],"tsc":["CC6.1","CC6.8","CC7.2","CC7.3"]},"agent":{"id":"000","name":"mail"},"manager":{"name":"mail"},"id":"1726422827.1936013","previous_output":"Sep 15 17:53:42 mail sshd[100734]: Invalid user user from 146.70.188.50 port 38336\nSep 15 17:53:38 mail sshd[100722]: Invalid user user from 146.70.188.50 port 47542\nSep 15 17:53:34 mail sshd[100720]: Invalid user user from 146.70.188.50 port 47532\nSep 15 17:53:30 mail sshd[100718]: Invalid user user from 146.70.188.50 port 38758\nSep 15 17:53:26 mail sshd[100716]: Invalid user user from 146.70.188.50 port 38746\nSep 15 17:53:22 mail sshd[100714]: Invalid user user from 146.70.188.50 port 38732\nSep 15 17:52:52 mail sshd[100711]: Invalid user user from 146.70.188.50 port 51720","full_log":"Sep 15 17:53:46 mail sshd[100736]: Invalid user user from 146.70.188.50 port 38352","predecoder":{"program_name":"sshd","timestamp":"Sep 15 17:53:46","hostname":"mail"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"146.70.188.50","srcport":"38352","srcuser":"user"},"location":"journald"},"program":"active-response/bin/host-deny"}}
2024/09/15 19:53:47 active-response/bin/host-deny: Ended
2024/09/15 19:53:47 active-response/bin/firewall-drop: Starting
2024/09/15 19:53:47 active-response/bin/firewall-drop: {"version":1,"origin":{"name":"node01","module":"wazuh-execd"},"command":"add","parameters":{"extra_args":[],"alert":{"timestamp":"2024-09-15T19:53:47.144+0200","rule":{"level":10,"description":"sshd: brute force trying to get access to the system. Non existent user.","id":"5712","mitre":{"id":["T1110"],"tactic":["Credential Access"],"technique":["Brute Force"]},"frequency":8,"firedtimes":1,"mail":true,"groups":["syslog","sshd","authentication_failures"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["SI.4","AU.14","AC.7"],"pci_dss":["11.4","10.2.4","10.2.5"],"tsc":["CC6.1","CC6.8","CC7.2","CC7.3"]},"agent":{"id":"000","name":"mail"},"manager":{"name":"mail"},"id":"1726422827.1936013","previous_output":"Sep 15 17:53:42 mail sshd[100734]: Invalid user user from 146.70.188.50 port 38336\nSep 15 17:53:38 mail sshd[100722]: Invalid user user from 146.70.188.50 port 47542\nSep 15 17:53:34 mail sshd[100720]: Invalid user user from 146.70.188.50 port 47532\nSep 15 17:53:30 mail sshd[100718]: Invalid user user from 146.70.188.50 port 38758\nSep 15 17:53:26 mail sshd[100716]: Invalid user user from 146.70.188.50 port 38746\nSep 15 17:53:22 mail sshd[100714]: Invalid user user from 146.70.188.50 port 38732\nSep 15 17:52:52 mail sshd[100711]: Invalid user user from 146.70.188.50 port 51720","full_log":"Sep 15 17:53:46 mail sshd[100736]: Invalid user user from 146.70.188.50 port 38352","predecoder":{"program_name":"sshd","timestamp":"Sep 15 17:53:46","hostname":"mail"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"146.70.188.50","srcport":"38352","srcuser":"user"},"location":"journald"},"program":"active-response/bin/firewall-drop"}}
2024/09/15 19:53:47 active-response/bin/firewall-drop: {"version":1,"origin":{"name":"firewall-drop","module":"active-response"},"command":"check_keys","parameters":{"keys":["146.70.188.50"]}}
2024/09/15 19:53:47 active-response/bin/firewall-drop: {"version":1,"origin":{"name":"node01","module":"wazuh-execd"},"command":"continue","parameters":{"extra_args":[],"alert":{"timestamp":"2024-09-15T19:53:47.144+0200","rule":{"level":10,"description":"sshd: brute force trying to get access to the system. Non existent user.","id":"5712","mitre":{"id":["T1110"],"tactic":["Credential Access"],"technique":["Brute Force"]},"frequency":8,"firedtimes":1,"mail":true,"groups":["syslog","sshd","authentication_failures"],"gdpr":["IV_35.7.d","IV_32.2"],"hipaa":["164.312.b"],"nist_800_53":["SI.4","AU.14","AC.7"],"pci_dss":["11.4","10.2.4","10.2.5"],"tsc":["CC6.1","CC6.8","CC7.2","CC7.3"]},"agent":{"id":"000","name":"mail"},"manager":{"name":"mail"},"id":"1726422827.1936013","previous_output":"Sep 15 17:53:42 mail sshd[100734]: Invalid user user from 146.70.188.50 port 38336\nSep 15 17:53:38 mail sshd[100722]: Invalid user user from 146.70.188.50 port 47542\nSep 15 17:53:34 mail sshd[100720]: Invalid user user from 146.70.188.50 port 47532\nSep 15 17:53:30 mail sshd[100718]: Invalid user user from 146.70.188.50 port 38758\nSep 15 17:53:26 mail sshd[100716]: Invalid user user from 146.70.188.50 port 38746\nSep 15 17:53:22 mail sshd[100714]: Invalid user user from 146.70.188.50 port 38732\nSep 15 17:52:52 mail sshd[100711]: Invalid user user from 146.70.188.50 port 51720","full_log":"Sep 15 17:53:46 mail sshd[100736]: Invalid user user from 146.70.188.50 port 38352","predecoder":{"program_name":"sshd","timestamp":"Sep 15 17:53:46","hostname":"mail"},"decoder":{"parent":"sshd","name":"sshd"},"data":{"srcip":"146.70.188.50","srcport":"38352","srcuser":"user"},"location":"journald"},"program":"active-response/bin/firewall-drop"}}
2024/09/15 19:53:47 active-response/bin/firewall-drop: Ended
- I can confirm that my IP address was added to the hosts.deny file and to my live iptables rules too
root@mail /var/ossec/logs # cat /etc/hosts.deny # /etc/hosts.deny: list of hosts that are _not_ allowed to access the system. # See the manual pages hosts_access(5) and hosts_options(5). # # Example: ALL: some.host.name, .some.domain # ALL EXCEPT in.fingerd: other.host.name, .other.domain # # If you're going to protect the portmapper use the name "rpcbind" for the # daemon name. See rpcbind(8) and rpc.mountd(8) for further information. # # The PARANOID wildcard matches any host whose name does not match its # address. # # You may wish to enable this to ensure any programs that don't # validate looked up hostnames still leave understandable logs. In past # versions of Debian this has been the default. # ALL: PARANOID ALL:146.70.188.50 root@mail /var/ossec/logs # root@mail /var/ossec/logs # iptables-save | grep -i DROP :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT DROP [0:0] -A INPUT -s 146.70.188.50/32 -j DROP -A INPUT -s 127.0.0.0/8 -d 127.0.0.0/8 -j DROP -A INPUT -j DROP -A FORWARD -s 146.70.188.50/32 -j DROP -A OUTPUT -j DROP root@mail /var/ossec/logs #
- wazuh is super-powerful, but I really only want it for [a] active-response temp-bans (shown above) and [b] email alerts
- email alerts aren't going to work because postfix isn't installed yet; let's use ansible to install & configure postfix now
- pre-test: mailutils is installed, but postfix is not
root@mail /var/ossec/logs # echo "this is just a test" | /usr/bin/mail -r noreply@opensourceecology.org -s "test from hetzner3" "michael@michaelaltfield.net" mail: Cannot open mailer: No such file or directory mail: cannot send message: No such file or directory root@mail /var/ossec/logs #
- now let's try to install postfix with ansible
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.postfix/tasks/main.yml, line
8, column 3, found a duplicate dict key (command). Using last defined value only.
PLAY [hetzner3] ************************************************************************
TASK [Gathering Facts] *****************************************************************
ok: [hetzner3]
TASK [maltfield.postfix : install postfix] *********************************************
changed: [hetzner3]
TASK [maltfield.postfix : generate dh parameters file] *********************************
[WARNING]: Consider using the file module with mode rather than running 'chmod'. If
you need to use command because file is insufficient you can add 'warn: false' to this
command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
fatal: [hetzner3]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": true, "cmd": ["chmod", "0400", "/etc/ssl/certs/dhparam.pem"], "delta": "0:00:00.003531", "end": "2024-09-15 20:03:32.776570", "msg": "non-zero return code", "rc": 1, "start": "2024-09-15 20:03:32.773039", "stderr": "chmod: cannot access '/etc/ssl/certs/dhparam.pem': No such file or directory", "stderr_lines": ["chmod: cannot access '/etc/ssl/certs/dhparam.pem': No such file or directory"], "stdout": "", "stdout_lines": []}
PLAY RECAP *****************************************************************************
hetzner3 : ok=2 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- hmm, it does appear to have been started
root@mail ~ # systemctl status postfix
● postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/lib/systemd/system/postfix.service; enabled; preset: enabled)
Active: active (exited) since Sun 2024-09-15 20:03:31 CEST; 2min 31s ago
Docs: man:postfix(1)
Process: 103774 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 103774 (code=exited, status=0/SUCCESS)
CPU: 1ms
Sep 15 20:03:31 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent...
Sep 15 20:03:31 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
root@mail ~ #
- but, yeah, it looks like it didn't create the dhparams file, which would explain why chmod says "no such file or directory"
root@mail ~ # ls -lah /etc/ssl/certs/ | grep -i dh root@mail ~ #
- here's the role tasks
user@ose:~/sandbox_local/ansible/hetzner3$ head -n15 roles/maltfield.postfix/tasks/main.yml
---
- name: install postfix
apt:
update_cache: yes
pkg:
- postfix
- name: generate dh parameters file
command: openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096
command: chown root:root /etc/ssl/certs/dhparam.pem
command: chmod 0400 /etc/ssl/certs/dhparam.pem
args:
creates: /etc/ssl/certs/dhparam.pem
- name: Whitelisted Domains
user@ose:~/sandbox_local/ansible/hetzner3$
- I executed these manually; well, it worked fine. not sure why ansible failed
root@mail ~ # time openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096 Generating DH parameters, 4096 bit long safe prime .............................................................+........................................................................................................................................................+....+...............................................................................................................................................................................................................................................+........................................................................................................................................................................................................................................................................................+..................................................................................................................................................................................................+...........................................................................................................................................................................+.................................................................................................................................................................+...................................................................................+...................................................+....................................................................................................................................................................................................................................................++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++*++* real 0m24.935s user 0m24.919s sys 0m0.016s root@mail ~ # root@mail ~ # chown root:root /etc/ssl/certs/dhparam.pem root@mail ~ # root@mail ~ # chmod 0400 /etc/ssl/certs/dhparam.pem root@mail ~ # root@mail ~ # ls -lah /etc/ssl/certs/ | grep -i dh -r-------- 1 root root 769 Sep 15 20:12 dhparam.pem root@mail ~ #
- i did a double-tap with ansible; it failed again when restarting postfix
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.postfix/tasks/main.yml, line
8, column 3, found a duplicate dict key (command). Using last defined value only.
PLAY [hetzner3] ************************************************************************
TASK [Gathering Facts] *****************************************************************
ok: [hetzner3]
TASK [maltfield.postfix : install postfix] *********************************************
ok: [hetzner3]
TASK [maltfield.postfix : generate dh parameters file] *********************************
[WARNING]: Consider using the file module with mode rather than running 'chmod'. If
you need to use command because file is insufficient you can add 'warn: false' to this
command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
ok: [hetzner3]
TASK [maltfield.postfix : Whitelisted Domains] *****************************************
changed: [hetzner3]
TASK [maltfield.postfix : Postfox main.cf] *********************************************
changed: [hetzner3]
TASK [maltfield.postfix : Postfox master.cf] *******************************************
changed: [hetzner3]
TASK [install basic essential packages] ************************************************
ok: [hetzner3]
RUNNING HANDLER [maltfield.postfix : restart postfix] **********************************
ERROR! [mux 57866] 13:13:28.475680 E mitogen.[ssh.144.76.164.201:32415.sudo.root]: raw pickle was: b'\x80\x02(X&\x00\x00\x00ose-57937-746bb1080740-215639e850q\x00X\x16\x00\x00\x00ansible_mitogen.targetq\x01NX\n\x00\x00\x00run_moduleq\x02)cmitogen.core\nKwargs\nq\x03}q\x04X\x06\x00\x00\x00kwargsq\x05}q\x06(X\x0b\x00\x00\x00runner_nameq\x07X\x0e\x00\x00\x00NewStyleRunnerq\x08X\x06\x00\x00\x00moduleq\tcansible.utils.unsafe_proxy\nAnsibleUnsafeText\nq\nX\x16\x00\x00\x00ansible.legacy.systemdq\x0b\x85q\x0c\x81q\rX\x04\x00\x00\x00pathq\x0eX9\x00\x00\x00/usr/lib/python3/dist-packages/ansible/modules/systemd.pyq\x0fX\t\x00\x00\x00json_argsq\x10XJ\x02\x00\x00{"name": "postfix", "state": "restarted", "_ansible_check_mode": false, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 0, "_ansible_version": "2.10.17", "_ansible_module_name": "ansible.legacy.systemd", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}q\x11X\x03\x00\x00\x00envq\x12}q\x13X\x14\x00\x00\x00interpreter_fragmentq\x14NX\t\x00\x00\x00is_pythonq\x15NX\n\x00\x00\x00module_mapq\x16}q\x17(X\x07\x00\x00\x00builtinq\x18]q\x19(X\x1a\x00\x00\x00ansible.module_utils._textq\x1aX\x1a\x00\x00\x00ansible.module_utils.basicq\x1bX\x1b\x00\x00\x00ansible.module_utils.commonq\x1cX/\x00\x00\x00ansible.module_utils.common._collections_compatq\x1dX(\x00\x00\x00ansible.module_utils.common._json_compatq\x1eX"\x00\x00\x00ansible.module_utils.common._utilsq\x1fX\'\x00\x00\x00ansible.module_utils.common.collectionsq X \x00\x00\x00ansible.module_utils.common.fileq!X&\x00\x00\x00ansible.module_utils.common.parametersq"X#\x00\x00\x00ansible.module_utils.common.processq#X$\x00\x00\x00ansible.module_utils.common.sys_infoq$X \x00\x00\x00ansible.module_utils.common.textq%X+\x00\x00\x00ansible.module_utils.common.text.convertersq&X+\x00\x00\x00ansible.module_utils.common.text.formattersq\'X&\x00\x00\x00ansible.module_utils.common.validationq(X$\x00\x00\x00ansible.module_utils.common.warningsq)X\x1b\x00\x00\x00ansible.module_utils.compatq*X\'\x00\x00\x00ansible.module_utils.compat._selectors2q+X%\x00\x00\x00ansible.module_utils.compat.selectorsq,X\x1b\x00\x00\x00ansible.module_utils.distroq-X#\x00\x00\x00ansible.module_utils.distro._distroq.X\x1a\x00\x00\x00ansible.module_utils.factsq/X,\x00\x00\x00ansible.module_utils.facts.ansible_collectorq0X$\x00\x00\x00ansible.module_utils.facts.collectorq1X!\x00\x00\x00ansible.module_utils.facts.compatq2X-\x00\x00\x00ansible.module_utils.facts.default_collectorsq3X#\x00\x00\x00ansible.module_utils.facts.hardwareq4X\'\x00\x00\x00ansible.module_utils.facts.hardware.aixq5X(\x00\x00\x00ansible.module_utils.facts.hardware.baseq6X*\x00\x00\x00ansible.module_utils.facts.hardware.darwinq7X-\x00\x00\x00ansible.module_utils.facts.hardware.dragonflyq8X+\x00\x00\x00ansible.module_utils.facts.hardware.freebsdq9X(\x00\x00\x00ansible.module_utils.facts.hardware.hpuxq:X(\x00\x00\x00ansible.module_utils.facts.hardware.hurdq;X)\x00\x00\x00ansible.module_utils.facts.hardware.linuxq<X*\x00\x00\x00ansible.module_utils.facts.hardware.netbsdq=X+\x00\x00\x00ansible.module_utils.facts.hardware.openbsdq>X)\x00\x00\x00ansible.module_utils.facts.hardware.sunosq?X$\x00\x00\x00ansible.module_utils.facts.namespaceq@X"\x00\x00\x00ansible.module_utils.facts.networkqAX&\x00\x00\x00ansible.module_utils.facts.network.aixqBX\'\x00\x00\x00ansible.module_utils.facts.network.baseqCX)\x00\x00\x00ansible.module_utils.facts.network.darwinqDX,\x00\x00\x00ansible.module_utils.facts.network.dragonflyqEX)\x00\x00\x00ansible.module_utils.facts.network.fc_wwnqFX*\x00\x00\x00ansible.module_utils.facts.network.freebsdqGX.\x00\x00\x00ansible.module_utils.facts.network.generic_bsdqHX\'\x00\x00\x00ansible.module_utils.facts.network.hpuxqIX\'\x00\x00\x00ansible.module_utils.facts.network.hurdqJX(\x00\x00\x00ansible.module_utils.facts.network.iscsiqKX(\x00\x00\x00ansible.module_utils.facts.network.linuxqLX)\x00\x00\x00ansible.module_utils.facts.network.netbsdqMX\'\x00\x00\x00ansible.module_utils.facts.network.nvmeqNX*\x00\x00\x00ansible.module_utils.facts.network.openbsdqOX(\x00\x00\x00ansible.module_utils.facts.network.sunosqPX \x00\x00\x00ansible.module_utils.facts.otherqQX\'\x00\x00\x00ansible.module_utils.facts.other.facterqRX%\x00\x00\x00ansible.module_utils.facts.other.ohaiqSX!\x00\x00\x00ansible.module_utils.facts.sysctlqTX!\x00\x00\x00ansible.module_utils.facts.systemqUX*\x00\x00\x00ansible.module_utils.facts.system.apparmorqVX&\x00\x00\x00ansible.module_utils.facts.system.capsqWX(\x00\x00\x00ansible.module_utils.facts.system.chrootqXX)\x00\x00\x00ansible.module_utils.facts.system.cmdlineqYX+\x00\x00\x00ansible.module_utils.facts.system.date_timeqZX.\x00\x00\x00ansible.module_utils.facts.system.distributionq[X%\x00\x00\x00ansible.module_utils.facts.system.dnsq\\X%\x00\x00\x00ansible.module_utils.facts.system.envq]X&\x00\x00\x00ansible.module_utils.facts.system.fipsq^X\'\x00\x00\x00ansible.module_utils.facts.system.localq_X%\x00\x00\x00ansible.module_utils.facts.system.lsbq`X)\x00\x00\x00ansible.module_utils.facts.system.pkg_mgrqaX*\x00\x00\x00ansible.module_utils.facts.system.platformqbX(\x00\x00\x00ansible.module_utils.facts.system.pythonqcX)\x00\x00\x00ansible.module_utils.facts.system.selinuxqdX-\x00\x00\x00ansible.module_utils.facts.system.service_mgrqeX.\x00\x00\x00ansible.module_utils.facts.system.ssh_pub_keysqfX&\x00\x00\x00ansible.module_utils.facts.system.userqgX"\x00\x00\x00ansible.module_utils.facts.timeoutqhX \x00\x00\x00ansible.module_utils.facts.utilsqiX"\x00\x00\x00ansible.module_utils.facts.virtualqjX\'\x00\x00\x00ansible.module_utils.facts.virtual.baseqkX,\x00\x00\x00ansible.module_utils.facts.virtual.dragonflyqlX*\x00\x00\x00ansible.module_utils.facts.virtual.freebsdqmX\'\x00\x00\x00ansible.module_utils.facts.virtual.hpuxqnX(\x00\x00\x00ansible.module_utils.facts.virtual.linuxqoX)\x00\x00\x00ansible.module_utils.facts.virtual.netbsdqpX*\x00\x00\x00ansible.module_utils.facts.virtual.openbsdqqX(\x00\x00\x00ansible.module_utils.facts.virtual.sunosqrX)\x00\x00\x00ansible.module_utils.facts.virtual.sysctlqsX\x1c\x00\x00\x00ansible.module_utils.parsingqtX)\x00\x00\x00ansible.module_utils.parsing.convert_boolquX\x1f\x00\x00\x00ansible.module_utils.pycompat24qvX\x1c\x00\x00\x00ansible.module_utils.serviceqwX\x18\x00\x00\x00ansible.module_utils.sixqxeX\x06\x00\x00\x00customqy]qzuX\x0e\x00\x00\x00py_module_nameq{X\x17\x00\x00\x00ansible.modules.systemdq|X\r\x00\x00\x00good_temp_dirq}X\x12\x00\x00\x00/root/.ansible/tmpq~X\x03\x00\x00\x00cwdq\x7fNX\t\x00\x00\x00extra_envq\x80NX\x0b\x00\x00\x00emulate_ttyq\x81\x88X\x0f\x00\x00\x00service_contextq\x82cmitogen.core\n_unpickle_context\nq\x83K\x00N\x86q\x84Rq\x85us\x85q\x86Rq\x87tq\x88.'
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: File "<stdin>", line 853, in _find_global
fatal: [hetzner3]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
RUNNING HANDLER [maltfield.postfix : rebuild whitelisted domains] **********************
NO MORE HOSTS LEFT *********************************************************************
PLAY RECAP *****************************************************************************
hetzner3 : ok=7 changed=3 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- I tried manual restart & status; it looks fine to me on the server-side..
root@mail ~ # systemctl restart postfix
root@mail ~ #
root@mail ~ # systemctl status postfix
● postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/lib/systemd/system/postfix.service; enabled; preset: enabled)
Active: active (exited) since Sun 2024-09-15 20:15:08 CEST; 5s ago
Docs: man:postfix(1)
Process: 105274 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 105274 (code=exited, status=0/SUCCESS)
CPU: 1ms
Sep 15 20:15:08 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent...
Sep 15 20:15:08 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
root@mail ~ #
root@mail ~ # journalctl -u postfix
Sep 15 20:03:31 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent...
Sep 15 20:03:31 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
Sep 15 20:15:06 mail systemd[1]: postfix.service: Deactivated successfully.
Sep 15 20:15:06 mail systemd[1]: Stopped postfix.service - Postfix Mail Transport Agent.
Sep 15 20:15:06 mail systemd[1]: Stopping postfix.service - Postfix Mail Transport Agent...
Sep 15 20:15:08 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent...
Sep 15 20:15:08 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
root@mail ~ #
- uhh, third times a charm? on running again, it didn't throw an error
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml [WARNING]: While constructing a mapping from /home/user/sandbox_local/ansible/hetzner3/roles/maltfield.postfix/tasks/main.yml, line 8, column 3, found a duplicate dict key (command). Using last defined value only. PLAY [hetzner3] ************************************************************************ TASK [Gathering Facts] ***************************************************************** ok: [hetzner3] TASK [maltfield.postfix : install postfix] ********************************************* ok: [hetzner3] TASK [maltfield.postfix : generate dh parameters file] ********************************* [WARNING]: Consider using the file module with mode rather than running 'chmod'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. ok: [hetzner3] TASK [maltfield.postfix : Whitelisted Domains] ***************************************** ok: [hetzner3] TASK [maltfield.postfix : Postfox main.cf] ********************************************* ok: [hetzner3] TASK [maltfield.postfix : Postfox master.cf] ******************************************* ok: [hetzner3] TASK [install basic essential packages] ************************************************ ok: [hetzner3] PLAY RECAP ***************************************************************************** hetzner3 : ok=7 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 user@ose:~/sandbox_local/ansible/hetzner3$
- unfortunately, I'm still not able to email myself
root@mail ~ # echo "this is just a test" | /usr/bin/mail -r noreply@opensourceecology.org -s "test from hetzner3" "michael@michaelaltfield.net" mail: cannot send message: Process exited with a non-zero status root@mail ~ #
- curiously, there's no mail.log, and I don't see any errors from postfix
root@mail /var/log # ls alternatives.log alternatives.log.1 apt audit btmp btmp.1 dpkg.log dpkg.log.1 faillog journal lastlog private README runit unattended-upgrades wtmp root@mail /var/log # root@mail /var/log # journalctl -u postfix -f Sep 15 20:03:31 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent... Sep 15 20:03:31 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent. Sep 15 20:15:06 mail systemd[1]: postfix.service: Deactivated successfully. Sep 15 20:15:06 mail systemd[1]: Stopped postfix.service - Postfix Mail Transport Agent. Sep 15 20:15:06 mail systemd[1]: Stopping postfix.service - Postfix Mail Transport Agent... Sep 15 20:15:08 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent... Sep 15 20:15:08 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
- ah, if I just don't specify the unit in `journalctl`, I see the error
root@mail /var/log # journalctl -f Sep 15 20:23:13 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.2 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=16909 DF PROTO=UDP SPT=47889 DPT=53 LEN=41 Sep 15 20:23:13 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=81 TC=0 HOPLIMIT=64 FLOWLBL=102047 PROTO=UDP SPT=37100 DPT=53 LEN=41 Sep 15 20:23:13 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.1 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=49769 DF PROTO=UDP SPT=56946 DPT=53 LEN=41 Sep 15 20:23:13 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=81 TC=0 HOPLIMIT=64 FLOWLBL=838135 PROTO=UDP SPT=56214 DPT=53 LEN=41 Sep 15 20:23:13 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=81 TC=0 HOPLIMIT=64 FLOWLBL=606878 PROTO=UDP SPT=48253 DPT=53 LEN=41 Sep 15 20:23:43 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.2 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=56850 DF PROTO=UDP SPT=40619 DPT=53 LEN=42 Sep 15 20:23:43 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=82 TC=0 HOPLIMIT=64 FLOWLBL=397680 PROTO=UDP SPT=36931 DPT=53 LEN=42 Sep 15 20:23:43 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.1 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=17522 DF PROTO=UDP SPT=57806 DPT=53 LEN=42 Sep 15 20:23:43 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.2 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=31763 DF PROTO=UDP SPT=57488 DPT=53 LEN=42 Sep 15 20:23:43 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=82 TC=0 HOPLIMIT=64 FLOWLBL=178850 PROTO=UDP SPT=53628 DPT=53 LEN=42 Sep 15 20:24:09 mail postfix/sendmail[106640]: fatal: parameter inet_interfaces: no local interface found for OBFUSCATED Sep 15 20:24:14 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.2 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=24249 DF PROTO=UDP SPT=37918 DPT=53 LEN=41 Sep 15 20:24:14 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=81 TC=0 HOPLIMIT=64 FLOWLBL=477425 PROTO=UDP SPT=47559 DPT=53 LEN=41 Sep 15 20:24:14 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=144.76.164.201 DST=185.12.64.1 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=26314 DF PROTO=UDP SPT=42086 DPT=53 LEN=41 Sep 15 20:24:14 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=81 TC=0 HOPLIMIT=64 FLOWLBL=604691 PROTO=UDP SPT=43822 DPT=53 LEN=41 Sep 15 20:24:14 mail kernel: iptablesIN= OUT=enp0s31f6 SRC=2a01:04f8:0200:40d7:0000:0000:0000:0002 DST=2a01:04ff:ff00:0000:0000:0000:0add:0001 LEN=81 TC=0 HOPLIMIT=64 FLOWLBL=67841 PROTO=UDP SPT=43091 DPT=53 LEN=41 ^C root@mail /var/log #
- Oh, looks like the issue was I had hard-coded an IP address into the config
- I converted these three postfix ansible role's "files" to "templates" and update the bind IP to Template:Ansible default ipv4.address and Template:Ansible default ipv6.address
- I pushed-out the configs with ansible (note I have to do a double-tap to bypass the errors on the first-go for some reason still)
- and now I don't get an error from this
root@mail ~ # echo "this is just a test" | /usr/bin/mail -r noreply@opensourceecology.org -s "test from hetzner3" "michael@michaelaltfield.net" root@mail ~ #
- unfortunately the mail doesn't come, but I'm thinking it's because I haven't updated the firewall rules to allow the postfix user to send messages
- I updated the firewall rules in 'provision.yml' and re-ran ansible
- ...but I'm having the same issue
- actually, I don't even see that postfix is running
root@mail ~ # ps -ef | grep -iE 'postfix' maltfie+ 103848 1 0 20:05 ? 00:00:00 SCREEN -S postfix maltfie+ 105249 105240 0 20:14 pts/0 00:00:00 screen -dr postfix root 113685 103864 0 20:58 pts/16 00:00:00 grep -iE postfix root@mail ~ #
- ok, this says "active" but "exited"
root@mail ~ # systemctl status postfix
● postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/lib/systemd/system/postfix.service; enabled; preset: enabled)
Active: active (exited) since Sun 2024-09-15 20:15:08 CEST; 41min ago
Docs: man:postfix(1)
Process: 105274 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
Main PID: 105274 (code=exited, status=0/SUCCESS)
CPU: 1ms
Sep 15 20:15:08 mail systemd[1]: Starting postfix.service - Postfix Mail Transport Agent...
Sep 15 20:15:08 mail systemd[1]: Finished postfix.service - Postfix Mail Transport Agent.
root@mail ~ #
- I tried to start it, and I got some errors to appear in journal
root@mail ~ # systemctl start postfix root@mail ~ #
- here's the other terminal with the errors
Sep 15 20:56:08 mail systemd[1]: Starting postfix@-.service - Postfix Mail Transport Agent (instance -)... Sep 15 20:56:08 mail postfix[113442]: Postfix is using backwards-compatible default settings Sep 15 20:56:08 mail postfix[113442]: See http://www.postfix.org/COMPATIBILITY_README.html for details Sep 15 20:56:08 mail postfix[113442]: To disable backwards compatibility use "postconf compatibility_level=3.6" and "postfix reload" Sep 15 20:56:08 mail postfix/postfix-script[113672]: starting the Postfix mail system Sep 15 20:56:08 mail postfix/master[113674]: warning: duplicate master.cf entry for service "trace" (private/trace) -- using the last entry Sep 15 20:56:08 mail postfix/master[113674]: daemon started -- version 3.7.11, configuration /etc/postfix Sep 15 20:56:08 mail systemd[1]: Started postfix@-.service - Postfix Mail Transport Agent (instance -). Sep 15 20:56:08 mail audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=postfix@- comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Sep 15 20:56:08 mail postfix/cleanup[113677]: error: open database /etc/postfix/virtual.db: No such file or directory Sep 15 20:56:08 mail postfix/pickup[113675]: 7CEA9B87FAF: uid=0 from=<noreply@opensourceecology.org> Sep 15 20:56:08 mail postfix/cleanup[113677]: 7CEA9B87FAF: message-id=<20240915185608.7CEA9B87FAF@mail.opensourceecology.org> Sep 15 20:56:08 mail postfix/cleanup[113677]: warning: hash:/etc/postfix/virtual is unavailable. open database /etc/postfix/virtual.db: No such file or directory Sep 15 20:56:08 mail postfix/cleanup[113677]: warning: hash:/etc/postfix/virtual lookup error for "michael@michaelaltfield.net" Sep 15 20:56:08 mail postfix/cleanup[113677]: warning: 7CEA9B87FAF: virtual_alias_maps map lookup problem for michael@michaelaltfield.net -- message not accepted, try again later Sep 15 20:56:08 mail postfix/pickup[113675]: 7E07BB87FAF: uid=0 from=<noreply@opensourceecology.org> ...
- so it seems pretty unhappy that we don't actually have our virtual.db defined yet.
- I actually decided that this is something that I don't want to have managed by ansible, because postfix lookup tables are actually stateful databases; they should be migrated like mariadb databases
- we do have a virtual file on hetzner2, but actually it's just all comments
[maltfield@opensourceecology ~]$ wc -l /etc/postfix/virtual 299 /etc/postfix/virtual [maltfield@opensourceecology ~]$ [maltfield@opensourceecology ~]$ head /etc/postfix/virtual # VIRTUAL(5) VIRTUAL(5) # # NAME # virtual - Postfix virtual alias table format # # SYNOPSIS # postmap /etc/postfix/virtual # # postmap -q "string" /etc/postfix/virtual # [maltfield@opensourceecology ~]$ [maltfield@opensourceecology ~]$ tail /etc/postfix/virtual # The Secure Mailer license must be distributed with this # software. # # AUTHOR(S) # Wietse Venema # IBM T.J. Watson Research # P.O. Box 704 # Yorktown Heights, NY 10598, USA # # VIRTUAL(5) [maltfield@opensourceecology ~]$ [maltfield@opensourceecology ~]$ grep -iE "^[^#]" /etc/postfix/virtual [maltfield@opensourceecology ~]$
- anyway, just to keep the comments, I copied this over to the new machine
- note that I'm using `-e 'ssh -p 31415'` to change the default port; otherwise it "just works"
[maltfield@opensourceecology ~]$ rsync -e 'ssh -p 32415' -av --progress /etc/postfix/virtual 144.76.164.201:
The authenticity of host '[144.76.164.201]:32415 ([144.76.164.201]:32415)' can't be established.
ECDSA key fingerprint is SHA256:hxzCaH018Wn8tEn7sZ22P/WrNMmZ29w5tK/w0KHq8eE.
ECDSA key fingerprint is MD5:dd:27:98:46:b8:be:af:e1:b5:7d:54:d6:15:72:38:53.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[144.76.164.201]:32415' (ECDSA) to the list of known hosts.
sending incremental file list
virtual
12,696 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/1)
sent 12,783 bytes received 35 bytes 5,127.20 bytes/sec
total size is 12,696 speedup is 0.99
[maltfield@opensourceecology ~]$
- back on hetzner3, I moved the file into place; note I had to change the owner
root@mail /etc/postfix # mv /home/maltfield/virtual /etc/postfix/ root@mail /etc/postfix # root@mail /etc/postfix # postmap /etc/postfix/virtual postmap: fatal: open database /etc/postfix/virtual.db: Permission denied root@mail /etc/postfix # root@mail /etc/postfix # ls -lah /etc/postfix/virtual -rw-r--r-- 1 maltfield maltfield 13K Apr 1 2020 /etc/postfix/virtual root@mail /etc/postfix # root@mail /etc/postfix # chown root:root /etc/postfix/virtual root@mail /etc/postfix # root@mail /etc/postfix # postmap /etc/postfix/virtual root@mail /etc/postfix # root@mail /etc/postfix # ls virtual* virtual virtual.db root@mail /etc/postfix #
- I tried to mail again; this time I got info messages suggesting it worked. And on my server, I see connections from the hetzner3 server. yay!
root@mail /etc/postfix # echo "this is just a test" | /usr/bin/mail --debug-level=remote -r noreply@opensourceecology.org -s "test from hetzner3" "michael@michaelaltfield.net" root@mail /etc/postfix #
- lol but I guess I've been spamming my personal server, so the hetzner3 logs are depressed that I'm not talking back to it
Sep 15 21:12:57 mail postfix/smtp[113810]: 3E06BB87E2F: host mail.michaelaltfield.net[2a01:4f8:c0c:95be::1] refused to talk to me: 421 4.7.0 mail.michaelaltfield.net Error: too many connections from 2a01:4f8:200:40d7::2
- but after some time, I do get a 250
Sep 15 21:12:57 mail postfix/smtp[113809]: 3B40CB87FAF: to=<michael@michaelaltfield.net>, relay=mail.michaelaltfield.net[2a01:4f8:c0c:95be::1]:25, delay=1691, delays=1689/0.66/1/0.03, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as BC4D8108)
- it looks like my server sent the auto-reply (michael@michaelaltfield.net is just a dumb fake email that auto-replies telling the user to go to email.michaelaltfield.net and solve a captcha to get my real/current email address), but my server is sending it to noreply@opensourceecology.org, which then goes to google's servers, per our MX records
2024-09-15T19:12:44.087871+00:00 mail postfix/smtp[1076271]: DEB7C131: to=<noreply@opensourceecology.org>, relay=aspmx.l.google.com[2a00:1450:400c:c07::1b]:25, delay=0.18, delays=0.02/0/0.13/0.04, dsn=4.3.0, status=deferred (bounce or trace service failure)
- I tried sending to my OSE gmail email address, but I got a bounce. Unfortunately, I can't see the exact error on Google's servers, but this is expected since we never auth'd this new server to send emails on behalf of opensourceecology.org
- it looks like we have ghandi for oswarehouse.org & opensourcewarehouse.org
- our other domains' nameservers are hosted on cloudflare
- I wasn't able to get past the cloudflare captcha in my normal web browser, but I was able to login on Tor Browser. Glad to see cloudflare isn't totally evil.
- Here's our current DNS SPF record
Type = TXT name = opensourceecology.org TTL = 2 min Content = v=spf1 a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ~all
- And a query for good measure
user@disp3433:~$ user@disp3433:~$ dig -t TXT opensourceecology.org ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t TXT opensourceecology.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18846 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;opensourceecology.org. IN TXT ;; ANSWER SECTION: opensourceecology.org. 120 IN TXT "v=spf1 a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ~all" ;; Query time: 188 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 14:39:06 -05 2024 ;; MSG SIZE rcvd: 138 user@disp3433:~$
- curiously, the IPv6 addresses are not listed. And I don't see any entries in cloudflare for openbuildinginstiture.org or other domains; I'll have to investigate that later..
- oh shit, I just realized -- we actually *don't* use our server for receiving emails; we only use it for sending emails (phplist).
- all OSE infrastructure uses GSuite (we're grandfathered-in for free) for "gmail" UI on @opensourceecology.org email addresses
- so, yeah, I updated the postfix config to only listen on localhost; it shouldn't bind to our ipv4 or ipv6 addresses
- and I updated the firewall to not allow incoming traffic on port 25
- And I updated the SPF record in CloudFlare
spf1=v a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ip4:144.76.164.201 ip6:2a01:4f8:200:40d7::2 ~all
- I was expecting it to take a few minutes, but it propagated immediately!
user@disp3433:~$ while true; do date; dig -t TXT opensourceecology.org; sleep 300; echo; done Sun Sep 15 02:53:58 PM -05 2024 ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t TXT opensourceecology.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8306 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;opensourceecology.org. IN TXT ;; ANSWER SECTION: opensourceecology.org. 120 IN TXT "spf1=v a mx include:_spf.google.com ip4:78.46.3.178 ip4:138.201.84.223 ip4:144.76.164.201 ip6:2a01:4f8:200:40d7::2 ~all" ;; Query time: 191 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 14:53:58 -05 2024 ;; MSG SIZE rcvd: 182
- I tried emailing myself again, but google still bounces it; maybe it'll take some time for them
- I went to update the RDNS in hetzner WUI
- I was surprised to see that the RDNS entry differs from our hostname
- old server hostname is just 'opensourceecology.org' (I thought it was 'mail.opensourceecology.org')
[root@opensourceecology ~]# hostname opensourceecology.org [root@opensourceecology ~]#
- and RDNS is 'mailer.opensourceecology.org' (I thought it was 'mail.opensourceecology.org')
user@disp3433:~$ dig -x 138.201.84.243 ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -x 138.201.84.243 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49686 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;243.84.201.138.in-addr.arpa. IN PTR ;; ANSWER SECTION: 243.84.201.138.in-addr.arpa. 86389 IN PTR mailer.opensourceecology.org. ;; Query time: 186 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 14:58:47 -05 2024 ;; MSG SIZE rcvd: 98 user@disp3433:~$
- ah, ok, apparently 'mail.opensourceecology.org' is already google's servers and so I made 'mailer.opensourceecology.org' hetzner2 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2019_Q1#Tue_Mar_19.2C_2019
- I guess if it doesn't really matter, I'm going to name everything 'hetzner3.opensourceecology.org' -- we'll see if it works
- I created two new DNS entries for 'hetzner3.opensourceecology.org'
A - 144.76.164.201 AAAA - 2a01:4f8:200:40d7::2
- that also was immediate; I remember now why we use cloudflare for DNS!
user@disp3433:~$ dig hetzner3.opensourceecology.org ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> hetzner3.opensourceecology.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55968 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;hetzner3.opensourceecology.org. IN A ;; ANSWER SECTION: hetzner3.opensourceecology.org. 120 IN A 144.76.164.201 ;; Query time: 189 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 15:05:43 -05 2024 ;; MSG SIZE rcvd: 75 user@disp3433:~$ dig -t AAAA hetzner3.opensourceecology.org ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t AAAA hetzner3.opensourceecology.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52394 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;hetzner3.opensourceecology.org. IN AAAA ;; ANSWER SECTION: hetzner3.opensourceecology.org. 120 IN AAAA 2a01:4f8:200:40d7::2 ;; Query time: 189 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 15:05:48 -05 2024 ;; MSG SIZE rcvd: 87 user@disp3433:~$
- ok, I used the hetzner wui to update RDNS too; it only took a few seconds to update
user@disp3433:~$ dig -x 144.76.164.201 ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -x 144.76.164.201 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17479 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;201.164.76.144.in-addr.arpa. IN PTR ;; ANSWER SECTION: 201.164.76.144.in-addr.arpa. 86400 IN PTR hetzner3.opensourceecology.org. ;; Query time: 723 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 15:07:35 -05 2024 ;; MSG SIZE rcvd: 100 user@disp3433:~$
- emails still getting bounced by Google Mail servers
- I also setup RDNS on IPv6
user@disp3433:~$ dig -x 2a01:4f8:200:40d7::2 ;; communications error to 10.139.1.1#53: timed out ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -x 2a01:4f8:200:40d7::2 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29353 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.d.0.4.0.0.2.0.8.f.4.0.1.0.a.2.ip6.arpa. IN PTR ;; ANSWER SECTION: 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.7.d.0.4.0.0.2.0.8.f.4.0.1.0.a.2.ip6.arpa. 86397 IN PTR hetzner3.opensourceecology.org. ;; Query time: 185 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 15:10:38 -05 2024 ;; MSG SIZE rcvd: 145 user@disp3433:~$
- I see that I did, in fact, update the hostname of the postfix config to be 'mailer.opensourceecology.org'
[root@opensourceecology ~]# grep -ir 'mailer.opensourceecology.org' /etc Binary file /etc/aliases.db matches /etc/postfix/main.cf:myhostname = mailer.opensourceecology.org [root@opensourceecology ~]#
- so I updated the hostname of main.cf on hetzner3 to be 'hetzner3.opensourceecology.org'
- I'm going to take lunch and see if it works after..
...
- after lunch I tried emailing michael@michaelaltfield.net again, and I got an error on hetzner3 complaining about my MX records?
Sep 16 00:06:19 mail postfix/smtp[117168]: 1EAA7B8809F: to=<michael@michaelaltfield.net>, relay=none, delay=10, delays=0.04/0.02/10/0, dsn=4.4.3, status=deferred (Host or domain name not found. Name service error for name=michaelaltfield.net type=MX: Host not found, try again)
- my server's DNS is fine
user@disp3433:~$ dig -t MX michaelaltfield.net ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t MX michaelaltfield.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50590 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;michaelaltfield.net. IN MX ;; ANSWER SECTION: michaelaltfield.net. 47 IN MX 10 mail.michaelaltfield.net. ;; Query time: 185 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 17:08:13 -05 2024 ;; MSG SIZE rcvd: 69 user@disp3433:~$ dig mail.michaelaltfield.net. ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> mail.michaelaltfield.net. ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11898 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;mail.michaelaltfield.net. IN A ;; ANSWER SECTION: mail.michaelaltfield.net. 1774 IN A 94.130.74.14 ;; Query time: 197 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) (UDP) ;; WHEN: Sun Sep 15 17:08:25 -05 2024 ;; MSG SIZE rcvd: 69 user@disp3433:~$
- oh shit, actually, looks like the server can't find it tho
root@mail /etc/postfix # sudo -u postfix dig -t MX michaelaltfield.net ; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -t MX michaelaltfield.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20574 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;michaelaltfield.net. IN MX ;; Query time: 4 msec ;; SERVER: ::1#53(::1) (UDP) ;; WHEN: Mon Sep 16 00:09:37 CEST 2024 ;; MSG SIZE rcvd: 48 root@mail /etc/postfix #
- ah, the above output says it's querying port 53 on localhost (ipv6). yeah, looks like ansible setup our DoT solution (stubby + unbound), but I guess it's broken. It's always DNS ;P
root@mail /etc/postfix # dpkg -l | grep -i stubby ii stubby 1.6.0-3+b1 amd64 modern asynchronous DNS API (stub resolver) root@mail /etc/postfix # dpkg -l | grep -i unbound ii libunbound8:amd64 1.17.1-2+deb12u2 amd64 library implementing DNS resolution and validation ii unbound 1.17.1-2+deb12u2 amd64 validating, recursive, caching DNS resolver root@mail /etc/postfix #
- unfortunately, re-running ansible to configure the 'maltifeld.dns' role gets stuck
- SE says I should figure out what process it's waiting-on and execute it myself; probably it's apt waiting for user input https://serverfault.com/questions/1020302/ansible-apt-module-hangs-process-sleeping
- I think mitogen (a performance "strategy" we use for optimizing ansible speed) is obfuscating what we're running
root@mail /etc/postfix # ps -ef | grep -i python
root 92168 1 0 Sep15 ? 00:00:16 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
wazuh 97237 1 0 Sep15 ? 00:00:34 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py
wazuh 97338 97237 0 Sep15 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py
wazuh 97341 97237 0 Sep15 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py
wazuh 97344 97237 0 Sep15 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py
maltfie+ 117202 117201 0 00:12 ? 00:00:00 /usr/bin/python3(mitogen:user@ose:63374)
maltfie+ 117205 117202 0 00:12 ? 00:00:00 /usr/bin/python3(mitogen:user@ose:63374)
root 117207 117202 0 00:12 pts/18 00:00:00 sudo -u root -H -- /usr/bin/python3 -c import codecs,os,sys;_=codecs.decode;exec(_(_("eNqFkcFPwyAUxs/jr+AGZGSDGtPY2ESzg/FgTBrjDtti6EqVSIHQbnX+9bJWXTsP3vjxPt73PV5Gl6mtZ045iQnwtB2QKmGA0vp3TBJwPBc7F2FGOWPkxBkdkg9V3vNW21ribAh+CMshtAQAAINlfQgBtGiCbwXTFKJC+FYZBIUpuqL8kNtdI3Itu/J8V/t5rszcHZo32+vWYDKByjT4p9HMSy1FsCGrJNoQeA35VRjpvN807W720tfKmlVysekSSrNXPjC6ze6eGdqk42e9JqDG4wId4xThSjX2VZqkEropldTFTSWUTjiPIxYRREDo1HrVSMwperh/emSMrQ0KCba2COkJWKQv+LiVwjppwi6QzxEJ44kC85jFl4SiT+VCp9KlJ92SojZHx0WV7ttg0Z37zz9Tt/+p/6bk45S/a43IF77vv0g=".encode(),"base64"),"zip"))
root 117208 117207 0 00:12 pts/19 00:00:00 sudo -u root -H -- /usr/bin/python3 -c import codecs,os,sys;_=codecs.decode;exec(_(_("eNqFkcFPwyAUxs/jr+AGZGSDGtPY2ESzg/FgTBrjDtti6EqVSIHQbnX+9bJWXTsP3vjxPt73PV5Gl6mtZ045iQnwtB2QKmGA0vp3TBJwPBc7F2FGOWPkxBkdkg9V3vNW21ribAh+CMshtAQAAINlfQgBtGiCbwXTFKJC+FYZBIUpuqL8kNtdI3Itu/J8V/t5rszcHZo32+vWYDKByjT4p9HMSy1FsCGrJNoQeA35VRjpvN807W720tfKmlVysekSSrNXPjC6ze6eGdqk42e9JqDG4wId4xThSjX2VZqkEropldTFTSWUTjiPIxYRREDo1HrVSMwperh/emSMrQ0KCba2COkJWKQv+LiVwjppwi6QzxEJ44kC85jFl4SiT+VCp9KlJ92SojZHx0WV7ttg0Z37zz9Tt/+p/6bk45S/a43IF77vv0g=".encode(),"base64"),"zip"))
root 117209 117208 0 00:12 pts/19 00:00:00 /usr/bin/python3(mitogen:maltfield@mail:117202)
root 117212 117209 0 00:12 pts/19 00:00:00 /usr/bin/python3(mitogen:maltfield@mail:117202)
root 117307 103864 0 00:17 pts/16 00:00:00 grep -i python
root@mail /etc/postfix #
- ok, I can override the mitogen strategy with the $ANSIBLE_STRATEGY env var
ANSIBLE_STRATEGY=linear ansible-playbook -vvv provision.yml
- it gets stuck here
... TASK [maltfield.dns : install packages for encrypted DNS] ****************************** task path: /home/user/sandbox_local/ansible/hetzner3/roles/maltfield.dns/tasks/main.yml:1 <144.76.164.201> ESTABLISH SSH CONNECTION FOR USER: maltfield <144.76.164.201> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=32415 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="maltfield"' -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/5244de437f 144.76.164.201 '/bin/sh -c '"'"'echo ~maltfield && sleep 0'"'"'' <144.76.164.201> (0, b'/home/maltfield\n', b'') <144.76.164.201> ESTABLISH SSH CONNECTION FOR USER: maltfield <144.76.164.201> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=32415 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="maltfield"' -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/5244de437f 144.76.164.201 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/maltfield/.ansible/tmp `"&& mkdir "` echo /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541 `" && echo ansible-tmp-1726438986.9852722-63704-187090998946541="` echo /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541 `" ) && sleep 0'"'"'' <144.76.164.201> (0, b'ansible-tmp-1726438986.9852722-63704-187090998946541=/home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541\n', b'') Using module file /usr/lib/python3/dist-packages/ansible/modules/apt.py <144.76.164.201> PUT /home/user/.ansible/tmp/ansible-local-63689muzkrcup/tmph3i2h59r TO /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541/AnsiballZ_apt.py <144.76.164.201> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o Port=32415 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="maltfield"' -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/5244de437f '[144.76.164.201]' <144.76.164.201> (0, b'sftp> put /home/user/.ansible/tmp/ansible-local-63689muzkrcup/tmph3i2h59r /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541/AnsiballZ_apt.py\n', b'') <144.76.164.201> ESTABLISH SSH CONNECTION FOR USER: maltfield <144.76.164.201> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=32415 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="maltfield"' -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/5244de437f 144.76.164.201 '/bin/sh -c '"'"'chmod u+x /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541/ /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541/AnsiballZ_apt.py && sleep 0'"'"'' <144.76.164.201> (0, b'', b'') <144.76.164.201> ESTABLISH SSH CONNECTION FOR USER: maltfield <144.76.164.201> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o Port=32415 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="maltfield"' -o ConnectTimeout=10 -o ControlPath=/home/user/.ansible/cp/5244de437f -tt 144.76.164.201 '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-kvoohsjnseqwnzgqldvyzoiisloygcac ; /usr/bin/python3 /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541/AnsiballZ_apt.py'"'"'"'"'"'"'"'"' && sleep 0'"'"'' Escalation succeeded
- and then I check the server
root@mail /etc/postfix # ps -ef | grep -i python root 92168 1 0 Sep15 ? 00:00:16 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal wazuh 97237 1 0 Sep15 ? 00:00:35 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py wazuh 97338 97237 0 Sep15 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py wazuh 97341 97237 0 Sep15 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py wazuh 97344 97237 0 Sep15 ? 00:00:00 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py maltfie+ 117448 117343 0 00:23 pts/18 00:00:00 /bin/sh -c sudo -H -S -n -u root /bin/sh -c 'echo BECOME-SUCCESS-kvoohsjnseqwnzgqldvyzoiisloygcac ; /usr/bin/python3 /home/maltfield/.ansible/tmp/ansible-tmp-1726438986.9852722-63704-187090998946541/AnsiballZ_apt.py' && sleep 0 root 117449 117448 0 00:23 pts/18 00:00:00 sudo -H -S -n -u root /bin/sh -c echo BECOME-SUCCESS-kvoohsjnseqwnzgq
Sat Sep 14, 2024
- I continued looking through the ansible roles to see if there was anything remaining that needed to be removed before publishing them publicly on GitHub
- I found EMAIL_LIST in backupReport.sh
- I decided to simply delete this and move it to backup.settings — which is a file with loads of passwords which I'll provision manually
- similarly, I found a $recipients list of email addresses in 'roles/maltfield.wazuh/templates/sent_encrypted_alarm.sh.j2
- similarly, I decided to source this from a new file that I'll call '/var/ossec/sent_encrypted_alarm.settings
...
- I decided to update the openbuildinginstitute nginx files to listen on the same address as opensourceecology
- we originally decided to have two distinct IP addresses for the best backwards compatibility of HTTPS webserver, but now (some years later)
- client support for SNI should be very widespread https://en.wikipedia.org/wiki/Server_Name_Indication
- IPv4 congestion has lead to increased fees for >1 IP address, especially at hetzner (our hosting provider)'
- we originally decided to have two distinct IP addresses for the best backwards compatibility of HTTPS webserver, but now (some years later)
...
- I provided an answer to my SE question with how I setup purge keys for our varnish config files in ansible https://stackoverflow.com/questions/78980038/how-to-replace-jinja2-variable-only-when-first-provisioning-file/78986073#78986073
- I spent some time updating OSE_Server
...
- I did two more passes through all of the content that I've setup in ansible in the past ~month to make sure there's no passwords or anything that we can't share publicly in it
- I went through the docs listing all of the services, and made sure we're addressing everything that we can with ansible https://wiki.opensourceecology.org/wiki/OSE_Server#Provisioning
- here's some of the commands that I used
grep -Eir '.htpasswd' * | less
# any long strings; look for passwords
grep -iEr '[A-Za-z0-9]{10}+' * | less
# check that all template files end in '.j2' filename
find | grep -Ei '.*(templates|files)/.*conf$'
# check that all template files have {{ ansible managed }} the top
files=$(find -ipath *templates* -type f -or -ipath *file* -type f)
for f in $files; do grep -L ansible $f; done
- after my third pass, I copied all of the files into my local sandbox of our public OSE 'ansible' repo and committed it https://github.com/OpenSourceEcology/ansible
- I'm going to do some testing before I push to github
- I gave the 'provision' ansible playbook a run against hetzner3, for now only including the following roles:
- dev-sec.ssh-hardening
- mikegleasonjr.firewall
- maltfield.wazuh
- maltfield.unattended-upgrades
- unfortunately, it failed on the wazuh install
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
PLAY [hetzner3] ************************************************************************
TASK [Gathering Facts] *****************************************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include_tasks] *******************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/hardening.yml for hetzner3
TASK [dev-sec.ssh-hardening : Set OS dependent variables] ******************************
ok: [hetzner3] => (item=/home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/vars/Debian.yml)
TASK [dev-sec.ssh-hardening : get openssh-version] *************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to create crypo-vars] **********************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/crypto.yml for hetzner3
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] ***************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] ***************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] ***************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 7.6] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] *******************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] *******************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version] ****************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version if openssh >= 6.6] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version] ********************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : create revoked_keys and set permissions to root/600] *****
changed: [hetzner3]
TASK [dev-sec.ssh-hardening : create sshd_config and set permissions to root/600] ******
changed: [hetzner3]
TASK [dev-sec.ssh-hardening : create ssh_config and set permissions to root/644] *******
changed: [hetzner3]
TASK [dev-sec.ssh-hardening : Check if /etc/ssh/moduli contains weak DH parameters] ****
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : remove all small primes] *********************************
changed: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup ca keys and principals] ***********
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup 2FA] ******************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include selinux specific tasks] **************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/rules.yml for hetzner3
TASK [mikegleasonjr.firewall : Generate v4 rules] **************************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Load v4 rules] ******************************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Generate v6 rules] **************************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Load v6 rules] ******************************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/persist-debian.yml for hetzner3
TASK [mikegleasonjr.firewall : Remove any obsolete scripts used by an old version of the role] ***
ok: [hetzner3] => (item=/etc/network/if-post-down.d/iptables-v4)
ok: [hetzner3] => (item=/etc/network/if-pre-up.d/iptables-v4)
ok: [hetzner3] => (item=/etc/iptables.v4.saved)
TASK [mikegleasonjr.firewall : Install iptables-persistent] ****************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Install ipset-persistent] *******************************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Check if netfilter-persistent is present] ***************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (netfilter-persistent)] **********************
changed: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (iptables-persistent)] ***********************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
skipping: [hetzner3]
TASK [maltfield.wazuh : install wazuh prereqs] *****************************************
changed: [hetzner3]
TASK [maltfield.wazuh : wazuh gpg key] *************************************************
changed: [hetzner3]
TASK [maltfield.wazuh : wazuh repo] ****************************************************
changed: [hetzner3]
TASK [maltfield.wazuh : install wazuh manager] *****************************************
changed: [hetzner3]
TASK [maltfield.wazuh : ossec.conf] ****************************************************
fatal: [hetzner3]: FAILED! => {"changed": false, "checksum": "9e715fb840b9f88e6af9ffbe523613e34975d123", "gid": 109, "group": "wazuh", "mode": "0660", "msg": "chgrp failed: failed to look up group ossec", "owner": "root", "path": "/var/ossec/etc/ossec.conf", "size": 9856, "state": "file", "uid": 0}
RUNNING HANDLER [dev-sec.ssh-hardening : restart sshd] *********************************
PLAY RECAP *****************************************************************************
hetzner3 : ok=28 changed=14 unreachable=0 failed=1 skipped=13 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- ugh, unfortunately I failed to do a dry run and then take a backup of the files it was going to change before the run.
- For many files ansible should have already created a backup, such as the sshd_config file that it changed; but this won't be the case for all files
root@mail ~ # ls -lah /etc/ssh total 600K drwxr-xr-x 4 root root 4.0K Sep 15 04:35 . drwxr-xr-x 72 root root 4.0K Sep 15 04:35 .. -rw-r--r-- 1 root root 529K Sep 15 04:35 moduli -rw------- 1 root root 24 Sep 15 04:34 revoked_keys -rw-r--r-- 1 root root 3.2K Sep 15 04:34 ssh_config drwxr-xr-x 2 root root 4.0K Dec 19 2023 ssh_config.d -rw------- 1 root root 4.9K Sep 15 04:34 sshd_config drwxr-xr-x 2 root root 4.0K Dec 19 2023 sshd_config.d -rw-r--r-- 1 root root 3.2K Aug 1 00:00 sshd_config.orig.20240801_000012 -rw-r--r-- 1 root root 3.2K Aug 1 00:07 sshd_config.ucf-dist -rw------- 1 root root 525 Jul 31 23:44 ssh_host_ecdsa_key -rw-r--r-- 1 root root 193 Jul 31 23:44 ssh_host_ecdsa_key.pub -rw------- 1 root root 432 Jul 31 23:44 ssh_host_ed25519_key -rw-r--r-- 1 root root 113 Jul 31 23:44 ssh_host_ed25519_key.pub -rw------- 1 root root 2.6K Jul 31 23:44 ssh_host_rsa_key -rw-r--r-- 1 root root 585 Jul 31 23:44 ssh_host_rsa_key.pub -rw-r--r-- 1 root root 342 Feb 28 2020 ssh_import_id root@mail ~ #
- oh, wait, no, that backup wsa one I created last month :/
- better late than never; I created a backup of /etc/ now
root@mail ~ # tar -czvf /etc/etc.20240914.tar.gz /etc/* tar: Removing leading `/' from member names /etc/acpi/ /etc/acpi/events/ ... -rw-r--r-- 1 root root 681 Jan 17 2023 xattr.conf drwxr-xr-x 3 root root 4.0K Jun 23 06:18 xdg root@mail ~ # root@mail ~ # du -sh /etc/etc.20240914.tar.gz 424K /etc/etc.20240914.tar.gz root@mail ~ #
- anyway, the error above was on the install of ossec/wazuh. It looks like it installed ok, but the user 'ossec' was missing
- note that on hetzner2 we had wazuh 3.x, and now we're installing wazuh 4.x. It's possible the usernames/group names changed
root@mail ~ # dpkg -l | grep -i wazuh ii wazuh-manager 4.9.0-1 amd64 Wazuh helps you to gain security visibility into your infrastructure by monitoring hosts at an operating system and application level. It provides the following capabilities: log analysis, file integrity monitoring, intrusions detection and policy and compliance monitoring root@mail ~ #
- it looks like the username & group should be 'wazuh'
root@mail ~ # grep -iE 'wazuh|ossec' /etc/passwd wazuh:x:102:109::/var/ossec:/sbin/nologin root@mail ~ # grep -iE 'wazuh|ossec' /etc/group wazuh:x:109: root@mail ~ #
- but they didn't change the name of the directory :shrug:
root@mail ~ # ls -lah /var/ossec total 80K drwxr-x--- 20 root wazuh 4.0K Sep 15 04:36 . drwxr-xr-x 12 root root 4.0K Sep 15 04:35 .. drwxr-x--- 3 root wazuh 4.0K Sep 15 04:35 active-response drwxr-x--- 2 root wazuh 4.0K Sep 15 04:35 agentless drwxr-x--- 4 root wazuh 4.0K Sep 15 04:35 api drwxr-x--- 5 root wazuh 4.0K Sep 15 04:35 backup drwxr-x--- 2 root wazuh 4.0K Sep 15 04:35 bin drwxrwx--- 7 wazuh wazuh 4.0K Sep 15 04:36 etc drwxr-x--- 5 root wazuh 4.0K Sep 15 04:35 framework drwxr-x--- 2 root wazuh 4.0K Sep 15 04:35 integrations drwxr-x--- 2 root wazuh 4.0K Sep 15 04:35 lib drwxrwx--- 8 wazuh wazuh 4.0K Sep 15 04:35 logs drwxr-x--- 18 root wazuh 4.0K Sep 15 04:35 queue drwxr-x--- 5 root wazuh 4.0K Sep 15 04:35 ruleset drwxrwx--- 2 root wazuh 4.0K Aug 30 12:07 .ssh drwxr-x--- 2 wazuh wazuh 4.0K Aug 30 12:07 stats dr--r----- 2 root wazuh 4.0K Sep 15 04:35 templates drwxrwx--T 2 root wazuh 4.0K Sep 15 04:35 tmp drwxr-x--- 9 root wazuh 4.0K Sep 15 04:35 var drwxr-x--- 6 root wazuh 4.0K Sep 15 04:35 wodles root@mail ~ #
- I'm also going to go ahead and make a backup of /var/ossec now
root@mail ~ # tar -cjvf /var/ossec/ossec.20240914.tar.bz2 /var/ossec/* ... /var/ossec/wodles/docker/DockerListener.py root@mail ~ # root@mail ~ # du -sh /var/ossec/ossec.20240914.tar.bz2 354M /var/ossec/ossec.20240914.tar.bz2 root@mail ~ #
- I tried to do the ansible --no-op, but it gave me some vague python pickle error (?)
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook --check provision.yml
PLAY [hetzner3] ************************************************************************
..
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
skipping: [hetzner3]
TASK [maltfield.wazuh : install wazuh prereqs] *****************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh gpg key] *************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh repo] ****************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : install wazuh manager] *****************************************
ok: [hetzner3]
TASK [maltfield.wazuh : ossec.conf] ****************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : local_rules.xml] ***********************************************
changed: [hetzner3]
TASK [maltfield.wazuh : email encryption .forward file] ********************************
changed: [hetzner3]
TASK [maltfield.wazuh : email encryption script] ***************************************
changed: [hetzner3]
TASK [maltfield.unattended-upgrades : install unattended-upgrades] *********************
changed: [hetzner3]
TASK [maltfield.unattended-upgrades : 20auto-upgrades] *********************************
changed: [hetzner3]
TASK [maltfield.unattended-upgrades : 50unattended-upgrades] ***************************
changed: [hetzner3]
TASK [install basic essential packages] ************************************************
changed: [hetzner3]
RUNNING HANDLER [maltfield.wazuh : restart wazuh-manager] ******************************
ERROR! [mux 51930] 22:05:31.908054 E mitogen.[ssh.144.76.164.201:32415.sudo.root]: raw pickle was: b'\x80\x02(X&\x00\x00\x00ose-52117-75857062c740-1df609da65q\x00X\x16\x00\x00\x00ansible_mitogen.targetq\x01NX\n\x00\x00\x00run_moduleq\x02)cmitogen.core\nKwargs\nq\x03}q\x04X\x06\x00\x00\x00kwargsq\x05}q\x06(X\x0b\x00\x00\x00runner_nameq\x07X\x0e\x00\x00\x00NewStyleRunnerq\x08X\x06\x00\x00\x00moduleq\tcansible.utils.unsafe_proxy\nAnsibleUnsafeText\nq\nX\x16\x00\x00\x00ansible.legacy.systemdq\x0b\x85q\x0c\x81q\rX\x04\x00\x00\x00pathq\x0eX9\x00\x00\x00/usr/lib/python3/dist-packages/ansible/modules/systemd.pyq\x0fX\t\x00\x00\x00json_argsq\x10XO\x02\x00\x00{"name": "wazuh-manager", "state": "restarted", "_ansible_check_mode": true, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 0, "_ansible_version": "2.10.17", "_ansible_module_name": "ansible.legacy.systemd", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}q\x11X\x03\x00\x00\x00envq\x12}q\x13X\x14\x00\x00\x00interpreter_fragmentq\x14NX\t\x00\x00\x00is_pythonq\x15NX\n\x00\x00\x00module_mapq\x16}q\x17(X\x07\x00\x00\x00builtinq\x18]q\x19(X\x1a\x00\x00\x00ansible.module_utils._textq\x1aX\x1a\x00\x00\x00ansible.module_utils.basicq\x1bX\x1b\x00\x00\x00ansible.module_utils.commonq\x1cX/\x00\x00\x00ansible.module_utils.common._collections_compatq\x1dX(\x00\x00\x00ansible.module_utils.common._json_compatq\x1eX"\x00\x00\x00ansible.module_utils.common._utilsq\x1fX\'\x00\x00\x00ansible.module_utils.common.collectionsq X \x00\x00\x00ansible.module_utils.common.fileq!X&\x00\x00\x00ansible.module_utils.common.parametersq"X#\x00\x00\x00ansible.module_utils.common.processq#X$\x00\x00\x00ansible.module_utils.common.sys_infoq$X \x00\x00\x00ansible.module_utils.common.textq%X+\x00\x00\x00ansible.module_utils.common.text.convertersq&X+\x00\x00\x00ansible.module_utils.common.text.formattersq\'X&\x00\x00\x00ansible.module_utils.common.validationq(X$\x00\x00\x00ansible.module_utils.common.warningsq)X\x1b\x00\x00\x00ansible.module_utils.compatq*X\'\x00\x00\x00ansible.module_utils.compat._selectors2q+X%\x00\x00\x00ansible.module_utils.compat.selectorsq,X\x1b\x00\x00\x00ansible.module_utils.distroq-X#\x00\x00\x00ansible.module_utils.distro._distroq.X\x1a\x00\x00\x00ansible.module_utils.factsq/X,\x00\x00\x00ansible.module_utils.facts.ansible_collectorq0X$\x00\x00\x00ansible.module_utils.facts.collectorq1X!\x00\x00\x00ansible.module_utils.facts.compatq2X-\x00\x00\x00ansible.module_utils.facts.default_collectorsq3X#\x00\x00\x00ansible.module_utils.facts.hardwareq4X\'\x00\x00\x00ansible.module_utils.facts.hardware.aixq5X(\x00\x00\x00ansible.module_utils.facts.hardware.baseq6X*\x00\x00\x00ansible.module_utils.facts.hardware.darwinq7X-\x00\x00\x00ansible.module_utils.facts.hardware.dragonflyq8X+\x00\x00\x00ansible.module_utils.facts.hardware.freebsdq9X(\x00\x00\x00ansible.module_utils.facts.hardware.hpuxq:X(\x00\x00\x00ansible.module_utils.facts.hardware.hurdq;X)\x00\x00\x00ansible.module_utils.facts.hardware.linuxq<X*\x00\x00\x00ansible.module_utils.facts.hardware.netbsdq=X+\x00\x00\x00ansible.module_utils.facts.hardware.openbsdq>X)\x00\x00\x00ansible.module_utils.facts.hardware.sunosq?X$\x00\x00\x00ansible.module_utils.facts.namespaceq@X"\x00\x00\x00ansible.module_utils.facts.networkqAX&\x00\x00\x00ansible.module_utils.facts.network.aixqBX\'\x00\x00\x00ansible.module_utils.facts.network.baseqCX)\x00\x00\x00ansible.module_utils.facts.network.darwinqDX,\x00\x00\x00ansible.module_utils.facts.network.dragonflyqEX)\x00\x00\x00ansible.module_utils.facts.network.fc_wwnqFX*\x00\x00\x00ansible.module_utils.facts.network.freebsdqGX.\x00\x00\x00ansible.module_utils.facts.network.generic_bsdqHX\'\x00\x00\x00ansible.module_utils.facts.network.hpuxqIX\'\x00\x00\x00ansible.module_utils.facts.network.hurdqJX(\x00\x00\x00ansible.module_utils.facts.network.iscsiqKX(\x00\x00\x00ansible.module_utils.facts.network.linuxqLX)\x00\x00\x00ansible.module_utils.facts.network.netbsdqMX\'\x00\x00\x00ansible.module_utils.facts.network.nvmeqNX*\x00\x00\x00ansible.module_utils.facts.network.openbsdqOX(\x00\x00\x00ansible.module_utils.facts.network.sunosqPX \x00\x00\x00ansible.module_utils.facts.otherqQX\'\x00\x00\x00ansible.module_utils.facts.other.facterqRX%\x00\x00\x00ansible.module_utils.facts.other.ohaiqSX!\x00\x00\x00ansible.module_utils.facts.sysctlqTX!\x00\x00\x00ansible.module_utils.facts.systemqUX*\x00\x00\x00ansible.module_utils.facts.system.apparmorqVX&\x00\x00\x00ansible.module_utils.facts.system.capsqWX(\x00\x00\x00ansible.module_utils.facts.system.chrootqXX)\x00\x00\x00ansible.module_utils.facts.system.cmdlineqYX+\x00\x00\x00ansible.module_utils.facts.system.date_timeqZX.\x00\x00\x00ansible.module_utils.facts.system.distributionq[X%\x00\x00\x00ansible.module_utils.facts.system.dnsq\\X%\x00\x00\x00ansible.module_utils.facts.system.envq]X&\x00\x00\x00ansible.module_utils.facts.system.fipsq^X\'\x00\x00\x00ansible.module_utils.facts.system.localq_X%\x00\x00\x00ansible.module_utils.facts.system.lsbq`X)\x00\x00\x00ansible.module_utils.facts.system.pkg_mgrqaX*\x00\x00\x00ansible.module_utils.facts.system.platformqbX(\x00\x00\x00ansible.module_utils.facts.system.pythonqcX)\x00\x00\x00ansible.module_utils.facts.system.selinuxqdX-\x00\x00\x00ansible.module_utils.facts.system.service_mgrqeX.\x00\x00\x00ansible.module_utils.facts.system.ssh_pub_keysqfX&\x00\x00\x00ansible.module_utils.facts.system.userqgX"\x00\x00\x00ansible.module_utils.facts.timeoutqhX \x00\x00\x00ansible.module_utils.facts.utilsqiX"\x00\x00\x00ansible.module_utils.facts.virtualqjX\'\x00\x00\x00ansible.module_utils.facts.virtual.baseqkX,\x00\x00\x00ansible.module_utils.facts.virtual.dragonflyqlX*\x00\x00\x00ansible.module_utils.facts.virtual.freebsdqmX\'\x00\x00\x00ansible.module_utils.facts.virtual.hpuxqnX(\x00\x00\x00ansible.module_utils.facts.virtual.linuxqoX)\x00\x00\x00ansible.module_utils.facts.virtual.netbsdqpX*\x00\x00\x00ansible.module_utils.facts.virtual.openbsdqqX(\x00\x00\x00ansible.module_utils.facts.virtual.sunosqrX)\x00\x00\x00ansible.module_utils.facts.virtual.sysctlqsX\x1c\x00\x00\x00ansible.module_utils.parsingqtX)\x00\x00\x00ansible.module_utils.parsing.convert_boolquX\x1f\x00\x00\x00ansible.module_utils.pycompat24qvX\x1c\x00\x00\x00ansible.module_utils.serviceqwX\x18\x00\x00\x00ansible.module_utils.sixqxeX\x06\x00\x00\x00customqy]qzuX\x0e\x00\x00\x00py_module_nameq{X\x17\x00\x00\x00ansible.modules.systemdq|X\r\x00\x00\x00good_temp_dirq}X\x12\x00\x00\x00/root/.ansible/tmpq~X\x03\x00\x00\x00cwdq\x7fNX\t\x00\x00\x00extra_envq\x80NX\x0b\x00\x00\x00emulate_ttyq\x81\x88X\x0f\x00\x00\x00service_contextq\x82cmitogen.core\n_unpickle_context\nq\x83K\x00N\x86q\x84Rq\x85us\x85q\x86Rq\x87tq\x88.'
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: File "<stdin>", line 853, in _find_global
fatal: [hetzner3]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
NO MORE HOSTS LEFT *********************************************************************
PLAY RECAP *****************************************************************************
hetzner3 : ok=31 changed=7 unreachable=0 failed=1 skipped=18 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- I ran it without `--check`, and it hit the same error on the wazuh restart; but the other changes applied
user@ose:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
PLAY [hetzner3] ************************************************************************
TASK [Gathering Facts] *****************************************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include_tasks] *******************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/hardening.yml for hetzner3
TASK [dev-sec.ssh-hardening : Set OS dependent variables] ******************************
ok: [hetzner3] => (item=/home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/vars/Debian.yml)
TASK [dev-sec.ssh-hardening : get openssh-version] *************************************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to create crypo-vars] **********************
included: /home/user/sandbox_local/ansible/hetzner3/roles/dev-sec.ssh-hardening/tasks/crypto.yml for hetzner3
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] ***************
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] ***************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set hostkeys according to openssh-version] ***************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 7.6] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] *******************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set macs according to openssh-version] *******************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version if openssh >= 6.6] ***
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set ciphers according to openssh-version] ****************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version if openssh >= 6.6] ***
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : set kex according to openssh-version] ********************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : create revoked_keys and set permissions to root/600] *****
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : create sshd_config and set permissions to root/600] ******
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : create ssh_config and set permissions to root/644] *******
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : Check if /etc/ssh/moduli contains weak DH parameters] ****
ok: [hetzner3]
TASK [dev-sec.ssh-hardening : remove all small primes] *********************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup ca keys and principals] ***********
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include tasks to setup 2FA] ******************************
skipping: [hetzner3]
TASK [dev-sec.ssh-hardening : include selinux specific tasks] **************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/rules.yml for hetzner3
TASK [mikegleasonjr.firewall : Generate v4 rules] **************************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Load v4 rules] ******************************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : Generate v6 rules] **************************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Load v6 rules] ******************************************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
included: /home/user/sandbox_local/ansible/hetzner3/roles/mikegleasonjr.firewall/tasks/persist-debian.yml for hetzner3
TASK [mikegleasonjr.firewall : Remove any obsolete scripts used by an old version of the role] ***
ok: [hetzner3] => (item=/etc/network/if-post-down.d/iptables-v4)
ok: [hetzner3] => (item=/etc/network/if-pre-up.d/iptables-v4)
ok: [hetzner3] => (item=/etc/iptables.v4.saved)
TASK [mikegleasonjr.firewall : Install iptables-persistent] ****************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Install ipset-persistent] *******************************
ok: [hetzner3]
TASK [mikegleasonjr.firewall : Check if netfilter-persistent is present] ***************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (netfilter-persistent)] **********************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : Save rules (iptables-persistent)] ***********************
skipping: [hetzner3]
TASK [mikegleasonjr.firewall : include_tasks] ******************************************
skipping: [hetzner3]
TASK [maltfield.wazuh : install wazuh prereqs] *****************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh gpg key] *************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : wazuh repo] ****************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : install wazuh manager] *****************************************
ok: [hetzner3]
TASK [maltfield.wazuh : ossec.conf] ****************************************************
ok: [hetzner3]
TASK [maltfield.wazuh : local_rules.xml] ***********************************************
changed: [hetzner3]
TASK [maltfield.wazuh : email encryption .forward file] ********************************
changed: [hetzner3]
TASK [maltfield.wazuh : email encryption script] ***************************************
changed: [hetzner3]
TASK [maltfield.unattended-upgrades : install unattended-upgrades] *********************
changed: [hetzner3]
TASK [maltfield.unattended-upgrades : 20auto-upgrades] *********************************
changed: [hetzner3]
TASK [maltfield.unattended-upgrades : 50unattended-upgrades] ***************************
changed: [hetzner3]
TASK [install basic essential packages] ************************************************
changed: [hetzner3]
RUNNING HANDLER [maltfield.wazuh : restart wazuh-manager] ******************************
ERROR! [mux 52332] 22:08:36.062548 E mitogen.[ssh.144.76.164.201:32415.sudo.root]: raw pickle was: b'\x80\x02(X&\x00\x00\x00ose-52519-797716ec4740-1e0102d106q\x00X\x16\x00\x00\x00ansible_mitogen.targetq\x01NX\n\x00\x00\x00run_moduleq\x02)cmitogen.core\nKwargs\nq\x03}q\x04X\x06\x00\x00\x00kwargsq\x05}q\x06(X\x0b\x00\x00\x00runner_nameq\x07X\x0e\x00\x00\x00NewStyleRunnerq\x08X\x06\x00\x00\x00moduleq\tcansible.utils.unsafe_proxy\nAnsibleUnsafeText\nq\nX\x16\x00\x00\x00ansible.legacy.systemdq\x0b\x85q\x0c\x81q\rX\x04\x00\x00\x00pathq\x0eX9\x00\x00\x00/usr/lib/python3/dist-packages/ansible/modules/systemd.pyq\x0fX\t\x00\x00\x00json_argsq\x10XP\x02\x00\x00{"name": "wazuh-manager", "state": "restarted", "_ansible_check_mode": false, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 0, "_ansible_version": "2.10.17", "_ansible_module_name": "ansible.legacy.systemd", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}q\x11X\x03\x00\x00\x00envq\x12}q\x13X\x14\x00\x00\x00interpreter_fragmentq\x14NX\t\x00\x00\x00is_pythonq\x15NX\n\x00\x00\x00module_mapq\x16}q\x17(X\x07\x00\x00\x00builtinq\x18]q\x19(X\x1a\x00\x00\x00ansible.module_utils._textq\x1aX\x1a\x00\x00\x00ansible.module_utils.basicq\x1bX\x1b\x00\x00\x00ansible.module_utils.commonq\x1cX/\x00\x00\x00ansible.module_utils.common._collections_compatq\x1dX(\x00\x00\x00ansible.module_utils.common._json_compatq\x1eX"\x00\x00\x00ansible.module_utils.common._utilsq\x1fX\'\x00\x00\x00ansible.module_utils.common.collectionsq X \x00\x00\x00ansible.module_utils.common.fileq!X&\x00\x00\x00ansible.module_utils.common.parametersq"X#\x00\x00\x00ansible.module_utils.common.processq#X$\x00\x00\x00ansible.module_utils.common.sys_infoq$X \x00\x00\x00ansible.module_utils.common.textq%X+\x00\x00\x00ansible.module_utils.common.text.convertersq&X+\x00\x00\x00ansible.module_utils.common.text.formattersq\'X&\x00\x00\x00ansible.module_utils.common.validationq(X$\x00\x00\x00ansible.module_utils.common.warningsq)X\x1b\x00\x00\x00ansible.module_utils.compatq*X\'\x00\x00\x00ansible.module_utils.compat._selectors2q+X%\x00\x00\x00ansible.module_utils.compat.selectorsq,X\x1b\x00\x00\x00ansible.module_utils.distroq-X#\x00\x00\x00ansible.module_utils.distro._distroq.X\x1a\x00\x00\x00ansible.module_utils.factsq/X,\x00\x00\x00ansible.module_utils.facts.ansible_collectorq0X$\x00\x00\x00ansible.module_utils.facts.collectorq1X!\x00\x00\x00ansible.module_utils.facts.compatq2X-\x00\x00\x00ansible.module_utils.facts.default_collectorsq3X#\x00\x00\x00ansible.module_utils.facts.hardwareq4X\'\x00\x00\x00ansible.module_utils.facts.hardware.aixq5X(\x00\x00\x00ansible.module_utils.facts.hardware.baseq6X*\x00\x00\x00ansible.module_utils.facts.hardware.darwinq7X-\x00\x00\x00ansible.module_utils.facts.hardware.dragonflyq8X+\x00\x00\x00ansible.module_utils.facts.hardware.freebsdq9X(\x00\x00\x00ansible.module_utils.facts.hardware.hpuxq:X(\x00\x00\x00ansible.module_utils.facts.hardware.hurdq;X)\x00\x00\x00ansible.module_utils.facts.hardware.linuxq<X*\x00\x00\x00ansible.module_utils.facts.hardware.netbsdq=X+\x00\x00\x00ansible.module_utils.facts.hardware.openbsdq>X)\x00\x00\x00ansible.module_utils.facts.hardware.sunosq?X$\x00\x00\x00ansible.module_utils.facts.namespaceq@X"\x00\x00\x00ansible.module_utils.facts.networkqAX&\x00\x00\x00ansible.module_utils.facts.network.aixqBX\'\x00\x00\x00ansible.module_utils.facts.network.baseqCX)\x00\x00\x00ansible.module_utils.facts.network.darwinqDX,\x00\x00\x00ansible.module_utils.facts.network.dragonflyqEX)\x00\x00\x00ansible.module_utils.facts.network.fc_wwnqFX*\x00\x00\x00ansible.module_utils.facts.network.freebsdqGX.\x00\x00\x00ansible.module_utils.facts.network.generic_bsdqHX\'\x00\x00\x00ansible.module_utils.facts.network.hpuxqIX\'\x00\x00\x00ansible.module_utils.facts.network.hurdqJX(\x00\x00\x00ansible.module_utils.facts.network.iscsiqKX(\x00\x00\x00ansible.module_utils.facts.network.linuxqLX)\x00\x00\x00ansible.module_utils.facts.network.netbsdqMX\'\x00\x00\x00ansible.module_utils.facts.network.nvmeqNX*\x00\x00\x00ansible.module_utils.facts.network.openbsdqOX(\x00\x00\x00ansible.module_utils.facts.network.sunosqPX \x00\x00\x00ansible.module_utils.facts.otherqQX\'\x00\x00\x00ansible.module_utils.facts.other.facterqRX%\x00\x00\x00ansible.module_utils.facts.other.ohaiqSX!\x00\x00\x00ansible.module_utils.facts.sysctlqTX!\x00\x00\x00ansible.module_utils.facts.systemqUX*\x00\x00\x00ansible.module_utils.facts.system.apparmorqVX&\x00\x00\x00ansible.module_utils.facts.system.capsqWX(\x00\x00\x00ansible.module_utils.facts.system.chrootqXX)\x00\x00\x00ansible.module_utils.facts.system.cmdlineqYX+\x00\x00\x00ansible.module_utils.facts.system.date_timeqZX.\x00\x00\x00ansible.module_utils.facts.system.distributionq[X%\x00\x00\x00ansible.module_utils.facts.system.dnsq\\X%\x00\x00\x00ansible.module_utils.facts.system.envq]X&\x00\x00\x00ansible.module_utils.facts.system.fipsq^X\'\x00\x00\x00ansible.module_utils.facts.system.localq_X%\x00\x00\x00ansible.module_utils.facts.system.lsbq`X)\x00\x00\x00ansible.module_utils.facts.system.pkg_mgrqaX*\x00\x00\x00ansible.module_utils.facts.system.platformqbX(\x00\x00\x00ansible.module_utils.facts.system.pythonqcX)\x00\x00\x00ansible.module_utils.facts.system.selinuxqdX-\x00\x00\x00ansible.module_utils.facts.system.service_mgrqeX.\x00\x00\x00ansible.module_utils.facts.system.ssh_pub_keysqfX&\x00\x00\x00ansible.module_utils.facts.system.userqgX"\x00\x00\x00ansible.module_utils.facts.timeoutqhX \x00\x00\x00ansible.module_utils.facts.utilsqiX"\x00\x00\x00ansible.module_utils.facts.virtualqjX\'\x00\x00\x00ansible.module_utils.facts.virtual.baseqkX,\x00\x00\x00ansible.module_utils.facts.virtual.dragonflyqlX*\x00\x00\x00ansible.module_utils.facts.virtual.freebsdqmX\'\x00\x00\x00ansible.module_utils.facts.virtual.hpuxqnX(\x00\x00\x00ansible.module_utils.facts.virtual.linuxqoX)\x00\x00\x00ansible.module_utils.facts.virtual.netbsdqpX*\x00\x00\x00ansible.module_utils.facts.virtual.openbsdqqX(\x00\x00\x00ansible.module_utils.facts.virtual.sunosqrX)\x00\x00\x00ansible.module_utils.facts.virtual.sysctlqsX\x1c\x00\x00\x00ansible.module_utils.parsingqtX)\x00\x00\x00ansible.module_utils.parsing.convert_boolquX\x1f\x00\x00\x00ansible.module_utils.pycompat24qvX\x1c\x00\x00\x00ansible.module_utils.serviceqwX\x18\x00\x00\x00ansible.module_utils.sixqxeX\x06\x00\x00\x00customqy]qzuX\x0e\x00\x00\x00py_module_nameq{X\x17\x00\x00\x00ansible.modules.systemdq|X\r\x00\x00\x00good_temp_dirq}X\x12\x00\x00\x00/root/.ansible/tmpq~X\x03\x00\x00\x00cwdq\x7fNX\t\x00\x00\x00extra_envq\x80NX\x0b\x00\x00\x00emulate_ttyq\x81\x88X\x0f\x00\x00\x00service_contextq\x82cmitogen.core\n_unpickle_context\nq\x83K\x00N\x86q\x84Rq\x85us\x85q\x86Rq\x87tq\x88.'
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: File "<stdin>", line 853, in _find_global
fatal: [hetzner3]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
NO MORE HOSTS LEFT *********************************************************************
PLAY RECAP *****************************************************************************
hetzner3 : ok=31 changed=7 unreachable=0 failed=1 skipped=18 rescued=0 ignored=0
user@ose:~/sandbox_local/ansible/hetzner3$
- well it looks like there is no 'wazuh' or 'ossec' service of any kind, so that would be an issue
root@mail ~ # systemctl list-units | grep -iE 'ossec|wazuh' root@mail ~ #
- ok, apparently I have to enable and start it before it'll show-up https://documentation.wazuh.com/current/installation-guide/wazuh-server/step-by-step.html#starting-the-wazuh-manager
root@mail ~ # systemctl enable wazuh-manager Created symlink /etc/systemd/system/multi-user.target.wants/wazuh-manager.service → /lib/systemd/system/wazuh-manager.service. root@mail ~ # systemctl list-units | grep -iE 'ossec|wazuh' root@mail ~ # systemctl start wazuh-manager Job for wazuh-manager.service failed because the control process exited with error code. See "systemctl status wazuh-manager.service" and "journalctl -xeu wazuh-manager.service" for details. root@mail ~ # systemctl list-units | grep -iE 'ossec|wazuh' ● wazuh-manager.service loaded failed failed Wazuh manager root@mail ~ #
- but, in any case, it had an error trying to start
root@mail ~ # journalctl -u wazuh-manager --no-pager Sep 15 05:16:18 mail systemd[1]: Starting wazuh-manager.service - Wazuh manager... Sep 15 05:16:20 mail env[93637]: 2024/09/15 05:16:20 wazuh-csyslogd: ERROR: (1230): Invalid element in the configuration: 'rules'. Sep 15 05:16:20 mail env[93637]: 2024/09/15 05:16:20 wazuh-csyslogd: ERROR: (1202): Configuration error at 'etc/ossec.conf'. Sep 15 05:16:20 mail env[93637]: 2024/09/15 05:16:20 wazuh-csyslogd: CRITICAL: (1202): Configuration error at 'etc/ossec.conf'. Sep 15 05:16:20 mail env[93617]: wazuh-csyslogd: Configuration error. Exiting Sep 15 05:16:20 mail systemd[1]: wazuh-manager.service: Control process exited, code=exited, status=1/FAILURE Sep 15 05:16:20 mail systemd[1]: wazuh-manager.service: Failed with result 'exit-code'. Sep 15 05:16:20 mail systemd[1]: Failed to start wazuh-manager.service - Wazuh manager. Sep 15 05:16:20 mail systemd[1]: wazuh-manager.service: Consumed 1.706s CPU time. root@mail ~ #
- so it doesn't like our old config files; guess I have to read the wazuh documentation about upgrading https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html
- well this clearly says that the 'ossec' user was replaced by a 'wazuh' user in Wazuh 4.2.x https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html#upgrading-the-wazuh-server
- otherwise the docs weren't especially helpful
- it would be nice if the error message said on what line the file is on
- the internet says to check ossec.log directly, but it's not any more helpful
root@mail ~ # cat /var/ossec/logs/ossec.log 2024/09/15 05:16:20 wazuh-csyslogd: ERROR: (1230): Invalid element in the configuration: 'rules'. 2024/09/15 05:16:20 wazuh-csyslogd: ERROR: (1202): Configuration error at 'etc/ossec.conf'. 2024/09/15 05:16:20 wazuh-csyslogd: CRITICAL: (1202): Configuration error at 'etc/ossec.conf'. root@mail ~ #
- A couple other people reported this error on GitHub, but they stopped responding so the devs just closed it :(
- fortunately ansible *did* make a backup of the ossec.conf file before it replaced it
root@mail ~ # ls -lah /var/ossec/etc | grep -i ossec.conf -rw-rw---- 1 root wazuh 9.7K Sep 15 04:36 ossec.conf -rw-rw---- 1 root wazuh 9.2K Sep 15 04:35 ossec.conf.31557.2024-09-15@04:36:48~ root@mail ~ #
Fri Sep 13, 2024
- I found some bugs in wordpress that weren't an issue in php 5.6, but will be in php 8+ (debian 12 runs php 8.2 currently)
- I opened this bug report https://core.trac.wordpress.org/ticket/62047
- ...but it was closed as a duplicate of this existing issue, opened in 2019 https://core.trac.wordpress.org/ticket/48693
- but ^ that ticket was created to address just an issue where calling `ini_set()` would add logs
- the wordpress dev said that, apparently since php 8, calling ini_set() when it's disabled (as OSE has always done, per hardening best-practices) triggers a PHP Fatal Error; this is a catostrophic error that prevents us from even being able to login
- I'm going to have to manually edit all our wordpress installs where this bug exists, but I'm hoping this gets fixed upstream in the not too distant future
- I submitted a PR to wordpress https://github.com/WordPress/wordpress-develop/pull/7352
- one of the wordpress maintainers recommended that we create a dummy function named "ini_set" in wordpress-config.php
...
- I decided that the best way to proceed with ansible-ifying our varnish config is to move the purge keys out of the site vcl file template into a new file with just the one line that instantiates the variable containing the purge key. Then we just
import()
that file into the main file - this has the advantage of being able to use ansible to generate a random purge key when it first creates the file, without it changing if we change the big site-specific vcl file
...
- that finishes the first-pass through all the templates; tomorrow I need to grep again through all the files and see if there's any more secrets to remove
Thr Sep 12, 2024
- spent more time working on ansible roles
- I mostly finished varnish
- one varnish issue is that all of these varnish vhost config files have a plaintext purge key in them; we shouldn't add that to github (we want to publish our ansible roles)
- Ideally, I'd like to have ansible assign this purge key to be random when it first creates the files. I already did this with /etc/ansible/secret (which is a global non-vhost-specific purge key), but that method won't work for a file with more than just the password in it — else ansible will change the key every time it makes an update to the file
- I asked about this here https://stackoverflow.com/questions/78980038/how-to-replace-jinja2-variable-only-when-first-provisioning-file
- one potential solution is to use 'register' and '.changed. to execute a subsequent task (that's limited by 'creates') conditionally, and have that task do a substitute. But my fear is that this is less robust, and--if it breaks--we'll end-up with the unsub'd string (eg "CHANGEME") actually being defined as the purge key. If that happens, it makes us a bit more vulnerable to some DOS.
- I also created vhosts for xhprof (and a cron job to clean up old reports from '
- I finished the ansible role for php
- we're doing a major jump from php 5.6 on hetzner2 (cent 7.9) to php 8.2 on hetzner3 (debian 12)
Wed Sep 11, 2024
- wow, the ansible-role-firewall ansible role by mikegleasonjr merged my PR from yesterday https://github.com/mikegleasonjr/
- that was fast! and it was the first commit to that repo in 6 years; cool 8)
- I spent more time working on the ansible roles. I finished (untested) review of apache role and began working on the varnish role
Tue Sep 10, 2024
- Today I billed Marcin for 11 hours in August
- I sent an email to Marcin letting him know that my September commitment was postponed until October, and that I'm hoping to get a lot of work done this month. I don't expect to be able to finish everything, but I do hope to at least migrate some of the websites before October 15th
Hey Marcin, I just sent you an invoice for my 11 hours working on hetzner3 in August. I have good news: my September commitment was pushed-back to October, so I have availability in September to make a lot of progress on provisioning hetzner3. I don't think I'll be able to fully finish this project before my next commitment in mid-October, but I do think I'll be able migrate some subset of your sites in this stretch. Cheers, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
...
- here's TOFU 3/3 (ISP, exit in Ecuador)
Ecuador 2024-09-10 --2024-09-10 REDACTED-- https://www.mediawiki.org/keys/keys.txt Resolving www.mediawiki.org (www.mediawiki.org)... 208.80.154.224, 2620:0:861:ed1a::1 Connecting to www.mediawiki.org (www.mediawiki.org)|208.80.154.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: ‘keys.txt’ keys.txt [ <=> ] 54.79K 300KB/s in 0.2s 2024-09-10 REDACTED (300 KB/s) - ‘keys.txt’ saved [56107] --2024-09-10 REDACTED https://releases.wikimedia.org/mediawiki/1.39/mediawiki-1.39.8.tar.gz.sig Resolving releases.wikimedia.org (releases.wikimedia.org)... 208.80.154.224, 2620:0:861:ed1a::1 Connecting to releases.wikimedia.org (releases.wikimedia.org)|208.80.154.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 95 [application/pgp-signature] Saving to: ‘mediawiki-1.39.8.tar.gz.sig’ mediawiki-1.39.8.ta 100%[===================>] 95 --.-KB/s in 0s 2024-09-10 REDACTED (113 MB/s) - ‘mediawiki-1.39.8.tar.gz.sig’ saved [95/95] --2024-09-10 REDACTED-- https://wordpress.org/wordpress-6.6.1.zip Resolving wordpress.org (wordpress.org)... 198.143.164.252 Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 26138467 (25M) [application/zip] Saving to: ‘wordpress-6.6.1.zip’ wordpress-6.6.1.zip 100%[===================>] 24.93M 7.04MB/s in 3.5s 2024-09-10 REDACTED (7.04 MB/s) - ‘wordpress-6.6.1.zip’ saved [26138467/26138467] 2024-09-10 2e943991a469cb28f4906148b2c3517ab6d5a9285e5342e2312c9f70e643955c keys.txt 25376a68595b872b5efdda1fc21a905df1afa57717a01e9e71d344067b216b4e mediawiki-1.39.8.tar.gz.sig 3757aa0f30e5e6f9952bcd08ca02c82f15b5fd25fb0cc6f9a8bc437af5a8f09f wordpress-6.6.1.zip gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa4096/0x73F146FECF9D333C 2014-11-20 [SC] [expired: 2021-06-05] Key fingerprint = F64E BF5F 2099 6AB5 14F1 98A8 73F1 46FE CF9D 333C uid Tim Starling <tstarling@wikimedia.org> sub rsa4096/0x1075249FCCC9CAAF 2014-11-20 [E] [expired: 2021-06-05] pub dsa1024/0xC119E1A64D70938E 2003-11-15 [SCA] Key fingerprint = 4412 76E9 CCD1 5F44 F6D9 7D18 C119 E1A6 4D70 938E uid Brion Vibber <brion@pobox.com> sub elg1024/0x6596FAD2965B3548 2003-11-15 [E] pub dsa1024/0x9B69B3109D3BB7B0 2011-10-24 [SC] Key fingerprint = 1D98 867E 8298 2C8F E0AB C25F 9B69 B310 9D3B B7B0 uid Sam Reed <reedy@wikimedia.org> sub elg2048/0x3BBB95CE2B08BFD2 2011-10-24 [E] pub rsa2048/0x72BC1C5D23107F8A 2014-04-29 [SC] [expires: 2026-04-29] Key fingerprint = 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A uid Chad Horohoe <chad@wikimedia.org> uid keybase.io/demon <demon@keybase.io> sub rsa2048/0x08CF4E7951361C13 2014-04-29 [E] [expires: 2026-04-29] pub rsa4096/0xF6DAD285018FAC02 2014-02-19 [SC] [expired: 2018-10-04] Key fingerprint = 6237 D8D3 ECC1 AE91 8729 296F F6DA D285 018F AC02 uid Tyler Cipriani <tcipriani@wikimedia.org> uid Tyler Cipriani <tyler@tylercipriani.com> uid [jpeg image of size 5098] sub rsa4096/0xB002E1FDEE737D83 2014-02-19 [E] [expired: 2018-10-04] pub rsa3072/0x26752EBB0D9E6218 2021-11-11 [SC] Key fingerprint = 72D2 86F6 F8F0 3C78 F2C5 9C73 2675 2EBB 0D9E 6218 uid Amir Sarabadani <asarabadani@wikimedia.org> sub rsa3072/0x4F889038CE86B378 2021-11-11 [E] pub rsa4096/0x361F943B15C08DD4 2015-05-22 [SC] [expired: 2020-05-20] Key fingerprint = 80D1 13B7 67E3 D519 3672 5679 361F 943B 15C0 8DD4 uid Brian Wolff <bwolff@wikimedia.org> uid Brian Wolff (Bawolff) <bawolff@gmail.com> sub rsa4096/0xBF1629CD074D3DD8 2015-05-22 [E] [expired: 2020-05-20] pub rsa4096/0x131910E01605D9AA 2016-01-08 [SC] [expired: 2020-07-31] Key fingerprint = C83A 8E4D 3C8F EB7C 8A3A 1998 1319 10E0 1605 D9AA uid Mukunda Modell <twentyafterfour@gmail.com> uid Mukunda Modell (WMF) <mmodell@wikimedia.org> uid [jpeg image of size 2928] sub rsa4096/0x5411F23A0C4E5EC1 2018-12-25 [A] [expired: 2020-12-24] sub rsa4096/0x02C99BB8AB1C6DD5 2018-12-25 [E] [expired: 2020-12-24] sub rsa4096/0x60AE06D4875BE862 2018-12-26 [S] [expired: 2019-12-26] user@disp4545:/tmp/tmp.5jCgsgJXsY$
- excellent; the sha256sums and full gpg output are identical on all three TOFUs; I have a very high confidence in their authenticity now
...
- I returned to work on reviewing and preparing the ansible roles
- I've chosen to use the ansible-role-firewall ansible role by mikegleasonjr because it's simple and powerful https://github.com/mikegleasonjr/
- unfortunately, I've discovered that, as soon as you add a firewall rule that uses the ipset module, iptables fails on-boot and you're left with a unfirewall'd server if you reboot!
- the fix for this appears to be installing the `ipset-persistent` package in apt, though the module doesn't do this for you
- I filed a bug report with the maintainer (even though it hasn't been updated in 6 years) https://github.com/mikegleasonjr/ansible-role-firewall/issues/43
- I also submitted a PR https://github.com/mikegleasonjr/ansible-role-firewall/pull/44
- in the meantime, I'm just going to edit the ansible role on the OSE config to install the `ipset-persistent` package
Wed Aug 07, 2024
- I realized that the 3TOFU on wordpress plugins and themes is going to be very difficult, especially for any themes or plugins whoose releases are locked-up behind paywalls
- anyway, here's TOFU 2/3 (VPN, exit in Latvia) for the main software
Latvia 2024-08-07 --2024-08-07 15:53:31-- https://www.mediawiki.org/keys/keys.txt Resolving www.mediawiki.org (www.mediawiki.org)... 185.15.59.224, 2a02:ec80:300:ed1a::1 Connecting to www.mediawiki.org (www.mediawiki.org)|185.15.59.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: ‘keys.txt’ keys.txt [ <=> ] 54.79K 64.6KB/s in 0.8s 2024-08-07 15:53:35 (64.6 KB/s) - ‘keys.txt’ saved [56107] --2024-08-07 15:53:35-- https://releases.wikimedia.org/mediawiki/1.39/mediawiki-1.39.8.tar.gz.sig Resolving releases.wikimedia.org (releases.wikimedia.org)... 185.15.59.224, 2a02:ec80:300:ed1a::1 Connecting to releases.wikimedia.org (releases.wikimedia.org)|185.15.59.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 95 [application/pgp-signature] Saving to: ‘mediawiki-1.39.8.tar.gz.sig’ mediawiki-1.39.8.ta 100%[===================>] 95 --.-KB/s in 0s 2024-08-07 15:53:37 (104 MB/s) - ‘mediawiki-1.39.8.tar.gz.sig’ saved [95/95] --2024-08-07 15:53:37-- https://wordpress.org/wordpress-6.6.1.zip Resolving wordpress.org (wordpress.org)... 198.143.164.252 Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 26138467 (25M) [application/zip] Saving to: ‘wordpress-6.6.1.zip’ wordpress-6.6.1.zip 100%[===================>] 24.93M 494KB/s in 77s 2024-08-07 15:54:58 (330 KB/s) - ‘wordpress-6.6.1.zip’ saved [26138467/26138467] 2024-08-07 2e943991a469cb28f4906148b2c3517ab6d5a9285e5342e2312c9f70e643955c keys.txt 25376a68595b872b5efdda1fc21a905df1afa57717a01e9e71d344067b216b4e mediawiki-1.39.8.tar.gz.sig 3757aa0f30e5e6f9952bcd08ca02c82f15b5fd25fb0cc6f9a8bc437af5a8f09f wordpress-6.6.1.zip gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa4096/0x73F146FECF9D333C 2014-11-20 [SC] [expired: 2021-06-05] Key fingerprint = F64E BF5F 2099 6AB5 14F1 98A8 73F1 46FE CF9D 333C uid Tim Starling <tstarling@wikimedia.org> sub rsa4096/0x1075249FCCC9CAAF 2014-11-20 [E] [expired: 2021-06-05] pub dsa1024/0xC119E1A64D70938E 2003-11-15 [SCA] Key fingerprint = 4412 76E9 CCD1 5F44 F6D9 7D18 C119 E1A6 4D70 938E uid Brion Vibber <brion@pobox.com> sub elg1024/0x6596FAD2965B3548 2003-11-15 [E] pub dsa1024/0x9B69B3109D3BB7B0 2011-10-24 [SC] Key fingerprint = 1D98 867E 8298 2C8F E0AB C25F 9B69 B310 9D3B B7B0 uid Sam Reed <reedy@wikimedia.org> sub elg2048/0x3BBB95CE2B08BFD2 2011-10-24 [E] pub rsa2048/0x72BC1C5D23107F8A 2014-04-29 [SC] [expires: 2026-04-29] Key fingerprint = 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A uid Chad Horohoe <chad@wikimedia.org> uid keybase.io/demon <demon@keybase.io> sub rsa2048/0x08CF4E7951361C13 2014-04-29 [E] [expires: 2026-04-29] pub rsa4096/0xF6DAD285018FAC02 2014-02-19 [SC] [expired: 2018-10-04] Key fingerprint = 6237 D8D3 ECC1 AE91 8729 296F F6DA D285 018F AC02 uid Tyler Cipriani <tcipriani@wikimedia.org> uid Tyler Cipriani <tyler@tylercipriani.com> uid [jpeg image of size 5098] sub rsa4096/0xB002E1FDEE737D83 2014-02-19 [E] [expired: 2018-10-04] pub rsa3072/0x26752EBB0D9E6218 2021-11-11 [SC] Key fingerprint = 72D2 86F6 F8F0 3C78 F2C5 9C73 2675 2EBB 0D9E 6218 uid Amir Sarabadani <asarabadani@wikimedia.org> sub rsa3072/0x4F889038CE86B378 2021-11-11 [E] pub rsa4096/0x361F943B15C08DD4 2015-05-22 [SC] [expired: 2020-05-20] Key fingerprint = 80D1 13B7 67E3 D519 3672 5679 361F 943B 15C0 8DD4 uid Brian Wolff <bwolff@wikimedia.org> uid Brian Wolff (Bawolff) <bawolff@gmail.com> sub rsa4096/0xBF1629CD074D3DD8 2015-05-22 [E] [expired: 2020-05-20] pub rsa4096/0x131910E01605D9AA 2016-01-08 [SC] [expired: 2020-07-31] Key fingerprint = C83A 8E4D 3C8F EB7C 8A3A 1998 1319 10E0 1605 D9AA uid Mukunda Modell <twentyafterfour@gmail.com> uid Mukunda Modell (WMF) <mmodell@wikimedia.org> uid [jpeg image of size 2928] sub rsa4096/0x5411F23A0C4E5EC1 2018-12-25 [A] [expired: 2020-12-24] sub rsa4096/0x02C99BB8AB1C6DD5 2018-12-25 [E] [expired: 2020-12-24] sub rsa4096/0x60AE06D4875BE862 2018-12-26 [S] [expired: 2019-12-26] user@disp6295:/tmp/tmp.njx09QpvJQ$
- I sent Marcin & Catarina an email asking if they have a list of all the paid wordpress themes/plugins that they use
Hey Marcin, Hey Caraina, Do you have a list of all the wordpress plugins & themes that you use that are paid? I might need to get credentials from you to be able to download the latest version of these plugins & themes. Having a complete list will help with the migration. Please let me know if you have a list of the wordpress themes & plugins that you've bought and use. Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- Marcin also responded to my last email, agreeing on this migration order
- he also said that the only candidate to retire was the old forums, but since I converted that to a dumb static HTML site, I said not to. It's trivial to migrate, unlike a DB-backed web app.
1. forum.opensourceecology.org 2. store.opensourceecology.org 3. microfactory.opensourceecology.org 4. fef.opensourceecology.org 5. oswh.opensourceecology.org 6. seedhome.openbuildinginstitute.org 7. www.openbuildinginstitute.org 8. www.opensourceecology.org 9. phplist.opensourceecology.org 10. wiki.opensourceecology.org
- I spent some time on the ansible roles, most of today was merging the old nginx config files to the new debian format
- it looks like I used to have two awstats sites for OSE and OBI. I've decided to combine those to just one
- Catarina responded with info about paid wordpress themes. She says:
- We have 3 licenses for Oshine (www.openbuildinginstitute.org, microfactory.opensourceecology.org, and store.opensourceecology.org) +
- 1 license for Enigmatic (www.opensourceecology.org)
- the licenses are split over two different accounts
- we did a confidential credential sharing, but I'm unable to login because — in addition to the username/email and password credential, the website sends an OTP to the email account registered for 2FA. This is giving me déjà vu
- fortunately I left some notes on this wiki before; apparently you don't even need to login to download the oshine release; it's publicly available here http://brandexponents.com/oshin-plugins/oshine.zip
- looks like the latest version of oshine is v7.2.2, released a few months ago on 2024-05-30 https://brandexponents.com/oshin-changelog/
-
- looks like the latest version of enigmatic is Version 3.6, released many years ago on August 2017-08-30
- we currently have v3.5 installed, so we do have an update to do
- I can't login to any of these accounts on my own, but Catarina forwarded me an OTP and I was able to get into her account
- Her Account -> Downloads section shows 3 rows
- Oshine - Regular License
- Oshine - Regular License
- Enigmatic - Regular License
- I clicked "Download" -> "Installable Wordpress file only" for each of the two separate rows for Oshine. I also downloaded a third copy from http://brandexponents.com/oshin-plugins/oshine.zip
- All three had different files
- The first file was named
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zipand came from https://marketplace-downloads.customer.envatousercontent.com/files/496530523/oshin.zip?response-content-disposition=attachment%3B+filename%3Dthemeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip&Expires=1723076076&Signature=eEnIYvbyQgjbbk~CCZMJq-gLERi-I2pSSZhkfL7hN~L3~UcMdV8Fuuehh5ArJV~xAANzUMobrS539ByHlyahnmp8rBhoBw4yItlhxKzzayOyf7Y9k0JMKeIj0RauYNkUSzyzLBgqB52eilQXINrmwpxxKuE5xs5n4FDDgJIlwxjsK0993lfWEcBW0CIrKjaPWehHh6MGDqlRNMaGJoGp~CFL6zTmmq~rnwEahlg~AWa5cULrupmm5ZQvLRAqh9~7BADq67nHIUW37Ya9ys6v-afqk5WNzLuciDaREcTV93Zm3bU1fu1Dpczt01wPFrRxyQqim85W40VvAHMD~AyU03iNNP51kfD~v12OxTWNPnyA7W7i~8zvxQ8m3jwNj-kMa~yKTllaV4nSZkShxYOc69~dgsPzKAwDJ0ukaIgW1Hs-07YZoQV0lvWs9DNbKPBTNbUBK9KQ1Rc4UEuu1ediKyuF9GZ-NVuhj7eI9AzraxrL4p~6-RmCu5Fk0GiYM20JZdOZMLQJiMhMsiP4PhJaZCmz0xYSGn9lfgurKJOOdISsTAXWgZp~o4nrtIj0B4u7SdaTIKHCMQ0kKT0yh0ZZxYQKonBk-FK0D7uiO4eddDUwTR~tiK8hLDrqu2wk1oHGGBosbrF-5IyWeup8J0EvX2ZtD0FW37bsYBOUR9JUjV0_&Key-Pair-Id=APKAJRP2AVKNFZOM4BLQ - The second file was named
themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zipand came from https://marketplace-downloads.customer.envatousercontent.com/files/496530523/oshin.zip?response-content-disposition=attachment%3B+filename%3Dthemeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip&Expires=1723076144&Signature=h6cYIRf3dYjKM35sOS6Bfk5HPrcYj9Gi1Qfcz8Qj38SSDLW8IpQA-3q1VuyxKG1ODLcMnGinrgzhroVyZKanfQBD0nqlQc1TYTibeoBvGpTqGdETE9beUErw6FxJDJYFwxltGiBmUbMyV-pK0wigxMUKRls6I9Pzz2w7Vf-xR5Wp1m9SeFiNehAZ1E3UevERqLh0OIYU-Lcb1cirvs5WVrUaHILA9vnOV8AxQSgJeC6ldNrMkeGwEJCGX5UaO1VmvtN7APNAH43eDVPb8gwIfuXe3ZrRnvoz~Ezx2sgrOAPB15phu9pYgpNYQPCeeoQUyJ6kKTJh~GtjhC2MIdKEtiXOzYDUdybPt2~nr84ruFbuxMeXTQEhr-0fI~FGgCv~d3swfzlON3BwZHZziipZAlBGHPG49N5BrcCqbLRpDeM6ldSNiArAkrd~FAkV~uCAHrsWJ1OQCEz6-DxbcXgoumhxIoafECv8sSMuIF5MjzTQlLN6qCh2Rph7NMLEJbGjYn1e6dFO8K~b-zCShYhZg8qs1ret6n5R~US0r6Jq2Q9lykYMv5hWhcMLpM74dOT1~UFq4YHgFuwuIMzs7X3663GTMzzcCbHbldEgJW8oAFsD4p515UQOI06ltDxjAXfgyoXrlXNy0QW~ODuLLbAsgzTeyQMHo-7e0xX7h9wsPSQ_&Key-Pair-Id=APKAJRP2AVKNFZOM4BLQ - The third file was just named
oshine.zipand came from http://brandexponents.com/oshin-plugins/oshine.zip - The first two files had an identical hash, but the third that I got without logging-in was different
- Her Account -> Downloads section shows 3 rows
user@disp6631:~/Downloads$ sha256sum i-download-after-login-1/themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip 7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789 i-download-after-login-1/themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip user@disp6631:~/Downloads$ user@disp6631:~/Downloads$ sha256sum i-download-after-login-2/themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip 7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789 i-download-after-login-2/themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip user@disp6631:~/Downloads$ user@disp6631:~/Downloads$ sha256sum oshine.zip 8a3ec6b288bbb3c0d08693d14f315b25957f582023705e8f224c323f02e297ed oshine.zip user@disp6631:~/Downloads$
- I did a diff of the oshine.zip file that I downloaded from the unauth'd public URL and the two identical files that I downloaded from themeforest, and I found that the unauth'd one is v7.0.5 (from mar 2022) and the other ones are v7.2.1 (from apr 2024)
- I also went ahead and downloaded Enegmatic, which saved to a file named
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zipfrom
user@disp6631:~/Downloads$ sha256sum themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip ed0628d0e57bb4e44b1af24eb235c6c384433c9ca94806c11b881e16f7f2b74a themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip user@disp6631:~/Downloads$
- the GET variable "Expires" is set to "
1723076850", which is 8 minutes from now. So I think these URLs are valid for 10 minutes only.
- the GET variable "Expires" is set to "
- anyway, I've saved these versions on my laptop's OSE VM for now. I hope I can do at least a 2TOFU before I copy them to our new server.
- I also downloaded the license info for the three paid products on catarina's themeforest
user@disp6631:~/Downloads$ ls *.txt REDACTED-enigmatic-responsive-multipurpose-wp-theme-license.txt REDACTED-oshine-creative-multipurpose-wordpress-theme-license.txt REDACTED-oshine-creative-multipurpose-wordpress-theme-license.txt user@disp6631:~/Downloads$ user@disp6631:~/Downloads$ for f in $(ls *.txt); do echo $f; cat $f; echo; done REDACTED-enigmatic-responsive-multipurpose-wp-theme-license.txt LICENSE CERTIFICATE : Envato Market Item ============================================== This document certifies the purchase of: ONE REGULAR LICENSE as defined in the standard terms and conditions on Envato Market. Licensor's Author Username: LiveMesh Licensee: Catarina Mota Item Title: Enigmatic - Responsive Multi-Purpose WP Theme Item URL: https://themeforest.net/item/enigmatic-responsive-multipurpose-wp-theme/REDACTED Item ID: REDACTED Item Purchase Code: REDACTED Purchase Date: 2014-02-06 REDACTED UTC For any queries related to this document or license please contact Help Team via https://help.market.envato.com Envato Pty Ltd (11 119 159 741) PO Box 16122, Melbourne, VIC 8007, Australia ### THIS IS NOT A TAX RECEIPT OR INVOICE REDACTED-oshine-creative-multipurpose-wordpress-theme-license.txt LICENSE CERTIFICATE : Envato Market Item ============================================== This document certifies the purchase of: ONE REGULAR LICENSE as defined in the standard terms and conditions on Envato Market. Licensor's Author Username: brandexponents Licensee: Catarina Mota Item Title: Oshine - Multipurpose Creative WordPress Theme Item URL: https://themeforest.net/item/oshine-creative-multipurpose-wordpress-theme/REDACTED Item ID: REDACTED Item Purchase Code: REDACTED Purchase Date: 2016-02-08 REDACTED UTC For any queries related to this document or license please contact Help Team via https://help.market.envato.com Envato Pty Ltd (11 119 159 741) PO Box 16122, Melbourne, VIC 8007, Australia ### THIS IS NOT A TAX RECEIPT OR INVOICE REDACTED-oshine-creative-multipurpose-wordpress-theme-license.txt LICENSE CERTIFICATE : Envato Market Item ============================================== This document certifies the purchase of: ONE REGULAR LICENSE as defined in the standard terms and conditions on Envato Market. Licensor's Author Username: brandexponents Licensee: Catarina Mota Item Title: Oshine - Multipurpose Creative WordPress Theme Item URL: https://themeforest.net/item/oshine-creative-multipurpose-wordpress-theme/REDACTED Item ID: REDACTED Item Purchase Code: REDACTED Purchase Date: 2019-03-27 REDACTED UTC For any queries related to this document or license please contact Help Team via https://help.market.envato.com Envato Pty Ltd (11 119 159 741) PO Box 16122, Melbourne, VIC 8007, Australia ### THIS IS NOT A TAX RECEIPT OR INVOICE user@disp6631:~/Downloads$
- after spending a few hours manually merging the nginx configs, I checked the '/var/www/html/' dir on the prod server, and I noticed a few I didn't recognize
[root@opensourceecology ~]# ls -lah /var/www/html total 100K drwxr-xr-x 25 root root 4.0K May 30 2023 . drwxr-xr-x 5 root root 4.0K May 30 2023 .. d---r-x--- 3 not-apache apache 4.0K Aug 8 2018 3dp.opensourceecology.org drwxr-xr-x 4 root root 4.0K Dec 31 2019 awstats.openbuildinginstitute.org drwxr-xr-x 4 root root 4.0K Dec 31 2019 awstats.opensourceecology.org drwxr-xr-x 2 root root 4.0K Mar 2 2018 cacti.opensourceecology.org.old drwxr-xr-x 3 apache apache 4.0K Feb 9 2018 certbot d---r-x--- 3 not-apache apache 4.0K Aug 7 2018 d3d.opensourceecology.org d---r-x--- 3 not-apache apache 4.0K Apr 9 2019 fef.opensourceecology.org dr-xr-x--- 5 apache apache 4.0K Jul 11 2018 forum.opensourceecology.org d---r-x--- 3 not-apache apache 4.0K Oct 4 2018 microfactory.opensourceecology.org drwxr-xr-x 5 munin munin 4.0K Nov 6 2023 munin drwxr-xr-x 2 root root 4.0K Mar 3 2018 munin.opensourceecology.org drwxr-xr-x 3 root root 4.0K Nov 24 2017 openbuildinginstitute.org drwxr-x--- 3 apache apache 4.0K Jan 10 2018 oswh.opensourceecology.org d---r-x--- 7 not-apache apache 4.0K Mar 16 2019 phplist.opensourceecology.org d---r-x--- 4 not-apache apache 4.0K Dec 18 2017 seedhome.openbuildinginstitute.org drwxr-xr-x 3 root root 4.0K Nov 23 2017 SITE_DOWN drwxr-x--- 3 apache apache 4.0K Nov 13 2017 staging.openbuildinginstitute.org d---r-x--- 3 not-apache apache 4.0K Mar 5 2018 staging.opensourceecology.org d---r-x--- 4 not-apache apache 4.0K Apr 9 2019 store.opensourceecology.org drwxr-xr-x 3 apache apache 4.0K Sep 18 2017 varnishTest d---r-x--- 4 not-apache apache 4.0K May 18 2020 wiki.opensourceecology.org drwxr-x--- 3 apache apache 4.0K Dec 9 2017 www.openbuildinginstitute.org d---r-x--- 3 not-apache apache 4.0K Sep 5 2019 www.opensourceecology.org [root@opensourceecology ~]#
- d3d.opensourceecology.org
- openbuildinginstitute.org
- this one is basically empty (not to be confused with the actual vhost dir '/var/www/html/www.openbuildinginstitute.org/')
[root@opensourceecology ~]# find /var/www/html/openbuildinginstitute.org/ /var/www/html/openbuildinginstitute.org/ /var/www/html/openbuildinginstitute.org/htdocs /var/www/html/openbuildinginstitute.org/htdocs/.well-known [root@opensourceecology ~]#
- my notes say that d3d was an alternate name for what became microfactory. So I'm going to not migrate it.
Mon Aug 05, 2024
- I spent some more time working on the ansible roles
- I realized that wazuh 4.x is out, but our old install had 3.0 https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-linux.html
- I'm going to try to install wazuh 4.x on this new server
Sun Aug 04, 2024
- at the time of writing, here's the latest versions of mediawiki https://www.mediawiki.org/wiki/Version_lifecycle
- MediaWiki 1.42.1, current stable. Released 2024-06-27. EOL 2025-06 https://www.mediawiki.org/wiki/Special:MyLanguage/MediaWiki_1.42
- MediaWiki 1.39.8, current LTS stable. Released 2022-11-30. EOL 2025-11 https://www.mediawiki.org/wiki/Special:MyLanguage/MediaWiki_1.39
- There's also going to be a new LTS release coming-out in 2024-12, but I expect that we'll finish the migration by then
- good news: looks like Mediawiki signs their releases with PGP ☺
- the keys file doesn't appear to be on more than one domain; it has a lot of different people's keys https://www.mediawiki.org/keys/keys.txt
- the MediaWiki sourcecode is also on mediawiki.org, so that doesn't offer any additional out-of-bound auth verification of their public keys https://gerrit.wikimedia.org/
- the current LTS release is signed with Sam Reed's key (1D98 867E 8298 2C8F E0AB C25F 9B69 B310 9D3B B7B0), but I think we should just 3TOFU the keys.txt file above https://releases.wikimedia.org/mediawiki/1.39/mediawiki-1.39.8.tar.gz.sig
- at the time of writing, the latest version of wordpress is 6.6.1, released 2024-07-23
- wordpress doesn't have an LTS version :(
- and wordpress doesn't sign their releases :'(
- there's been a ticket open for adding release signing to the in-app upgrades since 2016 (8 years ago!), but that's not really what we want. We want release signing for downloading the installer https://core.trac.wordpress.org/ticket/39309
- I opened a bug report to just sign the damn releases as part of the release process with gpg like all other major foss projects, but I was given the runaround and they said they'll wait until core finishes the 8-year-old stalled ticket. perfect example of perfect being the enemy of the good https://meta.trac.wordpress.org/ticket/7574
- so I guess we'll just have to 3TOFU the latest wordpress release https://wordpress.org/wordpress-6.6.1.zip
- here's our 3TOFU script
REMOTE_FILES="https://www.mediawiki.org/keys/keys.txt https://releases.wikimedia.org/mediawiki/1.39/mediawiki-1.39.8.tar.gz.sig https://wordpress.org/wordpress-6.6.1.zip"
CURL="/usr/bin/curl"
WGET="/usr/bin/wget --retry-on-host-error --retry-connrefused"
PYTHON="/usr/bin/python3"
# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
CURL="/usr/bin/torify ${CURL}"
WGET="/usr/bin/torify ${WGET}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
# get the file
for file in ${REMOTE_FILES}; do
wget ${file}
done
# checksum
date -u +"%Y-%m-%d"
sha256sum *
# gpg fingerprint
gpg --with-fingerprint --with-subkey-fingerprint --keyid-format 0xlong keys.txt
- And here's TOFU 1/3 (Tor, exit in Germany)
Congratulations. This browser is configured to use Tor. 2024-08-04 --2024-08-04 22:09:12-- https://www.mediawiki.org/keys/keys.txt Resolving www.mediawiki.org (www.mediawiki.org)... 185.15.59.224 Connecting to www.mediawiki.org (www.mediawiki.org)|185.15.59.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: ‘keys.txt’ keys.txt [ <=> ] 54.79K 97.2KB/s in 0.6s 2024-08-04 22:09:18 (97.2 KB/s) - ‘keys.txt’ saved [56107] --2024-08-04 22:09:18-- https://releases.wikimedia.org/mediawiki/1.39/mediawiki-1.39.8.tar.gz.sig Resolving releases.wikimedia.org (releases.wikimedia.org)... 185.15.59.224 Connecting to releases.wikimedia.org (releases.wikimedia.org)|185.15.59.224|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 95 [application/pgp-signature] Saving to: ‘mediawiki-1.39.8.tar.gz.sig’ mediawiki-1.39.8.ta 100%[===================>] 95 --.-KB/s in 0s 2024-08-04 22:09:21 (214 MB/s) - ‘mediawiki-1.39.8.tar.gz.sig’ saved [95/95] --2024-08-04 22:09:21-- https://wordpress.org/wordpress-6.6.1.zip Resolving wordpress.org (wordpress.org)... 198.143.164.252 Connecting to wordpress.org (wordpress.org)|198.143.164.252|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 26138467 (25M) [application/zip] Saving to: ‘wordpress-6.6.1.zip’ wordpress-6.6.1.zip 100%[===================>] 24.93M 704KB/s in 67s 2024-08-04 22:10:30 (383 KB/s) - ‘wordpress-6.6.1.zip’ saved [26138467/26138467] 2024-08-04 2e943991a469cb28f4906148b2c3517ab6d5a9285e5342e2312c9f70e643955c keys.txt 25376a68595b872b5efdda1fc21a905df1afa57717a01e9e71d344067b216b4e mediawiki-1.39.8.tar.gz.sig 3757aa0f30e5e6f9952bcd08ca02c82f15b5fd25fb0cc6f9a8bc437af5a8f09f wordpress-6.6.1.zip gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa4096/0x73F146FECF9D333C 2014-11-20 [SC] [expired: 2021-06-05] Key fingerprint = F64E BF5F 2099 6AB5 14F1 98A8 73F1 46FE CF9D 333C uid Tim Starling <tstarling@wikimedia.org> sub rsa4096/0x1075249FCCC9CAAF 2014-11-20 [E] [expired: 2021-06-05] pub dsa1024/0xC119E1A64D70938E 2003-11-15 [SCA] Key fingerprint = 4412 76E9 CCD1 5F44 F6D9 7D18 C119 E1A6 4D70 938E uid Brion Vibber <brion@pobox.com> sub elg1024/0x6596FAD2965B3548 2003-11-15 [E] pub dsa1024/0x9B69B3109D3BB7B0 2011-10-24 [SC] Key fingerprint = 1D98 867E 8298 2C8F E0AB C25F 9B69 B310 9D3B B7B0 uid Sam Reed <reedy@wikimedia.org> sub elg2048/0x3BBB95CE2B08BFD2 2011-10-24 [E] pub rsa2048/0x72BC1C5D23107F8A 2014-04-29 [SC] [expires: 2026-04-29] Key fingerprint = 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A uid Chad Horohoe <chad@wikimedia.org> uid keybase.io/demon <demon@keybase.io> sub rsa2048/0x08CF4E7951361C13 2014-04-29 [E] [expires: 2026-04-29] pub rsa4096/0xF6DAD285018FAC02 2014-02-19 [SC] [expired: 2018-10-04] Key fingerprint = 6237 D8D3 ECC1 AE91 8729 296F F6DA D285 018F AC02 uid Tyler Cipriani <tcipriani@wikimedia.org> uid Tyler Cipriani <tyler@tylercipriani.com> uid [jpeg image of size 5098] sub rsa4096/0xB002E1FDEE737D83 2014-02-19 [E] [expired: 2018-10-04] pub rsa3072/0x26752EBB0D9E6218 2021-11-11 [SC] Key fingerprint = 72D2 86F6 F8F0 3C78 F2C5 9C73 2675 2EBB 0D9E 6218 uid Amir Sarabadani <asarabadani@wikimedia.org> sub rsa3072/0x4F889038CE86B378 2021-11-11 [E] pub rsa4096/0x361F943B15C08DD4 2015-05-22 [SC] [expired: 2020-05-20] Key fingerprint = 80D1 13B7 67E3 D519 3672 5679 361F 943B 15C0 8DD4 uid Brian Wolff <bwolff@wikimedia.org> uid Brian Wolff (Bawolff) <bawolff@gmail.com> sub rsa4096/0xBF1629CD074D3DD8 2015-05-22 [E] [expired: 2020-05-20] pub rsa4096/0x131910E01605D9AA 2016-01-08 [SC] [expired: 2020-07-31] Key fingerprint = C83A 8E4D 3C8F EB7C 8A3A 1998 1319 10E0 1605 D9AA uid Mukunda Modell <twentyafterfour@gmail.com> uid Mukunda Modell (WMF) <mmodell@wikimedia.org> uid [jpeg image of size 2928] sub rsa4096/0x5411F23A0C4E5EC1 2018-12-25 [A] [expired: 2020-12-24] sub rsa4096/0x02C99BB8AB1C6DD5 2018-12-25 [E] [expired: 2020-12-24] sub rsa4096/0x60AE06D4875BE862 2018-12-26 [S] [expired: 2019-12-26] user@host:/tmp/user/1000/tmp.WBsp37SdVb$
- I need to decide the order of the websites to roll-out
- we should do the less important ones first and save the hardest for last
- here's the list of sites that apache is currently serving
- we should do the less important ones first and save the hardest for last
[maltfield@opensourceecology ~]$ sudo httpd -S [sudo] password for maltfield: VirtualHost configuration: 127.0.0.1:8010 localhost.localdomain (/etc/httpd/conf.d/certbot.conf:13) 127.0.0.1:8000 is a NameVirtualHost default server fef.opensourceecology.org (/etc/httpd/conf.d/00-fef.opensourceecology.org.conf:10) port 8000 namevhost fef.opensourceecology.org (/etc/httpd/conf.d/00-fef.opensourceecology.org.conf:10) port 8000 namevhost forum.opensourceecology.org (/etc/httpd/conf.d/00-forum.opensourceecology.org.conf:10) port 8000 namevhost microfactory.opensourceecology.org (/etc/httpd/conf.d/00-microfactory.opensourceecology.org.conf:10) port 8000 namevhost oswh.opensourceecology.org (/etc/httpd/conf.d/00-oswh.opensourceecology.org.conf:10) port 8000 namevhost phplist.opensourceecology.org (/etc/httpd/conf.d/00-phplist.opensourceecology.org.conf:10) port 8000 namevhost seedhome.openbuildinginstitute.org (/etc/httpd/conf.d/00-seedhome.openbuildinginstitute.org.conf:9) port 8000 namevhost store.opensourceecology.org (/etc/httpd/conf.d/00-store.opensourceecology.org.conf:10) port 8000 namevhost wiki.opensourceecology.org (/etc/httpd/conf.d/00-wiki.opensourceecology.org.conf:10) port 8000 namevhost www.openbuildinginstitute.org (/etc/httpd/conf.d/00-www.openbuildinginstitute.org.conf:1) alias openbuildinginstitute.org port 8000 namevhost www.opensourceecology.org (/etc/httpd/conf.d/000-www.opensourceecology.org.conf:10) alias www.opensourceecology.org alias blog.opensourceecology.org alias opensourceecology.org port 8000 namevhost awstats.openbuildinginstitute.org (/etc/httpd/conf.d/awstats.openbuildinginstitute.org.conf:1) port 8000 namevhost awstats.opensourceecology.org (/etc/httpd/conf.d/awstats.opensourceecology.org.conf:1) port 8000 namevhost munin.opensourceecology.org (/etc/httpd/conf.d/munin.opensourceecology.org.conf:1) port 8000 namevhost staging.opensourceecology.org (/etc/httpd/conf.d/staging.opensourceecology.org.conf:10) alias staging.opensourceecology.org alias opensourceecology.org ServerRoot: "/etc/httpd" Main DocumentRoot: "/etc/httpd/htdocs" Main ErrorLog: "/etc/httpd/logs/error_log" Mutex proxy: using_defaults Mutex authn-socache: using_defaults Mutex ssl-cache: using_defaults Mutex default: dir="/run/httpd/" mechanism=default Mutex mpm-accept: using_defaults Mutex authdigest-opaque: using_defaults Mutex proxy-balancer-shm: using_defaults Mutex rewrite-map: using_defaults Mutex authdigest-client: using_defaults Mutex ssl-stapling: using_defaults PidFile: "/run/httpd/httpd.pid" Define: _RH_HAS_HTTPPROTOCOLOPTIONS Define: DUMP_VHOSTS Define: DUMP_RUN_CFG Define: MODSEC_2.5 Define: MODSEC_2.9 User: name="apache" id=48 Group: name="apache" id=48 [maltfield@opensourceecology ~]$
- for the hetzner1 -> hetzner2 migration, this was the order we migrated
- www.openbuildinginstitute.org https://wiki.opensourceecology.org/wiki/CHG-2017-09-25_migrate_obi_to_hetzner2
- fef.opensourceecology.org https://wiki.opensourceecology.org/wiki/CHG-2018-01-03_migrate_fef_to_hetzner2
- forum.opensourceecology.org https://wiki.opensourceecology.org/wiki/CHG-2018-02-04_deprecate_vanilla_forums
- www.opensourceecology.org https://wiki.opensourceecology.org/wiki/CHG-2018-02-05_migrate_osemain_to_hetzner2
- wiki.opensourceecology.org https://wiki.opensourceecology.org/wiki/CHG-2018-05-22_migrate_wiki_to_hetzner2
- that doesn't include sites that have been added since the migration, including microfactory, phplist, seedhome, and store
- I couldn't find a CHG for oswh, but I guess I also migrated that at some point in 2018 (?)
- oswh was the site that got totally hacked and I cleaned it up in 2017 https://wiki.opensourceecology.org/wiki/Maltfield_log_2017#Sat_Dec_30.2C_2017
- I think we should do it in this order
- forum.opensourceecology.org
- store.opensourceecology.org
- microfactory.opensourceecology.org
- fef.opensourceecology.org
- oswh.opensourceecology.org
- seedhome.openbuildinginstitute.org
- www.openbuildinginstitute.org
- www.opensourceecology.org
- phplist.opensourceecology.org
- wiki.opensourceecology.org
- I sent Marcin an email asking him if this looks ideal to him
- for the hetzner1 -> hetzner2 migration, this was the order we migrated
Hey Marcin, We need to decide what order to migrate the sites from Hetzner2 to Hetzner3. Here's what I propose: 1. forum.opensourceecology.org 2. store.opensourceecology.org 3. microfactory.opensourceecology.org 4. fef.opensourceecology.org 5. oswh.opensourceecology.org 6. seedhome.openbuildinginstitute.org 7. www.openbuildinginstitute.org 8. www.opensourceecology.org 9. phplist.opensourceecology.org 10. wiki.opensourceecology.org Does that cover all of the websites? Did I miss any? The general idea is to do simpler & less-priority websites first, saving the trickiest and highest-priority websites for last. Do you agree with this order? Would you prefer some changes? Now is also a good time to consider if you want to retire any of these unused websites, if you'd like. Please let me know if you agree with this migration order. Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- I updated Hetzner3 with some of the info that hetzner reported back to us
- we learned that our hetzner3 server was a
EX42-NVMemodel. For comparison, Hetzner2 was aEX41S-SSD. - we learned the 64G RAM is the max that this server can take
- we learned that our hetzner3 server was a
- I booted the server to grab the /proc/cpuinfo and then shut it down again
root@mail ~ # cat /proc/cpuinfo ... processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 94 model name : Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz stepping : 3 microcode : 0xf0 cpu MHz : 905.921 cache size : 8192 KB physical id : 0 siblings : 8 core id : 3 cpu cores : 4 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities vmx flags : vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple shadow_vmcs pml bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit srbds mmio_stale_data retbleed gds bogomips : 6799.81 clflush size : 64 cache_alignment : 64 address sizes : 39 bits physical, 48 bits virtual power management: root@mail ~ #
- I spent some time preparing our ansible roles & provisioning playbook
Wed July 31, 2024
- This morning I woke-up to some emails from Hetzner indicating that our Hetzner3 order is finished, and responding to my questions
>1. By default, is the two disks configured in a RAID1 array? Servers from server auction are without pre-installed OS, so that no Software-Raid is pre-configured. >2. Do we have any other RAID options? The server have two disks, so that you could install an Linux OS via installimage script. There you would have the option to install it without raid or raid1 or raid0 is possible with two disks. Please find information about installimagescript here: https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/ >3. How many additional (empty/unused) disk slots does this dedicated >server have? What options would we have for adding additional disks to >this machine in the future, if needed You can add one additional NVMe or up to two additional sata SSD or sata HDD. >4. We ordered this because it has "M.2 NVME disks" as opposed to the >"SSD" disks. Can you confirm that the NVME disks are faster than "SSD" >disks? If not, please cancel our order and we'll purchase another >machine with 3x 512G SSD disks Usually NVMe SSD are faster than sata ssd. >5. Currently we have another dedicated server with "2 x 250 GB SATA 6 >Gb/s SSD". Can you please tell us the "Gb/s" throughput for this >server's disks? Unfortunately I don't have information about it. You should test it yourself on your server.
- so it looks like we did well snagging the auction for the last-available, lowest-price with "2x SSD M.2 NVMe 512 GB"
- I checked their server auction page again, and I do still see one server available at their lowest 37.72 EUR/mo price with 2x 512G NVMe disks, so I guess one listing doesn't necessarily mean that there's only one server available.
- after migration, we should end-up with a 25% full disk. If we ~triple our current disk usage before we retire this server, we have the ability to add two more non-NVMe SATA SSD disks in another RAID1, which we can partition-up as-needed for our backups, tmp files, etc (hopefully we can keep www and DB on the faster NVMe disks)
- their hardware page has info on the addon SSD disks that they lease, and their prices https://docs.hetzner.com/robot/dedicated-server/general-information/root-server-hardware/#drives
- it looks like their cheapest non-NVMe SSD is 8.50 EUR/mo for a 1T disk. Of course, we would need two for the RAID, so that means we can 4x our disk space in the future for an additional 17 EUR / mo. They also have a 3.84 TB SATA SSD for 37 EUR/mo, and they have both 16T and 22T SATA HDDs for 20.50 and 27.00 EUR/mo, respectfully.
- we also get a free 100GB "storage box" along with our purchase, which I guess is some external NFS mount. It's probably slow as hell, but we *could* use it for something like our two local copies of encrypted backup data. We also have this as part of our hetzner2 plan, but we don't use it.
- the free 100G storage box is named "BX10". We can even increase this to a BX11 (1T @ 3.20 EUR/mo), BX21 (5T @ 10.90 EUR/mo), BX31 (20.80 EUR/mo), or BX41 (40.60 EUR/mo)
- this is my first time setting-up a hetzner dedicated server (I inherited both hetzner1 & hetzner2)
- I was hoping for something like an in-browser KVM/VNC like SolusVM offers. Or to feed it 'cloudinit' like hetnzer cloud offers, but I don't see that as an option
- looks like they have scripts for installing a few disros. docs here https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/
- and here's the general docs on their dedicated servers https://docs.hetzner.com/robot/dedicated-server/getting-started/root-server-guide/
- ok, they do offer a KVM-over-IP for installing custom distros. Apparently a technician has to physically plug-in to the machine, and you're given 3 free hours. After that it's 8.40 EUR/hr https://docs.hetzner.com/robot/dedicated-server/maintainance/kvm-console/
- the new server is now listed on our "Hetzner Robot" Server Page https://robot.hetzner.com/server
- the old server is listed as "EX41S-SSD #XXXXXX"
- the new server is listed more simply as "Server Auction #XXXXXXX"
- I sent another support request to hetzner asking which type of hardware we have
Hi, I have another question about possible disk upgrades to our newly-purcahsed server "Server Auction #2443019". Can you please tell us what type of server we ordered? Is it an AX? DX? EX? GEX? PX? RX? SX? Or what? I ask because your documentation page on what disk upgrade options are available has a lot of caveats (eg "only for the following servers" or "not available for XYZ"), so to know what our options are I need to know which server type we have. * https://docs.hetzner.com/robot/dedicated-server/general-information/root-server-hardware/#drives Please let us know which server type we have, so that that I can figure out what disk upgrade options are available (and their prices). Thank you,
- I probably should have done this before, but I checked the cloud hetzner offerings
- I didn't check it before because I expected it to be memory bound. I do want to stick with 64G of RAM
- indeed, the cheapest server with 64G of RAM in the cloud is 95.99 EUR/mo — but that also gives us 16 dedicated cores (AMD) and 360 GB disk. And, of course, it's easier to upgrade. But too expensive
- as said above, we *could* probably get-by with 16G of RAM. That's 23.99 EUR/mo with dedicated vCPU. 32G is with dedicated vCPU is 47.99 EU/mo. With shared vCPU, we get 16G RAM for 15.90 EUR/mo or 32G for 31.90GB/mo. But we can't increase the RAM beyond 32GB/mo on the shared vCPU systems
- therefore, I do think going with the dedicated server is our best bet due to the value on the RAM that we get.
- I also asked hetzner sales about possible memory upgrades. I don't think we'll need more than 64G of RAM, but it would be good to know if upgrading is possible
Hi, I have another question about possible memory upgrades to our newly-purcahsed server "Server Auction #2443019". Can you please tell us if our current configuration with "4x RAM 16384 MB DDR4" is the maximum RAM that this system can accept? Does the server only have 4x RAM slots? Or are there some empty ones? Is it possible to increase one or more of the RAM slots with >16G RAM chips? Please let us know what options we have for future memory upgrades on this new dedicated server. Thank you,
- I tried to shut down the hetzner3 server (to eliminate it as a vector until I've hardened it), but there's only an option to reboot :(
- I gave hetzner my ssh public key at order-time, and — yep — it's setup with the root user by default :(
user@ose:~/tmp/ansible$ ssh root@144.76.164.201 Linux rescue 6.9.7 #1 SMP Thu Jun 27 15:07:37 UTC 2024 x86_64 -------------------- Welcome to the Hetzner Rescue System. This Rescue System is based on Debian GNU/Linux 12 (bookworm) with a custom kernel. You can install software like you would in a normal system. To install a new operating system from one of our prebuilt images, run 'installimage' and follow the instructions. Important note: Any data that was not written to the disks will be lost during a reboot. For additional information, check the following resources: Rescue System: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/hetzner-rescue-system Installimage: https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage Install custom software: https://docs.hetzner.com/robot/dedicated-server/operating-systems/installing-custom-images other articles: https://docs.hetzner.com/robot -------------------- Rescue System (via Legacy/CSM) up since 2024-07-31 09:16 +02:00 Hardware data: CPU1: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz (Cores 8) Memory: 64099 MB Disk /dev/nvme0n1: 512 GB (=> 476 GiB) doesn't contain a valid partition table Disk /dev/nvme1n1: 512 GB (=> 476 GiB) doesn't contain a valid partition table Total capacity 953 GiB with 2 Disks Network data: eth0 LINK: yes MAC: 90:1b:0e:c4:28:b4 IP: 144.76.164.201 IPv6: 2a01:4f8:200:40d7::2/64 Intel(R) PRO/1000 Network Driver root@rescue ~ #
- here's our disks info. So already only 476.9G. It'll be a bit less after we put a filesystem on it, I'm sure
root@rescue ~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 3.1G 1 loop nvme0n1 259:0 0 476.9G 0 disk nvme1n1 259:1 0 476.9G 0 disk root@rescue ~ #
- I'm reading through the guide on hetzner's `installimage` tool https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/
- the guide suggests a couple commands to see if we have a hardware raid. I didn't think we did, and this appears to confirm it
root@rescue ~ # megacli -LDInfo -Lall -Aall
Exit Code: 0x00
root@rescue ~ #
root@rescue ~ # arcconf GETCONFIG 1 LD
Controllers found: 0
Invalid controller number.
root@rescue ~ #
- it appears we don't have any software RAIDs already setup either
root@rescue ~ # ls /dev/md* ls: cannot access '/dev/md*': No such file or directory root@rescue ~ #
- quick disk tests show that we're getting 2 Gb/s disk read. shit, that's slower than the advertised "6 Gb/s" on our prod server (though admittedly I never tested this)
root@rescue ~ # hdparm -Ttv /dev/nvme0n1 /dev/nvme0n1: readonly = 0 (off) readahead = 256 (on) geometry = 488386/64/32, sectors = 1000215216, start = 0 Timing cached reads: 35298 MB in 1.97 seconds = 17911.22 MB/sec Timing buffered disk reads: 6992 MB in 3.00 seconds = 2330.11 MB/sec root@rescue ~ #
- ok, I ran installimage
root@rescue ~ # installimage
- I selected "Debian"
- I selected "Debian-1205-bookworm-amd64-base"
- it dumped me into a midnight commander editor and said I could save with F10
- it said that "by default all disks are used for software raid" — that sounds good
- the "standard config" file that it gave me was too long to try to copy & paste, but the default RAID looked like what we wanted
- the default partition layout had a 32G swap, 1G /boot, and the rest allocated to '/'
- hetzner2 has a 488M /boot that's currently 84% full (using 386M). 386/1024 = 38% full, which is much better. That sounds good.
- hetzner2 has a 32G swap. It does get used, but it's currently using <1G. 32G should be fine.
- I thought about setting up an LVM. It would be a better idea than having everything on one disk, but it would inevitable require more maintenance. For the sake of keeping things simple for a non-profit that has no sysadmins on staff, I'm going to stick to "just allocate the rest to '/'"
- oh, cool, the config file said what our disks are (if it can be trusted). It says: SAMSUNG MZVLB512HAJQ
- looks like they're from 2017 https://ssd.userbenchmark.com/SpeedTest/401452/SAMSUNG-MZVLB512HAJQ-000L2
- samsung advertises them as having 3.5 Gbps sequental read + 2.9 Gbps sequential write + 460K random read iops + 500k random write iops
- I decided to accept all these defaults and proceed with the install
- oh, except I did change the hostname line
- ah shit, how do I send an F10 command over ssh? stupid midnight commander editor...
- I pressed Ctrl+[[ and that seemed to work
- the install finished in a few minutes
Hetzner Online GmbH - installimage Your server will be installed now, this will take some minutes You can abort at any time with CTRL+C ... : Reading configuration done : Loading image file variables done : Loading debian specific functions done 1/16 : Deleting partitions done 2/16 : Test partition size done 3/16 : Creating partitions and /etc/fstab done 4/16 : Creating software RAID level 1 done 5/16 : Formatting partitions : formatting /dev/md/0 with swap done : formatting /dev/md/1 with ext3 done : formatting /dev/md/2 with ext4 done 6/16 : Mounting partitions done 7/16 : Sync time via ntp done : Importing public key for image validation done 8/16 : Validating image before starting extraction done 9/16 : Extracting image (local) done 10/16 : Setting up network config done 11/16 : Executing additional commands : Setting hostname done : Generating new SSH keys done : Generating mdadm config done : Generating ramdisk done : Generating ntp config done 12/16 : Setting up miscellaneous files done 13/16 : Configuring authentication : Fetching SSH keys done : Disabling root password done : Disabling SSH root login with password done : Copying SSH keys done 14/16 : Installing bootloader grub done 15/16 : Running some debian specific functions done 16/16 : Clearing log files done INSTALLATION COMPLETE You can now reboot and log in to your new system with the same credentials that you used to log into the rescue system. root@rescue ~ #
- I have to say that I was happy that it generated new ssh keys and that it said it was verifying some public key and doing some image verification.
- oh, and they disabled root password and ssh login with password. that's better than I expected from them. good.
- I ran `shutdown -h now`, and the server didn't come back. That's actually good. I wanted to see if I could shut the thing down (the safest machine is a machine that's off, especially before hardening), but the hetzner robot WUI didn't give an option to shutdown (only to reboot).
- after waiting 5 minutes with no pongs to my pings, I logged into the hetzner robot wui -> server -> reset tab -> Execute an automatic hardware reset, and I clicked "Send" https://robot.hetzner.com/server
- after a few more minutes with still no pongs to my pings, I logged into the hetzner robot wui -> server -> WOL tab -> clicked "Send WOL signal to server"
- after about 2 minutes, I started getting pings and was able to ssh-in as the 'root' user
- as soon as I got a shell from ssh, I quickly pasted-in my "jumpstart" provisioning and hardening commands to create a user for me, do basic ssh hardening, and setup a basic firewall to block everything except ssh
adduser maltfield --disabled-password --gecos '' groupadd sshaccess gpasswd -a maltfield sshaccess mkdir /home/maltfield/.ssh/ echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGNYjR7UKiJSAG/AbP+vlCBqNfQZ2yuSXfsEDuM7cEU8PQNJyuJnS7m0VcA48JRnpUpPYYCCB0fqtIEhpP+szpMg2LByfTtbU0vDBjzQD9mEfwZ0mzJsfzh1Nxe86l/d6h6FhxAqK+eG7ljYBElDhF4l2lgcMAl9TiSba0pcqqYBRsvJgQoAjlZOIeVEvM1lyfWfrmDaFK37jdUCBWq8QeJ98qpNDX4A76f9T5Y3q5EuSFkY0fcU+zwFxM71bGGlgmo5YsMMdSsW+89fSG0652/U4sjf4NTHCpuD0UaSPB876NJ7QzeDWtOgyBC4nhPpS8pgjsnl48QZuVm6FNDqbXr9bVk5BdntpBgps+gXdSL2j0/yRRayLXzps1LCdasMCBxCzK+lJYWGalw5dNaIDHBsEZiK55iwPp0W3lU9vXFO4oKNJGFgbhNmn+KAaW82NBwlTHo/tOlj2/VQD9uaK5YLhQqAJzIq0JuWZWFLUC2FJIIG0pJBIonNabANcN+vq+YJqjd+JXNZyTZ0mzuj3OAB/Z5zS6lT9azPfnEjpcOngFs46P7S/1hRIrSWCvZ8kfECpa8W+cTMus4rpCd40d1tVKzJA/n0MGJjEs2q4cK6lC08pXxq9zAyt7PMl94PHse2uzDFhrhh7d0ManxNZE+I5/IPWOnG1PJsDlOe4Yqw== maltfield@ose" > /home/maltfield/.ssh/authorized_keys chown -R maltfield:maltfield /home/maltfield/.ssh chmod -R 0600 /home/maltfield/.ssh chmod 0700 /home/maltfield/.ssh # without this, apt-get may get stuck export DEBIAN_FRONTEND=noninteractive apt-get update apt-get -y install iptables iptables-persistent apt-get -y purge nftables update-alternatives --set iptables /usr/sbin/iptables-legacy update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy update-alternatives --set arptables /usr/sbin/arptables-legacy update-alternatives --set ebtables /usr/sbin/ebtables-legacy iptables -A INPUT -i lo -j ACCEPT iptables -A INPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j DROP iptables -A INPUT -p icmp -j ACCEPT iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 32415 -j ACCEPT iptables -A INPUT -j DROP iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT iptables -A OUTPUT -m owner --uid-owner 0 -j ACCEPT iptables -A OUTPUT -m owner --uid-owner 42 -j ACCEPT iptables -A OUTPUT -m owner --uid-owner 1000 -j ACCEPT iptables -A OUTPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7 iptables -A OUTPUT -j DROP ip6tables -A INPUT -i lo -j ACCEPT ip6tables -A INPUT -s ::1/128 -d ::1/128 -j DROP ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ip6tables -A INPUT -j DROP ip6tables -A OUTPUT -s ::1/128 -d ::1/128 -j ACCEPT ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT ip6tables -A OUTPUT -m owner --uid-owner 0 -j ACCEPT ip6tables -A OUTPUT -m owner --uid-owner 42 -j ACCEPT ip6tables -A OUTPUT -m owner --uid-owner 1000 -j ACCEPT ip6tables -A OUTPUT -j DROP iptables-save > /etc/iptables/rules.v4 ip6tables-save > /etc/iptables/rules.v6 cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig.`date "+%Y%m%d_%H%M%S"` grep 'Port 32415' /etc/ssh/sshd_config || echo 'Port 32415' >> /etc/ssh/sshd_config grep 'AllowGroups sshaccess' /etc/ssh/sshd_config || echo 'AllowGroups sshaccess' >> /etc/ssh/sshd_config grep 'PermitRootLogin no' /etc/ssh/sshd_config || echo 'PermitRootLogin no' >> /etc/ssh/sshd_config grep 'PasswordAuthentication no' /etc/ssh/sshd_config || echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config systemctl restart sshd.service apt-get -y upgrade
- I added this new entry to my ose VM's /home/user/.ssh/config
Host hetzner3 Hostname 144.76.164.201 Port 32415 ForwardAgent yes User maltfield
- I then gave my user sudo permission
root@mail ~ # cp /etc/sudoers /etc/sudoers.20240731.orig root@mail ~ # root@mail ~ # visudo root@mail ~ # root@mail ~ # diff /etc/sudoers.20240731.orig /etc/sudoers 47a48 > maltfield ALL=(ALL:ALL) NOPASSWD:ALL root@mail ~ #
- alright, basic hardening is done.
- That's all I really wanted to achieve for now. Next I'd like to prepare some ansible playbooks to setup the rest of the basic hardening
- for now, I just want to leave this machine off in the meantime
- I attempted to shut it down again
- I left a ping open for 147 minutes, and I never got a pong back. So I'd say it's off. Great!
- it appears that I just have to trigger a WOL on hetzner robot WUI to turn it back on, which I'll do after I spend some time working on the ansible roles playbooks
Tue July 30, 2024
- Marcin gave me the go-ahead to order a "hetzner3" server and begin provisioning it with Debian in preparation to migrate all our sites from the CentOS7 hetzner2 server to this new server
- This is going to be an enormous project. When I did the hetzner1 -> hetzner2 migration, I inherited both systems (in 2017). For some reason the websites were split across both servers (plus dreamhost too iirc?). but I consolidated everything onto "hetzner2" and canceled "hetzner1" in 2018 https://wiki.opensourceecology.org/index.php?title=OSE_Server&oldid=298909#Assessment_of_Server_Options
- I'll be using ansible to assist in provisioning this server (and hopefully make it easier to provision future servers). Marcin expressed interest in lowering this barrier for others, as well.
- I noticed that 5 years ago I created a repo for OSE's ansible playbooks, but it's empty.
- I just added the LICENSE to this repo, and I plan to use it to publish our ansible roles/playbooks
- First thing I need to do is decide which server to buy from hetzner's dedicated serer offerings
- holy crap, not only are their server auctions *much* cheeper per month, they also don't have a one-time-setup fee (usually ~$50-$200?)
- I've written pretty extensively in the past about what specs I'd be looking to get in a future OSE Server migration https://wiki.opensourceecology.org/index.php?title=OSE_Server&oldid=298909#OSE_Server_and_Server_Requirements
- In 2018, I said we'd want min 2-4 cores
- In 2018, I said we'd want min 8-16 G RAM
- In 2018, I said we'd want min ~200G disk
- Honestly, I expect that the lowest offerings of a dedicated server in 2024 are probably going to suffice for us, but what I'm mostly concerned-about is the disk.
- even last week when I did the yum updates, I nearly filled the disk just by extracting a copy of our backups. Currently we have two 250G disks in a software RAID-1 (mirror) array. That give us a useable 197G
- it's also provisioned with all the data on '/'. It would be smart if we setup an LVM
- It's important to me that we double this at-least, but I'll see if there's any deals on 1TB disks or larger
- also what we currently have is a 6 Gb/s SSD, so I don't want to downgrade that by going to a spinning-disk HDD. NvME might be a welcome upgrade. I/O wait is probably a bottleneck, but not currently one that's causing us agony
- I spent some time reviewing the munin graphs
- load rarely ever touches 3. Most of the time it hovers between 0.2 - 1. So I agree that 4 cores is fine for us now.
- most of these auctions have a Intel Core i7-4770, which is a 4-core + 8 thread proc. That should be fine.
- somehow our varnish hits are way down. They used to average >80%, but currently they're down to 28-44%
- load rarely ever touches 3. Most of the time it hovers between 0.2 - 1. So I agree that 4 cores is fine for us now.
- I documented these charts and my findings on a new Hetzner3 page
- I looked through the listings in the server auctions
- I don't want one that's only 32G RAM (few of these are)
- It looks like some have "2 x SSD SATA 250 GB" and some have "2 x SSD M.2 NVMe 512 GB". If we can, let's get the NVMe disks with better io
- there is one with "2 x HDD SATA 2,0 TB Enterprise". More space would be nice, but not at the sacrifice of io
- questions I have for hetzner:
- how many disk slots are there? Can we add more disks in the future?
- by default, do all these systems have RAID-1? Do we have other RAID options?
- oh, actually, there was only one server available for less than 38 EUR/mo that had the 2x 512GB NVME
- I went ahead and ordered it
- I also sent a separate message to hetzner sales asking them for detailed info about the different read & write speeds of their HDD, SSD, and NVME offerings in dedicated servers
- I sent an email to Marcin
Hey Marcin, I just ordered a dedicated server from Hetzner with the following specs: * Intel Core i7-6700 * 2x SSD M.2 NVMe 512 GB * 4x RAM 16384 MB DDR4 * NIC 1 Gbit Intel I219-LM * Location: Germany, FSN1 * Rescue system (English) * 1 x Primary IPv4 While they had plenty of servers available with the i7-6700 and 16G of RAM, they only had one with 2x 512 GB NVMe disks (the others were just "SSD" disks). Those NVMe disks should give us a performance boost, so I snagged it while it was available. I did some reviews of our munin charts to determine our hetzner3 server's needs. For more info, see * https://wiki.opensourceecology.org/wiki/Hetzner3 Please let me know if you have any questions about this server. Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
Fri July 26, 2024
- I started the CHG-2024-07-26_yum_update today at 11:00
- pre-state proof shows we have lots of outdated system packages, as expected
[root@opensourceecology ~]# yum list updates ... xz-libs.x86_64 5.2.2-2.el7_9 updates yum.noarch 3.4.3-168.el7.centos base yum-cron.noarch 3.4.3-168.el7.centos base yum-plugin-fastestmirror.noarch 1.1.31-54.el7_8 base yum-utils.noarch 1.1.31-54.el7_8 base zlib.x86_64 1.2.7-21.el7_9 updates [root@opensourceecology ~]#
- I tried to check the backups log, but it was empty :/
[root@opensourceecology ~]# cat /var/log/backups/backup.log [root@opensourceecology ~]#
- ok, looks like it rotated already; this file shows a 20.424G backup file successfully uploaded to backblaze with rclone
[root@opensourceecology ~]# ls /var/log/backups/ backup.lo backup.log-20240628.gz backup.log-20240714.gz backup.log backup.log-20240629.gz backup.log-20240715.gz backup.log-20240615.gz backup.log-20240701.gz backup.log-20240716.gz backup.log-20240616.gz backup.log-20240702.gz backup.log-20240718.gz backup.log-20240617.gz backup.log-20240704.gz backup.log-20240719.gz backup.log-20240619.gz backup.log-20240706.gz backup.log-20240721.gz backup.log-20240621.gz backup.log-20240707.gz backup.log-20240722.gz backup.log-20240622.gz backup.log-20240708.gz backup.log-20240724.gz backup.log-20240623.gz backup.log-20240709.gz backup.log-20240725.gz backup.log-20240625.gz backup.log-20240711.gz backup.log-20240726 backup.log-20240626.gz backup.log-20240712.gz backup.log-20240627.gz backup.log-20240713.gz [root@opensourceecology ~]# [root@opensourceecology ~]# tail -n20 /var/log/backups/backup.log-20240726 * daily_hetzner2_20240726_072001.tar.gpg:100% /20.424G, 2.935M/s, - 2024/07/26 09:50:31 INFO : daily_hetzner2_20240726_072001.tar.gpg: Copied (new) 2024/07/26 09:50:31 INFO : Transferred: 20.424G / 20.424 GBytes, 100%, 2.979 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1h57m0.8s real 117m1.219s user 4m20.240s sys 2m9.432s + echo ================================================================================ ================================================================================ ++ date -u +%Y%m%d_%H%M%S + echo 'INFO: Finished Backup Run at 20240726_095031' INFO: Finished Backup Run at 20240726_095031 + echo ================================================================================ ================================================================================ + exit 0 [root@opensourceecology ~]#
- the query of b2 backup files also looks good
[root@opensourceecology ~]# sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups | grep `date "+%Y%m%d"` daily_hetzner2_20240726_072001.tar.gpg [root@opensourceecology ~]# date -u Fri Jul 26 16:03:55 UTC 2024 [root@opensourceecology ~]# sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups daily_hetzner2_20240724_072001.tar.gpg daily_hetzner2_20240725_072001.tar.gpg daily_hetzner2_20240726_072001.tar.gpg monthly_hetzner2_20230801_072001.tar.gpg monthly_hetzner2_20230901_072001.tar.gpg monthly_hetzner2_20231001_072001.tar.gpg monthly_hetzner2_20231101_072001.tar.gpg monthly_hetzner2_20231201_072001.tar.gpg monthly_hetzner2_20240201_072001.tar.gpg monthly_hetzner2_20240301_072001.tar.gpg monthly_hetzner2_20240401_072001.tar.gpg monthly_hetzner2_20240501_072001.tar.gpg monthly_hetzner2_20240601_072001.tar.gpg monthly_hetzner2_20240701_072001.tar.gpg weekly_hetzner2_20240708_072001.tar.gpg weekly_hetzner2_20240715_072001.tar.gpg weekly_hetzner2_20240722_072001.tar.gpg yearly_hetzner2_20190101_111520.tar.gpg yearly_hetzner2_20200101_072001.tar.gpg yearly_hetzner2_20210101_072001.tar.gpg yearly_hetzner2_20230101_072001.tar.gpg yearly_hetzner2_20240101_072001.tar.gpg [root@opensourceecology ~]#
- that backup is already 8 hours old; so let's bring down the webserver + stop the databases and take a real fresh backup before we do anything
- stopped nginx
[root@opensourceecology ~]# # create dir for logging the change [root@opensourceecology ~]# tmpDir="/var/tmp/CHG-2024-07-26_yum_update" [root@opensourceecology ~]# mkdir -p $tmpDir [root@opensourceecology ~]# [root@opensourceecology ~]# # begin to gracefully shutdown nginx in the background [root@opensourceecology ~]# time nice /sbin/nginx -s quit nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.openbuildinginstitute.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.openbuildinginstitute.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.openbuildinginstitute.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.openbuildinginstitute.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 real 0m0.078s user 0m0.038s sys 0m0.021s [root@opensourceecology ~]# [root@opensourceecology ~]# date -u Fri Jul 26 16:06:37 UTC 2024 [root@opensourceecology ~]#
- stopped DBs
[root@opensourceecology ~]# systemctl status mariadb ● mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2024-07-22 18:55:28 UTC; 3 days ago Process: 1230 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS) Process: 1099 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 1229 (mysqld_safe) CGroup: /system.slice/mariadb.service ├─1229 /bin/sh /usr/bin/mysqld_safe --basedir=/usr └─1704 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql ... Jul 22 18:55:25 opensourceecology.org systemd[1]: Starting MariaDB database s.... Jul 22 18:55:26 opensourceecology.org mariadb-prepare-db-dir[1099]: Database M... Jul 22 18:55:26 opensourceecology.org mariadb-prepare-db-dir[1099]: If this is... Jul 22 18:55:26 opensourceecology.org mysqld_safe[1229]: 240722 18:55:26 mysql... Jul 22 18:55:26 opensourceecology.org mysqld_safe[1229]: 240722 18:55:26 mysql... Jul 22 18:55:28 opensourceecology.org systemd[1]: Started MariaDB database se.... Hint: Some lines were ellipsized, use -l to show in full. [root@opensourceecology ~]# systemctl stop mariadb [root@opensourceecology ~]# systemctl status mariadb ● mariadb.service - MariaDB database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Active: inactive (dead) since Fri 2024-07-26 16:07:43 UTC; 3s ago Process: 1230 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=0/SUCCESS) Process: 1229 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS) Process: 1099 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS) Main PID: 1229 (code=exited, status=0/SUCCESS) Jul 22 18:55:25 opensourceecology.org systemd[1]: Starting MariaDB database s.... Jul 22 18:55:26 opensourceecology.org mariadb-prepare-db-dir[1099]: Database M... Jul 22 18:55:26 opensourceecology.org mariadb-prepare-db-dir[1099]: If this is... Jul 22 18:55:26 opensourceecology.org mysqld_safe[1229]: 240722 18:55:26 mysql... Jul 22 18:55:26 opensourceecology.org mysqld_safe[1229]: 240722 18:55:26 mysql... Jul 22 18:55:28 opensourceecology.org systemd[1]: Started MariaDB database se.... Jul 26 16:07:40 opensourceecology.org systemd[1]: Stopping MariaDB database s.... Jul 26 16:07:43 opensourceecology.org systemd[1]: Stopped MariaDB database se.... Hint: Some lines were ellipsized, use -l to show in full. [root@opensourceecology ~]#
- the backup is taking a long time. while I wait, I checked `top`, and I see `gzip` is using 80%-100%
- so it seems that gzip is bound by a single core. it could go much faster if it could be split across multiple cores (parallel processing)
- quick googling while I wait suggests that we could use `pigz` as a replacement to `gzip` to get this (admittedly low priority) performance boost https://stackoverflow.com/questions/12313242/utilizing-multi-core-for-targzip-bzip-compression-decompression
- there's other options too. apparently xz has native multi-treadded support since v5.2.0 https://askubuntu.com/a/858828
- there's also pbzip2 for bzip, and many others https://askubuntu.com/a/258228
- the other two commands that get stuck on one-core are `tar` and `gpg2`
- it looks like gpg also attempts to compress with xz. That gives us no benefits in our case because we're encrypting a tarball that just contains a bunch of already-compressed tarballs. So we could probably get some performance improvements by telling gpg to skip the xz compression with `--compression-algo none` https://stackoverflow.com/questions/46261024/how-to-do-large-file-parallel-encryption-using-gnupg-and-gnu-parallel
- finally (after ~30 min to generate the encrypted backup file), rclone is using >100% of CPU to upload it, so that's good. Our script does limit upload to 3 MB/s. I guess one improvement would be some argument to bypass that throttle
- it said the upload was going to take just under 2 hours, so I canceled it and manually ran the upload command (minus the throttle)
- upload speeds are now ~27-32 MB/s (so ~10x faster). It says it'll finish in just over 10 minutes.
- upload is done
[root@opensourceecology ~]# time sudo /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log ^C real 33m47.250s user 23m56.551s sys 2m2.866s [root@opensourceecology ~]# [root@opensourceecology ~]# /bin/sudo -u b2user /bin/rclone -v copy /home/b2user/sync/daily_hetzner2_20240726_160837.tar.gpg b2:ose-server-backups ... 2024/07/26 16:56:38 INFO : Transferred: 18.440G / 19.206 GBytes, 96%, 22.492 MBytes/s, ETA 34s Transferred: 0 / 1, 0% Elapsed time: 14m0.5s Transferring: * daily_hetzner2_20240726_160837.tar.gpg: 96% /19.206G, 21.268M/s, 36s 2024/07/26 16:57:36 INFO : daily_hetzner2_20240726_160837.tar.gpg: Copied (new) 2024/07/26 16:57:36 INFO : Transferred: 19.206G / 19.206 GBytes, 100%, 21.910 MBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 14m58.6s [root@opensourceecology ~]#
- ok, this very durable backup is uploaded; let's proceed
[root@opensourceecology ~]# sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups | grep `date "+%Y%m%d"` daily_hetzner2_20240726_072001.tar.gpg daily_hetzner2_20240726_160837.tar.gpg [root@opensourceecology ~]# date -u Fri Jul 26 16:58:11 UTC 2024 [root@opensourceecology ~]# sudo -u b2user /home/b2user/virtualenv/bin/b2 ls ose-server-backups daily_hetzner2_20240724_072001.tar.gpg daily_hetzner2_20240725_072001.tar.gpg daily_hetzner2_20240726_072001.tar.gpg daily_hetzner2_20240726_160837.tar.gpg monthly_hetzner2_20230801_072001.tar.gpg monthly_hetzner2_20230901_072001.tar.gpg monthly_hetzner2_20231001_072001.tar.gpg monthly_hetzner2_20231101_072001.tar.gpg monthly_hetzner2_20231201_072001.tar.gpg monthly_hetzner2_20240201_072001.tar.gpg monthly_hetzner2_20240301_072001.tar.gpg monthly_hetzner2_20240401_072001.tar.gpg monthly_hetzner2_20240501_072001.tar.gpg monthly_hetzner2_20240601_072001.tar.gpg monthly_hetzner2_20240701_072001.tar.gpg weekly_hetzner2_20240708_072001.tar.gpg weekly_hetzner2_20240715_072001.tar.gpg weekly_hetzner2_20240722_072001.tar.gpg yearly_hetzner2_20190101_111520.tar.gpg yearly_hetzner2_20200101_072001.tar.gpg yearly_hetzner2_20210101_072001.tar.gpg yearly_hetzner2_20230101_072001.tar.gpg yearly_hetzner2_20240101_072001.tar.gpg [root@opensourceecology ~]#
- we have a snapshot of the current state of packages
[root@opensourceecology ~]# time nice rpm -qa &> "${tmpDir}/before.log"
real 0m0.716s
user 0m0.678s
sys 0m0.037s
[root@opensourceecology ~]#
[root@opensourceecology ~]# echo $tmpDir
/var/tmp/CHG-2024-07-26_yum_update
[root@opensourceecology ~]#
[root@opensourceecology ~]# tail /var/tmp/CHG-2024-07-26_yum_update/before.log
libdb-utils-5.3.21-25.el7.x86_64
libuser-0.60-9.el7.x86_64
python-lxml-3.2.1-4.el7.x86_64
net-snmp-agent-libs-5.7.2-48.el7_8.x86_64
epel-release-7-14.noarch
perl-parent-0.225-244.el7.noarch
libstdc++-devel-4.8.5-39.el7.x86_64
libsodium13-1.0.5-1.el7.x86_64
ncurses-5.9-14.20130511.el7_4.x86_64
e2fsprogs-libs-1.42.9-17.el7.x86_64
[root@opensourceecology ~]#
- I kicked-off the updates. I got a bit of a freight at first when we got "404 Not Found" errors from 484 mirrors, but eventually `yum` found a server. I'm glad we did the updates now, before all the mirrors shutdown (centos was EOL some years ago, and will no longer be getting maintenance updates as of a few weeks ago)
[root@opensourceecology ~]# grep "Error 404" /var/tmp/CHG-2024-07-26_yum_update/update.log | wc -l 484 [root@opensourceecology ~]# [root@opensourceecology ~]# cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) [root@opensourceecology ~]#
- actually, it says it's updating 434 packages total. So I guess some dependencies got added to the 200-odd count before
- ok, the update command finished in just under 4 minutes of wall time
... real 3m56.410s user 2m1.833s sys 0m44.510s [root@opensourceecology ~]#
- post update info
[root@opensourceecology ~]# # log the post-state packages and versions
[root@opensourceecology ~]# time nice rpm -qa &> "${tmpDir}/after.log"
real 0m0.805s
user 0m0.769s
sys 0m0.036s
[root@opensourceecology ~]#
[root@opensourceecology ~]# time nice needs-restarting &> "${tmpDir}/needs-restarting.log"
real 0m8.156s
user 0m6.956s
sys 0m0.652s
[root@opensourceecology ~]# time nice needs-restarting -r &> "${tmpDir}/needs-reboot.log"
real 0m0.155s
user 0m0.104s
sys 0m0.051s
[root@opensourceecology ~]#
[root@opensourceecology ~]# cat /var/tmp/CHG-2024-07-26_yum_update/needs-reboot.log
Core libraries or services have been updated:
systemd -> 219-78.el7_9.9
dbus -> 1:1.10.24-15.el7
openssl-libs -> 1:1.0.2k-26.el7_9
linux-firmware -> 20200421-83.git78c0348.el7_9
kernel -> 3.10.0-1160.119.1.el7
glibc -> 2.17-326.el7_9.3
Reboot is required to ensure that your system benefits from these updates.
More information:
https://access.redhat.com/solutions/27943
[root@opensourceecology ~]#
[root@opensourceecology ~]# cat /var/tmp/CHG-2024-07-26_yum_update/needs-restarting.log
30842 : /usr/lib/systemd/systemd-udevd
13696 : sshd: maltfield@pts/0
27401 : /bin/bash
744 : /sbin/auditd
19086 : /bin/bash
13692 : sshd: maltfield [priv]
30672 : smtpd -n smtp -t inet -u
13699 : -bash
18035 : su -
27436 : less /root/backups/backup.sh
18036 : -bash
18030 : sudo su -
1484 : /var/ossec/bin/ossec-analysisd
24493 : /bin/bash
21581 : su -
21580 : sudo su -
21582 : -bash
797 : /usr/lib/systemd/systemd-logind
24476 : /bin/bash
1830 : qmgr -l -t unix -u
30673 : proxymap -t unix -u
19119 : sudo su -
24511 : /bin/bash
29833 : local -t unix
27417 : sudo su -
19130 : -bash
1 : /usr/lib/systemd/systemd --system --deserialize 23
29830 : cleanup -z -t unix -u
1500 : /var/ossec/bin/ossec-logcollector
24475 : SCREEN -S upgrade
2150 : /usr/sbin/varnishd -P /var/run/varnish.pid -f /etc/varnish/default.vcl -a 127.0.0.1:6081 -T 127.0.0.1:6082 -S /etc/varnish/secret -u varnish -g varnish -s malloc,40G
2152 : /usr/sbin/varnishd -P /var/run/varnish.pid -f /etc/varnish/default.vcl -a 127.0.0.1:6081 -T 127.0.0.1:6082 -S /etc/varnish/secret -u varnish -g varnish -s malloc,40G
29835 : bounce -z -t unix -u
775 : /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
27419 : -bash
585 : /usr/lib/systemd/systemd-journald
771 : /usr/sbin/irqbalance --foreground
770 : /usr/sbin/acpid
1170 : /sbin/agetty --noclear tty1 linux
30690 : smtp -t unix -u
778 : /usr/sbin/chronyd
8695 : gpg-agent --daemon --use-standard-socket
24529 : /bin/bash
2121 : /var/ossec/bin/ossec-syscheckd
1806 : /usr/libexec/postfix/master -w
19129 : su -
19065 : /bin/bash
2124 : /var/ossec/bin/ossec-monitord
29832 : trivial-rewrite -n rewrite -t unix -u
19044 : /bin/bash
30693 : smtp -t unix -u
30692 : smtp -t unix -u
30691 : cleanup -z -t unix -u
27418 : su -
1475 : /var/ossec/bin/ossec-execd
19025 : /bin/bash
19024 : SCREEN -S CHG-2024-07-26_yum_update
1458 : /var/ossec/bin/ossec-maild
19023 : screen -S CHG-2024-07-26_yum_update
[root@opensourceecology ~]#
- alright, time to reboot
[root@opensourceecology ~]# reboot Connection to opensourceecology.org closed by remote host. Connection to opensourceecology.org closed. user@ose:~$
- system came back in about 1 minute
- first attempt to load the wiki resulted in a 503 "Error 503 Backend fetch failed" from varnish
- it's not just warming-up; apache didn't come-up on start
[root@opensourceecology ~]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2024-07-26 17:09:47 UTC; 2min 7s ago Docs: man:httpd(8) man:apachectl(8) Process: 1094 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) Main PID: 1094 (code=exited, status=1/FAILURE) Jul 26 17:09:47 opensourceecology.org systemd[1]: Starting The Apache HTTP Se.... Jul 26 17:09:47 opensourceecology.org httpd[1094]: (98)Address already in use:... Jul 26 17:09:47 opensourceecology.org httpd[1094]: (98)Address already in use:... Jul 26 17:09:47 opensourceecology.org httpd[1094]: no listening sockets availa... Jul 26 17:09:47 opensourceecology.org httpd[1094]: AH00015: Unable to open logs Jul 26 17:09:47 opensourceecology.org systemd[1]: httpd.service: main process...E Jul 26 17:09:47 opensourceecology.org systemd[1]: Failed to start The Apache .... Jul 26 17:09:47 opensourceecology.org systemd[1]: Unit httpd.service entered .... Jul 26 17:09:47 opensourceecology.org systemd[1]: httpd.service failed. Hint: Some lines were ellipsized, use -l to show in full. [root@opensourceecology ~]#
- it says that the port is already in-use
[root@opensourceecology ~]# journalctl -u httpd --no-pager -- Logs begin at Fri 2024-07-26 17:09:34 UTC, end at Fri 2024-07-26 17:15:26 UTC. -- Jul 26 17:09:47 opensourceecology.org systemd[1]: Starting The Apache HTTP Server... Jul 26 17:09:47 opensourceecology.org httpd[1094]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:443 Jul 26 17:09:47 opensourceecology.org httpd[1094]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443 Jul 26 17:09:47 opensourceecology.org httpd[1094]: no listening sockets available, shutting down Jul 26 17:09:47 opensourceecology.org httpd[1094]: AH00015: Unable to open logs Jul 26 17:09:47 opensourceecology.org systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE Jul 26 17:09:47 opensourceecology.org systemd[1]: Failed to start The Apache HTTP Server. Jul 26 17:09:47 opensourceecology.org systemd[1]: Unit httpd.service entered failed state. Jul 26 17:09:47 opensourceecology.org systemd[1]: httpd.service failed. [root@opensourceecology ~]#
- before I start making changes, I'm going to initiate another backup (and wait at-least 30 minutes for the tar to finish)
- I'm going to want to diff the apache configs, so I made a copy of the backup that I made just before the updates and copied it into the temp CHG dir
[root@opensourceecology CHG-2024-07-26_yum_update]# mkdir backup_before [root@opensourceecology CHG-2024-07-26_yum_update]# rsync -av --progress /home/b2user/sync.old/daily_hetzner2_20240726_160837.tar.gpg backup_before/ sending incremental file list daily_hetzner2_20240726_160837.tar.gpg 20,622,312,871 100% 127.14MB/s 0:02:34 (xfr#1, to-chk=0/1) sent 20,627,347,744 bytes received 35 bytes 133,510,341.61 bytes/sec total size is 20,622,312,871 speedup is 1.00 [root@opensourceecology CHG-2024-07-26_yum_update]#
- well, unfortunately the wiki being down means I can't reference our docs on how to restore backups, but I managed to figure it out
[root@opensourceecology backup_before]# gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --decrypt daily_hetzner2_20240726_160837.tar.gpg > daily_hetzner2_20240726_160837.tar gpg: AES256 encrypted data gpg: encrypted with 1 passphrase [root@opensourceecology backup_before]# [root@opensourceecology backup_before]# du -sh * 20G daily_hetzner2_20240726_160837.tar 20G daily_hetzner2_20240726_160837.tar.gpg [root@opensourceecology backup_before]# [root@opensourceecology backup_before]# du -sh * 20G daily_hetzner2_20240726_160837.tar 20G daily_hetzner2_20240726_160837.tar.gpg [root@opensourceecology backup_before]# [root@opensourceecology backup_before]# tar -xf daily_hetzner2_20240726_160837.tar [root@opensourceecology backup_before]# [root@opensourceecology backup_before]# du -sh * 20G daily_hetzner2_20240726_160837.tar 20G daily_hetzner2_20240726_160837.tar.gpg 20G root [root@opensourceecology backup_before]# rm -f daily_hetzner2_20240726_160837.tar.gpg [root@opensourceecology backup_before]# [root@opensourceecology backup_before]# du -sh * 20G daily_hetzner2_20240726_160837.tar 20G root [root@opensourceecology backup_before]#
- to make this easier for the next person, I created a README directly in the backups dir
[root@opensourceecology backups]# cat /root/backups/README.txt 2024-07-26 ========== The process to restore from backups is documented on the wiki * https://wiki.opensourceecology.org/wiki/Backblaze#Restore_from_backups Oh, the wiki is down and you need to restore from backups to restore the wiki? Don't worry, I got you. All backups are stored on Backblaze B2. You can download them with rclone or just by logging into the Backblaze B2 WUI. First decrypt the main wrapper tar with `gpg` gpg --batch --passphrase-file <path-to-symmetric-encrypton-private-key> --decrypt <path-to-encrypted-tarball> > <path-to-decrypted-tarball> For example: gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --decrypt daily_hetzner2_20240726_160837.tar.gpg > daily_hetzner2_20240726_160837.tar Then you can untar the wrapper tarball and the compressed tarball inside of that. For example: tar -xf daily_hetzner2_20240726_160837.tar cd root/backups/sync/daily_hetzner2_20240726_160837/www/ tar -xf www.20240726_160837.tar.gz head var/www/html/www.opensourceecology.org/htdocs/index.php --Michael Altfield <https://michaelaltfield.net.> [root@opensourceecology backups]#
- and I was able to extract the www files from the backups prior to the update
[root@opensourceecology backup_before]# cd root/backups/sync/daily_hetzner2_20240726_160837/www/ [root@opensourceecology www]# [root@opensourceecology www]# ls www.20240726_160837.tar.gz [root@opensourceecology www]# [root@opensourceecology www]# tar -xf www.20240726_160837.tar.gz [root@opensourceecology www]# [root@opensourceecology www]# du -sh * 32G var 19G www.20240726_160837.tar.gz [root@opensourceecology www]#
- oh, actually I want the /ettc/ config file
[root@opensourceecology www]# cd ../etc [root@opensourceecology etc]# [root@opensourceecology etc]# tar -xf etc.20240726_160837.tar.gz [root@opensourceecology etc]# [root@opensourceecology etc]# du -sh * 46M etc 13M etc.20240726_160837.tar.gz [root@opensourceecology etc]#
- a diff of the pre-update configs and the current configs shows 4x new files
[root@opensourceecology etc]# diff -ril etc/httpd /etc/httpd diff: etc/httpd/logs: No such file or directory diff: etc/httpd/modules: No such file or directory Only in /etc/httpd/conf.d: autoindex.conf Only in /etc/httpd/conf.d: ssl.conf Only in /etc/httpd/conf.d: userdir.conf Only in /etc/httpd/conf.d: welcome.conf [root@opensourceecology etc]#
- I just moved these 4x files out (into our tmp change dir), and tried a restart; it came up
[root@opensourceecology CHG-2024-07-26_yum_update]# mkdir moved_from_etc_httpd [root@opensourceecology CHG-2024-07-26_yum_update]# mv /etc/httpd/conf.d/autoindex.conf moved_from_etc_httpd/ [root@opensourceecology CHG-2024-07-26_yum_update]# mv /etc/httpd/conf.d/ssl.conf moved_from_etc_httpd/ [root@opensourceecology CHG-2024-07-26_yum_update]# mv /etc/httpd/conf.d/userdir.conf moved_from_etc_httpd/ [root@opensourceecology CHG-2024-07-26_yum_update]# mv /etc/httpd/conf.d/welcome.conf moved_from_etc_httpd/ [root@opensourceecology CHG-2024-07-26_yum_update]# [root@opensourceecology CHG-2024-07-26_yum_update]# systemctl restart httpd [root@opensourceecology CHG-2024-07-26_yum_update]# [root@opensourceecology CHG-2024-07-26_yum_update]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2024-07-26 17:59:36 UTC; 4s ago Docs: man:httpd(8) man:apachectl(8) Main PID: 15908 (httpd) Status: "Processing requests..." CGroup: /system.slice/httpd.service ├─15908 /usr/sbin/httpd -DFOREGROUND ├─15910 /usr/sbin/httpd -DFOREGROUND ├─15911 /usr/sbin/httpd -DFOREGROUND ├─15912 /usr/sbin/httpd -DFOREGROUND ├─15913 /usr/sbin/httpd -DFOREGROUND ├─15914 /usr/sbin/httpd -DFOREGROUND ├─15921 /usr/sbin/httpd -DFOREGROUND ├─15927 /usr/sbin/httpd -DFOREGROUND ├─15928 /usr/sbin/httpd -DFOREGROUND ├─15936 /usr/sbin/httpd -DFOREGROUND ├─15937 /usr/sbin/httpd -DFOREGROUND ├─15938 /usr/sbin/httpd -DFOREGROUND └─15939 /usr/sbin/httpd -DFOREGROUND Jul 26 17:59:36 opensourceecology.org systemd[1]: Starting The Apache HTTP Se.... Jul 26 17:59:36 opensourceecology.org systemd[1]: Started The Apache HTTP Server. Hint: Some lines were ellipsized, use -l to show in full. [root@opensourceecology CHG-2024-07-26_yum_update]#
- I was able to load and edit the wiki; I spent some time adding some updates to the CHG article https://wiki.opensourceecology.org/wiki/CHG-2020-05-04_yum_update
- for some reason my browser keeps locking-up when all I'm trying to do is edit the text in the textarea for ^ this wiki article. I don't use the wysiwyg editor. I'm literally just editing text in a textarea; that shouldn't require any processing
- It took me ~20 minutes just to make a few changes to one wiki article because the page on firefox kept locking-up, sometimes displaying a spinning circle over the page
- I launched a new DispVM with *only* firefox running and *only* one tab open in firefox. The issue persisted, and the VM with the (idle) firefox on the edit page was taxed with 20-60% CPU usage; something is definitely wrong, but it's unclear if the bug is on our mediawiki server, my firefox client, or both
- anyway, I'm continuing with the validation steps
- I was successfully able to load the frontpage of all the 9x websites
- the logo at the top (and bottom) of https://oswh.opensourceecology.org/ was missing, but I'm not sure if that was the case before the updates or not
- I simply get a 404 on the image http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png
- I guess the domain is wrong; we don't appear to use opensourcewarehouse.org anymore, so I guess this was an issue that predates our updates now
- everything else looked good
- the logo at the top (and bottom) of https://oswh.opensourceecology.org/ was missing, but I'm not sure if that was the case before the updates or not
- I logged into the munin. It loads fine
- I do see some gaps in mysql charts where everything drops to 0 for a few hours, which I guess is before/why Marcin did reboots again. My job now isn't to investigate this now, but I'm just making a note here
- otherwise munin is working; validated.
- I logged into awstates. it loads fine
- I just quickly scanned the main pages for www.opensourceecology.org and wiki.opensourceecology.org; they look fine
- I already tested edits on wiki.opensourceecolgy.org; they're working (setting aside the client-side lag)
- I was successfully able to make a trivial change to the main wordpress site, and then revert that change https://www.opensourceecology.org/offline-wiki-zim-kiwix/
- the only thing left is the backups, which have been running the background since shortly after the reboot
- the backups finished being created successfully
- the backups are currently being uploaded at the rate-limited 3 MB/s. they're at 39% now, and estimated to finish uploading in 1h10m from now.
- the upload is the last step; that's good enough for me to consider the backups functional
- that completes our validation; I think it's safe to mark this change as successful
- I sent an update email to Marcin & Catarina
Hey Marcin & Catarina, I've finished updating the system packages on the hetzner2 server. It's a very good thing that we did this, because your server tried and failed to download its updates from 484 mirrors before it finally found a server that it could download its updates from. As I mentioned in Nov 2022, your server runs CentOS 7, which stopped receiving "Full Updates" by Red Hat in Aug 2020. As of Jun 2024, it is no longer going to be updated in any way (security, maintenance, etc). At some point in the future, I guess all of their update servers will go down too. We're lucky at least one was still online. * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2022#Wed_November_02.2C_2022 Today I was successfully able to update 434 system packages onto hetzner2. I did some quick validation of a subset of your websites, and I only found a couple minor errors 1. The header & footer images of oswh don't load https://oswh.opensourceecology.org/ 2. Editing the wiki sometimes causes my browser to lock-up; it's not clear if this is a pre-existing issue, or if the issue is caused by your server or my client I did not update your server's applications that cannot be updated by the package manager (eg wordpress, mediawiki, etc). If you don't detect any issues with your server, then I would recommend that we do the application upgrade simultaneously with a migration to a new server running Debian. I'd like to stress again the urgency of the need to migrate off of CentOS 7. Besides the obvious security risks of running a server that is no longer receiving security patches, at some point in the likely-not-too-distant future, your server is going to break and it will be extremely non-trivial to fix it. The deadline for migrating was in 2020. I highly recommend prioritizing a project to migrate your server to a new Debian server ASAP. Please spend some time testing your various websites, and let me know if you experience any issues. Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- I confirmed the list of updates on the server is now empty
[root@opensourceecology CHG-2024-07-26_yum_update]# yum list updates Loaded plugins: fastestmirror, replace Loading mirror speeds from cached hostfile * base: ftp.plusline.net * epel: mirrors.n-ix.net * extras: ftp.plusline.net * updates: mirror.checkdomain.de [root@opensourceecology CHG-2024-07-26_yum_update]#
- I'm considering the change successful
- looks like my tmp change dir pushed the disk to 86% capacity; let's clean that up
[root@opensourceecology CHG-2024-07-26_yum_update]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 17M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/md2 197G 161G 27G 86% / /dev/md1 488M 386M 77M 84% /boot tmpfs 6.3G 0 6.3G 0% /run/user/1005 [root@opensourceecology CHG-2024-07-26_yum_update]# ls after.log before.log needs-reboot.log update.log backup_before moved_from_etc_httpd needs-restarting.log [root@opensourceecology CHG-2024-07-26_yum_update]# du -sh * 28K after.log 70G backup_before 28K before.log 28K moved_from_etc_httpd 4.0K needs-reboot.log 4.0K needs-restarting.log 216K update.log [root@opensourceecology CHG-2024-07-26_yum_update]# ls before.log before.log [root@opensourceecology CHG-2024-07-26_yum_update]# [root@opensourceecology CHG-2024-07-26_yum_update]# ls backup_before/ daily_hetzner2_20240726_160837.tar root [root@opensourceecology CHG-2024-07-26_yum_update]# [root@opensourceecology CHG-2024-07-26_yum_update]# du -sh backup_before/* 20G backup_before/daily_hetzner2_20240726_160837.tar 51G backup_before/root [root@opensourceecology CHG-2024-07-26_yum_update]# [root@opensourceecology CHG-2024-07-26_yum_update]# ls backup_before/root backups [root@opensourceecology CHG-2024-07-26_yum_update]# [root@opensourceecology CHG-2024-07-26_yum_update]# rm -rf backup_before/root [root@opensourceecology CHG-2024-07-26_yum_update]#
- great, now we're down to 59%
[root@opensourceecology CHG-2024-07-26_yum_update]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /dev/shm tmpfs 32G 17M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/md2 197G 110G 78G 59% / /dev/md1 488M 386M 77M 84% /boot tmpfs 6.3G 0 6.3G 0% /run/user/1005 [root@opensourceecology CHG-2024-07-26_yum_update]#
Wed July 24, 2024
- Marcin contacted me a few days ago saying that the server needs reboots again
- I found that the last time we did a system packages update was in 2020, over 4 years ago. I strongly recommended that we update the system packages, and probably the web applications as well
- here's the link to the last time I updated the system packages in May 2020 https://wiki.opensourceecology.org/wiki/CHG-2020-05-04_yum_update
- I also noted that CentOS is now not only EOL, but it's also no longer receiving (security) updates.
- I warned Marcin about this approaching deadline in Nov 2022, and urged him to migrate to a new OS before 2024.
In my prior work at OSE, I've done my best to design your systems to be robust and "well oiled" so that they would run for as long as possible with as little maintenance as possible. However, code rots over time, and there's only so long you can hold-off before things fall apart. Python 2.7.5 was End-of-Life'd on 2020-01-01, and it no longer receives any updates. * https://en.wikipedia.org/wiki/History_of_Python CentOS 7.7 was released 2019-09-17. "Full Updates" stopped 2020-08-06, and it will no longer receive any maintenance updates after 2024-06-30. * https://wiki.centos.org/About/Product At some point, you're going to want to migrate to a new server with a new OS. I strongly recommend initiating this project before 2024.
- Here's the log entry https://wiki.opensourceecology.org/wiki/Maltfield_Log/2022#Wed_November_02.2C_2022
- I told Marcin to budget for ~$10,000 to migrate to a new server, as it's going to be a massive project that will likely require more than a month of full-time work to complete the migration
- Marcin said I should go ahead and prepare a CHG "ticket" article for the upgrade and schedule a time to do it
- I prepared a change ticket for updating the system packages on Friday https://wiki.opensourceecology.org/wiki/CHG-2024-07-26_yum_update
- I also noticed that I kept getting de-auth'd every few minutes on the wiki. That's annoying. Hopefully updates will help this (and other) issues go away.
- If we did a migration to debian, then we'd need to migrate to a new server
- previously when we migrated from hetzner1 to hetzner2, we got a 15x increase in RAM (from 4GB to 64GB). And the price of both servers was the same!
- I was expecting the next jump would have similar results: we'd migrate to a new server that costs the same for much better specs, but that's not looking like it's going to be the case :(
- Here's the currently-offered dedicated servers at hetzner https://www.hetzner.com/dedicated-rootserver/
- Currently we have 8-cores, 64G RAM, and two 250G disks in a RAID-1 software array. We pay 39 EUR/mo
- The cheapest dedicated server (EX44) currently is 46.41 EUR/month and comes with 14-cores, 64G RAM, and 2x 512G disks. That should meet our requirements https://www.hetzner.com/dedicated-rootserver/ex44/configurator/#/
- oh crap, we'd be downgrading the proc from the i7 (Intel® Core™ i7-6700) to an i5 (Intel® Core™ i5-13500)
- I'd have to check the munin charts, but I would be surprised if we ever break a load of 2, so that's still probably fine.
- I met with Marcin tonight to discuss [a] the system-level package upgrades, [b] the application (eg wordpress, mediawiki, etc) upgrades, and [c] the server migration
- I recommended that Marcin do the updates on staging, and described the risk of not doing it
- the problem is that the current staging environment is down, an d it may take a few days to restore it
- the risk is maybe a few days of downtime instead smaller change window during the update
- we agreed that I'll do the system-level package upgrades direct-to-production; Marcin accepted the risk of a few days of downtime
- Marcin also mentioned that Hetzner has a "server auction" page, which has some more servers that meet our needs at a slightly discounted price https://www.hetzner.com/sb/
- actually many of these are 37.72 EUR/mo, so they're actually *cheaper* than our current 39 EUR/mo. Great!
- there's >3 pages of servers for this 37.72 EUR/mo price. One of them has 2x 4TB drives (though it looks like spinning disks). This is a server graveyard built-to-spec for previous customers, it seems. We should be able to find one that meets our needs, so that means we'll easily double our disk and save ~15 EUR per year. Cool :)
- I recommended that Marcin do the updates on staging, and described the risk of not doing it
