Maltfield Log/2025 Q1

From Open Source Ecology
< Maltfield Log
Revision as of 16:58, 23 February 2025 by Maltfield (talk | contribs) (Feb 15)
Jump to navigation Jump to search

My work log from the first quarter of the year 2025. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Sat Feb 15, 2025

  1. Tom responded saying he got "Permission Denied" when trying to ssh into hetzner3
  2. I asked him to try again with '-vvv' and also send me his public key
  3. on the server's side, I saw this
root@hetzner3 ~ # journalctl -u ssh
...
Feb 15 02:08:19 hetzner3 sshd[1552191]: Connection from REDACTED port 40998 on 144.76.164.201 port 32415 rdomain ""
Feb 15 02:08:29 hetzner3 sshd[1552191]: Failed publickey for tgriffing from REDACTED port 40998 ssh2: RSA SHA256:OEad8nPjAY1Rwt73cPO7GxvX+I6scrraDZMYwQUldY4
Feb 15 02:08:29 hetzner3 sshd[1552191]: Connection closed by authenticating user tgriffing REDACTED port 40998 [preauth]
Feb 15 02:08:44 hetzner3 sshd[1552199]: Connection from REDACTED port 56426 on 144.76.164.201 port 32415 rdomain ""
Feb 15 02:08:47 hetzner3 sshd[1552199]: Failed publickey for tgriffing from REDACTED port 56426 ssh2: RSA SHA256:OEad8nPjAY1Rwt73cPO7GxvX+I6scrraDZMYwQUldY4
Feb 15 02:08:48 hetzner3 sshd[1552199]: Connection closed by authenticating user tgriffing REDACTED port 56426 [preauth]
  1. ...
  2. yesterday I setup a second vhost for obi running on a unique ipv6 address, but I couldn't get it fully working
  3. one issue was that my laptop might not have an ipv6 address; let's try from my personal server
  4. yesterday we setup hetzner3 to have two ipv6 addresses
    1. 2a01:4f8:200:40d7::2, which we had before
    2. 2a01:4f8:200:40d7::3, which is new
root@hetzner3 ~ # ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
	inet6 ::1/128 scope host noprefixroute 
	   valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
	inet6 2a01:4f8:200:40d7::3/64 scope global 
	   valid_lft forever preferred_lft forever
	inet6 2a01:4f8:200:40d7::2/64 scope global 
	   valid_lft forever preferred_lft forever
	inet6 fe80::921b:eff:fec4:28b4/64 scope link 
	   valid_lft forever preferred_lft forever
root@hetzner3 ~ #
  1. I confirmed that I can ping the server on both addresss; and it doesn't work (as expected) on '::4', which was never setup
maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::2
PING 2a01:4f8:200:40d7::2(2a01:4f8:200:40d7::2) 56 data bytes
64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=1 ttl=57 time=2.85 ms
64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=2 ttl=57 time=2.91 ms

--- 2a01:4f8:200:40d7::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.852/2.879/2.906/0.027 ms
maltfield@mail:~$ 

maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::3
PING 2a01:4f8:200:40d7::3(2a01:4f8:200:40d7::3) 56 data bytes
64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=1 ttl=57 time=3.19 ms
64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=2 ttl=57 time=2.88 ms

--- 2a01:4f8:200:40d7::3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.877/3.034/3.191/0.157 ms
maltfield@mail:~$ 

maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::4
PING 2a01:4f8:200:40d7::4(2a01:4f8:200:40d7::4) 56 data bytes

--- 2a01:4f8:200:40d7::4 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1011ms

maltfield@mail:~$
  1. yeah, I can't do this from my Qubes VM
user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::3
ping: connect: Network is unreachable
user@disp3993:~$ ping -c2 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=394 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=357 ms

--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 356.708/375.276/393.845/18.568 ms
user@disp3993:~$ 

user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::2
ping: connect: Network is unreachable
user@disp3993:~$ 

user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::3
ping: connect: Network is unreachable
user@disp3993:~$
  1. well, that's a good test. it's not worth continuing to pursue this if Marcin and Catarina can't use IPv6.
  2. I sent an email to them asking them to run a ping test
Hey Marcin,
Hey Catarina,

When you're at FeF, can you please run this command and send me the output?

  ping -c2 2a01:4f8:200:40d7::3

I'm trying to figure out if we can run two copies of osemain and obi (a static site + the broken/upgraded wordpress site). But we need an additional IP address to do this.

One way to do this (which would keep the domain name identical for both sites and therefore be best), would be to have each site bound to different IP addresses. With Hetnzer, we get just 1 IPv4 address and 616 IPv6 addresses.

But not every ISP (or router, or computer) provides IPv6 to their clients.  So this ping test will let me know if you're able to access a website that's only accessible over IPv6.

Please run the above ping command and send me the full output.


Thank you,
  1. if their test returns "network is unreachable", then I'll suggest we buy a second IPv4 address for ~2 EUR/mo (just to use until they finish the new/replacement websites)
  2. ...
  3. I checked-in on the mediawiki cron. It looks like it ran successfully this time
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
root@hetzner3 /usr/local/bin #

root@hetzner3 /usr/local/bin # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | head
total 2,8M
drwxrwx---  2 www-data www-data  72K Feb 15 05:04 .
drwxrwx--- 30 www-data www-data 4,0K Feb 13 19:48 ..
-rw-r--r--  1 www-data www-data 3,9K Feb 15 05:04 image_006de54f_46a605f583c811fa.png
-rw-r--r--  1 www-data www-data 4,6K Feb 15 05:04 image_00bb6228_3473277933896087.png
-rw-r--r--  1 www-data www-data 3,8K Feb 15 05:04 image_0289ccc4_158c15199e2053dd.png
-rw-r--r--  1 www-data www-data 4,3K Feb 15 05:04 image_02909476_69fb1a6ef8109be7.png
-rw-r--r--  1 www-data www-data 4,9K Feb 15 05:04 image_03bdc132_5d43e1a1df3307de.png
-rw-r--r--  1 www-data www-data 3,6K Feb 15 05:04 image_04f4de33_9b0075a6be38d1ad.png
-rw-r--r--  1 www-data www-data 3,9K Feb 15 05:04 image_0519550c_71efe71f45d70123.png
root@hetzner3 /usr/local/bin # 

root@hetzner3 /usr/local/bin # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | tail
-rw-r--r--  1 www-data www-data 3,2K Feb 15 05:04 image_fc79e722_7c8a08bcc570879b.png
-rw-r--r--  1 www-data www-data 3,5K Feb 15 05:04 image_fdef3933_121c56952a6c57d3.png
-rw-r--r--  1 www-data www-data 4,2K Feb 15 05:04 image_fe463544_629383d940de704d.png
-rw-r--r--  1 www-data www-data 4,0K Feb 15 05:04 image_fe56e8a6_d360bd714e425344.png
-rw-r--r--  1 www-data www-data 4,6K Feb 15 05:04 image_fe8301f9_4d8d17795ab35d70.png
-rw-r--r--  1 www-data www-data 4,2K Feb 15 05:04 image_feb29944_fda14c66d7316301.png
-rw-r--r--  1 www-data www-data 3,6K Feb 15 05:04 image_fef37631_1c47b152a8639807.png
-rw-r--r--  1 www-data www-data 3,9K Feb 15 05:04 image_ff128933_f9004121b7d624c0.png
-rw-r--r--  1 www-data www-data 3,7K Feb 15 05:04 image_ffb24016_bfc86a5d83776491.png
-rw-r--r--  1 www-data www-data 3,5K Feb 15 05:04 image_fff89283_ee1207b5b2d10c3e.png
root@hetzner3 /usr/local/bin # 

root@hetzner3 /usr/local/bin # ls /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | wc -l
500
root@hetzner3 /usr/local/bin # date -u
2025-02-15T20:54:29 UTC
root@hetzner3 /usr/local/bin #
  1. so there's exactly 500 images, and they were all generated today. perfect!
  2. the only issue I do see is that the log was cleared of messages from before, and it wasn't rotated
root@hetzner3 /usr/local/bin # ls -lah /var/log/cron
total 12K
drwxr-x---  2 www-data www-data 4,0K Feb 14 05:18 .
drwxr-xr-x 16 root     root     4,0K Feb 13 22:54 ..
-rw-r--r--  1 www-data www-data   86 Feb 15 05:04 mediawiki_generate_captchas.log
root@hetzner3 /usr/local/bin #
  1. looks like the cron is missing a second > sign (for append)
root@hetzner3 /usr/local/bin # cat /etc/cron.d/mediawiki_cron 
# Ansible managed
SHELL=/bin/bash

# regenerate new CAPTCHA images
20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log
root@hetzner3 /usr/local/bin #
  1. I fixed this in ansible and pushed it out https://github.com/OpenSourceEcology/ansible/commit/3aa97d5d27e538ce234aa0354a8a5ac1ae4bc082
user@ose:~/sandbox_local/ansible/hetzner3$ git diff HEAD^
diff --git a/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2 b/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2
index 28e5bf0..873a137 100644
--- a/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2
+++ b/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2
@@ -2,4 +2,4 @@
 SHELL=/bin/bash
 
 # regenerate new CAPTCHA images
-20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log
+20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &>> /var/log/cron/mediawiki_generate_captchas.log
user@ose:~/sandbox_local/ansible/hetzner3$
  1. ...
  2. another item I have in my TODO list was to confirm that APC is setup properly
  3. I never tested this. If APC isn't running, it'll greatly harm the performance of our PHP websites
  4. It looks like APC is no longer supported in PHP >5.4
  5. hetzner2 ran php v5.6 and hetzner3 runs v8.2.26, so I guess there's no change there
  6. the replacement for APC was APCu https://www.php.net/manual/en/book.apcu.php
  7. it's not clear to me when APCu was first supported, but they have releases going back to 2013https://pecl.php.net/package/apcu
  8. here's the mediawiki guide that describes apcu https://www.mediawiki.org/wiki/Manual:Performance_tuning
  9. it says to install php-acpu, which we've done (but I still want to test that it's actually enabled & working)
root@hetzner3 /usr/local/bin # apt-cache search apcu
apcupsd - APC UPS Power Management (daemon)
apcupsd-cgi - APC UPS Power Management (web interface)
apcupsd-doc - APC UPS Power Management (documentation/examples)
collectd-core - statistics collection and monitoring daemon (core system)
conky-all - highly configurable system monitor (all features enabled)
conky-cli - highly configurable system monitor (basic version)
conky-std - highly configurable system monitor (default version)
php-apcu - APC User Cache for PHP
php-apcu-all-dev - APC User Cache for PHP
php8.2-apcu - APC User Cache for PHP
php-symfony-polyfill-apcu - Symfony polyfill backporting apcu_* functions to lower PHP versions
root@hetzner3 /usr/local/bin #

root@hetzner3 /usr/local/bin # dpkg -l | grep -i apcu
ii  php-apcu                       5.1.22+4.0.11-2                         amd64        APC User Cache for PHP
ii  php8.2-apcu                    5.1.22+4.0.11-2                         amd64        APC User Cache for PHP
root@hetzner3 /usr/local/bin #
  1. according to the mediawiki docs, that should have installed some script 'apcu.php' that we can use to inspect the apc cache and confirm that mediawiki is using it
A script, apc.php is bundled with the APCu package which can be used to inspect the status of the cache, and also examine the contents of the user cache to verify that MediaWiki is correctly using it.
  1. uhh, that's just docs
root@hetzner3 /usr/local/bin # dpkg-query -L php-apcu
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/php-apcu
/usr/share/doc/php-apcu/README.Debian
/usr/share/doc/php-apcu/changelog.Debian.gz
/usr/share/doc/php-apcu/copyright
root@hetzner3 /usr/local/bin #
  1. ok, it's this one
root@hetzner3 /usr/local/bin # dpkg-query -L php8.2-apcu
/.
/etc
/etc/php
/etc/php/8.2
/etc/php/8.2/mods-available
/etc/php/8.2/mods-available/apcu.ini
/usr
/usr/include
/usr/include/php
/usr/include/php/20220829
/usr/include/php/20220829/ext
/usr/include/php/20220829/ext/apcu
/usr/include/php/20220829/ext/apcu/apc.h
/usr/include/php/20220829/ext/apcu/apc_api.h
/usr/include/php/20220829/ext/apcu/apc_arginfo.h
/usr/include/php/20220829/ext/apcu/apc_cache.h
/usr/include/php/20220829/ext/apcu/apc_globals.h
/usr/include/php/20220829/ext/apcu/apc_iterator.h
/usr/include/php/20220829/ext/apcu/apc_lock.h
/usr/include/php/20220829/ext/apcu/apc_mutex.h
/usr/include/php/20220829/ext/apcu/apc_serializer.h
/usr/include/php/20220829/ext/apcu/apc_sma.h
/usr/include/php/20220829/ext/apcu/apc_stack.h
/usr/include/php/20220829/ext/apcu/php_apc.h
/usr/include/php/20220829/ext/apcu/php_apc_legacy_arginfo.h
/usr/lib
/usr/lib/php
/usr/lib/php/20220829
/usr/lib/php/20220829/apcu.so
/usr/share
/usr/share/doc
/usr/share/doc/php8.2-apcu
/usr/share/doc/php8.2-apcu/changelog.Debian.gz
/usr/share/doc/php8.2-apcu/copyright
root@hetzner3 /usr/local/bin #
  1. ...but there's still no apcu.php file :(
  2. actually, we are missing a package; the MediaWiki docs say we also need this php-igbinary
root@hetzner3 /usr/local/bin # apt-cache search igbinary
php-igbinary - igbinary PHP serializer
php-igbinary-all-dev - igbinary PHP serializer
php8.2-igbinary - igbinary PHP serializer
You have mail in /var/mail/root
root@hetzner3 /usr/local/bin # dpkg -l | grep -i igbinary
root@hetzner3 /usr/local/bin #
  1. I added it to the depends listed in the ansible php role
  2. according to the docs, we're "strongly recommended" to set this
$wgMainCacheType = CACHE_ACCEL;
  1. I confirmed that we are using it
root@hetzner3 /usr/local/bin # grep -i cache /var/www/html/wiki.opensourceecology.org/LocalSettings.php 
$wgCategoryTreeDisableCache = false;
$wgCategoryTreeHTTPCache = true;
$egCache = true;
# sure that cached pages are cleared.
$wgCacheEpoch = max( $wgCacheEpoch, gmdate( 'YmdHis', @filemtime( FILE ) ) );
# VARNISH CACHE #
# INTERNAL MEDIAWIKI CACHE OPTIONS (DISTINCT FROM VARNISH)
# MainCache and MessageCache should use APCU per Aaron Schulz
$wgMainCacheType = CACHE_ACCEL;
# note that if message cache uses the db (per defaults), then it may make every
# not to cache the page (rightfully so), and the result is that varnish (which
# is our most important cache) is rendered useless. For more info, see:
$wgMessageCacheType = CACHE_ACCEL;
$wgUseLocalMessageCache = true;
# Parser Cache should still use the DB per Aaron Schulz
$wgParserCacheType = CACHE_DB;
$wgEnableSidebarCache = true;
# cache interface messages to files in this directory per Aaron Schulz
$wgCacheDirectory = "$IP/../cache";
root@hetzner3 /usr/local/bin #
  1. good to note that the mediawiki performance docs suggest using php-fpm instead of mod_php, something we've done for this migration from hetzner2 to hetzner3
  2. it also suggests using event MPM instead of prefork MPM
  3. crap, it looks like we're using prefork
root@hetzner3 /etc/apache2 # grep -irl 'mpm' *
magic
mods-available/mpm_event.conf
mods-available/pagespeed.conf
mods-available/mpm_worker.load
mods-available/mpm_prefork.conf
mods-available/mpm_prefork.load
mods-available/mpm_worker.conf
mods-available/php8.2.load
mods-available/mpm_prefork.conf.104647.2024-09-25@01:24:19~
mods-available/mpm_event.load
root@hetzner3 /etc/apache2 #

root@hetzner3 /etc/apache2 # ls -lah mods-enabled/
total 8,0K
drwxr-xr-x 2 root root 4,0K Dec 29 03:27 .
drwxr-xr-x 8 root root 4,0K Feb 13 20:02 ..
lrwxrwxrwx 1 root root   36 Sep 25 01:24 access_compat.load -> ../mods-available/access_compat.load
lrwxrwxrwx 1 root root   28 Sep 25 01:24 alias.conf -> ../mods-available/alias.conf
lrwxrwxrwx 1 root root   28 Sep 25 01:24 alias.load -> ../mods-available/alias.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 auth_basic.load -> ../mods-available/auth_basic.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 authn_core.load -> ../mods-available/authn_core.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 authn_file.load -> ../mods-available/authn_file.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 authz_core.load -> ../mods-available/authz_core.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 authz_host.load -> ../mods-available/authz_host.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 authz_user.load -> ../mods-available/authz_user.load
lrwxrwxrwx 1 root root   32 Sep 25 01:24 autoindex.conf -> ../mods-available/autoindex.conf
lrwxrwxrwx 1 root root   32 Sep 25 01:24 autoindex.load -> ../mods-available/autoindex.load
lrwxrwxrwx 1 root root   30 Sep 25 01:24 deflate.conf -> ../mods-available/deflate.conf
lrwxrwxrwx 1 root root   30 Sep 25 01:24 deflate.load -> ../mods-available/deflate.load
lrwxrwxrwx 1 root root   26 Sep 25 01:24 dir.conf -> ../mods-available/dir.conf
lrwxrwxrwx 1 root root   26 Sep 25 01:24 dir.load -> ../mods-available/dir.load
lrwxrwxrwx 1 root root   26 Sep 25 01:24 env.load -> ../mods-available/env.load
lrwxrwxrwx 1 root root   29 Sep 25 01:24 filter.load -> ../mods-available/filter.load
lrwxrwxrwx 1 root root   40 Sep 25 01:24 headers.load -> /etc/apache2/mods-available/headers.load
lrwxrwxrwx 1 root root   27 Sep 25 01:24 mime.conf -> ../mods-available/mime.conf
lrwxrwxrwx 1 root root   27 Sep 25 01:24 mime.load -> ../mods-available/mime.load
lrwxrwxrwx 1 root root   34 Dec 29 03:27 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf
lrwxrwxrwx 1 root root   34 Dec 29 03:27 mpm_prefork.load -> ../mods-available/mpm_prefork.load
lrwxrwxrwx 1 root root   34 Sep 25 01:24 negotiation.conf -> ../mods-available/negotiation.conf
lrwxrwxrwx 1 root root   34 Sep 25 01:24 negotiation.load -> ../mods-available/negotiation.load
lrwxrwxrwx 1 root root   29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf
lrwxrwxrwx 1 root root   29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load
lrwxrwxrwx 1 root root   43 Sep 26 00:06 proxy_fcgi.load -> /etc/apache2/mods-available/proxy_fcgi.load
lrwxrwxrwx 1 root root   38 Sep 26 00:11 proxy.load -> /etc/apache2/mods-available/proxy.load
lrwxrwxrwx 1 root root   41 Sep 25 01:24 remoteip.load -> /etc/apache2/mods-available/remoteip.load
lrwxrwxrwx 1 root root   33 Sep 25 01:24 reqtimeout.conf -> ../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root   33 Sep 25 01:24 reqtimeout.load -> ../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root   40 Sep 25 01:24 rewrite.load -> /etc/apache2/mods-available/rewrite.load
lrwxrwxrwx 1 root root   32 Sep 25 01:24 security2.conf -> ../mods-available/security2.conf
lrwxrwxrwx 1 root root   32 Sep 25 01:24 security2.load -> ../mods-available/security2.load
lrwxrwxrwx 1 root root   31 Sep 25 01:24 setenvif.conf -> ../mods-available/setenvif.conf
lrwxrwxrwx 1 root root   31 Sep 25 01:24 setenvif.load -> ../mods-available/setenvif.load
lrwxrwxrwx 1 root root   39 Sep 25 01:24 status.conf -> /etc/apache2/mods-available/status.conf
lrwxrwxrwx 1 root root   29 Sep 25 01:24 status.load -> ../mods-available/status.load
lrwxrwxrwx 1 root root   32 Sep 25 01:24 unique_id.load -> ../mods-available/unique_id.load
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # cat mods-enabled/mpm_prefork.load
# Conflicts: mpm_event mpm_worker
LoadModule mpm_prefork_module /usr/lib/apache2/modules/mod_mpm_prefork.so
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # cat mods-enabled/mpm_prefork.conf
# Ansible managed

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxRequestWorkers: maximum number of server processes allowed to start
# MaxConnectionsPerChild: maximum number of requests a server process serves

<IfModule mpm_prefork_module>
		StartServers                 5
		MinSpareServers              5
		MaxSpareServers             10
		MaxRequestWorkers          150
		MaxConnectionsPerChild    1000
</IfModule>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

root@hetzner3 /etc/apache2 #
  1. it looks like the last time I looked at prefork vs event, I realized we were still using mod_php, and that's when I switched us to php-fpm. But apparently I never revisited changing from MPM prefork to event https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Tue_Sep_24.2C_2024
  2. oh, no, I didn't find it in my logs, but I commented-out the prefork config in ansible on 2024-09-24 https://github.com/OpenSourceEcology/ansible/commit/763d9f996130edd579ad12388ec2861bc5c3cdc3
fix apache

This commit removes the mpm_prefork module load and config, since Debian 12 actually already ships with mpm_event with Apache 2.4. Note that mpm_event is generally considered more performant, but the downside is that it's not OK for modules that break when threads are used, such as mod_php

The solution to that is to switch to php-fpm, which is also more performant -- and thread safe.

When we had issues with Apache spawning runaway children and killing the whole server earlier this year (2024), my research suggested that we should use php-fpm, but I didn't make that change for simplicity. What we *did* change was to configure MaxServers in mpm_prefork, but that now conflicts with the defaults on Debian (before it was CentOS), because only one mpm can be selected.

Probably our apache configs in ansible will still need to be adjusted to direct php traffic to the cgi process (proxy?)
  1. unfortunately, I guess commenting it out wasn't sufficient to disable it.
  2. it looks like I need to remove the symlinks in 'mod-enabled/' from prefork and add these instead
root@hetzner3 /etc/apache2 # find . | grep -i event
./mods-available/mpm_event.conf
./mods-available/mpm_event.load
root@hetzner3 /etc/apache2 #
  1. here's the default config for mpm_event
root@hetzner3 /etc/apache2 # cat ./mods-available/mpm_event.conf
# event MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestWorkers: maximum number of worker threads
# MaxConnectionsPerChild: maximum number of requests a server process serves
StartServers            2
MinSpareThreads         25
MaxSpareThreads         75
ThreadLimit             64
ThreadsPerChild         25
MaxRequestWorkers       150
MaxConnectionsPerChild  0
You have mail in /var/mail/root
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # cat ./mods-available/mpm_event.load
# Conflicts: mpm_worker mpm_prefork
LoadModule mpm_event_module /usr/lib/apache2/modules/mod_mpm_event.so
root@hetzner3 /etc/apache2 #
  1. I tried to fix this (remove the symlinks for prefork and add new symlinks for event)

user@ose:~/sandbox_local/ansible/hetzner3$ git diff ... --- a/hetzner3/roles/maltfield.apache/tasks/main.yml +++ b/hetzner3/roles/maltfield.apache/tasks/main.yml @@ -140,29 +140,32 @@ 

  notify: restart apache

# we're using mpm_event, the default in Debian 12 with Apache 2.4

-#- name: enable module = mpm_prefork -# file: -# src: /etc/apache2/mods-available/mpm_prefork.load -# dest: /etc/apache2/mods-enabled/mpm_prefork.load -# state: link -# notify: restart apache -# -#- name: mpm_prefork.conf ...skipping... +- name: disable module = mpm_prefork + file: + path: /etc/apache2/mods-enabled/mpm_prefork.load + state: absent + notify: restart apache + +- name: mpm_prefork config symlink + file: + path: /etc/apache2/mods-enabled/mpm_prefork.conf + state: absent + notify: restart apache + +- name: enable module = mpm_event + file: + src: /etc/apache2/mods-available/mpm_event.load + dest: /etc/apache2/mods-enabled/mpm_event.load + state: link + notify: restart apache + +- name: mpm_event config symlink + file: + src: /etc/apache2/mods-available/mpm_event.conf + dest: /etc/apache2/mods-enabled/mpm_event.conf + state: link + notify: restart apache 

# prod mode config
- name: main apache4_prod.conf

diff --git a/hetzner3/roles/maltfield.php/tasks/main.yml b/hetzner3/roles/maltfield.php/tasks/main.yml index fecdcd6..75a806a 100644 --- a/hetzner3/roles/maltfield.php/tasks/main.yml +++ b/hetzner3/roles/maltfield.php/tasks/main.yml @@ -5,6 +5,7 @@ pkg: - php-fpm - php-apcu + - php-igbinary - php-curl 

  notify:

- restart php-fpm user@ose:~/sandbox_local/ansible/hetzner3$

  1. but apache failed on restart
root@hetzner3 /etc/apache2 # journalctl -u apache2 | less
...
root@hetzner3 /etc/apache2 # systemctl restart apache2
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # journalctl -u apache2 | less
...
Feb 15 21:50:31 hetzner3 apachectl[1909056]: [Sat Feb 15 21:50:31.965322 2025] [php:crit] [pid 1909056:tid 1909056] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
Feb 15 21:50:31 hetzner3 apachectl[1909056]: AH00013: Pre-configuration failed
Feb 15 21:50:31 hetzner3 apachectl[1909054]: Action 'graceful-stop' failed.
Feb 15 21:50:31 hetzner3 apachectl[1909054]: The Apache error log may have more information.
Feb 15 21:50:31 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'.
Feb 15 21:50:32 hetzner3 systemd[1]: Stopped apache2.service - The Apache HTTP Server.
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Consumed 8.836s CPU time.
Feb 15 21:50:32 hetzner3 systemd[1]: Starting apache2.service - The Apache HTTP Server...
Feb 15 21:50:32 hetzner3 apachectl[1909062]: [Sat Feb 15 21:50:32.073820 2025] [php:crit] [pid 1909062:tid 1909062] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
Feb 15 21:50:32 hetzner3 apachectl[1909062]: AH00013: Pre-configuration failed
Feb 15 21:50:32 hetzner3 apachectl[1909059]: Action 'start' failed.
Feb 15 21:50:32 hetzner3 apachectl[1909059]: The Apache error log may have more information.
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'.
Feb 15 21:50:32 hetzner3 systemd[1]: Failed to start apache2.service - The Apache HTTP Server.
You have mail in /var/mail/root
root@hetzner3 /etc/apache2 #
  1. looks like we're using event. please tell me we're not using mod_php still?
root@hetzner3 /etc/apache2 # ls -lah mods-enabled/ |grep -iE 'php|prefork|event'
lrwxrwxrwx 1 root root   42 Feb 15 21:40 mpm_event.conf -> /etc/apache2/mods-available/mpm_event.conf
lrwxrwxrwx 1 root root   42 Feb 15 21:40 mpm_event.load -> /etc/apache2/mods-available/mpm_event.load
lrwxrwxrwx 1 root root   29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf
lrwxrwxrwx 1 root root   29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load
root@hetzner3 /etc/apache2 #
  1. yeah, I think that this is mod_php; let's disable it
root@hetzner3 /etc/apache2 # cat mods-enabled/php8.2.conf 
# Using (?:pattern) instead of (pattern) is a small optimization that
# avoid capturing the matching pattern (as $1) which isn't used here
<FilesMatch ".+\.ph(?:ar|p|tml)$">
	SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch ".+\.phps$">
	SetHandler application/x-httpd-php-source
	# Deny access to raw php sources by default
	# To re-enable it's recommended to enable access to the files
	# only in specific virtual host or directory
	Require all denied
</FilesMatch>
# Deny access to files without filename (e.g. '.php')
<FilesMatch "^\.ph(?:ar|p|ps|tml)$">
	Require all denied
</FilesMatch>

# Running PHP scripts in user directories is disabled by default
# 
# To re-enable PHP in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
	<Directory /home/*/public_html>
		php_admin_flag engine Off
	</Directory>
</IfModule>
root@hetzner3 /etc/apache2 #
  1. shit, I'm not sure we're even using php-fpm
  2. oh, it looks like both are enabled? This is very annoying because mod_php is in "mods-enabled" and php-fpm is in "conf-enabled"
root@hetzner3 /etc/apache2 # ls -lah mods-available | grep -i php
-rw-r--r-- 1 root root  998 Jun 17  2024 php8.2.conf
-rw-r--r-- 1 root root  101 Jun 17  2024 php8.2.load
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # ls -lah conf-available | grep -i php
-rw-r--r-- 1 root root 1,4K Jun 17  2024 php8.2-fpm.conf
-rw-r--r-- 1 root root 1,2K Sep 25 01:46 phplist.virtualhost.include
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # ls -lah conf-enabled | grep -i php
lrwxrwxrwx 1 root root   43 Sep 25 23:53 php8.2-fpm.conf -> /etc/apache2/conf-available/php8.2-fpm.conf
root@hetzner3 /etc/apache2 #

root@hetzner3 /etc/apache2 # ls -lah mods-enabled | grep -i php
lrwxrwxrwx 1 root root   29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf
lrwxrwxrwx 1 root root   29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load
root@hetzner3 /etc/apache2 #
  1. anyway, I'll update ansible to remove
    1. before
root@hetzner3 /etc/apache2 # ls *-enabled/* | grep -i php
conf-enabled/php8.2-fpm.conf
mods-enabled/php8.2.conf
mods-enabled/php8.2.load
sites-enabled/phplist.opensourceecology.org.conf
root@hetzner3 /etc/apache2 #
    1. after
root@hetzner3 /etc/apache2 # ls *-enabled/* | grep -i php
conf-enabled/php8.2-fpm.conf
sites-enabled/phplist.opensourceecology.org.conf
root@hetzner3 /etc/apache2 #
  1. now it's working
root@hetzner3 /etc/apache2 # systemctl restart apache2
root@hetzner3 /etc/apache2 #
  1. I did a quick check of store and wiki, they both load fine
  2. wow, I'm glad I caught that. So now we're using the much more performant mpm event, and the (prerequisite) php-fpm
  3. but we still haven't confirmed apc is working!
  4. this answer on SE says to use 'apc.php' too, but it also provides an alternative https://stackoverflow.com/a/17631545/1174102
<pre><?php
print_r(apc_cache_info());
  1. I created a script for this on the wiki
cat > /var/www/html/wiki.opensourceecology.org/htdocs/apc.php <<'EOF'
<pre>
<?php
print_r(apc_cache_info());
?>

EOF

chown not-apache:www-data /var/www/html/wiki.opensourceecology.org/htdocs/apc.php chmod 0040 /var/www/html/wiki.opensourceecology.org/htdocs/apc.php

  1. crap, that page is empty. And this is in the apache logs
[Sat Feb 15 22:19:30.627296 2025] [proxy_fcgi:error] [pid 1917176:tid 1917223] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function apc_cache_info() in /var/www/html/wiki.opensourceecology.org/htdocs/apc.php:3\nStack trace:\n#0 {main}\n  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/apc.php on line 3'
  1. well, it looks like I'm not disabling that function; maybe it just doesn't exist anymore
root@hetzner3 /var/log/apache2 # grep -ir apc /etc/php/8.2/
/etc/php/8.2/mods-available/apcu.ini:extension=apcu.so
/etc/php/8.2/mods-available/igbinary.ini:; Use igbinary as serializer in APC cache (3.1.7 or later)
/etc/php/8.2/mods-available/igbinary.ini:;apc.serializer=igbinary
root@hetzner3 /var/log/apache2 #
  1. the php reference docs don't say the function has been retired. So maybe, in fact, apc isn't enabled? https://www.php.net/manual/en/function.apcu-cache-info.php
  2. oh, duh, ^ that one is 'acpu_' and the one I wrote to the file is 'apc_'
  3. let's try this
cat > /var/www/html/wiki.opensourceecology.org/htdocs/apc.php <<'EOF'
<pre>
<?php
print_r(apcu_cache_info());
?>

EOF

chown not-apache:www-data /var/www/html/wiki.opensourceecology.org/htdocs/apc.php chmod 0040 /var/www/html/wiki.opensourceecology.org/htdocs/apc.php

  1. that worked! It spat-out an array that spans 23,041 lines
  2. here's the top of it
		Array
		(
	[num_slots] => 4099
	[ttl] => 0
	[num_hits] => 1662
	[num_misses] => 1867
	[num_inserts] => 1701
	[num_entries] => 1665
	[expunges] => 0
	[start_time] => 1739654285
	[mem_size] => 2845632
	[memory_type] => mmap
	[cache_list] => Array
		(
  1. the SE post above says that if "num_hits" is increasing, then it's working
  2. I logged-in on the wiki and edited the main page
  3. then I refreshed the apc.php page, and now it says
Array
(
	[num_slots] => 4099
	[ttl] => 0
	[num_hits] => 1662
	[num_misses] => 1867
	[num_inserts] => 1701
	[num_entries] => 1665
	[expunges] => 0
	[start_time] => 1739654285
	[mem_size] => 2845632
	[memory_type] => mmap
	[cache_list] => Array
		(
  1. fuck; num_hits didn't increase
  2. wait, maybe it's cached. I tried "/apc.php?nocache=1"
Array
(
	[num_slots] => 4099
	[ttl] => 0
	[num_hits] => 6929
	[num_misses] => 2080
	[num_inserts] => 1870
	[num_entries] => 1777
	[expunges] => 0
	[start_time] => 1739654285
	[mem_size] => 3586816
	[memory_type] => mmap
	[cache_list] => Array
		(
  1. oh, that's a huge increase. I guess we can conclude that apc is working. great!
  2. let's cleanup that script
root@hetzner3 /var/www/html/wiki.opensourceecology.org # rm /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
  1. ...
  2. another item on my TODO list is to confirm that my various vhost hardenings are still working (since the upgrades, changes to php-fpm, etc...it's likely some of these broke)
    1. wp-login.php shouldn't be accessible
    2. phplist config.php shouldn't be accessible (I want to confirm because old had ".*" prefix and new doesn't)
    3. .php files in wp-content uploads dirs shouldn't be executable
    4. anything other than GET POST HEAD should be denied
    5. anything that starts with a dot (other than .well-known) should be denied (test .git & .htaccess)
    6. indexes on a dir without 'index.php' shouldn't show a directory listing
    7. .htaccess shouldn't allow changing things
  3. this claims to block access to wp-login.php, but I'm pretty sure it's just the wordpress plugin; you can see the varnish header suggest that it penetrated past nginx
user@disp3993:~$ curl -iL store.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 15 Feb 2025 22:34:14 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://store.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block

HTTP/1.1 403 Forbidden
Server: nginx
Date: Sat, 15 Feb 2025 22:34:17 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 214
Connection: keep-alive
X-Varnish: 29914517
Age: 0
Via: 1.1 varnish-v4

<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /wp-login.php
on this server.</p>
</body></html>
user@disp3993:~$
  1. also these configs should span apps, and we can see the wiki just gives a 404
user@disp3993:~$ curl -iL wiki.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 15 Feb 2025 22:36:54 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://wiki.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=1;includeSubDomains

HTTP/1.1 404 Not Found
Server: nginx
Date: Sat, 15 Feb 2025 22:36:56 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 16
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: strict-origin
X-Varnish: 427397 427395
Age: 107
Via: 1.1 varnish (Varnish/7.1)

File not found.
user@disp3993:~$
  1. oh, actually, I had an error in /etc/hosts. This is totally broken on hetnzer3
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 15 Feb 2025 23:01:17 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://store.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=1;includeSubDomains

HTTP/1.1 302 Found
Server: nginx
Date: Sat, 15 Feb 2025 23:01:20 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: /
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 460388
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 15 Feb 2025 23:01:21 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 85328
Connection: keep-alive
X-Pingback: https://store.opensourceecology.org/xmlrpc.php
Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 200920 460383
Age: 9
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

user@disp3993:~$
  1. I'm not sure if this broke or I intentionally disabled it
  2. one issue is that the plugin we had been using in wordpress to redirect away from wp-login.php is no longer available due to "Security Issue" https://wordpress.org/plugins/rename-wp-login/
  3. as a result, there's a narrow window during the migration of sites to hetzner3 where we need to login at wp-login.php and configure the replacement plugin to use a distinct login URL
  4. I also took a moment to update the plugins list on the ose wiki article for wordpress https://wiki.opensourceecology.org/wiki/Wordpress
  5. I think the best solution to this is to see if we can configure 'melapress-login-security' from the command-line after migrating, so that we never have to login at wp-login.php
  6. after some trial-and-error, I found the settings for this plugin are stored in the wordpress options table; there's 4 of them
MariaDB [store_db]> select option_name from wp_options where option_name like '%mls_%';
+--------------------+
| option_name        |
+--------------------+
| mls_activation     |
| mls_active_version |
| mls_options        |
| mls_setting        |
+--------------------+
4 rows in set (0,001 sec)

MariaDB [store_db]>
  1. the first two are a simple string
  2. the second two are long json strings
  3. well it looks like the min password length is set in 'wp_options' and the wp-login.php rename is set in 'mls_setting'
  4. unfortunately my email address got saved in wp_options, so I think I'm just going to set mls_setting (so that doesn't have to get stored in the wiki CHG migration commands/instructions)
  5. alright, here's what we have
root@hetzner3 ~ # echo "select option_value from wp_options where option_name = 'mls_setting';" | mysql -uroot -p$mysqlPass store_db
option_value
a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}
root@hetzner3 ~ #
  1. it appears that we can either update the db directly (dangerous) or use wp-cli https://developer.wordpress.org/cli/commands/option/
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get siteurl
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
https://store.opensourceecology.org
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
  1. first I tested if 'update' needs to have a key for it to work; it doesn't (that way we don't have to delete first)
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get thisdoesnotexist
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Error: Could not get 'thisdoesnotexist' option. Does it exist?
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update thisdoesnotexist 'somevalue'
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'thisdoesnotexist' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get thisdoesnotexist
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
somevalue
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
  1. so here's my plan to change the login from wp-login.php to ose-hidden-login
sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}'
  1. to test this, I deleted the settings on store and deactivated the plugin
  2. I reactivated it, and confirmed that the "login page hardening" was back at the default (with nothing specified for 'Login page URL'
  3. I deactivated the plugin again in the wui, and tried the rest with the cli
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" plugin activate melapress-login-security
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'melapress-login-security' activated.
Success: Activated 1 of 1 plugins.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # 

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}'
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'mls_setting' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
  1. I refreshed '/wp-admin/' and re-checked the settings page. crap, it's still empty? https://store.opensourceecology.org/wp-admin/admin.php?page=mls-hide-login
  2. I confirmed the setting is there
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get mls_setting
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
  1. idk, maybe somehow I have to set both
  2. I obfuscated my email address and tried this
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_options 'a:62:{s:13:"master_switch";s:3:"yes";s:26:"activate_password_policies";s:3:"yes";s:37:"activate_password_expiration_policies";s:2:"no";s:34:"activate_password_recycle_policies";s:2:"no";s:16:"enforce_password";s:2:"no";s:10:"min_length";s:2:"20";s:16:"password_history";s:1:"1";s:16:"inherit_policies";s:2:"no";s:15:"password_expiry";a:2:{s:5:"value";s:1:"0";s:4:"unit";s:6:"months";}s:8:"ui_rules";a:7:{s:7:"history";s:2:"no";s:8:"username";s:2:"no";s:6:"length";b:1;s:7:"numeric";s:2:"no";s:8:"mix_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:5:"rules";a:6:{s:6:"length";b:1;s:7:"numeric";s:2:"no";s:10:"upper_case";s:2:"no";s:10:"lower_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:23:"change_initial_password";s:2:"no";s:12:"timed_logins";s:2:"no";s:21:"timed_logins_schedule";a:7:{s:6:"monday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:7:"tuesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:9:"wednesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"thursday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"friday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"saturday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"sunday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}}s:22:"inactive_users_enabled";s:2:"no";s:21:"inactive_users_expiry";a:2:{s:5:"value";i:30;s:4:"unit";s:4:"days";}s:30:"inactive_users_reset_on_unlock";s:2:"no";s:29:"failed_login_policies_enabled";s:3:"yes";s:21:"failed_login_attempts";s:1:"5";s:27:"failed_login_reset_attempts";s:4:"1440";s:27:"failed_login_unlock_setting";s:5:"timed";s:24:"failed_login_reset_hours";s:2:"60";s:29:"failed_login_reset_on_unblock";s:2:"no";s:18:"disable_self_reset";s:3:"yes";s:26:"disable_self_reset_message";b:0;s:27:"deactivated_account_message";s:161:"Your WordPress user has been deactivated. Please contact the <a href="mailto:michael@michaelaltfield.net">website administrator</a> to activate back your user.";s:19:"timed_login_message";b:0;s:30:"locked_user_disable_self_reset";s:2:"no";s:38:"locked_user_disable_self_reset_message";b:0;s:17:"restrict_login_ip";s:2:"no";s:23:"restrict_login_ip_count";i:3;s:22:"restrict_login_message";s:0:"";s:22:"notify_password_expiry";s:2:"no";s:30:"notify_password_reset_on_login";b:0;s:27:"notify_password_expiry_days";i:0;s:27:"notify_password_expiry_unit";s:4:"days";s:26:"restrict_login_credentials";s:7:"default";s:34:"restrict_login_credentials_message";s:0:"";s:24:"enable_sessions_policies";b:0;s:23:"remember_session_expiry";a:2:{s:5:"value";i:14;s:4:"unit";s:4:"days";}s:22:"default_session_expiry";a:2:{s:5:"value";i:2;s:4:"unit";s:4:"days";}s:22:"enable_device_policies";b:0;s:35:"enable_device_policies_admin_alerts";s:2:"no";s:25:"enable_security_questions";b:0;s:17:"enabled_questions";a:0:{}s:36:"device_policies_prompt_email_content";s:0:"";s:41:"device_policies_admin_alert_email_content";s:0:"";s:36:"device_policies_prompt_email_subject";s:0:"";s:41:"device_policies_admin_alert_email_subject";s:0:"";s:25:"min_answered_needed_count";i:3;s:39:"password_reset_request_disabled_message";s:0:"";s:24:"password_expired_message";s:0:"";s:36:"inactive_user_account_locked_message";s:0:"";s:51:"inactive_user_account_locked_reset_disabled_message";s:0:"";s:38:"restrict_logins_prompt_failure_message";s:0:"";s:34:"timed_logins_login_blocked_message";s:0:"";s:39:"restrict_login_ip_login_blocked_message";s:0:"";s:35:"failed_logins_login_blocked_message";s:0:"";s:40:"security_prompt_response_failure_message";s:0:"";s:24:"timed_logins_auto_logout";b:0;s:13:"ppm-user-role";s:0:"";s:22:"excluded_special_chars";s:0:"";}'
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'mls_options' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
  1. refreshing the wui then lead to a critical error
==> apache2/store.opensourceecology.org/error.log <==
[Sun Feb 16 00:10:53.811257 2025] [proxy_fcgi:error] [pid 1950272:tid 1950284] [client 146.70.129.180:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php:446\nStack trace:\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/class-melapress-login-security.php(362): MLS\\MLS_Options->init()\n#1 [internal function]: MLS_Core->init()\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()\n#4 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()\n#5 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(700): do_action()\n#6 /var/www/html/store.opensourceecology.org/wp-config.php(119): require_once('...')\n#7 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(...'
  1. I think the problem is that this json is encoded with the length of the string for some reason. Yeah, the string before I edited it was 161 chars; I updated this to 159 chars (to reflect the shorter email address)
  2. crap, this is a catch-22; I can't fix the fatal error because of the fatal error
Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php:446
Stack trace:
#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/class-melapress-login-security.php(362): MLS\MLS_Options->init()
#1 [internal function]: MLS_Core->init()
#2 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()
#3 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()
#4 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()
#5 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(700): do_action()
#6 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1375): require('...')
#7 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1294): WP_CLI\Runner->load_wordpress()
#8 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(28): WP_CLI\Runner->start()
#9 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/bootstrap.php(83): WP_CLI\Bootstrap\LaunchRunner->process()
#10 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/wp-cli.php(32): WP_CLI\bootstrap()
#11 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/php/boot-phar.php(20): include('...')
#12 /home/wp/.wp-cli/wp-cli-2.11.0.phar(4): include('...')
#13 {main}
  thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php on line 446
Error: There has been a critical error on this website.Learn more about troubleshooting WordPress. There has been a critical error on this website.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # 

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_passwor

=Fri Feb 14, 2025=

# I checked on the mediawiki cron that I created yesterday and should have run for the first time last night; it failed :(
<pre>
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log 
www-data is not in the sudoers file.
This incident has been reported to the administrator.
www-data is not in the sudoers file.
This incident has been reported to the administrator.
root@hetzner3 /usr/local/bin #
  1. so it's important that we don't execute some python script located in our webroot as root, but I tried to de-escalate in two places (for layered security)
    1. the cron executes the script as www-data
    2. the script executes the captcha.py script as www-data with sudo
  2. that doesn't work because the www-data itself can't execute the 'sudo' command
  3. I could add that user to sudo, but that sounds risky
  4. it's probably best just to execute the man bash script as root
  5. I updated the script with a long comment explaining this
root@hetzner3 /usr/local/bin # diff mediawiki_generate_captchas.sh.20250214 mediawiki_generate_captchas.sh
5c5
< # Version: 0.1
---
> # Version: 0.2
8c8
< #           * 
---
> #           * https://www.mediawiki.org/wiki/Extension:ConfirmEdit#FancyCaptcha
11c11
< # Updated: 2025-02-13
---
> # Updated: 2025-02-14
42a43,61
> 
> ###########################
> # (!!) SECURITY NOTE (!!) #
> ###########################
> # It's critical that any scripts executed inside our web server's document root
> # are executed as a non-privileged user! This is because, if there's a
> # vulnerability in MediaWiki (or wordpress, phpList, etc), then an attacker
> # can modify the contents of such scripts (eg captcha.py). And if we execute
> # that script as root, then it would give the attacker the ability a way to
> # execute arbitrary code as root, escalating their privileges.
> #
> # Also note that this script itself is executed as root by cron. It's better to
> # do the de-escalation in this script, because a human might execute it as
> # root. Also note this script wouldn't work if run as 'www-data', because
> # 'www-data' isn't in sudoers.
> #
> # Therefore, make sure any executions of scripts in /var/www/html/*/htdocs are
> # prefixed with `sudo -u www-data -bash -c "<path to script>"`
> ###########################
root@hetzner3 /usr/local/bin #
  1. and I updated the cron to run the script as root.
  2. we'll have to check-in on this tomorrow to see if it's working https://github.com/OpenSourceEcology/ansible/commit/720dc468f8bc6ddc08f77fdcdaa31b2a61eb4b35
  3. ...
  4. so yesterday I finished my quick verification of the wiki
  5. we're still blocked on osemain and obi
  6. otherwise, all the vhost snapshots are done
  7. I still haven't finished setting up the base server, so let's do that
  8. first, I'm the only human user who can ssh into this server
maltfield@hetzner3:/home$ ls
b2user  maltfield  not-apache  wp
maltfield@hetzner3:/home$
  1. hetzner2 had a few additional users
    1. cataria
    2. marcin
    3. tgriffing
[maltfield@opensourceecology home]$ ls
b2user  cmota  crupp  Flipo  hart  jthomas  lberezhny  maltfield  marcin  not-apache  stagingsync  tgriffing  wp
[maltfield@opensourceecology home]$
  1. there's others, but I don't think they're active
  2. to protect the server from getting broken, it's important that Marcin and Catarina do *not* have root access
  3. to mitigate the bus factor, it's important that we have at least two users with root access: that's me and tgriffin
[root@opensourceecology ~]# grep wheel /etc/group
wheel:x:10:maltfield,crupp,tgriffing,root,jthomas
[root@opensourceecology ~]#
  1. it's not a huge risk, though. If marcin ever needed, he can just login on the hetzner WUI and reboot the server to a rescue distro, reset the root password, and get in that way
    1. and if he lost hetzner WUI access, he can KYC into it
  2. we *do* want Marcin and Catarina to have accounts, though. For ssh, access to webserver files (eg to scp a very large file that exceeds our https upload size limits), and keepass
[root@opensourceecology ~]# grep marcin /etc/group
marcin:x:1001:
apache:x:48:cmota,crupp,maltfield,wp,apache,marcin
sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas
keepass:x:993:maltfield,marcin,cmota,crupp
apache-admins:x:1012:cmota,maltfield,marcin,crupp,tgriffing,wp,apache
[root@opensourceecology ~]# 

[root@opensourceecology ~]# grep cmota /etc/group
apache:x:48:cmota,crupp,maltfield,wp,apache,marcin
cmota:x:1002:
sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas
keepass:x:993:maltfield,marcin,cmota,crupp
apache-admins:x:1012:cmota,maltfield,marcin,crupp,tgriffing,wp,apache
[root@opensourceecology ~]#
  1. I decided against provisioning users with ansible; I'll just do it manually
  2. The only thing I do need to keep in ansible is to poke a hole in the firewall for these new users
  3. I confirmed that ssh has been hardened with the 'sshaccess' groups already
root@hetzner3 /usr/local/bin # grep -i group /etc/ssh/sshd_config
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
# In case you don't use PAM (`UsePAM no`), you can alternatively restrict users and groups here. For key-based authentication this is not necessary, since all keys must be explicitely enabled.
AllowGroups sshaccess
root@hetzner3 /usr/local/bin #
  1. currently it's just me
root@hetzner3 /usr/local/bin # grep sshaccess /etc/group
sshaccess:x:1001:maltfield
root@hetzner3 /usr/local/bin #
  1. the keepass group doesn't yet exist, but the apache-admins group does
root@hetzner3 /usr/local/bin # grep keepass /etc/group
root@hetzner3 /usr/local/bin # 
root@hetzner3 /usr/local/bin # grep apache /etc/group
users:x:100:maltfield,wp,not-apache
apache-admins:x:1004:www-data,wp
not-apache:x:1006:
root@hetzner3 /usr/local/bin #
  1. I updated https://wiki.opensourceecology.org/wiki/Hetzner3
  2. here's the pubkeys we have on hetzner2 for these new users
[root@opensourceecology ~]# cat /home/marcin/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500
[root@opensourceecology ~]# 

[root@opensourceecology ~]# cat /home/cmota/.ssh/authorized_keys 
#ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDOhf/CGM3t3sklCPAiwaAl7NQPiyDwy5WwV2oxO6rCsDC3jmwn2JCJ6LZ4jXI9NvQQXwg+1kOqsSv2ZFS6RUq1Rs/EW8jR5lFxy9+FgRFpAIlzglAWIBbjpamX0duCMAwpOH2bK1oEDokNM37TtBynFyNytJqLgIGI48IR45KIU4Ib/IN8KzcQRdfPMPFW3EcO1jHJCMUl5SIKt36rgFvhc6JmHF9sl2fCPZ6Jmbi21DOz8fVQDdxQNAc2ubkAS2Sn27LZS7ww6I1VPK43IKBZlZXm3/FQ5HyHrXsGelIDCwdrwM/GTffClDc9EX1ujEHXZIILBDDVLVbwzI+3QHsHvaU9pY+pXBC1dw4LN8InmVazGeV9MoH6SeSeEx4g4KS+2qMiF6fjViPzpLeDW7hIRiTc2+8pjoivNKcgO0xCIBJ/dvTpTExCOPDBVJisnU+3Psq6Wn0Nb7ts7A0tyCtGOG2rxQEYCRmIy8AvQ6WlKztGFvqrfdHaRO9v7WY0hMtyD7kh0tHhHgxSFNcWSIVAIf8JSfhm3f/Orcsg2JDliT4wJzMz7OAYGOQmpqDI+qYa/ULgp9qlf/XBQWeMrQwAU0YeepRJjx3rz9xMxOLj2ddV3JSS8wHg+BGLpPwvqaoFrVIXIgHkW0AbBoXHmtSIZSZ5t5+odWlT1aLb5GmlXw== catarinamota@catarinas-air.pk5001z
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer
[root@opensourceecology ~]# 

[root@opensourceecology ~]# cat //home/tgriffing/.ssh/authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0
[root@opensourceecology ~]#
  1. here's the vars
USERNAME='marcin'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500'

USERNAME='cmota'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer'

USERNAME='tgriffing'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0'
  1. and here's the commands I'll use to create these users
# create user if it doesn't yet exist
adduser ${USERNAME} --disabled-password --gecos ''

# add ssh pubkey
mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
  1. here's Marcin
root@hetzner3 ~ # USERNAME='marcin'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500'
root@hetzner3 ~ # 
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `marcin' ...
Adding new group `marcin' (1007) ...
Adding new user `marcin' (1007) with group `marcin (1007)' ...
Creating home directory `/home/marcin' ...
Copying files from `/etc/skel' ...
Adding new user `marcin' to supplemental / extra groups `users' ...
Adding user `marcin' to group `users' ...
root@hetzner3 ~ # 
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
  1. And Catarina
root@hetzner3 ~ # USERNAME='cmota'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer'
root@hetzner3 ~ # 
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `cmota' ...
Adding new group `cmota' (1008) ...
Adding new user `cmota' (1008) with group `cmota (1008)' ...
Creating home directory `/home/cmota' ...
Copying files from `/etc/skel' ...
Adding new user `cmota' to supplemental / extra groups `users' ...
Adding user `cmota' to group `users' ...
root@hetzner3 ~ # 

root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
  1. And Tom
root@hetzner3 ~ # USERNAME='tgriffing'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0'
root@hetzner3 ~ # 

root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `tgriffing' ...
Adding new group `tgriffing' (1009) ...
Adding new user `tgriffing' (1009) with group `tgriffing (1009)' ...
Creating home directory `/home/tgriffing' ...
Copying files from `/etc/skel' ...
Adding new user `tgriffing' to supplemental / extra groups `users' ...
Adding user `tgriffing' to group `users' ...
root@hetzner3 ~ # 

root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys       
root@hetzner3 ~ #
  1. then I added the users to the groups
gpasswd -a marcin sshaccess
gpasswd -a cmota sshaccess
gpasswd -a tgriffing sshaccess

gpasswd -a marcin www-data
gpasswd -a cmota www-data
gpasswd -a tgriffing www-data

gpasswd -a tgriffing apache-admins
  1. then I created the keepass group and added everyone to it
groupadd keepass

gpasswd -a maltfield keepass
gpasswd -a marcin keepass
gpasswd -a cmota keepass
gpasswd -a tgriffing keepass
  1. finally, I added Tom to the sudoers group
gpasswd -a tgriffing sudo
  1. I updated the ansible playbook to permit these human user accounts internet access https://github.com/OpenSourceEcology/ansible/commit/f294f3384f5d4bc47ae55a1631c264950e0e1355
user@ose:~/sandbox_local/ansible/hetzner3$ git diff
diff --git a/hetzner3/provision.yml b/hetzner3/provision.yml
index 0f8e428..9ae54a1 100644
--- a/hetzner3/provision.yml
+++ b/hetzner3/provision.yml
@@ -85,8 +85,11 @@
		 - -A OUTPUT -m owner --uid-owner root -j ACCEPT
	   820 allow apt:
		 - -A OUTPUT -m owner --uid-owner _apt -j ACCEPT
-      830 allow maltfield:
+      830 allow human users internet access:
		 - -A OUTPUT -m owner --uid-owner maltfield -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner marcin -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner cmota -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner tgriffing -j ACCEPT
	   840 allow postfix:
		 - -A OUTPUT -m owner --uid-owner postfix -j ACCEPT
	   850 allow stubby:
@@ -128,8 +131,11 @@
		 - -A OUTPUT -m owner --uid-owner root -j ACCEPT
	   820 allow apt:
		 - -A OUTPUT -m owner --uid-owner _apt -j ACCEPT
-      830 allow maltfield:
+      830 allow human users internet access:
		 - -A OUTPUT -m owner --uid-owner maltfield -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner marcin -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner cmota -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner tgriffing -j ACCEPT
	   840 allow postfix:
		 - -A OUTPUT -m owner --uid-owner postfix -j ACCEPT
	   850 allow stubby:
user@ose:~/sandbox_local/ansible/hetzner3$
  1. I sent an email to Marcin
Hi Marcin,

Can you please confirm that ssh access to our new OSE server (hetzner3) is working?

We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details:

User: marcin
Server: 144.76.164.201
Port: 32415

For example, you should be able to login with the following command:

  ssh -p 32415 marcin@144.76.164.201

The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2.

After you login, please set a password for your account with the following command:

  passwd

Please confirm that you were able to login to hetzner3 with ssh and update your password.
  1. I sent an email to Catarina
Hi Catarina,

Can you please confirm that ssh access to our new OSE server (hetzner3) is working?

We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details:

User: cmota
Server: 144.76.164.201
Port: 32415

For example, you should be able to login with the following command:

  ssh -p 32415 cmota@144.76.164.201

The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2.

After you login, please set a password for your account with the following command:

  passwd

Please confirm that you were able to login to hetzner3 with ssh and update your password.
  1. I sent an email to Tom
Hi Tom,

Can you please confirm that ssh access to our new OSE server (hetzner3) is working?

We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details:

User: tgriffing
Server: 144.76.164.201
Port: 32415

For example, you should be able to login with the following command:

  ssh -p 32415 tgriffing@144.76.164.201

The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2.

After you login, please set a password for your account with the following command:

  passwd

After updating your password, you should be able to become root with sudo.

  sudo su -

Please confirm that you were able to login to hetzner3 with ssh, 
update your password, and become root.
  1. that's all the human accounts; I'll just have to wait for their confirmations
  2. ...
  3. another thing I wanted to do was to diff the postfix configs on hetzner2 & hetzner3. I did some basic postfix setup just to get wazuh functional. And it appears to work for phpList too, but I never checked the postfix config to make sure it's sane
  4. hetzner2
[root@opensourceecology ~]# postconf > /home/maltfield/postconf_hetzner2_20250214
[root@opensourceecology ~]# chown maltfield /home/maltfield/postconf_hetzner2_20250214 
[root@opensourceecology ~]#
  1. hetzner3
root@hetzner3 ~ # postconf > /home/maltfield/postconf_hetzner3_20250214
root@hetzner3 ~ # chown maltfield /home/maltfield/postconf_hetzner3_20250214 
root@hetzner3 ~ #
  1. and I copied them locally for a visual diff in meld
user@ose:~$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/user/.ssh/id_rsa: 
Identity added: /home/user/.ssh/id_rsa (guttersnipe@guttersnipe)
user@ose:~$ cd tmp/hetzner3/
user@ose:~/tmp/hetzner3$ mkdir postfix
user@ose:~/tmp/hetzner3$ cd postfix/
user@ose:~/tmp/hetzner3/postfix$ 
user@ose:~/tmp/hetzner3/postfix$ rsync -av --progress opensourceecology.org:postconf_hetzner2_20250214 .
receiving incremental file list
postconf_hetzner2_20250214
		 32,676 100%   31.16MB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 43 bytes  received 32,799 bytes  4,378.93 bytes/sec
total size is 32,676  speedup is 0.99
user@ose:~/tmp/hetzner3/postfix$ 
user@ose:~/tmp/hetzner3/postfix$ rsync -av --progress hetzner3:postconf_hetzner3_20250214 .
receiving incremental file list
postconf_hetzner3_20250214
		 40,776 100%   38.89MB/s    0:00:00 (xfr#1, to-chk=0/1)

sent 43 bytes  received 40,908 bytes  7,445.64 bytes/sec
total size is 40,776  speedup is 1.00
user@ose:~/tmp/hetzner3/postfix$
  1. here's the diff
user@ose:~/tmp/hetzner3/postfix$ diff postconf_hetzner2_20250214 postconf_hetzner3_20250214 
11c11,12
< address_verify_poll_count = ${stress?1}${stress:3}
---
> address_verify_pending_request_limit = 5000
> address_verify_poll_count = ${stress?{1}:{3}}
37c38
< append_dot_mydomain = yes
---
> append_dot_mydomain = ${{$compatibility_level} <level {1} ? {yes} : {no}}
56a58
> cleanup_replace_stray_cr_lf = yes
61a64
> compatibility_level = 0
62a66
> confirm_delay_cleared = no
69c73
< daemon_directory = /usr/libexec/postfix
---
> daemon_directory = /usr/lib/postfix/sbin/
79a84
> default_delivery_status_filter =
95a101
> default_transport_rate_delay = 0s
111c117,118
< disable_vrfy_command = no
---
> disable_vrfy_command = yes
> dns_ncache_ttl_fix_enable = no
113a121
> dnssec_probe = ns:.
117a126
> empty_address_local_login_sender_maps_lookup_key = <>
119a129
> enable_idna2003_compatibility = no
121a132
> enable_threaded_bounces = no
138a150
> error_transport_rate_delay = $default_transport_rate_delay
157a170
> header_from_format = standard
164c177
< import_environment = MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ XAUTHORITY DISPLAY LANG=C
---
> import_environment = MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ XAUTHORITY DISPLAY LANG=C POSTLOG_SERVICE POSTLOG_HOSTNAME
166c179
< inet_interfaces = localhost
---
> inet_interfaces = 127.0.0.1, [::1], localhost
167a181
> info_log_address_format = external
173a188
> known_tcp_ports = lmtp=24, smtp=25, smtps=submissions=465, submission=587
174a190
> lmdb_map_size = 16777216
175a192
> lmtp_address_verify_target = rcpt
176a194
> lmtp_balance_inet_protocols = yes
178a197
> lmtp_bind_address_enforce = no
184a204
> lmtp_connection_reuse_count_limit = 0
192a213
> lmtp_delivery_status_filter = $default_delivery_status_filter
200a222
> lmtp_dns_reply_filter =
201a224
> lmtp_dns_support_level =
203a227
> lmtp_fallback_relay =
212a237
> lmtp_min_data_rate = 500
217c242
< lmtp_per_record_deadline = no
---
> lmtp_per_request_deadline = ${lmtp_per_record_deadline?{$lmtp_per_record_deadline}:{no}}
253c278,280
< lmtp_tls_ciphers = export
---
> lmtp_tls_chain_files =
> lmtp_tls_ciphers = medium
> lmtp_tls_connection_reuse = no
261c288,289
< lmtp_tls_fingerprint_digest = md5
---
> lmtp_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> lmtp_tls_force_insecure_host_tlsa_lookup = no
266c294
< lmtp_tls_mandatory_protocols = !SSLv2
---
> lmtp_tls_mandatory_protocols = !SSLv2, !SSLv3
270c298
< lmtp_tls_protocols = !SSLv2
---
> lmtp_tls_protocols = !SSLv2, !SSLv3
273a302
> lmtp_tls_servername =
275a305
> lmtp_tls_trust_anchor_file =
276a307,308
> lmtp_tls_wrappermode = no
> lmtp_transport_rate_delay = $default_transport_rate_delay
282a315
> local_delivery_status_filter = $default_delivery_status_filter
291a325
> local_login_sender_maps = static:*
297a332
> local_transport_rate_delay = $default_transport_rate_delay
301c336
< mail_release_date = 20130622
---
> mail_release_date = 20240304
303,304c338,339
< mail_version = 2.10.1
< mailbox_command = /usr/bin/procmail
---
> mail_version = 3.7.11
> mailbox_command =
309a345,348
> maillog_file =
> maillog_file_compressor = gzip
> maillog_file_prefixes = /var, /dev/stdout
> maillog_file_rotate_suffix = %Y%m%d-%H%M%S
321a361
> message_drop_headers = bcc, content-length, resent-bcc, return-path
324a365
> meta_directory = /etc/postfix
326c367
< milter_connect_macros = j {daemon_name} v
---
> milter_connect_macros = j {daemon_name} {daemon_addr} v _
335a377
> milter_macro_defaults =
353,354c395,396
< myhostname = mailer.opensourceecology.org
< mynetworks = 127.0.0.1/32
---
> myhostname = hetzner3.opensourceecology.org
> mynetworks = 127.0.0.1/32 [::1]/128
361a404
> openssl_path = openssl
365a409
> pipe_delivery_status_filter = $default_delivery_status_filter
366a411,412
> postlog_service_name = postlog
> postlogd_watchdog_timeout = 10s
370a417
> postscreen_allowlist_interfaces = ${postscreen_whitelist_interfaces?{$postscreen_whitelist_interfaces}:{static:all}}
374d420
< postscreen_blacklist_action = ignore
381c427,428
< postscreen_command_time_limit = ${stress?10}${stress:300}s
---
> postscreen_command_time_limit = ${stress?{10}:{300}}s
> postscreen_denylist_action = ${postscreen_blacklist_action?{$postscreen_blacklist_action}:{ignore}}
385a433,435
> postscreen_dnsbl_allowlist_threshold = ${postscreen_dnsbl_whitelist_threshold?{$postscreen_dnsbl_whitelist_threshold}:{0}}
> postscreen_dnsbl_max_ttl = ${postscreen_dnsbl_ttl?{$postscreen_dnsbl_ttl}:{1}}h
> postscreen_dnsbl_min_ttl = 60s
389c439
< postscreen_dnsbl_ttl = 1h
---
> postscreen_dnsbl_timeout = 10s
396c446
< postscreen_greet_wait = ${stress?2}${stress:6}s
---
> postscreen_greet_wait = ${stress?{2}:{6}}s
406a457
> postscreen_reject_footer_maps = $smtpd_reject_footer_maps
412,414c463,464
< postscreen_whitelist_interfaces = static:all
< prepend_delivered_header = command, file, forward
< process_id = 5305
---
> prepend_delivered_header =
> process_id = 1460591
419c469
< proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps $alias_maps
---
> proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps $alias_maps $smtpd_client_restrictions $smtpd_helo_restrictions $smtpd_sender_restrictions $smtpd_relay_restrictions $smtpd_recipient_restrictions $address_verify_sender_dependent_default_transport_maps $address_verify_sender_dependent_relayhost_maps $address_verify_transport_maps $fallback_transport_maps $lmtp_discard_lhlo_keyword_address_maps $lmtp_pix_workaround_maps $lmtp_sasl_password_maps $lmtp_tls_policy_maps $mailbox_command_maps $mailbox_transport_maps $postscreen_discard_ehlo_keyword_address_maps $rbl_reply_maps $sender_dependent_default_transport_maps $sender_dependent_relayhost_maps $smtp_discard_ehlo_keyword_address_maps $smtp_pix_workaround_maps $smtp_sasl_password_maps $smtp_tls_policy_maps $smtpd_discard_ehlo_keyword_address_maps $smtpd_milter_maps $virtual_gid_maps $virtual_uid_maps $local_login_sender_maps $postscreen_reject_footer_maps $smtpd_reject_footer_maps $tls_server_sni_maps $tlsproxy_client_policy_maps $default_delivery_status_filter $lmtp_delivery_status_filter $lmtp_dns_reply_filter $lmtp_reply_filter $local_delivery_status_filter $pipe_delivery_status_filter $postscreen_command_filter $smtp_delivery_status_filter $smtp_dns_reply_filter $smtp_reply_filter $smtpd_command_filter $smtpd_dns_reply_filter $virtual_delivery_status_filter $body_checks $header_checks $lmtp_body_checks $lmtp_header_checks $lmtp_mime_header_checks $lmtp_nested_header_checks $milter_header_checks $mime_header_checks $nested_header_checks $smtp_body_checks $smtp_header_checks $smtp_mime_header_checks $smtp_nested_header_checks
458c508
< relay_domains = $mydestination
---
> relay_domains = ${{$compatibility_level} <level {2} ? {$mydestination} : {}}
467a518
> relay_transport_rate_delay = $default_transport_rate_delay
475a527
> respectful_logging = ${{$compatibility_level} <level {3.6} ? {no} : {yes}}
490a543
> retry_transport_rate_delay = $default_transport_rate_delay
500a554
> service_name =
502a557
> shlib_directory = /usr/lib/postfix
505a561
> smtp_address_verify_target = rcpt
506a563
> smtp_balance_inet_protocols = yes
508a566
> smtp_bind_address_enforce = no
514a573
> smtp_connection_reuse_count_limit = 0
522a582
> smtp_delivery_status_filter = $default_delivery_status_filter
530a591
> smtp_dns_reply_filter =
531a593
> smtp_dns_support_level =
543a606
> smtp_min_data_rate = 500
549c612
< smtp_per_record_deadline = no
---
> smtp_per_request_deadline = ${smtp_per_record_deadline?{$smtp_per_record_deadline}:{no}}
579a643
> smtp_tcp_port = smtp
584c648,651
< smtp_tls_ciphers = export
---
> smtp_tls_chain_files =
> smtp_tls_ciphers = medium
> smtp_tls_connection_reuse = no
> smtp_tls_dane_insecure_mx_policy = ${{$smtp_tls_security_level} == {dane} ? {dane} : {may}}
592c659,660
< smtp_tls_fingerprint_digest = md5
---
> smtp_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> smtp_tls_force_insecure_host_tlsa_lookup = no
594c662
< smtp_tls_loglevel = 0
---
> smtp_tls_loglevel = 1
597,598c665,666
< smtp_tls_mandatory_protocols = !SSLv2
< smtp_tls_note_starttls_offer = no
---
> smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
> smtp_tls_note_starttls_offer = yes
601c669
< smtp_tls_protocols = !SSLv2
---
> smtp_tls_protocols = !SSLv2, !SSLv3
604c672,673
< smtp_tls_security_level =
---
> smtp_tls_security_level = may
> smtp_tls_servername =
606a676
> smtp_tls_trust_anchor_file =
607a678,679
> smtp_tls_wrappermode = no
> smtp_transport_rate_delay = $default_transport_rate_delay
614,617c686,690
< smtpd_client_connection_count_limit = 50
< smtpd_client_connection_rate_limit = 0
< smtpd_client_event_limit_exceptions = ${smtpd_client_connection_limit_exceptions:$mynetworks}
< smtpd_client_message_rate_limit = 0
---
> smtpd_client_auth_rate_limit = 0
> smtpd_client_connection_count_limit = 4
> smtpd_client_connection_rate_limit = 120
> smtpd_client_event_limit_exceptions = $mynetworks
> smtpd_client_message_rate_limit = 120
620,621c693,694
< smtpd_client_recipient_rate_limit = 0
< smtpd_client_restrictions =
---
> smtpd_client_recipient_rate_limit = 120
> smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated, check_helo_access hash:/etc/postfix/whitelisted_domains, reject_unknown_client_hostname
627a701
> smtpd_dns_reply_filter =
630c704
< smtpd_error_sleep_time = 1s
---
> smtpd_error_sleep_time = 4s
633,636c707,714
< smtpd_forbidden_commands = CONNECT GET POST
< smtpd_hard_error_limit = ${stress?1}${stress:20}
< smtpd_helo_required = no
< smtpd_helo_restrictions =
---
> smtpd_forbid_bare_newline = no
> smtpd_forbid_bare_newline_exclusions = $mynetworks
> smtpd_forbid_bare_newline_reject_code = 550
> smtpd_forbid_unauth_pipelining = no
> smtpd_forbidden_commands = CONNECT GET POST regexp:{{/^[^A-Z]/ Bogus}}
> smtpd_hard_error_limit = 10
> smtpd_helo_required = yes
> smtpd_helo_restrictions = permit_mynetworks, check_helo_access hash:/etc/postfix/whitelisted_domains, permit_sasl_authenticated
638c716
< smtpd_junk_command_limit = ${stress?1}${stress:100}
---
> smtpd_junk_command_limit = ${stress?{1}:{100}}
639a718
> smtpd_milter_maps =
640a720
> smtpd_min_data_rate = 500
644c724,725
< smtpd_per_record_deadline = ${stress?yes}${stress:no}
---
> smtpd_per_request_deadline = ${smtpd_per_record_deadline?{$smtpd_per_record_deadline}:{${stress?{yes}:{no}}}}
> smtpd_policy_service_default_action = 451 4.3.5 Server configuration problem
646a728,730
> smtpd_policy_service_policy_context =
> smtpd_policy_service_request_limit = 0
> smtpd_policy_service_retry_delay = 1s
647a732
> smtpd_policy_service_try_limit = 2
654c739
< smtpd_recipient_restrictions =
---
> smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, check_recipient_access hash:/etc/postfix/whitelisted_domains, reject_unauth_destination
655a741
> smtpd_reject_footer_maps =
658c744,745
< smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination
---
> smtpd_relay_before_recipient_restrictions = ${{$compatibility_level} <level {3.6} ? {no} : {yes}}
> smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
663a751
> smtpd_sasl_mechanism_filter = !external, static:rest
664a753
> smtpd_sasl_response_limit = 12288
665a755
> smtpd_sasl_service = smtp
669c759
< smtpd_sender_restrictions =
---
> smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/whitelisted_domains, reject_unknown_sender_domain
671,673c761,763
< smtpd_soft_error_limit = 10
< smtpd_starttls_timeout = ${stress?10}${stress:300}s
< smtpd_timeout = ${stress?10}${stress:300}s
---
> smtpd_soft_error_limit = 4
> smtpd_starttls_timeout = ${stress?{10}:{300}}s
> smtpd_timeout = ${stress?{10}:{300}}s
678c768
< smtpd_tls_auth_only = no
---
> smtpd_tls_auth_only = yes
680,681c770,772
< smtpd_tls_cert_file =
< smtpd_tls_ciphers = export
---
> smtpd_tls_cert_file = /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
> smtpd_tls_chain_files =
> smtpd_tls_ciphers = medium
683,684c774,775
< smtpd_tls_dh1024_param_file =
< smtpd_tls_dh512_param_file =
---
> smtpd_tls_dh1024_param_file = /etc/ssl/certs/dhparam.pem
> smtpd_tls_dh512_param_file = /etc/ssl/certs/dhparam.pem
688c779
< smtpd_tls_eecdh_grade = strong
---
> smtpd_tls_eecdh_grade = ultra
690,693c781,784
< smtpd_tls_fingerprint_digest = md5
< smtpd_tls_key_file = $smtpd_tls_cert_file
< smtpd_tls_loglevel = 0
< smtpd_tls_mandatory_ciphers = medium
---
> smtpd_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> smtpd_tls_key_file = /etc/letsencrypt/live/opensourceecology.org/privkey.pem
> smtpd_tls_loglevel = 1
> smtpd_tls_mandatory_ciphers = high
695,696c786,787
< smtpd_tls_mandatory_protocols = !SSLv2
< smtpd_tls_protocols =
---
> smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
> smtpd_tls_protocols = !SSLv2, !SSLv3
699c790
< smtpd_tls_security_level =
---
> smtpd_tls_security_level = may
705a797,798
> smtputf8_autodetect_classes = sendmail, verify
> smtputf8_enable = ${{$compatibility_level} <level {1} ? {no} : {yes}}
714a808
> strict_smtputf8 = no
718c812
< syslog_name = ${multi_instance_name:postfix}${multi_instance_name?$multi_instance_name}
---
> syslog_name = ${multi_instance_name?{$multi_instance_name}:{postfix}}
720a815,816
> tls_config_file = default
> tls_config_name =
721a818
> tls_dane_digests = sha512 sha256
722a820
> tls_eecdh_auto_curves = X25519 X448 prime256v1 secp521r1 secp384r1
725,726c823,825
< tls_export_cipherlist = aNULL:-aNULL:ALL:+RC4:@STRENGTH
< tls_high_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH
---
> tls_export_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:LOW:EXPORT:+RC4:@STRENGTH
> tls_fast_shutdown_enable = yes
> tls_high_cipherlist = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA
728,729c827,828
< tls_low_cipherlist = aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH
< tls_medium_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH
---
> tls_low_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:LOW:+RC4:@STRENGTH
> tls_medium_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH
736a836,858
> tls_server_sni_maps =
> tls_session_ticket_cipher = aes-256-cbc
> tls_ssl_options = NO_COMPRESSION
> tls_wildcard_matches_multiple_labels = yes
> tlsmgr_service_name = tlsmgr
> tlsproxy_client_CAfile = $smtp_tls_CAfile
> tlsproxy_client_CApath = $smtp_tls_CApath
> tlsproxy_client_cert_file = $smtp_tls_cert_file
> tlsproxy_client_chain_files = $smtp_tls_chain_files
> tlsproxy_client_dcert_file = $smtp_tls_dcert_file
> tlsproxy_client_dkey_file = $smtp_tls_dkey_file
> tlsproxy_client_eccert_file = $smtp_tls_eccert_file
> tlsproxy_client_eckey_file = $smtp_tls_eckey_file
> tlsproxy_client_enforce_tls = $smtp_enforce_tls
> tlsproxy_client_fingerprint_digest = $smtp_tls_fingerprint_digest
> tlsproxy_client_key_file = $smtp_tls_key_file
> tlsproxy_client_loglevel = $smtp_tls_loglevel
> tlsproxy_client_loglevel_parameter = smtp_tls_loglevel
> tlsproxy_client_per_site = $smtp_tls_per_site
> tlsproxy_client_policy_maps = ${tlsproxy_client_policy:$smtp_tls_policy_maps}
> tlsproxy_client_scert_verifydepth = $smtp_tls_scert_verifydepth
> tlsproxy_client_security_level = ${tlsproxy_client_level:$smtp_tls_security_level}
> tlsproxy_client_use_tls = $smtp_use_tls
744a867
> tlsproxy_tls_chain_files = $smtpd_tls_chain_files
763d885
< tlsproxy_tls_session_cache_timeout = $smtpd_tls_session_cache_timeout
789c911,912
< virtual_alias_domains = $virtual_alias_maps
---
> virtual_alias_address_length_limit = 1000
> virtual_alias_domains = openbuildinginstitute.org osedev.org
791c914
< virtual_alias_maps = $virtual_maps
---
> virtual_alias_maps = hash:/etc/postfix/virtual
795a919
> virtual_delivery_status_filter = $default_delivery_status_filter
815a940
> virtual_transport_rate_delay = $default_transport_rate_delay
user@ose:~/tmp/hetzner3/postfix$
  1. here's what I noticed
    1. disable_vrfy_command changed from 'no' to 'yes'
      1. docs say enabling this "stops some techniques used to harvest email addresses.", so that's probably good https://www.postfix.org/postconf.5.html
    2. inet_interfaces now includes 127.0.0.1 and [::1]
      1. Not sure why that was missing before, but I don't see any issues with it
      2. note that it's not bound to any external IPs. This seems problematic, but it's as-desired. OSE doesn't use postfix for receiving mail. Our MX records point to our free Google Workplace account. We just use postfix for *sending* mail, which greatly reduces a lot of the risk and complexity.
    3. lmtp_tls_ciphers changed from "export" to "medium"
      1. according to the docs " In Postfix ≥ 3.8 this [export] cipher grade is always identical to "medium"" https://www.postfix.org/postconf.5.html#smtp_tls_policy_maps
      2. I'm guessing this is a shadow of when the US had laws limiting the export of some crypto. Jesus, CentOS 7 is old. That shit changed in, what, the 90s?
      3. anyway, there's no change here
      4. the dirty part of TLS encryption in emails is that it's only opportunistic and always vulnerable to downgrade attacks (eg sslstrip). To require encryption between our server and other email servers would be going against the RFC standard, and it would impair our (eg phpList) email delivery :( so we'll leave this at the default
    4. lmtp_tls_fingerprint_digest changed from md5 to sha256, if possible
      1. that's good
    5. lmtp_tls_mandatory_protocols changed from disallowing SSLv2 to disallowing both SSLv2 and SSLv3
      1. that's good. we should only be using TLS
    6. same as ^ with lmtp_tls_protocols
    7. mailbox_command changed from '/usr/bin/procmail' to
      1. idk what that means, but I don't think it matters for us
    8. myhostname changed from 'mailer.opensourceecology.org' to 'hetzner3.opensourceecology.org'
      1. yeah, I named the old server 'mail' hoping it would decrease risk of email bounces, but that had some weird consequences. 'hetzner3' seems to work, afaict
    9. 'mynetworks' changed from '127.0.0.1/32' to '127.0.0.1/32 [::1]/128'
      1. I see no harm in adding local ipv6
    10. postscreen_blacklist_action disappeared
      1. docs say this was renamed to postscreen_denylist_action https://www.postfix.org/postconf.5.html#postscreen_blacklist_action
      2. looks like it's still 'ignore' in both cases
    11. postscreen_dnsbl_ttl changed from '1h' to '60s'
      1. that's a big change
      2. docs say default is 'https://www.postfix.org/postconf.5.html#postscreen_dnsbl_ttl
      3. docs say this is basically a cache: "The amount of time that postscreen(8) remembers that a client IP address passed a DNS-based reputation test, before it is required to pass that test again. "
      4. I see zero impact on OSE's postfix server, which isn't even exposed to the public internet
      5. docs say
    12. same for postscreen_dnsbl_ttl
    13. smtp_tls_fingerprint_digest similar to above, possibly use sha256
    14. smtp_tls_loglevel changed from 0 to 1
      1. the default is 0, but I guess debian changed this? Or maybe I did. Anyway, docs say it's ok to use anything <2
    15. smtp_tls_note_starttls_offer changed from 'no' to 'yes'
      1. also just affects logging. default is 'no'. So now we just log the hostname. Should be fine.
    16. smtp_tls_protocols changed, same as above, to disable SSLv3 too
    17. smtpd_client_message_rate_limit changed from 0 to 120
      1. docs say the default is 0. We probably would be fine to have this at 0 for OSE's purposes. But I don't see an issue at 120 either.
    18. smtpd_client_restrictions changed from to 'permit_mynetworks, permit_sasl_authenticated, check_helo_access hash:/etc/postfix/whitelisted_domains, reject_unknown_client_hostname'
      1. that should be fine
    19. smtpd_error_sleep_time changed from 1s to 4s
      1. I think that's fine
    20. smtpd_helo_required changed from 'no' to 'yes'
      1. shouldn't matter, and probably a good change
    21. smtpd_relay_restrictions changed from 'permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination' to 'permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination'
      1. so 'defer' changed to 'reject'. I guess that's fine.
    22. smtpd_soft_error_limit changed from 10 to 4
      1. docs suggest this is a tarpit config. After this number of errors, it slows down processing of requests from that client
      2. again, shouldn't affect us
    23. smtpd_tls_auth_only changed from 'no' to 'yes'
      1. this apparently only affects sasl auth, which is forced to use TLS now
      2. should be fine; we're not doing any SASL auth
    24. smtpd_tls_auth_only changed from to '/etc/letsencrypt/live/opensourceecology.org/fullchain.pem'
      1. wow I can't believe this was empty before. So I guess we were one of those servers that was sending all smtp messages without encryption? That or snake oil.
      2. this is a good change
    25. smtpd_tls_dh1024_param_file and smtpd_tls_dh512_param_file got set to our ansible-made dhparam file; that's good
    26. smtpd_tls_eecdh_grade changed from 'strong' to 'ultra'
      1. docs say that's an increase from 128 bits to 192 bits. It says it's twice as slow. should be fine.
    27. smtpd_tls_mandatory_ciphers changed from 'medium' to 'high'
      1. that's probably for the best
    28. smtpd_tls_security_level changed from to 'may'
      1. yeah, apparently empty means "don't use TLS". So this is a very good change.
      2. again, 'encrypt' enforces TLS, but that's a violation of RFC 2487 https://tools.ietf.org/html/rfc2487
    29. tls_export_cipherlist changed from 'aNULL:-aNULL:ALL:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:LOW:EXPORT:+RC4:@STRENGTH'
      1. looks like "ALL" was removed ,so that's probably good
    30. tls_high_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH' to 'EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA'
      1. ALL was removed. I don't like that it lists SSLv3
      2. usually my go-to is https://bettercrypto.org/ but it looks like they took the site down saying it's not longer updated :(
      3. the docs say that you are "strongly encouraged to not change" this setting
      4. well, it appears that we are changing it
user@ose:~/sandbox_local/ansible/hetzner3$ grep -ir cipherlist *
roles/maltfield.postfix/templates/main.cf.j2:tls_high_cipherlist=EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA
roles/maltfield.postfix/templates/master.cf.j2:  -o tls_preempt_cipherlist=yes
user@ose:~/sandbox_local/ansible/hetzner3$
      1. here we can view the default settings. that's not very helpful
root@hetzner3 ~ # postconf -d | grep -i tls_high_cipherlist
tls_high_cipherlist = aNULL:-aNULL:HIGH:@STRENGTH
root@hetzner3 ~ #
      1. this SE answer cites bettercrypto.org, and it includes SSLv3 too; I guess we'll leave it as-is https://serverfault.com/a/670359
    2. tls_low_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:LOW:+RC4:@STRENGTH'
    3. tls_medium_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH'
    4. tlsproxy_tls_session_cache_timeout disappeared
      1. docs say it's obsolete and replaced by smtpd_tls_session_cache_timeout
    5. virtual_alias_domains changed from '$virtual_alias_maps' to 'openbuildinginstitute.org osedev.org'
    6. virtual_alias_maps changed from '$virtual_maps' to 'hash:/etc/postfix/virtual'
      1. note that '$virtual_maps' isn't set on hetzner2, so I don't think any alias domains were defined
      2. it might actually be better to use an external maps file (rather than just hardcoding it), but I don't want to break it; let's leave it as-is
  2. there were lots of other changes, but those were the ones that stood-out to me
  3. in general, I'd say the changes were good: err-ing on the side of "more" secure
  4. ok, postfix looks fine to me
  5. SPF records and rDNS were already set
  6. probably after the migration we'd want to remove the old server, but there's nothing to do now until we migrate all the webservers
  7. oh, actually, one thing I never setup was SPF records for non-ose domains (eg obi, osedev, oswh)
  8. I opened the shared ose keepass, and I saw 2 domains in ghandi:
    1. oswarehouse.org
user@disp5497:~$ dig +short -t txt oswarehouse.org
"v=spf1 include:_mailcust.gandi.net ?all"
user@disp5497:~$
    1. opensourcewarehouse.org
  2. I think those are completely unused; the nameserver for the former is ghandi and the nameserver for the later is set to dreamhost
  3. oh, there *is* an spf record for the ghandi one (oswarehouse.org), but it's not something we ever set = ""v=spf1 include:_mailcust.gandi.net ?all""
  4. I didn't log into dreamhost, but a query shows the same: we never set it
user@disp5497:~$ dig +short -t txt opensourcewarehouse.org
"v=spf1 mx include:netblocks.dreamhost.com include:relay.mailchannels.net -all"
user@disp5497:~$
  1. I'm not going to worry about email records (MX, spf) for these domains
  2. I think the others are on cloudflare? let's check
  3. I logged into cloudflare, but I only see 1 domain = opensourceecology.org
  4. yeah, I think this is why I left a note for myself to research further; I don't know where is OBI
  5. ah, dig says the nameserver is dreamhost
user@disp5497:~$ dig NS openbuildinginstitute.org

; <<>> DiG 9.16.50-Debian <<>> NS openbuildinginstitute.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7804
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openbuildinginstitute.org.	IN	NS

;; ANSWER SECTION:
openbuildinginstitute.org. 14400 IN	NS	ns1.dreamhost.com.
openbuildinginstitute.org. 14400 IN	NS	ns2.dreamhost.com.
openbuildinginstitute.org. 14400 IN	NS	ns3.dreamhost.com.

;; Query time: 116 msec
;; SERVER: 10.139.1.1#53(10.139.1.1)
;; WHEN: Fri Feb 14 17:19:06 -05 2025
;; MSG SIZE  rcvd: 121

user@disp5497:~$
  1. I tried to login to dreamhost, but I couldn't. I entered the creds in the shared ose keepass, then it demanded an OTP sent to some email.
  2. I decided to create a new gapps account for this, to fix this problem in the future
  3. I sent an email to Marcin asking him to change the email address with that account from his personal email to this new, shared email.
  4. I guess the domain/spf work is paused until I can gain access to dreamhost again
  5. looks like the spf records for obi don't matter either
user@disp5497:~$ dig -t txt openbuildinginstitute.org

; <<>> DiG 9.16.50-Debian <<>> -t txt openbuildinginstitute.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10607
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openbuildinginstitute.org.	IN	TXT

;; ANSWER SECTION:
openbuildinginstitute.org. 300	IN	TXT	"v=spf1 mx include:netblocks.dreamhost.com include:relay.mailchannels.net -all"

;; Query time: 133 msec
;; SERVER: 10.139.1.1#53(10.139.1.1)
;; WHEN: Fri Feb 14 17:40:26 -05 2025
;; MSG SIZE  rcvd: 144

user@disp5497:~$
  1. ...but I'll still need access to dreamhost, anyway – to update the A and AAAA records for the migration to hetzner3
  2. ...
  3. another unknown that I wanted to figure-out was if we could host the same domain (eg for obi and osemain) on two distinct IP addresses
    1. but we only have one IPv4 address. We could pay for a second one, but we have access to tons of free IPv6 addreses. The question is: does that work with /etc/hosts? Will it work for Marcin and Catarina?
  4. first I checked the IP addresses we *do* have
root@hetzner3 /etc/nginx/sites-enabled # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
	link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
	inet 127.0.0.1/8 scope host lo
	   valid_lft forever preferred_lft forever
	inet6 ::1/128 scope host noprefixroute 
	   valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
	link/ether 90:1b:0e:c4:28:b4 brd ff:ff:ff:ff:ff:ff
	inet 144.76.164.201/27 brd 144.76.164.223 scope global enp0s31f6
	   valid_lft forever preferred_lft forever
	inet6 2a01:4f8:200:40d7::2/64 scope global 
	   valid_lft forever preferred_lft forever
	inet6 fe80::921b:eff:fec4:28b4/64 scope link 
	   valid_lft forever preferred_lft forever
root@hetzner3 /etc/nginx/sites-enabled #
  1. woah, that says we have a '/27' IP address. Is that a misconfig? /32 would be 1 IP. A /27 would be 32 IPs https://www.freecodecamp.org/news/subnet-cheat-sheet-24-subnet-mask-30-26-27-29-and-other-ip-address-cidr-network-references/
  2. I really, really doubt hetzner gave us 32 IP addresses, but we should check
  3. this subnet calculator says "144.76.164.201/27" would mean that we have "144.76.164.193 - 144.76.164.222" https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=27&cip=144.76.164.201&ctype=ipv4&x=Calculate
  4. I logged into the hetzner site for managing hetzner 3 (and hetzner2) https://ro
Retrieved from "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2025_Q1&oldid=303833"