Maltfield Log/2025 Q1: Difference between revisions
Jump to navigation
Jump to search
Site Under Maintenance
Sorry, this website is temporarily down for maintenance. Please check back later.
root@mail:/etc/apache2#
(Feb 16) |
(Feb 20) |
||
| Line 7: | Line 7: | ||
# [[User:Maltfield]] | # [[User:Maltfield]] | ||
# [[Special:Contributions/Maltfield]] | # [[Special:Contributions/Maltfield]] | ||
=Thr Feb 20, 2025= | |||
# Marcin sent me an email saying that a wiki user just maliciously deleted a bunch of articles on the wiki, and asked me if there's a way to easily UNdelete in batches | |||
# I asked about this on SE | |||
## https://webmasters.stackexchange.com/questions/146516/mass-undelete-on-mediawiki-unnuke-restore | |||
## https://webapps.stackexchange.com/questions/180833/mass-undelete-on-mediawiki-unnuke-restore | |||
# But previously I had looked at these two extensions, and found the third. Worth looking into. | |||
## https://www.mediawiki.org/wiki/Extension:DeleteBatch | |||
## https://www.mediawiki.org/wiki/Extension:SmiteSpam | |||
## https://www.mediawiki.org/wiki/Extension:Moderation | |||
=Sun Feb 16, 2025= | =Sun Feb 16, 2025= | ||
Revision as of 17:09, 23 February 2025
My work log from the first quarter of the year 2025. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
See Also
Thr Feb 20, 2025
- Marcin sent me an email saying that a wiki user just maliciously deleted a bunch of articles on the wiki, and asked me if there's a way to easily UNdelete in batches
- I asked about this on SE
- But previously I had looked at these two extensions, and found the third. Worth looking into.
Sun Feb 16, 2025
- hmm, I still see only one entry for the mediawiki cron log, despite the fact that it's definitely set to append now
root@hetzner3 /etc/nginx # cat /etc/cron.d/mediawiki_cron # Ansible managed SHELL=/bin/bash # regenerate new CAPTCHA images 20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &>> /var/log/cron/mediawiki_generate_captchas.log root@hetzner3 /etc/nginx # cat /var/log/cron/mediawiki_generate_captchas.log Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads... root@hetzner3 /etc/nginx #
- I updated the script to print the datestamp; hopefully that'll help us debug this now and in the future
rroot@hetzner3 /usr/local/bin # diff mediawiki_generate_captchas.sh.20250216 mediawiki_generate_captchas.sh
5c5
< # Version: 0.2
---
> # Version: 0.3
11c11
< # Updated: 2025-02-14
---
> # Updated: 2025-02-16
43a44,48
> stamp=`date -u +%Y%m%d_%H%M%S`
> echo "================================================================================"
> echo "INFO: Beginning mediawiki_generate_captchas.sh Run on ${stamp}"
> echo "================================================================================"
>
64c69,70
< sudo -u www-data bash -c "python3 '${CAPTCHA_SCRIPT_PATH}' --wordlist=${WORDLIST_FILE_PATH} --key='${CAPTCHA_SECRET}' --output='${CAPTCHA_DIR_PATH}' --font='${FONT_FILE_PATH}' --count='${NUM_FILES}'"
---
> echo "INFO: generating ${NUM_FILES} new captcha images"
> sudo -u www-data bash -c "time nice python3 '${CAPTCHA_SCRIPT_PATH}' --wordlist=${WORDLIST_FILE_PATH} --key='${CAPTCHA_SECRET}' --output='${CAPTCHA_DIR_PATH}' --font='${FONT_FILE_PATH}' --count='${NUM_FILES}'"
68c74,75
< sudo -u www-data bash -c "ls -t ${CAPTCHA_DIR_PATH}/* | sed -e '1,${NUM_FILES}d' | xargs rm"
---
> echo "INFO: deleting all but the most-recent ${NUM_FILES} captcha images"
> sudo -u www-data bash -c "time nice ls -t ${CAPTCHA_DIR_PATH}/* | sed -e '1,${NUM_FILES}d' | xargs rm"
69a77
> echo "INFO: exiting at $(date -u +%Y%m%d_%H%M%S)"
root@hetzner3 /usr/local/bin #
root@hetzner3 /usr/local/bin # ./mediawiki_generate_captchas.sh
================================================================================
INFO: Beginning mediawiki_generate_captchas.sh Run on 20250216_213845
================================================================================
INFO: generating 500 new captcha images
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
real 0m1,430s
user 0m1,385s
sys 0m0,049s
INFO: deleting all but the most-recent 500 captcha images
real 0m0,028s
user 0m0,020s
sys 0m0,011s
INFO: exiting at 20250216_213846
root@hetzner3 /usr/local/bin #
- note I also added 'time' and 'nice' and a timestamp at the end.
- ...
- yesterday we fixed a lot of performance configurations of php & apache
- then we began looking at verifying the nginx security configs, which we already found an issue: wp-login.php isn't being blocked
user@disp3993:~$ curl -IL https://store.opensourceecology.org/wp-login.php HTTP/1.1 302 Found Server: nginx Date: Sun, 16 Feb 2025 21:02:14 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: / X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Referrer-Policy: no-referrer-when-downgrade X-Varnish: 560106 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Sun, 16 Feb 2025 21:02:14 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 85328 Connection: keep-alive X-Pingback: https://store.opensourceecology.org/xmlrpc.php Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink X-Frame-Options: SAMEORIGIN Vary: Accept-Encoding X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Referrer-Policy: no-referrer-when-downgrade X-Varnish: 331395 460383 Age: 79262 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" user@disp3993:~$
- ah, looks like I commented it out 4 months ago when I first realized that the 'rename-wp-login' plugin was removed by wordpress due to security issues
user@ose:~/sandbox_local/ansible/hetzner3$ tail roles/maltfield.apache/templates/security.virtualhost.include.j2 # block access to 'wp-login.php' from brute-forcers; # see wp plugin 'rename-wp-login' # TODO: 2024-10: we need to re-enable this after we find a replacement for the # (now-deprecated) 'rename-wp-login' wordpress plugin # * https://wordpress.org/plugins/rename-wp-login/ # <LocationMatch ".*wp-login.php"> # Require all denied # </LocationMatch> user@ose:~/sandbox_local/ansible/hetzner3$
- I'm actually surprised it's a config in apache, not nginx
- but, anyway, I uncommented it in ansible and pushed it out https://github.com/OpenSourceEcology/ansible/commit/12257eee3bd5134f29af86b11e104a1245144ef3
- after restarting apache and clearing the varnish cache, it's now working as-desired
user@disp3993:~$ curl -IL https://store.opensourceecology.org/wp-login.php HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:12:35 GMT Content-Type: text/html; charset=iso-8859-1 Content-Length: 199 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Varnish: 560114 Age: 0 Via: 1.1 varnish (Varnish/7.1) user@disp3993:~$ curl -IL store.opensourceecology.org/wp-login.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Sun, 16 Feb 2025 21:12:43 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://store.opensourceecology.org/wp-login.php X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=1;includeSubDomains HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:12:43 GMT Content-Type: text/html; charset=iso-8859-1 Content-Length: 199 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Varnish: 35975 Age: 0 Via: 1.1 varnish (Varnish/7.1) user@disp3993:~$
- actually, I do have a config that should have blocked it in nginx
root@hetzner3 /etc/nginx # cat conf.d/secure.include
# Ansible managed
################################################################################
# File: secure.include
# Version: 0.2
# Purpose: Basic security settings that couldn't be put in the main nginx.conf.
# This should be included in the server{} blocks nginx vhosts.
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2017-11-23
# Updated: 2024-08-07
################################################################################
# whitelist requests to disable TRACE & DELETE
if ($request_method !~ ^(GET|HEAD|POST)$ ) {
# note: 444 is a meta code; it doesn't return anything, actually
# it just logs, drops, & closes the connection (useful
# against malware)
return 444;
}
## block some bot's useragents (may need to remove some, if impacts SEO)
# 2020-05-03: I'm removing this from my config for my migration to hetzner
# Logic: If a bot is playing bad, they'll be blocked by DOS rate
# limits, anyway. Otherwise, let 'em scrape.
#if ($blockedagent) {
# return 403;
#}
# prevent access to any dirs/files starting with a dot (.)
location ~ /\.(?!well\-known) { access_log off; log_not_found off; deny all; }
# prevent access to any config files
location ~ config\. { access_log off; log_not_found off; deny all; }
root@hetzner3 /etc/nginx #
- looks like it's included. what happened?
root@hetzner3 ~ # cat /etc/nginx/sites-enabled/store.opensourceecology.org.conf
# Ansible managed
################################################################################
# File: store.opensourceecology.org.conf
# Version: 0.3
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
# Updated: 2024-10-04
################################################################################
server {
access_log /var/log/nginx/store.opensourceecology.org/access.log main;
error_log /var/log/nginx/store.opensourceecology.org/error.log;
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
listen 127.0.0.1:443;
listen [::1]:443;
listen 144.76.164.201:443;
listen [2a01:4f8:200:40d7::3]:443;
server_name store.opensourceecology.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
# root /var/www/html/SITE_DOWN/htdocs/;
# index index.html index.htm;
#
# # force all requests to load exactly this page
# location / {
# try_files $uri /index.html;
# }
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
}
}
root@hetzner3 ~ #
- oh, duh, wp-login.php != wp-config.php
- let's test that, though
- allright, that's working perfect
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-config.php HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:27:53 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive user@disp3993:~$ curl -IL store.opensourceecology.org/config.php HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:27:57 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive user@disp3993:~$ curl -IL store.opensourceecology.org/wp-config.php.bak HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:28:06 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive user@disp3993:~$ curl -IL store.opensourceecology.org/config.php.bak HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:28:10 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive user@disp3993:~$ curl -IL store.opensourceecology.org/config.bak.php HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:28:13 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive user@disp3993:~$ curl -IL store.opensourceecology.org/wp-config.bak.php HTTP/1.1 403 Forbidden Server: nginx Date: Sun, 16 Feb 2025 21:28:16 GMT Content-Type: text/html Content-Length: 146 Connection: keep-alive user@disp3993:~$
- here's the list of things to test
- wp-login.php shouldn't be accessible
- phplist config.php shouldn't be accessible (I want to confirm because old had ".*" prefix and new doesn't)
- .php files in wp-content uploads dirs shouldn't be executable
- anything other than GET POST HEAD should be denied
- anything that starts with a dot (other than .well-known) should be denied (test .git & .htaccess)
- indexes on a dir without 'index.php' shouldn't show a directory listing
- .htaccess shouldn't allow changing things
- so the first two are done
- let's test for php executability in wp, phplist, and mediawiki
- first we create some php script and put it in all the dirs that are writeable by the web server
dirs="/var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/tmp/ /var/www/html/phplist.opensourceecology.org/public_html/uploadimages/ /var/www/html/wiki.opensourceecology.org/htdocs/images/"
for d in $dirs; do
cat > ${d}/test.php <<'EOF'
<pre>
<?php
echo "if this php was processed, that's very bad!";
?>
EOF
chown not-apache:www-data ${d}/test.php chmod 0040 ${d}/test.php
done
- I actually expected it to dump back the raw php, but it looks like it returns 403 for any files that end in .php; that's better
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/test.php<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/doesnotexist.php <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$
- this looks good
user@disp3993:~$ curl -L phplist.opensourceecology.org/uploadimages/test.php <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ user@disp3993:~$ curl -L wiki.opensourceecology.org/images/test.php <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$
- oh crap, I did find one that worked!
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/test.php <pre> if this php was processed, that's very bad!
user@disp3993:~$
- this is actually a good opportunity. Before, when we used mod_php, we wouldn't just 403 refuse to serve it on nginx, but we'd also configure apache not to pass it to mod_php. So let's take this opportunity to add that layered security at the apache layer too
- we're just using the default php-fpm config file; let's create a backup and see if we can harden it to *not* pass files in these dirs back to the php-fpm daemon/proxy
root@hetzner3 /etc/apache2/conf-available # cp php8.2-fpm.conf php8.2-fpm.conf.20250216.conf root@hetzner3 /etc/apache2/conf-available #
- oh, looks like I already configured this here, but for some reason the 'tmp' dir is omitted
root@hetzner3 /etc/apache2/conf-available # head -n30 security.virtualhost.include
- Ansible managed
- File: security.virtualhost.include
- Version: 0.4
- Purpose: File includes some common security-hardening that's intended to be
- Include()d into other vhost files' <VirtualHost> blocks
- Author: Michael Altfield <michael@michaelaltfield.net>
- Created: 2024-09-14
- Updated: 2025-02-16
# don't execute any php files inside uploads directories <LocationMatch "/wp-content/uploads/"> SetHandler ! </LocationMatch> <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> Require all denied </LocationMatch>
- <LocationMatch "/wp-content/tmp/">
- SetHandler !
- </LocationMatch>
- <LocationMatch "/wp-content/tmp/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
- Require all denied
- </LocationMatch>
<LocationMatch "/uploadimages/"> SetHandler ! </LocationMatch> root@hetzner3 /etc/apache2/conf-available #
- ok, so what we needed to test here was that 'SetHandler' would successfully prevent it from being executed by php
- I uncommented just this one block, restarted apache, and cleared the cache
root@hetzner3 /etc/apache2/conf-available # head -n30 security.virtualhost.include # Ansible managed ################################################################################ # File: security.virtualhost.include # Version: 0.4 # Purpose: File includes some common security-hardening that's intended to be # Include()d into other vhost files' <VirtualHost> blocks # Author: Michael Altfield <michael@michaelaltfield.net> # Created: 2024-09-14 # Updated: 2025-02-16 ################################################################################ # don't execute any php files inside uploads directories <LocationMatch "/wp-content/uploads/"> SetHandler ! </LocationMatch> <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> Require all denied </LocationMatch> <LocationMatch "/wp-content/tmp/"> SetHandler ! </LocationMatch> # <LocationMatch "/wp-content/tmp/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> # Require all denied # </LocationMatch> <LocationMatch "/uploadimages/"> SetHandler ! </LocationMatch> root@hetzner3 /etc/apache2/conf-available #
- this confirms that that config block is working as-expected
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/test.php <pre> <?php echo "if this php was processed, that's very bad!"; ?>
user@disp3993:~$
- now we uncomment the second block, and it's totally blocked; perfect
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/test.php <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$
- I updated this in ansible, pushed it out, and re-confirmed it's working as-expected https://github.com/OpenSourceEcology/ansible/commit/d523d0ef4b218fd7189d336f1c8f8fea6dc3dbe8
- the next verification is to make sure only POST, HEAD, and GET requests are allowed
- it looks like we do have this set in our /etc/nginx/conf.d/secure.include
root@hetzner3 /etc/nginx # grep -ir POST *
conf.d/secure.include: if ($request_method !~ ^(GET|HEAD|POST)$ ) {
mime.types: application/postscript ps eps ai;
nginx.conf: # allow large posts for image uploads
nginx.conf.1282157.2024-09-28@23:10:52~: # allow large posts for image uploads
nginx.conf.3932816.2024-10-05@03:45:22~: # allow large posts for image uploads
root@hetzner3 /etc/nginx #
You have mail in /var/mail/root
root@hetzner3 /etc/nginx # cat conf.d/secure.include
# Ansible managed
################################################################################
# File: secure.include
# Version: 0.2
# Purpose: Basic security settings that couldn't be put in the main nginx.conf.
# This should be included in the server{} blocks nginx vhosts.
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2017-11-23
# Updated: 2024-08-07
################################################################################
# whitelist requests to disable TRACE & DELETE
if ($request_method !~ ^(GET|HEAD|POST)$ ) {
# note: 444 is a meta code; it doesn't return anything, actually
# it just logs, drops, & closes the connection (useful
# against malware)
return 444;
}
## block some bot's useragents (may need to remove some, if impacts SEO)
# 2020-05-03: I'm removing this from my config for my migration to hetzner
# Logic: If a bot is playing bad, they'll be blocked by DOS rate
# limits, anyway. Otherwise, let 'em scrape.
#if ($blockedagent) {
# return 403;
#}
# prevent access to any dirs/files starting with a dot (.)
location ~ /\.(?!well\-known) { access_log off; log_not_found off; deny all; }
# prevent access to any config files
location ~ config\. { access_log off; log_not_found off; deny all; }
root@hetzner3 /etc/nginx #
- quick test shows the 444 functioning as expected
user@disp3993:~$ curl --request OPTIONS www.opensourceecology.org curl: (52) Empty reply from server user@disp3993:~$ curl --request OPTIONS www.openbuildinginstitute.org curl: (52) Empty reply from server user@disp3993:~$ curl --request OPTIONS store.opensourceecology.org curl: (52) Empty reply from server user@disp3993:~$ curl --request OPTIONS phplist.opensourceecology.org curl: (52) Empty reply from server user@disp3993:~$ curl --request OPTIONS wiki.opensourceecology.org curl: (52) Empty reply from server user@disp3993:~$
- curiously I don't get the same from CONNECT, but I guess this is something built-into nginx?
user@disp3993:~$ curl --request CONNECT wiki.opensourceecology.org <html> <head><title>405 Not Allowed</title></head> <body> <center><h1>405 Not Allowed</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl --request CONNECT store.opensourceecology.org <html> <head><title>405 Not Allowed</title></head> <body> <center><h1>405 Not Allowed</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$
- a full list of HTTP methods can be found here https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods
- curiously our varnish configs seem to allow many more types – for some reason. But that's fine.
user@ose:~/sandbox_local/ansible/hetzner3$ grep -ir HEAD * | grep -i POST | grep -i GET
roles/maltfield.varnish/templates/www.openbuildinginstitute.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/oswh.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/fef.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/forum.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/www.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/store.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/phplist.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/microfactory.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/seedhome.openbuildinginstitute.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.nginx/templates/secure.include.j2: if ($request_method !~ ^(GET|HEAD|POST)$ ) {
roles/maltfield.apache/templates/security.directory.include.j2: <LimitExcept GET POST HEAD>
user@ose:~/sandbox_local/ansible/hetzner3$
- as that shows, apache also limits it
root@hetzner3 /etc/apache2 # cat conf-available/security.directory.include # Ansible managed ################################################################################ # File: security.directory.include # Version: 0.1 # Purpose: File includes some common security-hardening that's intended to be # Include()d into other vhost files' <Directory> blocks # Author: Michael Altfield <michael@michaelaltfield.net> # Created: 2024-09-14 # Updated: 2024-09-14 ################################################################################ #Options -Indexes -Includes Options None AllowOverride None # we can't do this when using mod_remoteip #Require ip 127.0.0.1 # Harden vhost docroot by blocking all request types except the 3 essentials # Note: The debian wiki actually says not to use this since it can result in # disabling auth modules, so be careful # * https://wiki.debian.org/Apache/Hardening <LimitExcept GET POST HEAD> Require all denied </LimitExcept> root@hetzner3 /etc/apache2 #
- our next test is that anything starting with a dot is refused (eg .git, .htaccess, .htpasswd, .svn, etc)
- this looks good
user@disp3993:~$ curl store.opensourceecology.org/.git <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl store.opensourceecology.org/.htaccess <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl store.opensourceecology.org/.htpasswd <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl store.opensourceecology.org/.svn <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl store.opensourceecology.org/somedir/.svn <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl store.opensourceecology.org/somedir/.git <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl wiki.opensourceecology.org/somedir/.git <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$ curl wiki.opensourceecology.org/.htpasswd <html> <head><title>403 Forbidden</title></head> <body> <center><h1>403 Forbidden</h1></center> <hr><center>nginx</center> </body> </html> user@disp3993:~$
- and .well-known is the only one that worked
user@disp3993:~$ curl -L wiki.opensourceecology.org/.well-known <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> user@disp3993:~$
- ok, next test is to make sure Indexes aren't working
- I guess for this, let's just access the uploads dirs; all of these work as expected
user@disp3993:~$ curl -L wiki.opensourceecology.org/images/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> user@disp3993:~$
- lastly, we need to test that .htaccess files don't work
- I'll create one in all the upload dirs again
dirs="/var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/tmp/ /var/www/html/phplist.opensourceecology.org/public_html/uploadimages/ /var/www/html/wiki.opensourceecology.org/htdocs/images/"
for d in $dirs; do
cat > ${d}/.htaccess <<'EOF'
Options +Indexes
EOF
chown not-apache:www-data ${d}/test.php
chmod 0040 ${d}/test.php
done
- spot check after running
root@hetzner3 /etc/apache2 # ls -lah /var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/.htaccess -rw-r--r-- 1 root root 17 Feb 16 22:47 /var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/.htaccess root@hetzner3 /etc/apache2 # cat /var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/.htaccess Options +Indexes root@hetzner3 /etc/apache2 #
- looks like it's ignoring the .htaccess files; perfect
user@disp3993:~$ curl -L wiki.opensourceecology.org/images/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> user@disp3993:~$
- well, that's all the hardening verification I had on my TODO list. Glad we did that; we fixed the wordpress tmp/ dir issue.
- ...
- I'm still blocked on most of the hetzner3 work, but I have two more items on my TODO list:
- setup the staging server with hetnzer3
- that's a huge & high-hanging-fruit task. last I checked the keys expired.
- migrate a script that I use on my personal site for taking down all of my sites = /usr/local/bin/website_maintance.sh
- this basically just moves the main apache2.conf file to apache2_prod.conf, and then we create a symlink named apache2.conf, which points to it.
- And we have a new apache2_maint.conf file, which points all of the websites to some simple index.html file saying that the server is undergoing maintenance.
- then the script just swaps symlinks as-needed
- setup the staging server with hetnzer3
- ah, this is elucidating; it just swaps the includes for all the vhost config files for a single config file
root@mail:/etc/apache2# diff apache2_prod.conf apache2_maint.conf 230c230 < IncludeOptional sites-enabled/*.conf --- > IncludeOptional sites-available/000-maintenance.conf root@mail:/etc/apache2#
- simple
root@mail:/etc/apache2# cat sites-available/000-maintenance.conf <VirtualHost *:8000>
DocumentRoot /var/www/html/maintenance/htdocs/
CustomLog ${APACHE_LOG_DIR}/access.log combined ErrorLog ${APACHE_LOG_DIR}/error.log
Include 'conf-available/security.virtualhost.include'
<Directory "/var/www/html/maintenance/htdocs/"> Include 'conf-available/security.directory.include' Options +FollowSymLinks </Directory>
# redirects everything to "/index.html" RewriteEngine On RewriteRule ^(.*) /index.html
</VirtualHost> root@mail:/etc/apache2# root@mail:/etc/apache2# ls -lah /var/www/html/maintenance/htdocs/ total 12K d---r-x--- 2 root www-data 4,0K May 18 2022 . d---r-x--- 3 root www-data 4,0K May 18 2022 ..
r----- 1 root www-data 407 May 18 2022 index.html
root@mail:/etc/apache2# cat /var/www/html/maintenance/htdocs/index.html <!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
- apparently I had already mostly set this up with ansible, I just needed to add the binary
root@hetzner3 /etc/apache2 # cat sites-available/000-maintenance.conf
# Ansible managed
################################################################################
# File: 000-maintenance.conf
# Version: 0.1
# Purpose: vhost of a simple "site currently under maintenance" landing page
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-10
# Updated: 2024-09-10
################################################################################
<VirtualHost *:8000>
DocumentRoot /var/www/html/SITE_DOWN/htdocs/
CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorLog ${APACHE_LOG_DIR}/error.log
Include 'conf-available/security.virtualhost.include'
<Directory "/var/www/html/SITE_DOWN/htdocs/">
Include 'conf-available/security.directory.include'
Options +FollowSymLinks
</Directory>
# redirects everything to "/index.html"
RewriteEngine On
RewriteRule ^(.*) /index.html
</VirtualHost>
root@hetzner3 /etc/apache2 #
- I added the binary, but it looks like I'm missing the vhost
root@hetzner3 /etc/apache2 # website_maintenance.sh onYou have mail in /var/mail/root INFO: Enabling Maintenance Mode AH00112: Warning: DocumentRoot [/var/www/html/SITE_DOWN/htdocs/] does not exist Syntax OK INFO: Maintenance Mode is now on root@hetzner3 /etc/apache2 #
- it's on hetzner2
[maltfield@opensourceecology ~]$ ls -lah /var/www/html/SITE_DOWN/ total 12K drwxr-xr-x 3 root root 4.0K Nov 23 2017 . drwxr-xr-x 25 root root 4.0K May 30 2023 .. drwxr-xr-x 2 root root 4.0K Nov 23 2017 htdocs [maltfield@opensourceecology ~]$ ls -lah /var/www/html/SITE_DOWN/htdocs/ total 12K drwxr-xr-x 2 root root 4.0K Nov 23 2017 . drwxr-xr-x 3 root root 4.0K Nov 23 2017 .. -rw-r--r-- 1 root root 303 Nov 23 2017 index.html [maltfield@opensourceecology ~]$ [maltfield@opensourceecology ~]$ cat /var/www/html/SITE_DOWN/htdocs/index.html <!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta charset="UTF-8"> </head> <body> <pre>Sorry, we're down for maintenance. Please check back later.
</body>
</html> [maltfield@opensourceecology ~]$
- but not hetzner3
root@hetzner3 /etc/apache2 # ls -lah /var/www/html total 56K d---r-x--- 13 not-apache www-data 4,0K Feb 14 23:31 . drwxr-xr-x 3 root root 4,0K Sep 25 02:37 .. d---r-x--- 3 not-apache www-data 4,0K Dec 31 01:38 fef.opensourceecology.org d---r-x--- 5 not-apache www-data 4,0K Jul 11 2018 forum.opensourceecology.org ----r----- 1 not-apache www-data 138 Mar 3 2018 .htpasswd d---r-x--- 3 not-apache www-data 4,0K Jan 28 23:55 microfactory.opensourceecology.org drwxr-xr-x 3 root root 4,0K Feb 14 23:31 new-obi d---r-x--- 3 not-apache www-data 4,0K Jan 30 04:18 oswh.opensourceecology.org d---r-x--- 6 not-apache www-data 4,0K Feb 12 23:30 phplist.opensourceecology.org d---r-x--- 4 not-apache www-data 4,0K Dec 26 20:44 seedhome.openbuildinginstitute.org d---r-x--- 4 not-apache www-data 4,0K Feb 16 00:56 store.opensourceecology.org d---r-x--- 4 not-apache www-data 4,0K Feb 13 23:05 wiki.opensourceecology.org d---r-x--- 3 not-apache www-data 4,0K Jan 30 22:18 www.openbuildinginstitute.org d---r-x--- 3 not-apache www-data 4,0K Feb 1 00:23 www.opensourceecology.org root@hetzner3 /etc/apache2 #
- probably we could create some nice-looking SITE_DOWN page with OSE logos, but I'm just going to keep it simple now. it would be trivial to update later.
root@hetzner3 /etc/apache2 # mkdir -p /var/www/html/SITE_DOWN/htdocs/ root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # cat > /var/www/html/SITE_DOWN/htdocs/index.html <<'EOF' <!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta charset="UTF-8"> </head> <body> <pre>Sorry, we're down for maintenance. Please check back later.
</body>
</html>
EOF root@hetzner3 /etc/apache2 #
root@hetzner3 /etc/apache2 # time fix_web_permissions.sh
- cool, it's working
user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/ <!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta charset="UTF-8"> </head> <body> <pre>Sorry, we're down for maintenance. Please check back later.
</body>
</html>
user@disp3993:~$
- and I turn it off on the server
root@hetzner3 /usr/local/bin # website_maintenance.sh off INFO: Disabling Maintenance Mode Syntax OK INFO: Maintenance Mode is now off You have mail in /var/mail/root root@hetzner3 /usr/local/bin #
- and the curl works as-expected again
user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/ <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL was not found on this server.</p> </body></html> user@disp3993:~$
- I updated the wiki, describing how to use this script https://wiki.opensourceecology.org/wiki/OSE_Server#Maintenance_Mode
- ...
- ok, the only task on my TODO list that I'm *not* blocked-on is to fix the sync with the staging server
- the staging server lives on the prod server, and it's only accessible by connecting to the VPN running on the dev server https://wiki.opensourceecology.org/wiki/OSE_Staging_Server
- some time ago the keys expired and everything broke, even between hetzner2 and the staging server
[root@opensourceecology ~]# ip a show tun0 Device "tun0" does not exist. [root@opensourceecology ~]#
- I tried to connect to the VPN, following the documentation on the wiki; it also failed saying my cert expired
user@ose:~/openvpn$ sudo openvpn client.conf 2025-02-16 18:28:17 Note: Kernel support for ovpn-dco missing, disabling data channel offload. 2025-02-16 18:28:17 WARNING: file '/home/user/openvpn/username.txt' is group or others accessible 2025-02-16 18:28:17 OpenVPN 2.6.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO] 2025-02-16 18:28:17 library versions: OpenSSL 3.0.15 3 Sep 2024, LZO 2.10 2025-02-16 18:28:17 DCO version: N/A Enter Auth Password: (no echo) 2025-02-16 18:28:51 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Enter Private Key Password: *************** 2025-02-16 18:29:16 WARNING: Your certificate has expired! 2025-02-16 18:29:16 TCP/UDP: Preserving recently used remote address: [AF_INET]195.201.233.113:1194 2025-02-16 18:29:16 Socket Buffers: R=[212992->212992] S=[212992->212992] 2025-02-16 18:29:16 UDPv4 link local: (not bound) 2025-02-16 18:29:16 UDPv4 link remote: [AF_INET]195.201.233.113:1194 2025-02-16 18:29:17 TLS: Initial packet from [AF_INET]195.201.233.113:1194, sid=ef36b092 c9f96236 2025-02-16 18:29:17 VERIFY OK: depth=1, CN=osedev1 2025-02-16 18:29:17 VERIFY ERROR: depth=0, error=certificate has expired: CN=server, serial=17809834222100140758844598132739229483 2025-02-16 18:29:17 OpenSSL: error:0A000086:SSL routines::certificate verify failed 2025-02-16 18:29:17 TLS_ERROR: BIO read tls_read_plaintext error 2025-02-16 18:29:17 TLS Error: TLS object -> incoming plaintext read error 2025-02-16 18:29:17 TLS Error: TLS handshake failed
- looks like the dev server is still online; I was able to ssh into it
user@ose:~/openvpn$ ssh osedev1 Enter passphrase for key '/home/user/.ssh/id_rsa': Last login: Fri Oct 28 20:42:19 2022 from tor-exit-35.for-privacy.net [maltfield@osedev1 ~]$
- it's been online for 1,707 days. not bad!
[maltfield@osedev1 ~]$ uptime 00:31:57 up 1707 days, 23:41, 1 user, load average: 1.31, 0.88, 0.50 [maltfield@osedev1 ~]$
- poor little server's load is high; looks like it's swapping. mostly it's running just ossec
- ah, gross, it's a centos server; yum cron is using the most RAM
- damn, it's CentOS7
[maltfield@osedev1 ~]$ cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [maltfield@osedev1 ~]$
- I don't know why I didn't think about this before, but this server is now EOL and a risk
- moreover, I'm not sure it's even going to work as-desired because it's going to be so different from prod
- I'm going to have to talk to Marcin about this. it's going to be a lot of work to rebuild it, and I doubt he's going to want to pay for that. so probably we should just retire the ose dev server (and therefore the staging server that runs on it) :'(
- I made a list of items to discuss with Marcin. Until we have this conversation, I can't continue with hetzner3
Sat Feb 15, 2025
- Tom responded saying he got "Permission Denied" when trying to ssh into hetzner3
- I asked him to try again with '-vvv' and also send me his public key
- on the server's side, I saw this
root@hetzner3 ~ # journalctl -u ssh ... Feb 15 02:08:19 hetzner3 sshd[1552191]: Connection from REDACTED port 40998 on 144.76.164.201 port 32415 rdomain "" Feb 15 02:08:29 hetzner3 sshd[1552191]: Failed publickey for tgriffing from REDACTED port 40998 ssh2: RSA SHA256:OEad8nPjAY1Rwt73cPO7GxvX+I6scrraDZMYwQUldY4 Feb 15 02:08:29 hetzner3 sshd[1552191]: Connection closed by authenticating user tgriffing REDACTED port 40998 [preauth] Feb 15 02:08:44 hetzner3 sshd[1552199]: Connection from REDACTED port 56426 on 144.76.164.201 port 32415 rdomain "" Feb 15 02:08:47 hetzner3 sshd[1552199]: Failed publickey for tgriffing from REDACTED port 56426 ssh2: RSA SHA256:OEad8nPjAY1Rwt73cPO7GxvX+I6scrraDZMYwQUldY4 Feb 15 02:08:48 hetzner3 sshd[1552199]: Connection closed by authenticating user tgriffing REDACTED port 56426 [preauth]
- ...
- yesterday I setup a second vhost for obi running on a unique ipv6 address, but I couldn't get it fully working
- one issue was that my laptop might not have an ipv6 address; let's try from my personal server
- yesterday we setup hetzner3 to have two ipv6 addresses
- 2a01:4f8:200:40d7::2, which we had before
- 2a01:4f8:200:40d7::3, which is new
root@hetzner3 ~ # ip -6 a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000 inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000 inet6 2a01:4f8:200:40d7::3/64 scope global valid_lft forever preferred_lft forever inet6 2a01:4f8:200:40d7::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::921b:eff:fec4:28b4/64 scope link valid_lft forever preferred_lft forever root@hetzner3 ~ #
- I confirmed that I can ping the server on both addresss; and it doesn't work (as expected) on '::4', which was never setup
maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::2 PING 2a01:4f8:200:40d7::2(2a01:4f8:200:40d7::2) 56 data bytes 64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=1 ttl=57 time=2.85 ms 64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=2 ttl=57 time=2.91 ms --- 2a01:4f8:200:40d7::2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 2.852/2.879/2.906/0.027 ms maltfield@mail:~$ maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::3 PING 2a01:4f8:200:40d7::3(2a01:4f8:200:40d7::3) 56 data bytes 64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=1 ttl=57 time=3.19 ms 64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=2 ttl=57 time=2.88 ms --- 2a01:4f8:200:40d7::3 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 2.877/3.034/3.191/0.157 ms maltfield@mail:~$ maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::4 PING 2a01:4f8:200:40d7::4(2a01:4f8:200:40d7::4) 56 data bytes --- 2a01:4f8:200:40d7::4 ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1011ms maltfield@mail:~$
- yeah, I can't do this from my Qubes VM
user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::3 ping: connect: Network is unreachable user@disp3993:~$ ping -c2 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=394 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=357 ms --- 1.1.1.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 356.708/375.276/393.845/18.568 ms user@disp3993:~$ user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::2 ping: connect: Network is unreachable user@disp3993:~$ user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::3 ping: connect: Network is unreachable user@disp3993:~$
- well, that's a good test. it's not worth continuing to pursue this if Marcin and Catarina can't use IPv6.
- I sent an email to them asking them to run a ping test
Hey Marcin, Hey Catarina, When you're at FeF, can you please run this command and send me the output? ping -c2 2a01:4f8:200:40d7::3 I'm trying to figure out if we can run two copies of osemain and obi (a static site + the broken/upgraded wordpress site). But we need an additional IP address to do this. One way to do this (which would keep the domain name identical for both sites and therefore be best), would be to have each site bound to different IP addresses. With Hetnzer, we get just 1 IPv4 address and 616 IPv6 addresses. But not every ISP (or router, or computer) provides IPv6 to their clients. So this ping test will let me know if you're able to access a website that's only accessible over IPv6. Please run the above ping command and send me the full output. Thank you,
- if their test returns "network is unreachable", then I'll suggest we buy a second IPv4 address for ~2 EUR/mo (just to use until they finish the new/replacement websites)
- ...
- I checked-in on the mediawiki cron. It looks like it ran successfully this time
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads... root@hetzner3 /usr/local/bin # root@hetzner3 /usr/local/bin # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | head total 2,8M drwxrwx--- 2 www-data www-data 72K Feb 15 05:04 . drwxrwx--- 30 www-data www-data 4,0K Feb 13 19:48 .. -rw-r--r-- 1 www-data www-data 3,9K Feb 15 05:04 image_006de54f_46a605f583c811fa.png -rw-r--r-- 1 www-data www-data 4,6K Feb 15 05:04 image_00bb6228_3473277933896087.png -rw-r--r-- 1 www-data www-data 3,8K Feb 15 05:04 image_0289ccc4_158c15199e2053dd.png -rw-r--r-- 1 www-data www-data 4,3K Feb 15 05:04 image_02909476_69fb1a6ef8109be7.png -rw-r--r-- 1 www-data www-data 4,9K Feb 15 05:04 image_03bdc132_5d43e1a1df3307de.png -rw-r--r-- 1 www-data www-data 3,6K Feb 15 05:04 image_04f4de33_9b0075a6be38d1ad.png -rw-r--r-- 1 www-data www-data 3,9K Feb 15 05:04 image_0519550c_71efe71f45d70123.png root@hetzner3 /usr/local/bin # root@hetzner3 /usr/local/bin # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | tail -rw-r--r-- 1 www-data www-data 3,2K Feb 15 05:04 image_fc79e722_7c8a08bcc570879b.png -rw-r--r-- 1 www-data www-data 3,5K Feb 15 05:04 image_fdef3933_121c56952a6c57d3.png -rw-r--r-- 1 www-data www-data 4,2K Feb 15 05:04 image_fe463544_629383d940de704d.png -rw-r--r-- 1 www-data www-data 4,0K Feb 15 05:04 image_fe56e8a6_d360bd714e425344.png -rw-r--r-- 1 www-data www-data 4,6K Feb 15 05:04 image_fe8301f9_4d8d17795ab35d70.png -rw-r--r-- 1 www-data www-data 4,2K Feb 15 05:04 image_feb29944_fda14c66d7316301.png -rw-r--r-- 1 www-data www-data 3,6K Feb 15 05:04 image_fef37631_1c47b152a8639807.png -rw-r--r-- 1 www-data www-data 3,9K Feb 15 05:04 image_ff128933_f9004121b7d624c0.png -rw-r--r-- 1 www-data www-data 3,7K Feb 15 05:04 image_ffb24016_bfc86a5d83776491.png -rw-r--r-- 1 www-data www-data 3,5K Feb 15 05:04 image_fff89283_ee1207b5b2d10c3e.png root@hetzner3 /usr/local/bin # root@hetzner3 /usr/local/bin # ls /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | wc -l 500 root@hetzner3 /usr/local/bin # date -u 2025-02-15T20:54:29 UTC root@hetzner3 /usr/local/bin #
- so there's exactly 500 images, and they were all generated today. perfect!
- the only issue I do see is that the log was cleared of messages from before, and it wasn't rotated
root@hetzner3 /usr/local/bin # ls -lah /var/log/cron total 12K drwxr-x--- 2 www-data www-data 4,0K Feb 14 05:18 . drwxr-xr-x 16 root root 4,0K Feb 13 22:54 .. -rw-r--r-- 1 www-data www-data 86 Feb 15 05:04 mediawiki_generate_captchas.log root@hetzner3 /usr/local/bin #
- looks like the cron is missing a second > sign (for append)
root@hetzner3 /usr/local/bin # cat /etc/cron.d/mediawiki_cron # Ansible managed SHELL=/bin/bash # regenerate new CAPTCHA images 20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log root@hetzner3 /usr/local/bin #
- I fixed this in ansible and pushed it out https://github.com/OpenSourceEcology/ansible/commit/3aa97d5d27e538ce234aa0354a8a5ac1ae4bc082
user@ose:~/sandbox_local/ansible/hetzner3$ git diff HEAD^ diff --git a/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2 b/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2 index 28e5bf0..873a137 100644 --- a/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2 +++ b/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2 @@ -2,4 +2,4 @@ SHELL=/bin/bash # regenerate new CAPTCHA images -20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log +20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &>> /var/log/cron/mediawiki_generate_captchas.log user@ose:~/sandbox_local/ansible/hetzner3$
- ...
- another item I have in my TODO list was to confirm that APC is setup properly
- I never tested this. If APC isn't running, it'll greatly harm the performance of our PHP websites
- It looks like APC is no longer supported in PHP >5.4
- hetzner2 ran php v5.6 and hetzner3 runs v8.2.26, so I guess there's no change there
- the replacement for APC was APCu https://www.php.net/manual/en/book.apcu.php
- it's not clear to me when APCu was first supported, but they have releases going back to 2013https://pecl.php.net/package/apcu
- here's the mediawiki guide that describes apcu https://www.mediawiki.org/wiki/Manual:Performance_tuning
- it says to install php-acpu, which we've done (but I still want to test that it's actually enabled & working)
root@hetzner3 /usr/local/bin # apt-cache search apcu apcupsd - APC UPS Power Management (daemon) apcupsd-cgi - APC UPS Power Management (web interface) apcupsd-doc - APC UPS Power Management (documentation/examples) collectd-core - statistics collection and monitoring daemon (core system) conky-all - highly configurable system monitor (all features enabled) conky-cli - highly configurable system monitor (basic version) conky-std - highly configurable system monitor (default version) php-apcu - APC User Cache for PHP php-apcu-all-dev - APC User Cache for PHP php8.2-apcu - APC User Cache for PHP php-symfony-polyfill-apcu - Symfony polyfill backporting apcu_* functions to lower PHP versions root@hetzner3 /usr/local/bin # root@hetzner3 /usr/local/bin # dpkg -l | grep -i apcu ii php-apcu 5.1.22+4.0.11-2 amd64 APC User Cache for PHP ii php8.2-apcu 5.1.22+4.0.11-2 amd64 APC User Cache for PHP root@hetzner3 /usr/local/bin #
- according to the mediawiki docs, that should have installed some script 'apcu.php' that we can use to inspect the apc cache and confirm that mediawiki is using it
A script, apc.php is bundled with the APCu package which can be used to inspect the status of the cache, and also examine the contents of the user cache to verify that MediaWiki is correctly using it.
- uhh, that's just docs
root@hetzner3 /usr/local/bin # dpkg-query -L php-apcu /. /usr /usr/share /usr/share/doc /usr/share/doc/php-apcu /usr/share/doc/php-apcu/README.Debian /usr/share/doc/php-apcu/changelog.Debian.gz /usr/share/doc/php-apcu/copyright root@hetzner3 /usr/local/bin #
- ok, it's this one
root@hetzner3 /usr/local/bin # dpkg-query -L php8.2-apcu /. /etc /etc/php /etc/php/8.2 /etc/php/8.2/mods-available /etc/php/8.2/mods-available/apcu.ini /usr /usr/include /usr/include/php /usr/include/php/20220829 /usr/include/php/20220829/ext /usr/include/php/20220829/ext/apcu /usr/include/php/20220829/ext/apcu/apc.h /usr/include/php/20220829/ext/apcu/apc_api.h /usr/include/php/20220829/ext/apcu/apc_arginfo.h /usr/include/php/20220829/ext/apcu/apc_cache.h /usr/include/php/20220829/ext/apcu/apc_globals.h /usr/include/php/20220829/ext/apcu/apc_iterator.h /usr/include/php/20220829/ext/apcu/apc_lock.h /usr/include/php/20220829/ext/apcu/apc_mutex.h /usr/include/php/20220829/ext/apcu/apc_serializer.h /usr/include/php/20220829/ext/apcu/apc_sma.h /usr/include/php/20220829/ext/apcu/apc_stack.h /usr/include/php/20220829/ext/apcu/php_apc.h /usr/include/php/20220829/ext/apcu/php_apc_legacy_arginfo.h /usr/lib /usr/lib/php /usr/lib/php/20220829 /usr/lib/php/20220829/apcu.so /usr/share /usr/share/doc /usr/share/doc/php8.2-apcu /usr/share/doc/php8.2-apcu/changelog.Debian.gz /usr/share/doc/php8.2-apcu/copyright root@hetzner3 /usr/local/bin #
- ...but there's still no apcu.php file :(
- actually, we are missing a package; the MediaWiki docs say we also need this php-igbinary
root@hetzner3 /usr/local/bin # apt-cache search igbinary php-igbinary - igbinary PHP serializer php-igbinary-all-dev - igbinary PHP serializer php8.2-igbinary - igbinary PHP serializer You have mail in /var/mail/root root@hetzner3 /usr/local/bin # dpkg -l | grep -i igbinary root@hetzner3 /usr/local/bin #
- I added it to the depends listed in the ansible php role
- according to the docs, we're "strongly recommended" to set this
$wgMainCacheType = CACHE_ACCEL;
- I confirmed that we are using it
root@hetzner3 /usr/local/bin # grep -i cache /var/www/html/wiki.opensourceecology.org/LocalSettings.php $wgCategoryTreeDisableCache = false; $wgCategoryTreeHTTPCache = true; $egCache = true; # sure that cached pages are cleared. $wgCacheEpoch = max( $wgCacheEpoch, gmdate( 'YmdHis', @filemtime( FILE ) ) ); # VARNISH CACHE # # INTERNAL MEDIAWIKI CACHE OPTIONS (DISTINCT FROM VARNISH) # MainCache and MessageCache should use APCU per Aaron Schulz $wgMainCacheType = CACHE_ACCEL; # note that if message cache uses the db (per defaults), then it may make every # not to cache the page (rightfully so), and the result is that varnish (which # is our most important cache) is rendered useless. For more info, see: $wgMessageCacheType = CACHE_ACCEL; $wgUseLocalMessageCache = true; # Parser Cache should still use the DB per Aaron Schulz $wgParserCacheType = CACHE_DB; $wgEnableSidebarCache = true; # cache interface messages to files in this directory per Aaron Schulz $wgCacheDirectory = "$IP/../cache"; root@hetzner3 /usr/local/bin #
- good to note that the mediawiki performance docs suggest using php-fpm instead of mod_php, something we've done for this migration from hetzner2 to hetzner3
- it also suggests using event MPM instead of prefork MPM
- crap, it looks like we're using prefork
root@hetzner3 /etc/apache2 # grep -irl 'mpm' * magic mods-available/mpm_event.conf mods-available/pagespeed.conf mods-available/mpm_worker.load mods-available/mpm_prefork.conf mods-available/mpm_prefork.load mods-available/mpm_worker.conf mods-available/php8.2.load mods-available/mpm_prefork.conf.104647.2024-09-25@01:24:19~ mods-available/mpm_event.load root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah mods-enabled/ total 8,0K drwxr-xr-x 2 root root 4,0K Dec 29 03:27 . drwxr-xr-x 8 root root 4,0K Feb 13 20:02 .. lrwxrwxrwx 1 root root 36 Sep 25 01:24 access_compat.load -> ../mods-available/access_compat.load lrwxrwxrwx 1 root root 28 Sep 25 01:24 alias.conf -> ../mods-available/alias.conf lrwxrwxrwx 1 root root 28 Sep 25 01:24 alias.load -> ../mods-available/alias.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 auth_basic.load -> ../mods-available/auth_basic.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 authn_core.load -> ../mods-available/authn_core.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 authn_file.load -> ../mods-available/authn_file.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 authz_core.load -> ../mods-available/authz_core.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 authz_host.load -> ../mods-available/authz_host.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 authz_user.load -> ../mods-available/authz_user.load lrwxrwxrwx 1 root root 32 Sep 25 01:24 autoindex.conf -> ../mods-available/autoindex.conf lrwxrwxrwx 1 root root 32 Sep 25 01:24 autoindex.load -> ../mods-available/autoindex.load lrwxrwxrwx 1 root root 30 Sep 25 01:24 deflate.conf -> ../mods-available/deflate.conf lrwxrwxrwx 1 root root 30 Sep 25 01:24 deflate.load -> ../mods-available/deflate.load lrwxrwxrwx 1 root root 26 Sep 25 01:24 dir.conf -> ../mods-available/dir.conf lrwxrwxrwx 1 root root 26 Sep 25 01:24 dir.load -> ../mods-available/dir.load lrwxrwxrwx 1 root root 26 Sep 25 01:24 env.load -> ../mods-available/env.load lrwxrwxrwx 1 root root 29 Sep 25 01:24 filter.load -> ../mods-available/filter.load lrwxrwxrwx 1 root root 40 Sep 25 01:24 headers.load -> /etc/apache2/mods-available/headers.load lrwxrwxrwx 1 root root 27 Sep 25 01:24 mime.conf -> ../mods-available/mime.conf lrwxrwxrwx 1 root root 27 Sep 25 01:24 mime.load -> ../mods-available/mime.load lrwxrwxrwx 1 root root 34 Dec 29 03:27 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf lrwxrwxrwx 1 root root 34 Dec 29 03:27 mpm_prefork.load -> ../mods-available/mpm_prefork.load lrwxrwxrwx 1 root root 34 Sep 25 01:24 negotiation.conf -> ../mods-available/negotiation.conf lrwxrwxrwx 1 root root 34 Sep 25 01:24 negotiation.load -> ../mods-available/negotiation.load lrwxrwxrwx 1 root root 29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf lrwxrwxrwx 1 root root 29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load lrwxrwxrwx 1 root root 43 Sep 26 00:06 proxy_fcgi.load -> /etc/apache2/mods-available/proxy_fcgi.load lrwxrwxrwx 1 root root 38 Sep 26 00:11 proxy.load -> /etc/apache2/mods-available/proxy.load lrwxrwxrwx 1 root root 41 Sep 25 01:24 remoteip.load -> /etc/apache2/mods-available/remoteip.load lrwxrwxrwx 1 root root 33 Sep 25 01:24 reqtimeout.conf -> ../mods-available/reqtimeout.conf lrwxrwxrwx 1 root root 33 Sep 25 01:24 reqtimeout.load -> ../mods-available/reqtimeout.load lrwxrwxrwx 1 root root 40 Sep 25 01:24 rewrite.load -> /etc/apache2/mods-available/rewrite.load lrwxrwxrwx 1 root root 32 Sep 25 01:24 security2.conf -> ../mods-available/security2.conf lrwxrwxrwx 1 root root 32 Sep 25 01:24 security2.load -> ../mods-available/security2.load lrwxrwxrwx 1 root root 31 Sep 25 01:24 setenvif.conf -> ../mods-available/setenvif.conf lrwxrwxrwx 1 root root 31 Sep 25 01:24 setenvif.load -> ../mods-available/setenvif.load lrwxrwxrwx 1 root root 39 Sep 25 01:24 status.conf -> /etc/apache2/mods-available/status.conf lrwxrwxrwx 1 root root 29 Sep 25 01:24 status.load -> ../mods-available/status.load lrwxrwxrwx 1 root root 32 Sep 25 01:24 unique_id.load -> ../mods-available/unique_id.load root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # cat mods-enabled/mpm_prefork.load # Conflicts: mpm_event mpm_worker LoadModule mpm_prefork_module /usr/lib/apache2/modules/mod_mpm_prefork.so root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # cat mods-enabled/mpm_prefork.conf # Ansible managed # prefork MPM # StartServers: number of server processes to start # MinSpareServers: minimum number of server processes which are kept spare # MaxSpareServers: maximum number of server processes which are kept spare # MaxRequestWorkers: maximum number of server processes allowed to start # MaxConnectionsPerChild: maximum number of requests a server process serves <IfModule mpm_prefork_module> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxRequestWorkers 150 MaxConnectionsPerChild 1000 </IfModule> # vim: syntax=apache ts=4 sw=4 sts=4 sr noet root@hetzner3 /etc/apache2 #
- it looks like the last time I looked at prefork vs event, I realized we were still using mod_php, and that's when I switched us to php-fpm. But apparently I never revisited changing from MPM prefork to event https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Tue_Sep_24.2C_2024
- oh, no, I didn't find it in my logs, but I commented-out the prefork config in ansible on 2024-09-24 https://github.com/OpenSourceEcology/ansible/commit/763d9f996130edd579ad12388ec2861bc5c3cdc3
fix apache This commit removes the mpm_prefork module load and config, since Debian 12 actually already ships with mpm_event with Apache 2.4. Note that mpm_event is generally considered more performant, but the downside is that it's not OK for modules that break when threads are used, such as mod_php The solution to that is to switch to php-fpm, which is also more performant -- and thread safe. When we had issues with Apache spawning runaway children and killing the whole server earlier this year (2024), my research suggested that we should use php-fpm, but I didn't make that change for simplicity. What we *did* change was to configure MaxServers in mpm_prefork, but that now conflicts with the defaults on Debian (before it was CentOS), because only one mpm can be selected. Probably our apache configs in ansible will still need to be adjusted to direct php traffic to the cgi process (proxy?)
- unfortunately, I guess commenting it out wasn't sufficient to disable it.
- it looks like I need to remove the symlinks in 'mod-enabled/' from prefork and add these instead
root@hetzner3 /etc/apache2 # find . | grep -i event ./mods-available/mpm_event.conf ./mods-available/mpm_event.load root@hetzner3 /etc/apache2 #
- here's the default config for mpm_event
root@hetzner3 /etc/apache2 # cat ./mods-available/mpm_event.conf # event MPM # StartServers: initial number of server processes to start # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestWorkers: maximum number of worker threads # MaxConnectionsPerChild: maximum number of requests a server process serves StartServers 2 MinSpareThreads 25 MaxSpareThreads 75 ThreadLimit 64 ThreadsPerChild 25 MaxRequestWorkers 150 MaxConnectionsPerChild 0 You have mail in /var/mail/root root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # cat ./mods-available/mpm_event.load # Conflicts: mpm_worker mpm_prefork LoadModule mpm_event_module /usr/lib/apache2/modules/mod_mpm_event.so root@hetzner3 /etc/apache2 #
- I tried to fix this (remove the symlinks for prefork and add new symlinks for event)
user@ose:~/sandbox_local/ansible/hetzner3$ git diff ... --- a/hetzner3/roles/maltfield.apache/tasks/main.yml +++ b/hetzner3/roles/maltfield.apache/tasks/main.yml @@ -140,29 +140,32 @@
notify: restart apache # we're using mpm_event, the default in Debian 12 with Apache 2.4
-#- name: enable module = mpm_prefork -# file: -# src: /etc/apache2/mods-available/mpm_prefork.load -# dest: /etc/apache2/mods-enabled/mpm_prefork.load -# state: link -# notify: restart apache -# -#- name: mpm_prefork.conf ...skipping... +- name: disable module = mpm_prefork + file: + path: /etc/apache2/mods-enabled/mpm_prefork.load + state: absent + notify: restart apache + +- name: mpm_prefork config symlink + file: + path: /etc/apache2/mods-enabled/mpm_prefork.conf + state: absent + notify: restart apache + +- name: enable module = mpm_event + file: + src: /etc/apache2/mods-available/mpm_event.load + dest: /etc/apache2/mods-enabled/mpm_event.load + state: link + notify: restart apache + +- name: mpm_event config symlink + file: + src: /etc/apache2/mods-available/mpm_event.conf + dest: /etc/apache2/mods-enabled/mpm_event.conf + state: link + notify: restart apache
# prod mode config - name: main apache4_prod.conf
diff --git a/hetzner3/roles/maltfield.php/tasks/main.yml b/hetzner3/roles/maltfield.php/tasks/main.yml index fecdcd6..75a806a 100644 --- a/hetzner3/roles/maltfield.php/tasks/main.yml +++ b/hetzner3/roles/maltfield.php/tasks/main.yml @@ -5,6 +5,7 @@ pkg: - php-fpm - php-apcu + - php-igbinary - php-curl
notify:
- restart php-fpm user@ose:~/sandbox_local/ansible/hetzner3$
- but apache failed on restart
root@hetzner3 /etc/apache2 # journalctl -u apache2 | less ... root@hetzner3 /etc/apache2 # systemctl restart apache2 Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details. root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # journalctl -u apache2 | less ... Feb 15 21:50:31 hetzner3 apachectl[1909056]: [Sat Feb 15 21:50:31.965322 2025] [php:crit] [pid 1909056:tid 1909056] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP. Feb 15 21:50:31 hetzner3 apachectl[1909056]: AH00013: Pre-configuration failed Feb 15 21:50:31 hetzner3 apachectl[1909054]: Action 'graceful-stop' failed. Feb 15 21:50:31 hetzner3 apachectl[1909054]: The Apache error log may have more information. Feb 15 21:50:31 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'. Feb 15 21:50:32 hetzner3 systemd[1]: Stopped apache2.service - The Apache HTTP Server. Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Consumed 8.836s CPU time. Feb 15 21:50:32 hetzner3 systemd[1]: Starting apache2.service - The Apache HTTP Server... Feb 15 21:50:32 hetzner3 apachectl[1909062]: [Sat Feb 15 21:50:32.073820 2025] [php:crit] [pid 1909062:tid 1909062] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP. Feb 15 21:50:32 hetzner3 apachectl[1909062]: AH00013: Pre-configuration failed Feb 15 21:50:32 hetzner3 apachectl[1909059]: Action 'start' failed. Feb 15 21:50:32 hetzner3 apachectl[1909059]: The Apache error log may have more information. Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'. Feb 15 21:50:32 hetzner3 systemd[1]: Failed to start apache2.service - The Apache HTTP Server. You have mail in /var/mail/root root@hetzner3 /etc/apache2 #
- looks like we're using event. please tell me we're not using mod_php still?
root@hetzner3 /etc/apache2 # ls -lah mods-enabled/ |grep -iE 'php|prefork|event' lrwxrwxrwx 1 root root 42 Feb 15 21:40 mpm_event.conf -> /etc/apache2/mods-available/mpm_event.conf lrwxrwxrwx 1 root root 42 Feb 15 21:40 mpm_event.load -> /etc/apache2/mods-available/mpm_event.load lrwxrwxrwx 1 root root 29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf lrwxrwxrwx 1 root root 29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load root@hetzner3 /etc/apache2 #
- yeah, I think that this is mod_php; let's disable it
root@hetzner3 /etc/apache2 # cat mods-enabled/php8.2.conf # Using (?:pattern) instead of (pattern) is a small optimization that # avoid capturing the matching pattern (as $1) which isn't used here <FilesMatch ".+\.ph(?:ar|p|tml)$"> SetHandler application/x-httpd-php </FilesMatch> <FilesMatch ".+\.phps$"> SetHandler application/x-httpd-php-source # Deny access to raw php sources by default # To re-enable it's recommended to enable access to the files # only in specific virtual host or directory Require all denied </FilesMatch> # Deny access to files without filename (e.g. '.php') <FilesMatch "^\.ph(?:ar|p|ps|tml)$"> Require all denied </FilesMatch> # Running PHP scripts in user directories is disabled by default # # To re-enable PHP in user directories comment the following lines # (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it # prevents .htaccess files from disabling it. <IfModule mod_userdir.c> <Directory /home/*/public_html> php_admin_flag engine Off </Directory> </IfModule> root@hetzner3 /etc/apache2 #
- shit, I'm not sure we're even using php-fpm
- oh, it looks like both are enabled? This is very annoying because mod_php is in "mods-enabled" and php-fpm is in "conf-enabled"
root@hetzner3 /etc/apache2 # ls -lah mods-available | grep -i php -rw-r--r-- 1 root root 998 Jun 17 2024 php8.2.conf -rw-r--r-- 1 root root 101 Jun 17 2024 php8.2.load root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah conf-available | grep -i php -rw-r--r-- 1 root root 1,4K Jun 17 2024 php8.2-fpm.conf -rw-r--r-- 1 root root 1,2K Sep 25 01:46 phplist.virtualhost.include root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah conf-enabled | grep -i php lrwxrwxrwx 1 root root 43 Sep 25 23:53 php8.2-fpm.conf -> /etc/apache2/conf-available/php8.2-fpm.conf root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah mods-enabled | grep -i php lrwxrwxrwx 1 root root 29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf lrwxrwxrwx 1 root root 29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load root@hetzner3 /etc/apache2 #
- anyway, I'll update ansible to remove
- before
root@hetzner3 /etc/apache2 # ls *-enabled/* | grep -i php conf-enabled/php8.2-fpm.conf mods-enabled/php8.2.conf mods-enabled/php8.2.load sites-enabled/phplist.opensourceecology.org.conf root@hetzner3 /etc/apache2 #
- after
root@hetzner3 /etc/apache2 # ls *-enabled/* | grep -i php conf-enabled/php8.2-fpm.conf sites-enabled/phplist.opensourceecology.org.conf root@hetzner3 /etc/apache2 #
- now it's working
root@hetzner3 /etc/apache2 # systemctl restart apache2 root@hetzner3 /etc/apache2 #
- I did a quick check of store and wiki, they both load fine
- wow, I'm glad I caught that. So now we're using the much more performant mpm event, and the (prerequisite) php-fpm
- but we still haven't confirmed apc is working!
- this answer on SE says to use 'apc.php' too, but it also provides an alternative https://stackoverflow.com/a/17631545/1174102
<pre><?php print_r(apc_cache_info());
- I created a script for this on the wiki
cat > /var/www/html/wiki.opensourceecology.org/htdocs/apc.php <<'EOF' <pre> <?php print_r(apc_cache_info()); ?>
EOF
chown not-apache:www-data /var/www/html/wiki.opensourceecology.org/htdocs/apc.php chmod 0040 /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
- crap, that page is empty. And this is in the apache logs
[Sat Feb 15 22:19:30.627296 2025] [proxy_fcgi:error] [pid 1917176:tid 1917223] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function apc_cache_info() in /var/www/html/wiki.opensourceecology.org/htdocs/apc.php:3\nStack trace:\n#0 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/apc.php on line 3'
- well, it looks like I'm not disabling that function; maybe it just doesn't exist anymore
root@hetzner3 /var/log/apache2 # grep -ir apc /etc/php/8.2/ /etc/php/8.2/mods-available/apcu.ini:extension=apcu.so /etc/php/8.2/mods-available/igbinary.ini:; Use igbinary as serializer in APC cache (3.1.7 or later) /etc/php/8.2/mods-available/igbinary.ini:;apc.serializer=igbinary root@hetzner3 /var/log/apache2 #
- the php reference docs don't say the function has been retired. So maybe, in fact, apc isn't enabled? https://www.php.net/manual/en/function.apcu-cache-info.php
- oh, duh, ^ that one is 'acpu_' and the one I wrote to the file is 'apc_'
- let's try this
cat > /var/www/html/wiki.opensourceecology.org/htdocs/apc.php <<'EOF' <pre> <?php print_r(apcu_cache_info()); ?>
EOF
chown not-apache:www-data /var/www/html/wiki.opensourceecology.org/htdocs/apc.php chmod 0040 /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
- that worked! It spat-out an array that spans 23,041 lines
- here's the top of it
Array ( [num_slots] => 4099 [ttl] => 0 [num_hits] => 1662 [num_misses] => 1867 [num_inserts] => 1701 [num_entries] => 1665 [expunges] => 0 [start_time] => 1739654285 [mem_size] => 2845632 [memory_type] => mmap [cache_list] => Array (
- the SE post above says that if "num_hits" is increasing, then it's working
- I logged-in on the wiki and edited the main page
- then I refreshed the apc.php page, and now it says
Array ( [num_slots] => 4099 [ttl] => 0 [num_hits] => 1662 [num_misses] => 1867 [num_inserts] => 1701 [num_entries] => 1665 [expunges] => 0 [start_time] => 1739654285 [mem_size] => 2845632 [memory_type] => mmap [cache_list] => Array (
- fuck; num_hits didn't increase
- wait, maybe it's cached. I tried "/apc.php?nocache=1"
Array ( [num_slots] => 4099 [ttl] => 0 [num_hits] => 6929 [num_misses] => 2080 [num_inserts] => 1870 [num_entries] => 1777 [expunges] => 0 [start_time] => 1739654285 [mem_size] => 3586816 [memory_type] => mmap [cache_list] => Array (
- oh, that's a huge increase. I guess we can conclude that apc is working. great!
- let's cleanup that script
root@hetzner3 /var/www/html/wiki.opensourceecology.org # rm /var/www/html/wiki.opensourceecology.org/htdocs/apc.php root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- ...
- another item on my TODO list is to confirm that my various vhost hardenings are still working (since the upgrades, changes to php-fpm, etc...it's likely some of these broke)
- wp-login.php shouldn't be accessible
- phplist config.php shouldn't be accessible (I want to confirm because old had ".*" prefix and new doesn't)
- .php files in wp-content uploads dirs shouldn't be executable
- anything other than GET POST HEAD should be denied
- anything that starts with a dot (other than .well-known) should be denied (test .git & .htaccess)
- indexes on a dir without 'index.php' shouldn't show a directory listing
- .htaccess shouldn't allow changing things
- this claims to block access to wp-login.php, but I'm pretty sure it's just the wordpress plugin; you can see the varnish header suggest that it penetrated past nginx
user@disp3993:~$ curl -iL store.opensourceecology.org/wp-login.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 15 Feb 2025 22:34:14 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://store.opensourceecology.org/wp-login.php X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block HTTP/1.1 403 Forbidden Server: nginx Date: Sat, 15 Feb 2025 22:34:17 GMT Content-Type: text/html; charset=iso-8859-1 Content-Length: 214 Connection: keep-alive X-Varnish: 29914517 Age: 0 Via: 1.1 varnish-v4 <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /wp-login.php on this server.</p> </body></html> user@disp3993:~$
- also these configs should span apps, and we can see the wiki just gives a 404
user@disp3993:~$ curl -iL wiki.opensourceecology.org/wp-login.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 15 Feb 2025 22:36:54 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://wiki.opensourceecology.org/wp-login.php X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=1;includeSubDomains HTTP/1.1 404 Not Found Server: nginx Date: Sat, 15 Feb 2025 22:36:56 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 16 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Referrer-Policy: strict-origin X-Varnish: 427397 427395 Age: 107 Via: 1.1 varnish (Varnish/7.1) File not found. user@disp3993:~$
- oh, actually, I had an error in /etc/hosts. This is totally broken on hetnzer3
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-login.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 15 Feb 2025 23:01:17 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://store.opensourceecology.org/wp-login.php X-Frame-Options: SAMEORIGIN X-XSS-Protection: 1; mode=block Strict-Transport-Security: max-age=1;includeSubDomains HTTP/1.1 302 Found Server: nginx Date: Sat, 15 Feb 2025 23:01:20 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: / X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Referrer-Policy: no-referrer-when-downgrade X-Varnish: 460388 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Sat, 15 Feb 2025 23:01:21 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 85328 Connection: keep-alive X-Pingback: https://store.opensourceecology.org/xmlrpc.php Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink X-Frame-Options: SAMEORIGIN Vary: Accept-Encoding X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Referrer-Policy: no-referrer-when-downgrade X-Varnish: 200920 460383 Age: 9 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" user@disp3993:~$
- I'm not sure if this broke or I intentionally disabled it
- one issue is that the plugin we had been using in wordpress to redirect away from wp-login.php is no longer available due to "Security Issue" https://wordpress.org/plugins/rename-wp-login/
- as a result, there's a narrow window during the migration of sites to hetzner3 where we need to login at wp-login.php and configure the replacement plugin to use a distinct login URL
- I also took a moment to update the plugins list on the ose wiki article for wordpress https://wiki.opensourceecology.org/wiki/Wordpress
- I think the best solution to this is to see if we can configure 'melapress-login-security' from the command-line after migrating, so that we never have to login at wp-login.php
- after some trial-and-error, I found the settings for this plugin are stored in the wordpress options table; there's 4 of them
MariaDB [store_db]> select option_name from wp_options where option_name like '%mls_%'; +--------------------+ | option_name | +--------------------+ | mls_activation | | mls_active_version | | mls_options | | mls_setting | +--------------------+ 4 rows in set (0,001 sec) MariaDB [store_db]>
- the first two are a simple string
- the second two are long json strings
- well it looks like the min password length is set in 'wp_options' and the wp-login.php rename is set in 'mls_setting'
- unfortunately my email address got saved in wp_options, so I think I'm just going to set mls_setting (so that doesn't have to get stored in the wiki CHG migration commands/instructions)
- alright, here's what we have
root@hetzner3 ~ # echo "select option_value from wp_options where option_name = 'mls_setting';" | mysql -uroot -p$mysqlPass store_db
option_value
a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}
root@hetzner3 ~ #
- it appears that we can either update the db directly (dangerous) or use wp-cli https://developer.wordpress.org/cli/commands/option/
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get siteurl
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
https://store.opensourceecology.org
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
- first I tested if 'update' needs to have a key for it to work; it doesn't (that way we don't have to delete first)
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get thisdoesnotexist
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Error: Could not get 'thisdoesnotexist' option. Does it exist?
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update thisdoesnotexist 'somevalue'
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'thisdoesnotexist' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get thisdoesnotexist
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
somevalue
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
- so here's my plan to change the login from wp-login.php to ose-hidden-login
sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}'
- to test this, I deleted the settings on store and deactivated the plugin
- I reactivated it, and confirmed that the "login page hardening" was back at the default (with nothing specified for 'Login page URL'
- I deactivated the plugin again in the wui, and tried the rest with the cli
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" plugin activate melapress-login-security
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'melapress-login-security' activated.
Success: Activated 1 of 1 plugins.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}'
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'mls_setting' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
- I refreshed '/wp-admin/' and re-checked the settings page. crap, it's still empty? https://store.opensourceecology.org/wp-admin/admin.php?page=mls-hide-login
- I confirmed the setting is there
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get mls_setting
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
- idk, maybe somehow I have to set both
- I obfuscated my email address and tried this
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_options 'a:62:{s:13:"master_switch";s:3:"yes";s:26:"activate_password_policies";s:3:"yes";s:37:"activate_password_expiration_policies";s:2:"no";s:34:"activate_password_recycle_policies";s:2:"no";s:16:"enforce_password";s:2:"no";s:10:"min_length";s:2:"20";s:16:"password_history";s:1:"1";s:16:"inherit_policies";s:2:"no";s:15:"password_expiry";a:2:{s:5:"value";s:1:"0";s:4:"unit";s:6:"months";}s:8:"ui_rules";a:7:{s:7:"history";s:2:"no";s:8:"username";s:2:"no";s:6:"length";b:1;s:7:"numeric";s:2:"no";s:8:"mix_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:5:"rules";a:6:{s:6:"length";b:1;s:7:"numeric";s:2:"no";s:10:"upper_case";s:2:"no";s:10:"lower_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:23:"change_initial_password";s:2:"no";s:12:"timed_logins";s:2:"no";s:21:"timed_logins_schedule";a:7:{s:6:"monday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:7:"tuesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:9:"wednesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"thursday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"friday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"saturday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"sunday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}}s:22:"inactive_users_enabled";s:2:"no";s:21:"inactive_users_expiry";a:2:{s:5:"value";i:30;s:4:"unit";s:4:"days";}s:30:"inactive_users_reset_on_unlock";s:2:"no";s:29:"failed_login_policies_enabled";s:3:"yes";s:21:"failed_login_attempts";s:1:"5";s:27:"failed_login_reset_attempts";s:4:"1440";s:27:"failed_login_unlock_setting";s:5:"timed";s:24:"failed_login_reset_hours";s:2:"60";s:29:"failed_login_reset_on_unblock";s:2:"no";s:18:"disable_self_reset";s:3:"yes";s:26:"disable_self_reset_message";b:0;s:27:"deactivated_account_message";s:161:"Your WordPress user has been deactivated. Please contact the <a href="mailto:michael@michaelaltfield.net">website administrator</a> to activate back your user.";s:19:"timed_login_message";b:0;s:30:"locked_user_disable_self_reset";s:2:"no";s:38:"locked_user_disable_self_reset_message";b:0;s:17:"restrict_login_ip";s:2:"no";s:23:"restrict_login_ip_count";i:3;s:22:"restrict_login_message";s:0:"";s:22:"notify_password_expiry";s:2:"no";s:30:"notify_password_reset_on_login";b:0;s:27:"notify_password_expiry_days";i:0;s:27:"notify_password_expiry_unit";s:4:"days";s:26:"restrict_login_credentials";s:7:"default";s:34:"restrict_login_credentials_message";s:0:"";s:24:"enable_sessions_policies";b:0;s:23:"remember_session_expiry";a:2:{s:5:"value";i:14;s:4:"unit";s:4:"days";}s:22:"default_session_expiry";a:2:{s:5:"value";i:2;s:4:"unit";s:4:"days";}s:22:"enable_device_policies";b:0;s:35:"enable_device_policies_admin_alerts";s:2:"no";s:25:"enable_security_questions";b:0;s:17:"enabled_questions";a:0:{}s:36:"device_policies_prompt_email_content";s:0:"";s:41:"device_policies_admin_alert_email_content";s:0:"";s:36:"device_policies_prompt_email_subject";s:0:"";s:41:"device_policies_admin_alert_email_subject";s:0:"";s:25:"min_answered_needed_count";i:3;s:39:"password_reset_request_disabled_message";s:0:"";s:24:"password_expired_message";s:0:"";s:36:"inactive_user_account_locked_message";s:0:"";s:51:"inactive_user_account_locked_reset_disabled_message";s:0:"";s:38:"restrict_logins_prompt_failure_message";s:0:"";s:34:"timed_logins_login_blocked_message";s:0:"";s:39:"restrict_login_ip_login_blocked_message";s:0:"";s:35:"failed_logins_login_blocked_message";s:0:"";s:40:"security_prompt_response_failure_message";s:0:"";s:24:"timed_logins_auto_logout";b:0;s:13:"ppm-user-role";s:0:"";s:22:"excluded_special_chars";s:0:"";}'
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'mls_options' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
- refreshing the wui then lead to a critical error
==> apache2/store.opensourceecology.org/error.log <==
[Sun Feb 16 00:10:53.811257 2025] [proxy_fcgi:error] [pid 1950272:tid 1950284] [client 146.70.129.180:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php:446\nStack trace:\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/class-melapress-login-security.php(362): MLS\\MLS_Options->init()\n#1 [internal function]: MLS_Core->init()\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()\n#4 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()\n#5 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(700): do_action()\n#6 /var/www/html/store.opensourceecology.org/wp-config.php(119): require_once('...')\n#7 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(...'
- I think the problem is that this json is encoded with the length of the string for some reason. Yeah, the string before I edited it was 161 chars; I updated this to 159 chars (to reflect the shorter email address)
- crap, this is a catch-22; I can't fix the fatal error because of the fatal error
Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php:446
Stack trace:
#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/class-melapress-login-security.php(362): MLS\MLS_Options->init()
#1 [internal function]: MLS_Core->init()
#2 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()
#3 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()
#4 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()
#5 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(700): do_action()
#6 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1375): require('...')
#7 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1294): WP_CLI\Runner->load_wordpress()
#8 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(28): WP_CLI\Runner->start()
#9 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/bootstrap.php(83): WP_CLI\Bootstrap\LaunchRunner->process()
#10 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/wp-cli.php(32): WP_CLI\bootstrap()
#11 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/php/boot-phar.php(20): include('...')
#12 /home/wp/.wp-cli/wp-cli-2.11.0.phar(4): include('...')
#13 {main}
thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php on line 446
Error: There has been a critical error on this website.Learn more about troubleshooting WordPress. There has been a critical error on this website.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_passwor
=Fri Feb 14, 2025=
# I checked on the mediawiki cron that I created yesterday and should have run for the first time last night; it failed :(
<pre>
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log
www-data is not in the sudoers file.
This incident has been reported to the administrator.
www-data is not in the sudoers file.
This incident has been reported to the administrator.
root@hetzner3 /usr/local/bin #
- so it's important that we don't execute some python script located in our webroot as root, but I tried to de-escalate in two places (for layered security)
- the cron executes the script as www-data
- the script executes the captcha.py script as www-data with sudo
- that doesn't work because the www-data itself can't execute the 'sudo' command
- I could add that user to sudo, but that sounds risky
- it's probably best just to execute the man bash script as root
- I updated the script with a long comment explaining this
root@hetzner3 /usr/local/bin # diff mediawiki_generate_captchas.sh.20250214 mediawiki_generate_captchas.sh 5c5 < # Version: 0.1 --- > # Version: 0.2 8c8 < # * --- > # * https://www.mediawiki.org/wiki/Extension:ConfirmEdit#FancyCaptcha 11c11 < # Updated: 2025-02-13 --- > # Updated: 2025-02-14 42a43,61 > > ########################### > # (!!) SECURITY NOTE (!!) # > ########################### > # It's critical that any scripts executed inside our web server's document root > # are executed as a non-privileged user! This is because, if there's a > # vulnerability in MediaWiki (or wordpress, phpList, etc), then an attacker > # can modify the contents of such scripts (eg captcha.py). And if we execute > # that script as root, then it would give the attacker the ability a way to > # execute arbitrary code as root, escalating their privileges. > # > # Also note that this script itself is executed as root by cron. It's better to > # do the de-escalation in this script, because a human might execute it as > # root. Also note this script wouldn't work if run as 'www-data', because > # 'www-data' isn't in sudoers. > # > # Therefore, make sure any executions of scripts in /var/www/html/*/htdocs are > # prefixed with `sudo -u www-data -bash -c "<path to script>"` > ########################### root@hetzner3 /usr/local/bin #
- and I updated the cron to run the script as root.
- we'll have to check-in on this tomorrow to see if it's working https://github.com/OpenSourceEcology/ansible/commit/720dc468f8bc6ddc08f77fdcdaa31b2a61eb4b35
- ...
- so yesterday I finished my quick verification of the wiki
- we're still blocked on osemain and obi
- otherwise, all the vhost snapshots are done
- I still haven't finished setting up the base server, so let's do that
- first, I'm the only human user who can ssh into this server
maltfield@hetzner3:/home$ ls b2user maltfield not-apache wp maltfield@hetzner3:/home$
- hetzner2 had a few additional users
- cataria
- marcin
- tgriffing
[maltfield@opensourceecology home]$ ls b2user cmota crupp Flipo hart jthomas lberezhny maltfield marcin not-apache stagingsync tgriffing wp [maltfield@opensourceecology home]$
- there's others, but I don't think they're active
- to protect the server from getting broken, it's important that Marcin and Catarina do *not* have root access
- to mitigate the bus factor, it's important that we have at least two users with root access: that's me and tgriffin
[root@opensourceecology ~]# grep wheel /etc/group wheel:x:10:maltfield,crupp,tgriffing,root,jthomas [root@opensourceecology ~]#
- it's not a huge risk, though. If marcin ever needed, he can just login on the hetzner WUI and reboot the server to a rescue distro, reset the root password, and get in that way
- and if he lost hetzner WUI access, he can KYC into it
- we *do* want Marcin and Catarina to have accounts, though. For ssh, access to webserver files (eg to scp a very large file that exceeds our https upload size limits), and keepass
[root@opensourceecology ~]# grep marcin /etc/group marcin:x:1001: apache:x:48:cmota,crupp,maltfield,wp,apache,marcin sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas keepass:x:993:maltfield,marcin,cmota,crupp apache-admins:x:1012:cmota,maltfield,marcin,crupp,tgriffing,wp,apache [root@opensourceecology ~]# [root@opensourceecology ~]# grep cmota /etc/group apache:x:48:cmota,crupp,maltfield,wp,apache,marcin cmota:x:1002: sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas keepass:x:993:maltfield,marcin,cmota,crupp apache-admins:x:1012:cmota,maltfield,marcin,crupp,tgriffing,wp,apache [root@opensourceecology ~]#
- I decided against provisioning users with ansible; I'll just do it manually
- The only thing I do need to keep in ansible is to poke a hole in the firewall for these new users
- I confirmed that ssh has been hardened with the 'sshaccess' groups already
root@hetzner3 /usr/local/bin # grep -i group /etc/ssh/sshd_config KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 # In case you don't use PAM (`UsePAM no`), you can alternatively restrict users and groups here. For key-based authentication this is not necessary, since all keys must be explicitely enabled. AllowGroups sshaccess root@hetzner3 /usr/local/bin #
- currently it's just me
root@hetzner3 /usr/local/bin # grep sshaccess /etc/group sshaccess:x:1001:maltfield root@hetzner3 /usr/local/bin #
- the keepass group doesn't yet exist, but the apache-admins group does
root@hetzner3 /usr/local/bin # grep keepass /etc/group root@hetzner3 /usr/local/bin # root@hetzner3 /usr/local/bin # grep apache /etc/group users:x:100:maltfield,wp,not-apache apache-admins:x:1004:www-data,wp not-apache:x:1006: root@hetzner3 /usr/local/bin #
- I updated https://wiki.opensourceecology.org/wiki/Hetzner3
- here's the pubkeys we have on hetzner2 for these new users
[root@opensourceecology ~]# cat /home/marcin/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500 [root@opensourceecology ~]# [root@opensourceecology ~]# cat /home/cmota/.ssh/authorized_keys #ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDOhf/CGM3t3sklCPAiwaAl7NQPiyDwy5WwV2oxO6rCsDC3jmwn2JCJ6LZ4jXI9NvQQXwg+1kOqsSv2ZFS6RUq1Rs/EW8jR5lFxy9+FgRFpAIlzglAWIBbjpamX0duCMAwpOH2bK1oEDokNM37TtBynFyNytJqLgIGI48IR45KIU4Ib/IN8KzcQRdfPMPFW3EcO1jHJCMUl5SIKt36rgFvhc6JmHF9sl2fCPZ6Jmbi21DOz8fVQDdxQNAc2ubkAS2Sn27LZS7ww6I1VPK43IKBZlZXm3/FQ5HyHrXsGelIDCwdrwM/GTffClDc9EX1ujEHXZIILBDDVLVbwzI+3QHsHvaU9pY+pXBC1dw4LN8InmVazGeV9MoH6SeSeEx4g4KS+2qMiF6fjViPzpLeDW7hIRiTc2+8pjoivNKcgO0xCIBJ/dvTpTExCOPDBVJisnU+3Psq6Wn0Nb7ts7A0tyCtGOG2rxQEYCRmIy8AvQ6WlKztGFvqrfdHaRO9v7WY0hMtyD7kh0tHhHgxSFNcWSIVAIf8JSfhm3f/Orcsg2JDliT4wJzMz7OAYGOQmpqDI+qYa/ULgp9qlf/XBQWeMrQwAU0YeepRJjx3rz9xMxOLj2ddV3JSS8wHg+BGLpPwvqaoFrVIXIgHkW0AbBoXHmtSIZSZ5t5+odWlT1aLb5GmlXw== catarinamota@catarinas-air.pk5001z ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer [root@opensourceecology ~]# [root@opensourceecology ~]# cat //home/tgriffing/.ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0 [root@opensourceecology ~]#
- here's the vars
USERNAME='marcin' PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500' USERNAME='cmota' PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer' USERNAME='tgriffing' PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0'
- and here's the commands I'll use to create these users
# create user if it doesn't yet exist
adduser ${USERNAME} --disabled-password --gecos ''
# add ssh pubkey
mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
- here's Marcin
root@hetzner3 ~ # USERNAME='marcin'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500'
root@hetzner3 ~ #
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `marcin' ...
Adding new group `marcin' (1007) ...
Adding new user `marcin' (1007) with group `marcin (1007)' ...
Creating home directory `/home/marcin' ...
Copying files from `/etc/skel' ...
Adding new user `marcin' to supplemental / extra groups `users' ...
Adding user `marcin' to group `users' ...
root@hetzner3 ~ #
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
- And Catarina
root@hetzner3 ~ # USERNAME='cmota'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer'
root@hetzner3 ~ #
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `cmota' ...
Adding new group `cmota' (1008) ...
Adding new user `cmota' (1008) with group `cmota (1008)' ...
Creating home directory `/home/cmota' ...
Copying files from `/etc/skel' ...
Adding new user `cmota' to supplemental / extra groups `users' ...
Adding user `cmota' to group `users' ...
root@hetzner3 ~ #
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
- And Tom
root@hetzner3 ~ # USERNAME='tgriffing'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0'
root@hetzner3 ~ #
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `tgriffing' ...
Adding new group `tgriffing' (1009) ...
Adding new user `tgriffing' (1009) with group `tgriffing (1009)' ...
Creating home directory `/home/tgriffing' ...
Copying files from `/etc/skel' ...
Adding new user `tgriffing' to supplemental / extra groups `users' ...
Adding user `tgriffing' to group `users' ...
root@hetzner3 ~ #
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
- then I added the users to the groups
gpasswd -a marcin sshaccess gpasswd -a cmota sshaccess gpasswd -a tgriffing sshaccess gpasswd -a marcin www-data gpasswd -a cmota www-data gpasswd -a tgriffing www-data gpasswd -a tgriffing apache-admins
- then I created the keepass group and added everyone to it
groupadd keepass gpasswd -a maltfield keepass gpasswd -a marcin keepass gpasswd -a cmota keepass gpasswd -a tgriffing keepass
- finally, I added Tom to the sudoers group
gpasswd -a tgriffing sudo
- I updated the ansible playbook to permit these human user accounts internet access https://github.com/OpenSourceEcology/ansible/commit/f294f3384f5d4bc47ae55a1631c264950e0e1355
user@ose:~/sandbox_local/ansible/hetzner3$ git diff diff --git a/hetzner3/provision.yml b/hetzner3/provision.yml index 0f8e428..9ae54a1 100644 --- a/hetzner3/provision.yml +++ b/hetzner3/provision.yml @@ -85,8 +85,11 @@ - -A OUTPUT -m owner --uid-owner root -j ACCEPT 820 allow apt: - -A OUTPUT -m owner --uid-owner _apt -j ACCEPT - 830 allow maltfield: + 830 allow human users internet access: - -A OUTPUT -m owner --uid-owner maltfield -j ACCEPT + - -A OUTPUT -m owner --uid-owner marcin -j ACCEPT + - -A OUTPUT -m owner --uid-owner cmota -j ACCEPT + - -A OUTPUT -m owner --uid-owner tgriffing -j ACCEPT 840 allow postfix: - -A OUTPUT -m owner --uid-owner postfix -j ACCEPT 850 allow stubby: @@ -128,8 +131,11 @@ - -A OUTPUT -m owner --uid-owner root -j ACCEPT 820 allow apt: - -A OUTPUT -m owner --uid-owner _apt -j ACCEPT - 830 allow maltfield: + 830 allow human users internet access: - -A OUTPUT -m owner --uid-owner maltfield -j ACCEPT + - -A OUTPUT -m owner --uid-owner marcin -j ACCEPT + - -A OUTPUT -m owner --uid-owner cmota -j ACCEPT + - -A OUTPUT -m owner --uid-owner tgriffing -j ACCEPT 840 allow postfix: - -A OUTPUT -m owner --uid-owner postfix -j ACCEPT 850 allow stubby: user@ose:~/sandbox_local/ansible/hetzner3$
- I sent an email to Marcin
Hi Marcin, Can you please confirm that ssh access to our new OSE server (hetzner3) is working? We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details: User: marcin Server: 144.76.164.201 Port: 32415 For example, you should be able to login with the following command: ssh -p 32415 marcin@144.76.164.201 The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2. After you login, please set a password for your account with the following command: passwd Please confirm that you were able to login to hetzner3 with ssh and update your password.
- I sent an email to Catarina
Hi Catarina, Can you please confirm that ssh access to our new OSE server (hetzner3) is working? We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details: User: cmota Server: 144.76.164.201 Port: 32415 For example, you should be able to login with the following command: ssh -p 32415 cmota@144.76.164.201 The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2. After you login, please set a password for your account with the following command: passwd Please confirm that you were able to login to hetzner3 with ssh and update your password.
- I sent an email to Tom
Hi Tom, Can you please confirm that ssh access to our new OSE server (hetzner3) is working? We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details: User: tgriffing Server: 144.76.164.201 Port: 32415 For example, you should be able to login with the following command: ssh -p 32415 tgriffing@144.76.164.201 The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2. After you login, please set a password for your account with the following command: passwd After updating your password, you should be able to become root with sudo. sudo su - Please confirm that you were able to login to hetzner3 with ssh, update your password, and become root.
- that's all the human accounts; I'll just have to wait for their confirmations
- ...
- another thing I wanted to do was to diff the postfix configs on hetzner2 & hetzner3. I did some basic postfix setup just to get wazuh functional. And it appears to work for phpList too, but I never checked the postfix config to make sure it's sane
- hetzner2
[root@opensourceecology ~]# postconf > /home/maltfield/postconf_hetzner2_20250214 [root@opensourceecology ~]# chown maltfield /home/maltfield/postconf_hetzner2_20250214 [root@opensourceecology ~]#
- hetzner3
root@hetzner3 ~ # postconf > /home/maltfield/postconf_hetzner3_20250214 root@hetzner3 ~ # chown maltfield /home/maltfield/postconf_hetzner3_20250214 root@hetzner3 ~ #
- and I copied them locally for a visual diff in meld
user@ose:~$ ssh-add ~/.ssh/id_rsa Enter passphrase for /home/user/.ssh/id_rsa: Identity added: /home/user/.ssh/id_rsa (guttersnipe@guttersnipe) user@ose:~$ cd tmp/hetzner3/ user@ose:~/tmp/hetzner3$ mkdir postfix user@ose:~/tmp/hetzner3$ cd postfix/ user@ose:~/tmp/hetzner3/postfix$ user@ose:~/tmp/hetzner3/postfix$ rsync -av --progress opensourceecology.org:postconf_hetzner2_20250214 . receiving incremental file list postconf_hetzner2_20250214 32,676 100% 31.16MB/s 0:00:00 (xfr#1, to-chk=0/1) sent 43 bytes received 32,799 bytes 4,378.93 bytes/sec total size is 32,676 speedup is 0.99 user@ose:~/tmp/hetzner3/postfix$ user@ose:~/tmp/hetzner3/postfix$ rsync -av --progress hetzner3:postconf_hetzner3_20250214 . receiving incremental file list postconf_hetzner3_20250214 40,776 100% 38.89MB/s 0:00:00 (xfr#1, to-chk=0/1) sent 43 bytes received 40,908 bytes 7,445.64 bytes/sec total size is 40,776 speedup is 1.00 user@ose:~/tmp/hetzner3/postfix$
- here's the diff
user@ose:~/tmp/hetzner3/postfix$ diff postconf_hetzner2_20250214 postconf_hetzner3_20250214
11c11,12
< address_verify_poll_count = ${stress?1}${stress:3}
---
> address_verify_pending_request_limit = 5000
> address_verify_poll_count = ${stress?{1}:{3}}
37c38
< append_dot_mydomain = yes
---
> append_dot_mydomain = ${{$compatibility_level} <level {1} ? {yes} : {no}}
56a58
> cleanup_replace_stray_cr_lf = yes
61a64
> compatibility_level = 0
62a66
> confirm_delay_cleared = no
69c73
< daemon_directory = /usr/libexec/postfix
---
> daemon_directory = /usr/lib/postfix/sbin/
79a84
> default_delivery_status_filter =
95a101
> default_transport_rate_delay = 0s
111c117,118
< disable_vrfy_command = no
---
> disable_vrfy_command = yes
> dns_ncache_ttl_fix_enable = no
113a121
> dnssec_probe = ns:.
117a126
> empty_address_local_login_sender_maps_lookup_key = <>
119a129
> enable_idna2003_compatibility = no
121a132
> enable_threaded_bounces = no
138a150
> error_transport_rate_delay = $default_transport_rate_delay
157a170
> header_from_format = standard
164c177
< import_environment = MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ XAUTHORITY DISPLAY LANG=C
---
> import_environment = MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ XAUTHORITY DISPLAY LANG=C POSTLOG_SERVICE POSTLOG_HOSTNAME
166c179
< inet_interfaces = localhost
---
> inet_interfaces = 127.0.0.1, [::1], localhost
167a181
> info_log_address_format = external
173a188
> known_tcp_ports = lmtp=24, smtp=25, smtps=submissions=465, submission=587
174a190
> lmdb_map_size = 16777216
175a192
> lmtp_address_verify_target = rcpt
176a194
> lmtp_balance_inet_protocols = yes
178a197
> lmtp_bind_address_enforce = no
184a204
> lmtp_connection_reuse_count_limit = 0
192a213
> lmtp_delivery_status_filter = $default_delivery_status_filter
200a222
> lmtp_dns_reply_filter =
201a224
> lmtp_dns_support_level =
203a227
> lmtp_fallback_relay =
212a237
> lmtp_min_data_rate = 500
217c242
< lmtp_per_record_deadline = no
---
> lmtp_per_request_deadline = ${lmtp_per_record_deadline?{$lmtp_per_record_deadline}:{no}}
253c278,280
< lmtp_tls_ciphers = export
---
> lmtp_tls_chain_files =
> lmtp_tls_ciphers = medium
> lmtp_tls_connection_reuse = no
261c288,289
< lmtp_tls_fingerprint_digest = md5
---
> lmtp_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> lmtp_tls_force_insecure_host_tlsa_lookup = no
266c294
< lmtp_tls_mandatory_protocols = !SSLv2
---
> lmtp_tls_mandatory_protocols = !SSLv2, !SSLv3
270c298
< lmtp_tls_protocols = !SSLv2
---
> lmtp_tls_protocols = !SSLv2, !SSLv3
273a302
> lmtp_tls_servername =
275a305
> lmtp_tls_trust_anchor_file =
276a307,308
> lmtp_tls_wrappermode = no
> lmtp_transport_rate_delay = $default_transport_rate_delay
282a315
> local_delivery_status_filter = $default_delivery_status_filter
291a325
> local_login_sender_maps = static:*
297a332
> local_transport_rate_delay = $default_transport_rate_delay
301c336
< mail_release_date = 20130622
---
> mail_release_date = 20240304
303,304c338,339
< mail_version = 2.10.1
< mailbox_command = /usr/bin/procmail
---
> mail_version = 3.7.11
> mailbox_command =
309a345,348
> maillog_file =
> maillog_file_compressor = gzip
> maillog_file_prefixes = /var, /dev/stdout
> maillog_file_rotate_suffix = %Y%m%d-%H%M%S
321a361
> message_drop_headers = bcc, content-length, resent-bcc, return-path
324a365
> meta_directory = /etc/postfix
326c367
< milter_connect_macros = j {daemon_name} v
---
> milter_connect_macros = j {daemon_name} {daemon_addr} v _
335a377
> milter_macro_defaults =
353,354c395,396
< myhostname = mailer.opensourceecology.org
< mynetworks = 127.0.0.1/32
---
> myhostname = hetzner3.opensourceecology.org
> mynetworks = 127.0.0.1/32 [::1]/128
361a404
> openssl_path = openssl
365a409
> pipe_delivery_status_filter = $default_delivery_status_filter
366a411,412
> postlog_service_name = postlog
> postlogd_watchdog_timeout = 10s
370a417
> postscreen_allowlist_interfaces = ${postscreen_whitelist_interfaces?{$postscreen_whitelist_interfaces}:{static:all}}
374d420
< postscreen_blacklist_action = ignore
381c427,428
< postscreen_command_time_limit = ${stress?10}${stress:300}s
---
> postscreen_command_time_limit = ${stress?{10}:{300}}s
> postscreen_denylist_action = ${postscreen_blacklist_action?{$postscreen_blacklist_action}:{ignore}}
385a433,435
> postscreen_dnsbl_allowlist_threshold = ${postscreen_dnsbl_whitelist_threshold?{$postscreen_dnsbl_whitelist_threshold}:{0}}
> postscreen_dnsbl_max_ttl = ${postscreen_dnsbl_ttl?{$postscreen_dnsbl_ttl}:{1}}h
> postscreen_dnsbl_min_ttl = 60s
389c439
< postscreen_dnsbl_ttl = 1h
---
> postscreen_dnsbl_timeout = 10s
396c446
< postscreen_greet_wait = ${stress?2}${stress:6}s
---
> postscreen_greet_wait = ${stress?{2}:{6}}s
406a457
> postscreen_reject_footer_maps = $smtpd_reject_footer_maps
412,414c463,464
< postscreen_whitelist_interfaces = static:all
< prepend_delivered_header = command, file, forward
< process_id = 5305
---
> prepend_delivered_header =
> process_id = 1460591
419c469
< proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps $alias_maps
---
> proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps $alias_maps $smtpd_client_restrictions $smtpd_helo_restrictions $smtpd_sender_restrictions $smtpd_relay_restrictions $smtpd_recipient_restrictions $address_verify_sender_dependent_default_transport_maps $address_verify_sender_dependent_relayhost_maps $address_verify_transport_maps $fallback_transport_maps $lmtp_discard_lhlo_keyword_address_maps $lmtp_pix_workaround_maps $lmtp_sasl_password_maps $lmtp_tls_policy_maps $mailbox_command_maps $mailbox_transport_maps $postscreen_discard_ehlo_keyword_address_maps $rbl_reply_maps $sender_dependent_default_transport_maps $sender_dependent_relayhost_maps $smtp_discard_ehlo_keyword_address_maps $smtp_pix_workaround_maps $smtp_sasl_password_maps $smtp_tls_policy_maps $smtpd_discard_ehlo_keyword_address_maps $smtpd_milter_maps $virtual_gid_maps $virtual_uid_maps $local_login_sender_maps $postscreen_reject_footer_maps $smtpd_reject_footer_maps $tls_server_sni_maps $tlsproxy_client_policy_maps $default_delivery_status_filter $lmtp_delivery_status_filter $lmtp_dns_reply_filter $lmtp_reply_filter $local_delivery_status_filter $pipe_delivery_status_filter $postscreen_command_filter $smtp_delivery_status_filter $smtp_dns_reply_filter $smtp_reply_filter $smtpd_command_filter $smtpd_dns_reply_filter $virtual_delivery_status_filter $body_checks $header_checks $lmtp_body_checks $lmtp_header_checks $lmtp_mime_header_checks $lmtp_nested_header_checks $milter_header_checks $mime_header_checks $nested_header_checks $smtp_body_checks $smtp_header_checks $smtp_mime_header_checks $smtp_nested_header_checks
458c508
< relay_domains = $mydestination
---
> relay_domains = ${{$compatibility_level} <level {2} ? {$mydestination} : {}}
467a518
> relay_transport_rate_delay = $default_transport_rate_delay
475a527
> respectful_logging = ${{$compatibility_level} <level {3.6} ? {no} : {yes}}
490a543
> retry_transport_rate_delay = $default_transport_rate_delay
500a554
> service_name =
502a557
> shlib_directory = /usr/lib/postfix
505a561
> smtp_address_verify_target = rcpt
506a563
> smtp_balance_inet_protocols = yes
508a566
> smtp_bind_address_enforce = no
514a573
> smtp_connection_reuse_count_limit = 0
522a582
> smtp_delivery_status_filter = $default_delivery_status_filter
530a591
> smtp_dns_reply_filter =
531a593
> smtp_dns_support_level =
543a606
> smtp_min_data_rate = 500
549c612
< smtp_per_record_deadline = no
---
> smtp_per_request_deadline = ${smtp_per_record_deadline?{$smtp_per_record_deadline}:{no}}
579a643
> smtp_tcp_port = smtp
584c648,651
< smtp_tls_ciphers = export
---
> smtp_tls_chain_files =
> smtp_tls_ciphers = medium
> smtp_tls_connection_reuse = no
> smtp_tls_dane_insecure_mx_policy = ${{$smtp_tls_security_level} == {dane} ? {dane} : {may}}
592c659,660
< smtp_tls_fingerprint_digest = md5
---
> smtp_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> smtp_tls_force_insecure_host_tlsa_lookup = no
594c662
< smtp_tls_loglevel = 0
---
> smtp_tls_loglevel = 1
597,598c665,666
< smtp_tls_mandatory_protocols = !SSLv2
< smtp_tls_note_starttls_offer = no
---
> smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
> smtp_tls_note_starttls_offer = yes
601c669
< smtp_tls_protocols = !SSLv2
---
> smtp_tls_protocols = !SSLv2, !SSLv3
604c672,673
< smtp_tls_security_level =
---
> smtp_tls_security_level = may
> smtp_tls_servername =
606a676
> smtp_tls_trust_anchor_file =
607a678,679
> smtp_tls_wrappermode = no
> smtp_transport_rate_delay = $default_transport_rate_delay
614,617c686,690
< smtpd_client_connection_count_limit = 50
< smtpd_client_connection_rate_limit = 0
< smtpd_client_event_limit_exceptions = ${smtpd_client_connection_limit_exceptions:$mynetworks}
< smtpd_client_message_rate_limit = 0
---
> smtpd_client_auth_rate_limit = 0
> smtpd_client_connection_count_limit = 4
> smtpd_client_connection_rate_limit = 120
> smtpd_client_event_limit_exceptions = $mynetworks
> smtpd_client_message_rate_limit = 120
620,621c693,694
< smtpd_client_recipient_rate_limit = 0
< smtpd_client_restrictions =
---
> smtpd_client_recipient_rate_limit = 120
> smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated, check_helo_access hash:/etc/postfix/whitelisted_domains, reject_unknown_client_hostname
627a701
> smtpd_dns_reply_filter =
630c704
< smtpd_error_sleep_time = 1s
---
> smtpd_error_sleep_time = 4s
633,636c707,714
< smtpd_forbidden_commands = CONNECT GET POST
< smtpd_hard_error_limit = ${stress?1}${stress:20}
< smtpd_helo_required = no
< smtpd_helo_restrictions =
---
> smtpd_forbid_bare_newline = no
> smtpd_forbid_bare_newline_exclusions = $mynetworks
> smtpd_forbid_bare_newline_reject_code = 550
> smtpd_forbid_unauth_pipelining = no
> smtpd_forbidden_commands = CONNECT GET POST regexp:{{/^[^A-Z]/ Bogus}}
> smtpd_hard_error_limit = 10
> smtpd_helo_required = yes
> smtpd_helo_restrictions = permit_mynetworks, check_helo_access hash:/etc/postfix/whitelisted_domains, permit_sasl_authenticated
638c716
< smtpd_junk_command_limit = ${stress?1}${stress:100}
---
> smtpd_junk_command_limit = ${stress?{1}:{100}}
639a718
> smtpd_milter_maps =
640a720
> smtpd_min_data_rate = 500
644c724,725
< smtpd_per_record_deadline = ${stress?yes}${stress:no}
---
> smtpd_per_request_deadline = ${smtpd_per_record_deadline?{$smtpd_per_record_deadline}:{${stress?{yes}:{no}}}}
> smtpd_policy_service_default_action = 451 4.3.5 Server configuration problem
646a728,730
> smtpd_policy_service_policy_context =
> smtpd_policy_service_request_limit = 0
> smtpd_policy_service_retry_delay = 1s
647a732
> smtpd_policy_service_try_limit = 2
654c739
< smtpd_recipient_restrictions =
---
> smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, check_recipient_access hash:/etc/postfix/whitelisted_domains, reject_unauth_destination
655a741
> smtpd_reject_footer_maps =
658c744,745
< smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination
---
> smtpd_relay_before_recipient_restrictions = ${{$compatibility_level} <level {3.6} ? {no} : {yes}}
> smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
663a751
> smtpd_sasl_mechanism_filter = !external, static:rest
664a753
> smtpd_sasl_response_limit = 12288
665a755
> smtpd_sasl_service = smtp
669c759
< smtpd_sender_restrictions =
---
> smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/whitelisted_domains, reject_unknown_sender_domain
671,673c761,763
< smtpd_soft_error_limit = 10
< smtpd_starttls_timeout = ${stress?10}${stress:300}s
< smtpd_timeout = ${stress?10}${stress:300}s
---
> smtpd_soft_error_limit = 4
> smtpd_starttls_timeout = ${stress?{10}:{300}}s
> smtpd_timeout = ${stress?{10}:{300}}s
678c768
< smtpd_tls_auth_only = no
---
> smtpd_tls_auth_only = yes
680,681c770,772
< smtpd_tls_cert_file =
< smtpd_tls_ciphers = export
---
> smtpd_tls_cert_file = /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
> smtpd_tls_chain_files =
> smtpd_tls_ciphers = medium
683,684c774,775
< smtpd_tls_dh1024_param_file =
< smtpd_tls_dh512_param_file =
---
> smtpd_tls_dh1024_param_file = /etc/ssl/certs/dhparam.pem
> smtpd_tls_dh512_param_file = /etc/ssl/certs/dhparam.pem
688c779
< smtpd_tls_eecdh_grade = strong
---
> smtpd_tls_eecdh_grade = ultra
690,693c781,784
< smtpd_tls_fingerprint_digest = md5
< smtpd_tls_key_file = $smtpd_tls_cert_file
< smtpd_tls_loglevel = 0
< smtpd_tls_mandatory_ciphers = medium
---
> smtpd_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> smtpd_tls_key_file = /etc/letsencrypt/live/opensourceecology.org/privkey.pem
> smtpd_tls_loglevel = 1
> smtpd_tls_mandatory_ciphers = high
695,696c786,787
< smtpd_tls_mandatory_protocols = !SSLv2
< smtpd_tls_protocols =
---
> smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
> smtpd_tls_protocols = !SSLv2, !SSLv3
699c790
< smtpd_tls_security_level =
---
> smtpd_tls_security_level = may
705a797,798
> smtputf8_autodetect_classes = sendmail, verify
> smtputf8_enable = ${{$compatibility_level} <level {1} ? {no} : {yes}}
714a808
> strict_smtputf8 = no
718c812
< syslog_name = ${multi_instance_name:postfix}${multi_instance_name?$multi_instance_name}
---
> syslog_name = ${multi_instance_name?{$multi_instance_name}:{postfix}}
720a815,816
> tls_config_file = default
> tls_config_name =
721a818
> tls_dane_digests = sha512 sha256
722a820
> tls_eecdh_auto_curves = X25519 X448 prime256v1 secp521r1 secp384r1
725,726c823,825
< tls_export_cipherlist = aNULL:-aNULL:ALL:+RC4:@STRENGTH
< tls_high_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH
---
> tls_export_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:LOW:EXPORT:+RC4:@STRENGTH
> tls_fast_shutdown_enable = yes
> tls_high_cipherlist = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA
728,729c827,828
< tls_low_cipherlist = aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH
< tls_medium_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH
---
> tls_low_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:LOW:+RC4:@STRENGTH
> tls_medium_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH
736a836,858
> tls_server_sni_maps =
> tls_session_ticket_cipher = aes-256-cbc
> tls_ssl_options = NO_COMPRESSION
> tls_wildcard_matches_multiple_labels = yes
> tlsmgr_service_name = tlsmgr
> tlsproxy_client_CAfile = $smtp_tls_CAfile
> tlsproxy_client_CApath = $smtp_tls_CApath
> tlsproxy_client_cert_file = $smtp_tls_cert_file
> tlsproxy_client_chain_files = $smtp_tls_chain_files
> tlsproxy_client_dcert_file = $smtp_tls_dcert_file
> tlsproxy_client_dkey_file = $smtp_tls_dkey_file
> tlsproxy_client_eccert_file = $smtp_tls_eccert_file
> tlsproxy_client_eckey_file = $smtp_tls_eckey_file
> tlsproxy_client_enforce_tls = $smtp_enforce_tls
> tlsproxy_client_fingerprint_digest = $smtp_tls_fingerprint_digest
> tlsproxy_client_key_file = $smtp_tls_key_file
> tlsproxy_client_loglevel = $smtp_tls_loglevel
> tlsproxy_client_loglevel_parameter = smtp_tls_loglevel
> tlsproxy_client_per_site = $smtp_tls_per_site
> tlsproxy_client_policy_maps = ${tlsproxy_client_policy:$smtp_tls_policy_maps}
> tlsproxy_client_scert_verifydepth = $smtp_tls_scert_verifydepth
> tlsproxy_client_security_level = ${tlsproxy_client_level:$smtp_tls_security_level}
> tlsproxy_client_use_tls = $smtp_use_tls
744a867
> tlsproxy_tls_chain_files = $smtpd_tls_chain_files
763d885
< tlsproxy_tls_session_cache_timeout = $smtpd_tls_session_cache_timeout
789c911,912
< virtual_alias_domains = $virtual_alias_maps
---
> virtual_alias_address_length_limit = 1000
> virtual_alias_domains = openbuildinginstitute.org osedev.org
791c914
< virtual_alias_maps = $virtual_maps
---
> virtual_alias_maps = hash:/etc/postfix/virtual
795a919
> virtual_delivery_status_filter = $default_delivery_status_filter
815a940
> virtual_transport_rate_delay = $default_transport_rate_delay
user@ose:~/tmp/hetzner3/postfix$
- here's what I noticed
- disable_vrfy_command changed from 'no' to 'yes'
- docs say enabling this "stops some techniques used to harvest email addresses.", so that's probably good https://www.postfix.org/postconf.5.html
- inet_interfaces now includes 127.0.0.1 and [::1]
- Not sure why that was missing before, but I don't see any issues with it
- note that it's not bound to any external IPs. This seems problematic, but it's as-desired. OSE doesn't use postfix for receiving mail. Our MX records point to our free Google Workplace account. We just use postfix for *sending* mail, which greatly reduces a lot of the risk and complexity.
- lmtp_tls_ciphers changed from "export" to "medium"
- according to the docs " In Postfix ≥ 3.8 this [export] cipher grade is always identical to "medium"" https://www.postfix.org/postconf.5.html#smtp_tls_policy_maps
- I'm guessing this is a shadow of when the US had laws limiting the export of some crypto. Jesus, CentOS 7 is old. That shit changed in, what, the 90s?
- anyway, there's no change here
- the dirty part of TLS encryption in emails is that it's only opportunistic and always vulnerable to downgrade attacks (eg sslstrip). To require encryption between our server and other email servers would be going against the RFC standard, and it would impair our (eg phpList) email delivery :( so we'll leave this at the default
- lmtp_tls_fingerprint_digest changed from md5 to sha256, if possible
- that's good
- lmtp_tls_mandatory_protocols changed from disallowing SSLv2 to disallowing both SSLv2 and SSLv3
- that's good. we should only be using TLS
- same as ^ with lmtp_tls_protocols
- mailbox_command changed from '/usr/bin/procmail' to
- idk what that means, but I don't think it matters for us
- myhostname changed from 'mailer.opensourceecology.org' to 'hetzner3.opensourceecology.org'
- yeah, I named the old server 'mail' hoping it would decrease risk of email bounces, but that had some weird consequences. 'hetzner3' seems to work, afaict
- 'mynetworks' changed from '127.0.0.1/32' to '127.0.0.1/32 [::1]/128'
- I see no harm in adding local ipv6
- postscreen_blacklist_action disappeared
- docs say this was renamed to postscreen_denylist_action https://www.postfix.org/postconf.5.html#postscreen_blacklist_action
- looks like it's still 'ignore' in both cases
- postscreen_dnsbl_ttl changed from '1h' to '60s'
- that's a big change
- docs say default is 'https://www.postfix.org/postconf.5.html#postscreen_dnsbl_ttl
- docs say this is basically a cache: "The amount of time that postscreen(8) remembers that a client IP address passed a DNS-based reputation test, before it is required to pass that test again. "
- I see zero impact on OSE's postfix server, which isn't even exposed to the public internet
- docs say
- same for postscreen_dnsbl_ttl
- smtp_tls_fingerprint_digest similar to above, possibly use sha256
- smtp_tls_loglevel changed from 0 to 1
- the default is 0, but I guess debian changed this? Or maybe I did. Anyway, docs say it's ok to use anything <2
- smtp_tls_note_starttls_offer changed from 'no' to 'yes'
- also just affects logging. default is 'no'. So now we just log the hostname. Should be fine.
- smtp_tls_protocols changed, same as above, to disable SSLv3 too
- smtpd_client_message_rate_limit changed from 0 to 120
- docs say the default is 0. We probably would be fine to have this at 0 for OSE's purposes. But I don't see an issue at 120 either.
- smtpd_client_restrictions changed from to 'permit_mynetworks, permit_sasl_authenticated, check_helo_access hash:/etc/postfix/whitelisted_domains, reject_unknown_client_hostname'
- that should be fine
- smtpd_error_sleep_time changed from 1s to 4s
- I think that's fine
- smtpd_helo_required changed from 'no' to 'yes'
- shouldn't matter, and probably a good change
- smtpd_relay_restrictions changed from 'permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination' to 'permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination'
- so 'defer' changed to 'reject'. I guess that's fine.
- smtpd_soft_error_limit changed from 10 to 4
- docs suggest this is a tarpit config. After this number of errors, it slows down processing of requests from that client
- again, shouldn't affect us
- smtpd_tls_auth_only changed from 'no' to 'yes'
- this apparently only affects sasl auth, which is forced to use TLS now
- should be fine; we're not doing any SASL auth
- smtpd_tls_auth_only changed from to '/etc/letsencrypt/live/opensourceecology.org/fullchain.pem'
- wow I can't believe this was empty before. So I guess we were one of those servers that was sending all smtp messages without encryption? That or snake oil.
- this is a good change
- smtpd_tls_dh1024_param_file and smtpd_tls_dh512_param_file got set to our ansible-made dhparam file; that's good
- smtpd_tls_eecdh_grade changed from 'strong' to 'ultra'
- docs say that's an increase from 128 bits to 192 bits. It says it's twice as slow. should be fine.
- smtpd_tls_mandatory_ciphers changed from 'medium' to 'high'
- that's probably for the best
- smtpd_tls_security_level changed from to 'may'
- yeah, apparently empty means "don't use TLS". So this is a very good change.
- again, 'encrypt' enforces TLS, but that's a violation of RFC 2487 https://tools.ietf.org/html/rfc2487
- tls_export_cipherlist changed from 'aNULL:-aNULL:ALL:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:LOW:EXPORT:+RC4:@STRENGTH'
- looks like "ALL" was removed ,so that's probably good
- tls_high_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH' to 'EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA'
- ALL was removed. I don't like that it lists SSLv3
- usually my go-to is https://bettercrypto.org/ but it looks like they took the site down saying it's not longer updated :(
- the docs say that you are "strongly encouraged to not change" this setting
- well, it appears that we are changing it
- disable_vrfy_command changed from 'no' to 'yes'
user@ose:~/sandbox_local/ansible/hetzner3$ grep -ir cipherlist * roles/maltfield.postfix/templates/main.cf.j2:tls_high_cipherlist=EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA roles/maltfield.postfix/templates/master.cf.j2: -o tls_preempt_cipherlist=yes user@ose:~/sandbox_local/ansible/hetzner3$
- here we can view the default settings. that's not very helpful
root@hetzner3 ~ # postconf -d | grep -i tls_high_cipherlist tls_high_cipherlist = aNULL:-aNULL:HIGH:@STRENGTH root@hetzner3 ~ #
- this SE answer cites bettercrypto.org, and it includes SSLv3 too; I guess we'll leave it as-is https://serverfault.com/a/670359
- tls_low_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:LOW:+RC4:@STRENGTH'
- tls_medium_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH'
- tlsproxy_tls_session_cache_timeout disappeared
- docs say it's obsolete and replaced by smtpd_tls_session_cache_timeout
- virtual_alias_domains changed from '$virtual_alias_maps' to 'openbuildinginstitute.org osedev.org'
- virtual_alias_maps changed from '$virtual_maps' to 'hash:/etc/postfix/virtual'
- note that '$virtual_maps' isn't set on hetzner2, so I don't think any alias domains were defined
- it might actually be better to use an external maps file (rather than just hardcoding it), but I don't want to break it; let's leave it as-is
- there were lots of other changes, but those were the ones that stood-out to me
- in general, I'd say the changes were good: err-ing on the side of "more" secure
- ok, postfix looks fine to me
- SPF records and rDNS were already set
- probably after the migration we'd want to remove the old server, but there's nothing to do now until we migrate all the webservers
- oh, actually, one thing I never setup was SPF records for non-ose domains (eg obi, osedev, oswh)
- I opened the shared ose keepass, and I saw 2 domains in ghandi:
- oswarehouse.org
user@disp5497:~$ dig +short -t txt oswarehouse.org "v=spf1 include:_mailcust.gandi.net ?all" user@disp5497:~$
- opensourcewarehouse.org
- I think those are completely unused; the nameserver for the former is ghandi and the nameserver for the later is set to dreamhost
- oh, there *is* an spf record for the ghandi one (oswarehouse.org), but it's not something we ever set = ""v=spf1 include:_mailcust.gandi.net ?all""
- I didn't log into dreamhost, but a query shows the same: we never set it
user@disp5497:~$ dig +short -t txt opensourcewarehouse.org "v=spf1 mx include:netblocks.dreamhost.com include:relay.mailchannels.net -all" user@disp5497:~$
- I'm not going to worry about email records (MX, spf) for these domains
- I think the others are on cloudflare? let's check
- I logged into cloudflare, but I only see 1 domain = opensourceecology.org
- yeah, I think this is why I left a note for myself to research further; I don't know where is OBI
- ah, dig says the nameserver is dreamhost
user@disp5497:~$ dig NS openbuildinginstitute.org ; <<>> DiG 9.16.50-Debian <<>> NS openbuildinginstitute.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7804 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;openbuildinginstitute.org. IN NS ;; ANSWER SECTION: openbuildinginstitute.org. 14400 IN NS ns1.dreamhost.com. openbuildinginstitute.org. 14400 IN NS ns2.dreamhost.com. openbuildinginstitute.org. 14400 IN NS ns3.dreamhost.com. ;; Query time: 116 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Fri Feb 14 17:19:06 -05 2025 ;; MSG SIZE rcvd: 121 user@disp5497:~$
- I tried to login to dreamhost, but I couldn't. I entered the creds in the shared ose keepass, then it demanded an OTP sent to some email.
- I decided to create a new gapps account for this, to fix this problem in the future
- I sent an email to Marcin asking him to change the email address with that account from his personal email to this new, shared email.
- I guess the domain/spf work is paused until I can gain access to dreamhost again
- looks like the spf records for obi don't matter either
user@disp5497:~$ dig -t txt openbuildinginstitute.org ; <<>> DiG 9.16.50-Debian <<>> -t txt openbuildinginstitute.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10607 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;openbuildinginstitute.org. IN TXT ;; ANSWER SECTION: openbuildinginstitute.org. 300 IN TXT "v=spf1 mx include:netblocks.dreamhost.com include:relay.mailchannels.net -all" ;; Query time: 133 msec ;; SERVER: 10.139.1.1#53(10.139.1.1) ;; WHEN: Fri Feb 14 17:40:26 -05 2025 ;; MSG SIZE rcvd: 144 user@disp5497:~$
- ...but I'll still need access to dreamhost, anyway – to update the A and AAAA records for the migration to hetzner3
- ...
- another unknown that I wanted to figure-out was if we could host the same domain (eg for obi and osemain) on two distinct IP addresses
- but we only have one IPv4 address. We could pay for a second one, but we have access to tons of free IPv6 addreses. The question is: does that work with /etc/hosts? Will it work for Marcin and Catarina?
- first I checked the IP addresses we *do* have
root@hetzner3 /etc/nginx/sites-enabled # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 90:1b:0e:c4:28:b4 brd ff:ff:ff:ff:ff:ff inet 144.76.164.201/27 brd 144.76.164.223 scope global enp0s31f6 valid_lft forever preferred_lft forever inet6 2a01:4f8:200:40d7::2/64 scope global valid_lft forever preferred_lft forever inet6 fe80::921b:eff:fec4:28b4/64 scope link valid_lft forever preferred_lft forever root@hetzner3 /etc/nginx/sites-enabled #
- woah, that says we have a '/27' IP address. Is that a misconfig? /32 would be 1 IP. A /27 would be 32 IPs https://www.freecodecamp.org/news/subnet-cheat-sheet-24-subnet-mask-30-26-27-29-and-other-ip-address-cidr-network-references/
- I really, really doubt hetzner gave us 32 IP addresses, but we should check
- this subnet calculator says "144.76.164.201/27" would mean that we have "144.76.164.193 - 144.76.164.222" https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=27&cip=144.76.164.201&ctype=ipv4&x=Calculate
- I logged into the hetzner site for managing hetzner 3 (and hetzner2) https://ro