My work log from the fourth quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

Fri Dec 13, 2024

here's TOFU 2/3 (VPN, exit in Sweden) for the new wordpress plugins

Sweden
2024-12-13
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . . . . . . . . jq: error (at <stdin>:0): Cannot index array with string "1.0.17"
. . . . . . . . . . . . . . . . . . . . . 


WARNING: Failed to download plugin woo-multi-currency
	null
	null

WARNING: Failed to download plugin woo-multi-currency
	null
	null

https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.4.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.25.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
2024-12-13
8b1f9a708838b8710b4198da1116689197e0a6134e0a1a5e786500576383034f  activitypub.4.4.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
1a53bdcd1ddb160d5807dc17a0f9e474402e22c899b3a9af486c9d5f0d2c4b36  extensions-leaflet-map.4.4.zip
27f1ab1e3f5274335d48d0cadaabdef98284880b0324771890d36a1f562fb44a  hcaptcha-for-forms-and-more.4.8.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
db016ec3c115ec20c1f0fba87b48b5eddee3a11f30d573b8a266a01077ee7ee1  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
ebd87841f73bb7946216ae4827a413dcc97fc5094cee2f8ddb6dea7eff356358  smart-slider-3.3.5.1.25.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
f9ce7a98840dd4bf490d955320a68ac553c767ba7f0eeae6e4f067be5a927ef3  wordpress-seo.24.0.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
user@disp7639:/tmp/tmp.FcejZlvblB$

...
I checked munin to see if my changes yesterday have filled the empty charts with data
3 out of 5 of the mysql charts now have data. the two that are empty still are "MySQL InnoDB free tablespace" and "MySQL slow queries". It's quite possible that this idle server has no data because – it's idle
I also wanted to see about adding all the other mysql charts possible, so I made a backup of the munin dir, deleted all the pluings/ symlinks for mysql, and recreated them

root@hetzner3 /usr/share/munin/plugins # tar -czf /var/tmp/munin.20241213.tar.gz /etc/munin/*
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
root@hetzner3 /usr/share/munin/plugins # du -sh /var/tmp/munin.20241213.tar.gz 
28K     /var/tmp/munin.20241213.tar.gz
root@hetzner3 /usr/share/munin/plugins #

shit, I just accidentally deleted all the mysql plugins
I was *trying* to delete the symlinks from /etc/munin/plugins. And to be safe, I even created a backup of this dir first. But then I was in the wrong screen session and I didn't realize my pwd

root@hetzner3 /usr/share/munin/plugins # tar -czf /var/tmp/munin.20241213.tar.gz /etc/munin/*
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
root@hetzner3 /usr/share/munin/plugins #

root@hetzner3 /usr/share/munin/plugins # du -sh /var/tmp/munin.20241213.tar.gz
28K     /var/tmp/munin.20241213.tar.gz
root@hetzner3 /usr/share/munin/plugins # 

root@hetzner3 /usr/share/munin/plugins # rm -f mysql_*
root@hetzner3 /usr/share/munin/plugins # ls -lah | grep -i mysql
root@hetzner3 /usr/share/munin/plugins #

fortunately I was able to restore them using apt. Note the package isn't 'munin' but 'munin-plugins-core'

root@hetzner3 /usr/share/munin/plugins # ls -lah | grep -i mysql
root@hetzner3 /usr/share/munin/plugins #

root@hetzner3 /usr/share/munin/plugins # sudo apt-get -o Dpkg::Options::="--force-confmiss" install --reinstall munin-plugins-core
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 71 not upgraded.
Need to get 0 B/242 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 66512 files and directories currently installed.)
Preparing to unpack .../munin-plugins-core_2.0.73-1_all.deb ...
Unpacking munin-plugins-core (2.0.73-1) over (2.0.73-1) ...
Setting up munin-plugins-core (2.0.73-1) ...
root@hetzner3 /usr/share/munin/plugins # 

root@hetzner3 /usr/share/munin/plugins # ls -lah | grep -i mysql                                      -rwxr-xr-x 1 root root  43K Mar 21  2023 mysql_
-rwxr-xr-x 1 root root 1,8K Mar 21  2023 mysql_bytes
-rwxr-xr-x 1 root root 5,6K Mar 21  2023 mysql_innodb
-rwxr-xr-x 1 root root 2,6K Mar 21  2023 mysql_queries
-rwxr-xr-x 1 root root 1,5K Mar 21  2023 mysql_slowqueries
-rwxr-xr-x 1 root root 1,6K Mar 21  2023 mysql_threads
root@hetzner3 /usr/share/munin/plugins # 
You have new mail in /var/mail/root

ok, now I *actually* deleted the symlinks and recreated them to be the complete set of possible charts

root@hetzner3 /usr/share/munin/plugins # tar -czf /var/tmp/munin.20241213.tar.gz /etc/munin/*
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
root@hetzner3 /usr/share/munin/plugins #

root@hetzner3 /usr/share/munin/plugins # du -sh /var/tmp/munin.20241213.tar.gz
28K     /var/tmp/munin.20241213.tar.gz
root@hetzner3 /usr/share/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql
lrwxrwxrwx 1 root root   31 Sep 25 01:47 mysql_ -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   36 Sep 25 01:47 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes
lrwxrwxrwx 1 root root   37 Sep 25 01:47 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb
lrwxrwxrwx 1 root root   42 Sep 25 01:47 mysql_isam_space_ -> /usr/share/munin/plugins/mysql_isam_space_
lrwxrwxrwx 1 root root   38 Sep 25 01:47 mysql_queries -> /usr/share/munin/plugins/mysql_queries
lrwxrwxrwx 1 root root   42 Sep 25 01:47 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries
lrwxrwxrwx 1 root root   38 Sep 25 01:47 mysql_threads -> /usr/share/munin/plugins/mysql_threads
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # rm -f mysql*
root@hetzner3 /etc/munin/plugins # for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -sf /usr/share/munin/plugins/mysql_ $i; done
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql
lrwxrwxrwx 1 root root   31 Dec 13 20:09 binlog_groupcommit -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 bin_relay_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 commands -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 files_tables -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_bpool -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_bpool_act -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_insert_buf -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_io -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_io_pend -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_rows -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_semaphores -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_tnx -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 myisam_indexes -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 qcache -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 qcache_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 replication -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 select_types -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 slow -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 sorts -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 table_locks -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 tmp_tables -> /usr/share/munin/plugins/mysql_
root@hetzner3 /etc/munin/plugins #

I applied and regenerated munin

root@hetzner3 /etc/munin/plugins # service munin-node restart
root@hetzner3 /etc/munin/plugins # sudo -u munin /usr/bin/munin-cron
root@hetzner3 /etc/munin/plugins #

well now I only see 1 chart on the munin wui = "MySQL InnoDB free tablespace" :( And it's still empty too
let's wait and see if the others come back?
actually, it looks like there's a better way to do this https://www.thesysadmin.rocks/2020/06/24/installing-munin-on-ubuntu-20-04-with-mysql-plugin/

munin-node-configure  --shell | grep mysql

the above command should output a bunch of `ln` commands with exactly what we need to copy & paste & create the mysql symlinks
note this might just be all the "subcharts" for 'mysql_', but it'll miss all the other 'mysql_*' files (eg 'mysql_queries' links to 'mysql_queries', not 'mysql_' like the rest)
to get those last ones, we'd want something like

ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins

after some hours, the whole mysql section in munin disappeared; let's try that again

root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql
lrwxrwxrwx 1 root root   31 Dec 13 20:09 binlog_groupcommit -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 bin_relay_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 commands -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 files_tables -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_bpool -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_bpool_act -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_insert_buf -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_io -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_io_pend -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_rows -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_semaphores -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 innodb_tnx -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 myisam_indexes -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 qcache -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 qcache_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 replication -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 select_types -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 slow -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 sorts -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 table_locks -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 13 20:09 tmp_tables -> /usr/share/munin/plugins/mysql_
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql | awk '{print $9}'
binlog_groupcommit
bin_relay_log
commands
connections
files_tables
innodb_bpool
innodb_bpool_act
innodb_insert_buf
innodb_io
innodb_io_pend
innodb_log
innodb_rows
innodb_semaphores
innodb_tnx
myisam_indexes
network_traffic
qcache
qcache_mem
replication
select_types
slow
sorts
table_locks
tmp_tables
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # for file in $(ls -lah | grep -i mysql | awk '{print $9}'); do rm -f $file; done
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql
lrwxrwxrwx 1 root root   31 Dec 14 01:18 mysql_ -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   36 Dec 14 01:18 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes
lrwxrwxrwx 1 root root   37 Dec 14 01:18 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb
lrwxrwxrwx 1 root root   38 Dec 14 01:18 mysql_queries -> /usr/share/munin/plugins/mysql_queries
lrwxrwxrwx 1 root root   42 Dec 14 01:18 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries
lrwxrwxrwx 1 root root   38 Dec 14 01:18 mysql_threads -> /usr/share/munin/plugins/mysql_threads
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # munin-node-configure  --shell 2>&1 | grep mysql
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # munin-node-configure  --suggest 2>&1 | grep mysql
mysql_                     | yes  | no [DBI connect('mysql;mysql_read_default_file=/etc/mysql/debian.cnf;mysql_connect_timeout=5','munin',...) failed: Access denied for user 'munin'@'localhost' (using password: NO)]
root@hetzner3 /etc/munin/plugins #

unfortunately `munin-node-configure` failed there at the end (it had no ouptut), so I'm just going to redo what I did before, but this time actually add the missing 'mysql_' prefix to the symlink name

root@hetzner3 /etc/munin/plugins # for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -sf /usr/share/munin/plugins/mysql_ mysql_$i; done
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql
lrwxrwxrwx 1 root root   31 Dec 14 01:18 mysql_ -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_binlog_groupcommit -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_bin_relay_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   36 Dec 14 01:18 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_commands -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_files_tables -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   37 Dec 14 01:18 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_bpool -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_bpool_act -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_insert_buf -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_io -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_io_pend -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_rows -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_semaphores -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_innodb_tnx -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_myisam_indexes -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_qcache -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_qcache_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   38 Dec 14 01:18 mysql_queries -> /usr/share/munin/plugins/mysql_queries
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_replication -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_select_types -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_slow -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   42 Dec 14 01:18 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_sorts -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_table_locks -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   38 Dec 14 01:18 mysql_threads -> /usr/share/munin/plugins/mysql_threads
lrwxrwxrwx 1 root root   31 Dec 14 01:27 mysql_tmp_tables -> /usr/share/munin/plugins/mysql_
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # service munin-node restart
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # sudo -u munin /usr/bin/munin-cron
root@hetzner3 /etc/munin/plugins #

well now the UI has a "mysql" section again, but there's only 5 charts.
As before, 3 out of 5 of the charts have data (but now with a big chunk of the last several hours of data missing)
for some reason all our new charts are missing too
I guess I'll wait until tomorrow. If they show up, then I'll want to document how I created these extra charts on 'hetzner3' – or just add that to ansible. If they don't show-up, then forget about it, these 5 charts are probably good enough.
I confirmed that the two varnish uptime charts are now visible and have data; that's fixed
I also checked the "process info" charts, which now has data for the "apache2" charts, but the "mysqld" ones are empty. I'm pretty sure this is because the process name is "mariadb" (not "mysqld")
I filled this out to more processes, so it now reads

[proc]
env.procname apache2|mariadbd|nginx|varnishd|varnishlog|varnishncsa|wazuh-db|wazuh-remoted|wazuh-syscheckd|wazuh-analysisd|wazuh-authd|python3|journalctl|systemd|kworker|unattended-upgrade

I'll have to check in on this tomorrow to see if the charts have updated
...
I wanted to check-in on the glacier inventory job that I kicked-off yesterday, but – as with the one I kicked off a few months ago – it shows no record of it!

user@disp4042:~$ aws glacier get-job-output --account-id OBFUSCATED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS
user@disp4042:~$ aws glacier list-jobs --account-id OBFUSCATED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": []
}
user@disp4042:~$

oh wait, that was the job ID from a few months ago. But I get the same result from the one from yesterday

aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN" ./output.json

jesus christ, SO says that the jobs may expire after 24 hours https://stackoverflow.com/questions/45112105/aws-glacier-job-id-was-not-found
so the jobs can take many days to run, and they expire within 24 hours. They really do make this as hard as fucking possible to delete vaults! It really should be criminal how hard they make it. This is madness!
oh, one other comment in ^ that thread worth revisting:

@OrestGulman Are you sure that you have data stored in the Amazon Glacier service? Please note that this is different to storing data in Amazon S3 with a 'Glacier' class

we actually are trying to delete data in Amazon S3 in the glacier class. I think? Anyway I found the vault in the "Glacier S3" Service of the Console WUI. Could that be adding a complication?
anyway, for now, I just kicked-off another job

user@disp4042:~$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"location": "/REDACTED/vaults/deleteMeIn2020/jobs/Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL",
"jobId": "Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL"
}
user@disp4042:~$ 

user@disp4042:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": [
	{
		"JobId": "Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL",
		"Action": "InventoryRetrieval",
		"VaultARN": "arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020",
		"CreationDate": "2024-12-14T01:59:59.138Z",
		"Completed": false,
		"StatusCode": "InProgress",
		"InventoryRetrievalParameters": {
			"Format": "JSON"
		}
	}
]
}
user@disp4042:~$

Thr Dec 12, 2024

I wanted to follow-up with the glacier deletion task
A couple months ago (on Oct 4), I documented that Marcin gave me auth to delete the 285.3 GB 'deleteMeIn2020' bucket from our amazon glacier account, which has been costing us $1.03/mo https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Fri_Oct_04.2C_2024
It's, like, unbelievably complicated to delete this bucket
First, before I can delete it, Amazon forces you to generate a recent inventory
An inventory can only be created via the API, and it takes time for the job to finish
I last checked on the inventory (that I initiated on Oct 4) on Oct 6, and it wasn't yet ready https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Sun_Oct_06.2C_2024
ok, here's our commands to get the inventory job's result

sudo apt-get -y install awscli

  aws configure set aws_access_key_id 'REDACTED'
  aws configure set aws_secret_access_key 'REDACTED'

aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

unfortunately, it says the keys that I used (which I extracted from hetzner2:/root/backups/glacierTest.py) don't have permission to see the job's output –- even though those are the keys that I used to initiate the job!

user@disp5233:~$ sudo apt-get -y install awscli
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
awscli is already the newest version (2.9.19-1).
The following packages were automatically installed and are no longer required:
  chromium-common chromium-sandbox libc++1-16 libc++abi1-16
  libcommons-compress-java librnp0 libunwind-16 libwpe-1.0-1
  libwpebackend-fdo-1.0-1 linux-image-6.1.0-10-amd64
  linux-image-6.1.0-11-amd64 linux-image-6.1.0-13-amd64
  linux-image-6.1.0-17-amd64 linux-image-6.1.0-18-amd64
  linux-image-6.1.0-20-amd64 linux-image-6.1.0-21-amd64
  linux-image-6.1.0-22-amd64 linux-image-6.1.0-23-amd64
  linux-image-6.1.0-25-amd64 linux-image-6.1.0-26-amd64
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
user@disp5233:~$  

user@disp5233:~$   aws configure set aws_access_key_id 'REDACTED'
user@disp5233:~$   aws configure set aws_secret_access_key 'REDACTED'
user@disp5233:~$ 

user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

An error occurred (AccessDeniedException) when calling the GetJobOutput operation: User: arn:aws:iam::REDACTED:user/backup-cron is not authorized to perform: glacier:GetJobOutput on resource: arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020
user@disp5233:~$

I also found some aws cred file in the root user's home, but it had the exact same creds as the glacierTest.py file :(

[root@opensourceecology ~]# cat /root/.aws/credentials 
[default]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED
[root@opensourceecology ~]#

I logged into the aws console as my personal 'maltfield' user (using creds in my personal keepass)
I clicked on my username in the top-right -> "Security Credentials"
the WUI showed that I have one set of Access Keys that were created "2449 days ago"
I checked the notes of my keepass and found a set of keys with the same key id
I reconfigured my aws cli on my local dispVM to use these creds, but I got the same error (though this time for user = 'maltfield')

user@disp5233:~$   aws configure set aws_access_key_id 'REDACTED'
user@disp5233:~$   aws configure set aws_secret_access_key 'REDACTED'
user@disp5233:~$ 

user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

An error occurred (AccessDeniedException) when calling the GetJobOutput operation: User: arn:aws:iam::REDACTED:user/maltfield is not authorized to perform: glacier:GetJobOutput on resource: arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020
user@disp5233:~$

oh, I realized that I was executing the above command literally with 'REDACTED' as the '--account-id' – when I swapped that with the actual numerical account ID, I got the same results for the 'maltfield' user
...but when I reverted back to the original creds (for the 'backup-cron' user), I got a different error saying that the job ID was not found. Is it possible that there's a very narrow window where the job needs to be queried? This is impossible!

user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS
user@disp5233:~$

in the WUI, I switched to the s3 Glacier service https://us-west-2.console.aws.amazon.com/glacier/home?region=us-west-2#/vaults
I clicked-on the 'deleteMeIn2020' bucket, and it still says the last inventory date is "August 1, 2018, 02:41:31 (UTC-05:00)". wtf?
well a query for all jobs shows no jobs

user@disp5233:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
	"JobList": []
}
user@disp5233:~$

well, I initiated a new inventory job AGAIN

user@disp5233:~$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
	"location": "/REDACTED/vaults/deleteMeIn2020/jobs/tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN",
	"jobId": "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN"
}
user@disp5233:~$ 

user@disp5233:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
	"JobList": [
		{
			"JobId": "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN",
			"Action": "InventoryRetrieval",
			"VaultARN": "arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020",
			"CreationDate": "2024-12-12T18:07:24.156Z",
			"Completed": false,
			"StatusCode": "InProgress",
			"InventoryRetrievalParameters": {
				"Format": "JSON"
			}
		}
	]
}
user@disp5233:~$ 

user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN" ./output.json

An error occurred (InvalidParameterValueException) when calling the GetJobOutput operation: The job is not currently available for download: tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN
user@disp5233:~$

I guess now we wait some days and hopefully it doesn't disappear again before we have a chance to check up on it?
...
returning to store.opensourceecology.org, the theme issues are now resolved
the site is still pretty broken, but it was never really working anyway, so that's as good as it'll get
what does remain, however, is finding a replacement for our now-unavailable security plugins ' rename-wp-login' and 'force-strong-passwords'
currently the login is exposed on the normal wp-login.php, which isn't good https://store.opensourceecology.org/wp-login.php
I recently setup a wordpress website which uses two plugins
1. melapress-login-security https://wordpress.org/plugins/melapress-login-security/
2. wp-2fa https://wordpress.org/plugins/wp-2fa/
it looks like I did install wps-hide-login, but I ended up deactivating it because 'melapress-login-security' includes this (as well as forcing strong passwords, feeding two birds with one scone)
one thing I didn't like so much about the 'wp-2fa' plugin was that it doesn't have a "relaxed" mode.
Currently we're using a 'google-authenticator' plugin https://wordpress.org/plugins/google-authenticator/
1. this plugin hasn't been updated in 2 years, but it has >30,000 active installations, and it appears to be working fine still
2. what I really like about it is that it has a checkbox for "Relaxed mode allows for more time drifting on your phone clock (±4 min)."
  1. this setting is found on the Users -> Profile
  2. in my experience, 2FA tends to cause security issues with websites due to loss of availability because users constantly get locked out due to time sync issues on their phone. For our purposes, I think the happy-medium between security-and-convenience is found where several codes in the past (and future) few minutes are ok. I think there's very little risk in this, but it's surprising how few TOTP implementations allow this
  3. oh, actually, it's been years since they've done a release, but the last commit was only 11 months ago, so that's not terrible https://github.com/ivankruchkoff/google-authenticator
  4. I'm not a huge fan of this plugin otherwise; it's not actively updated. But if I could find an alternative that's equally as lightweight
I figured I'd go ahead now and open a feature request to add a "relaxed" mode to the alternative 'wp-2fa' that I've already tested, but I couldn't fucking find the forge!
1. I opened a support ticket on their wordpress plugin page asking for a link to their vcs forge (eg github) https://wordpress.org/support/topic/where-is-the-vcs-forge/
anyway, I'll go ahead and 3TOFU all of them, but now I'm leaning towards not using melapress' 'wp-2fa', but yes using melapress' 'melapress-login-security'
in poking around melapress, I saw they also have a plugin for hCaptcha = 'advanced-nocaptcha-recaptcha' https://wordpress.org/plugins/advanced-nocaptcha-recaptcha/
1. I hate reCAPTCHA and cloudflare's often fails, but I've had pretty good experiences with hCaptcha. This supports all of them
2. previously I've used 'hcaptcha-for-forms-and-more' and had a good experience, so I'll add both to the 3TOFU https://wordpress.org/plugins/hcaptcha-for-forms-and-more/
3. in the site where I used 'hcaptcha-for-forms-and-more' before, they used 'wpforms-lite', so I'll add that too https://wordpress.org/plugins/wpforms-lite/
  1. looks like OSE is using 'contact-form-7', which is actively developed still https://wordpress.org/plugins/contact-form-7/
  2. despite the name, I found that 'wpforms-lite' actually is *less* lightweight than 'contact-form-7' in benchmark tests
    1. https://wphive.com/plugins/wpforms-lite/
    2. https://wphive.com/plugins/contact-form-7/
  3. contact-form-7 is a bit more popular (10+ million active installs vs 6+ million), but 'wpforms-lite' has a bit better reviews (4.9/5 vs 4/5)
  4. oh, it looks like contact-form-7 only supports reCAPTCHA whereas wpforms-lite has built-in support for hCaptcha and cf too
  5. I'll go ahead and add them both to the 3TOFU just incase we need it, but it's probably best to stick to contact-form-7
let's to a new 3TOFU of some candidate plugins, after which I can demo them and see what we want to use. I'm also going to add some others that we may or may not use
1. wps-hide-login
2. melapress-login-security
3. activitypub
4. aurora-heatmap
5. raw-html
6. related-posts-by-taxonomy
7. smart-slider-3
8. spam-destroyer
9. coinpayments-payment-gateway-for-woocommerce
10. woocommerce-gateway-stripe
11. wpfront-notification-bar
12. wordpress-seo
13. wp-pgp-encrypted-emails
14. woo-multi-currency
15. woocommerce-multilingual
16. include-mastodon-feed
17. bulk-media-register
18. enable-media-replace
19. regenerate-thumbnails
20. wp-qrcode
21. wp-pgp-encrypted-emails
22. woo-multi-currency
23. woocommerce-multilingual
24. include-mastodon-feed
25. wp-2fa
26. advanced-nocaptcha-recaptcha
27. hcaptcha-for-forms-and-more
28. leaflet-map
29. extensions-leaflet-map
30. wpforms-lite
in the past few months, I recently had to do an upgrade of my personal wordpress site
to do this securely, I wrote a script that spits-out a script that can be used for 3TOFU of all the themes & plugins installed

user@host:~$ cat /usr/local/bin/wordpress_3tofu.sh 
#!/bin/bash
#set -x
################################################################################
# File:    wordpress_3tofu.sh
# Version: 0.1
# Purpose: Generates a list of 3TOFU commands to verfiy the latest versions of 
#          all currently-installed wordpress themes and plugins
#           * https://tech.michaelaltfied.net/3tofu
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-28
# Updated: 2024-09-28
################################################################################

################################################################################
#                                  SETTINGS                                    #
################################################################################

################################################################################
#                                  FUNCTIONS                                   #
################################################################################

################################################################################
#                                  MAIN BODY                                   #
################################################################################

#####################
# DECLARE VARIABLES #
#####################

# space-delimited list of URLs for 3TOFU
REMOTE_FILES=''

CURL=$(which curl) || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

########
# CORE #
########

###########
# PLUGINS #
###########

# get list of plugins
plugins=$(sudo -u wp -i wp --path=/var/www/html/wordpress/htdocs --format=csv plugin list | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")

##########
# THEMES #
##########

# get list of themes
themes=$(sudo -u wp -i wp --path=/var/www/html/wordpress/htdocs --format=csv theme list | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")

###################
# OUTPUT COMMANDS #
###################

# HEADER

cat <<EOF

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: $(date -u --rfc-3339=seconds)
################################################################################

EOF

cat <<'EOF'
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

EOF

# CORE

cat <<'EOF'
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

EOF

# PLUGINS

echo "plugins='${plugins}'"

cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

EOF


# THEMES

echo "themes='${themes}'"

cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

EOF

# WARNINGS

cat <<'EOF'
echo -e "${WARNINGS}"
echo

EOF

# DOWNLOAD PAYLOADS

cat <<'EOF'
# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

EOF

# FINISH

cat <<'EOF'
# checksum
date -u +"%Y-%m-%d"
sha256sum *

EOF

exit 0
user@host:~$

this script works fine on my personal server, which uses a single wp multisite install, but it needs to be updated for OSE's multiple independent wordpress installs

root@hetzner3 /usr/local/bin # sudo -u wp -i wp --path=/var/www/html/wordpress/htdocs --format=csv plugin list | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " "
-bash: -vE: command not found
Error: This does not seem to be a WordPress installation.
The used path is: /var/www/html/wordpress/htdocs/
Pass --path=`path/to/wordpress` or run `wp core download`.
root@hetzner3 /usr/local/bin #

ignoring the spam of errors, this does work

root@hetzner3 /usr/local/bin # sudo -u wp -i wp --path="/var/www/html/store.opensourceecology.org/htdocs" --format=csv plugin list | grep -vE '^name,' | cut -d, -f1 | tr "\n" " "
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
PHP Warning:  wp_update_plugins(): An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/forums/">support forums</a>. (WordPress could not establish a secure connection to WordPress.org. Please contact your server administrator.) in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/functions.php on line 6085
Warning: wp_update_plugins(): An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/forums/">support forums</a>. (WordPress could not establish a secure connection to WordPress.org. Please contact your server administrator.) in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/functions.php on line 6085
akismet classic-editor contact-form-7 google-authenticator-encourage-user-activation google-authenticator hello meta-box ssl-insecure-content-fixer vcaching woocommerce coingate-for-woocommerce root@hetzner3 /usr/local/bin #

we can throw it in a loop that dynamically gets all of the sites

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do
	wp_docroot="$(dirname "${wordpress_site}")"
	echo $wp_docroot
done

here's an execution (currently we've restored only 1 wordpress vhost on hetzner3)

root@hetzner3 /usr/local/bin # wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do
		wp_docroot="$(dirname "${wordpress_site}")"
		echo $wp_docroot
done
/var/www/html/store.opensourceecology.org/htdocs
root@hetzner3 /usr/local/bin #

we can update this loop to get all the plugins for every site as follows

plugins=''
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do
	echo $wordpress_site;

	wp_docroot="$(dirname "${wordpress_site}")"
	vhost_dir="$(dirname "${wp_docroot}")"

	echo $wp_docroot
	echo $vhost_dir

	plugins="${plugins} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list 2>/dev/null | grep -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done

echo ${plugins}

here's an execution

root@hetzner3 /usr/local/bin # plugins=''
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do
		echo $wordpress_site;

		wp_docroot="$(dirname "${wordpress_site}")"
		vhost_dir="$(dirname "${wp_docroot}")"

		echo $wp_docroot
		echo $vhost_dir

		plugins="${plugins} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list 2>/dev/null | grep -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done

echo ${plugins}
/var/www/html/store.opensourceecology.org/htdocs/wp-content
/var/www/html/store.opensourceecology.org/htdocs
/var/www/html/store.opensourceecology.org
akismet classic-editor contact-form-7 google-authenticator-encourage-user-activation google-authenticator hello meta-box ssl-insecure-content-fixer vcaching woocommerce coingate-for-woocommerce
root@hetzner3 /usr/local/bin #

ok, here's the updated script

maltfield@hetzner3:~$ cat /usr/local/bin/wordpress_3tofu.sh 
#!/bin/bash
#set -x
################################################################################
# File:    wordpress_3tofu.sh
# Version: 0.2
# Purpose: Generates a list of 3TOFU commands to verfiy the latest versions of 
#          all currently-installed wordpress themes and plugins
#           * https://tech.michaelaltfied.net/3tofu
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-28
# Updated: 2024-12-12
################################################################################

################################################################################
#                                  SETTINGS                                    #
################################################################################

################################################################################
#                                  FUNCTIONS                                   #
################################################################################

################################################################################
#                                  MAIN BODY                                   #
################################################################################

#####################
# DECLARE VARIABLES #
#####################

# space-delimited list of URLs for 3TOFU
REMOTE_FILES=''

CURL=$(which curl) || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

########
# CORE #
########

###########
# PLUGINS #
###########

# get list of plugins
plugins=''
for wordpress_site in $wordpress_sites; do
	wp_docroot="$(dirname "${wordpress_site}")"

	plugins="${plugins} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list 2>/dev/null | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done

##########
# THEMES #
##########

# get list of themes
themes=''
for wordpress_site in $wordpress_sites; do
	wp_docroot="$(dirname "${wordpress_site}")"

	themes="${themes} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv theme list 2>/dev/null | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done

###################
# OUTPUT COMMANDS #
###################

# HEADER

cat <<EOF

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: $(date -u --rfc-3339=seconds)
################################################################################

EOF

cat <<'EOF'
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

EOF

# CORE

cat <<'EOF'
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

EOF

# PLUGINS

echo "plugins='${plugins}'"

cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

EOF


# THEMES

echo "themes='${themes}'"

cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

EOF

# WARNINGS

cat <<'EOF'
echo -e "${WARNINGS}"
echo

EOF

# DOWNLOAD PAYLOADS

cat <<'EOF'
# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

EOF

# FINISH

cat <<'EOF'
# checksum
date -u +"%Y-%m-%d"
sha256sum *

EOF

exit 0
maltfield@hetzner3:~$

and an execution shows it's working (again, this script outputs another script that we'll copy-and-paste onto some VM for 3TOFU

maltfield@hetzner3:~$ sudo /usr/local/bin/wordpress_3tofu.sh 

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-12-12 21:14:30+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins=' akismet classic-editor contact-form-7 google-authenticator-encourage-user-activation google-authenticator hello meta-box ssl-insecure-content-fixer vcaching woocommerce coingate-for-woocommerce '
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

themes=' oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve '
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
sha256sum *

maltfield@hetzner3:~$

ok, so that's great in the future when we need to 3TOFU updates to plugins & themes that are found already-installed on the server, but let's hack it with a manually-defined list of plugins (above) for 3TOFU of these new plugins

#!/bin/bash
#set -x
################################################################################
# File:    wordpress_3tofu.sh
# Version: 0.1
# Purpose: Generates a list of 3TOFU commands to verfiy the latest versions of 
#          all currently-installed wordpress themes and plugins
#           * https://tech.michaelaltfied.net/3tofu
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-28
# Updated: 2024-09-28
################################################################################

################################################################################
#                                  SETTINGS                                    #
################################################################################

################################################################################
#                                  FUNCTIONS                                   #
################################################################################

################################################################################
#                                  MAIN BODY                                   #
################################################################################

#####################
# DECLARE VARIABLES #
#####################

# space-delimited list of URLs for 3TOFU
REMOTE_FILES=''

CURL=$(which curl) || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

########
# CORE #
########

###########
# PLUGINS #
###########

# get list of plugins
plugins="wps-hide-login melapress-login-security activitypub aurora-heatmap raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"

###################
# OUTPUT COMMANDS #
###################

# HEADER

cat <<EOF

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: $(date -u --rfc-3339=seconds)
################################################################################

EOF

cat <<'EOF'
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

EOF

# CORE

cat <<'EOF'
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

EOF

# PLUGINS

echo "plugins='${plugins}'"

cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

EOF

# WARNINGS

cat <<'EOF'
echo -e "${WARNINGS}"
echo

EOF

# DOWNLOAD PAYLOADS

cat <<'EOF'
# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

EOF

# FINISH

cat <<'EOF'
# checksum
date -u +"%Y-%m-%d"
sha256sum *

EOF

exit 0

I created ^ this script on some DispVM and executed it; it spat-out our 3TOFU script

user@disp1594:~$ vim wordpress_3tofu.sh
user@disp1594:~$ 
user@disp1594:~$ chmod +x wordpress_3tofu.sh 
user@disp1594:~$ ./wordpress_3tofu.sh 

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-12-12 20:45:59+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='wps-hide-login melapress-login-security activitypub aurora-heatmap raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
sha256sum *

user@disp1594:~$

I copied and pasted ^ that script into a whonix dispVM
here's our TOFU 1/3 (Tor, exit in Poland)

Congratulations. This browser is configured to use Tor.
2024-12-12
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . . . . . . . . jq: error (at <stdin>:0): Cannot index array with string "1.0.17"
. . . . . . . . . . . . . . . . . . . . . 


WARNING: Failed to download plugin woo-multi-currency
	null
	null

WARNING: Failed to download plugin woo-multi-currency
	null
	null

https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.4.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.25.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
2024-12-12
8b1f9a708838b8710b4198da1116689197e0a6134e0a1a5e786500576383034f  activitypub.4.4.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
1a53bdcd1ddb160d5807dc17a0f9e474402e22c899b3a9af486c9d5f0d2c4b36  extensions-leaflet-map.4.4.zip
27f1ab1e3f5274335d48d0cadaabdef98284880b0324771890d36a1f562fb44a  hcaptcha-for-forms-and-more.4.8.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
923f38397284dceda1028a12c01e78bde22e0d0fecfdd8b95e52cfcc04e47342  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
ebd87841f73bb7946216ae4827a413dcc97fc5094cee2f8ddb6dea7eff356358  smart-slider-3.3.5.1.25.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
f9ce7a98840dd4bf490d955320a68ac553c767ba7f0eeae6e4f067be5a927ef3  wordpress-seo.24.0.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
user@host:/tmp/user/1000/tmp.THquHNCCMu$

well that looks good, except for the failure of the 'woo-multi-currency' plugin
1. looks like the URL for the "latest version" is missing

user@host:~$ json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/woo-multi-currency.json)
user@host:~$ echo $json

user@host:~$ ls
Desktop    Downloads  Pictures     Public     Videos
Documents  Music      plugin.json  Templates
user@host:~$ 

user@host:~$ cat plugin.json | jq -r .version
2.2.4
user@host:~$ 

user@host:~$ cat plugin.json | jq -r ".versions.2.2.4"
jq: error: Invalid numeric literal at EOF at line 1, column 6 (while parsing '.2.2.4') at <top-level>, line 1:
.versions.2.2.4         
jq: error: syntax error, unexpected LITERAL, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.versions.2.2.4         
jq: 2 compile errors
user@host:~$ 

user@host:~$ cat plugin.json | jq -r ".versions"
{
  "2.1.12": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.12.zip",
  "2.1.14": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.14.zip",
  "2.1.7": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.7.zip",
  "2.1.8": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.8.zip",
  "2.1.9": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.9.zip"
}
user@host:~$

whatever, this plugin wasn't so important. Let's just ignore & skip this plugin
by next week we should have all 3 TOFUs. If they match, I'll go ahead and copy them to hetzner3, figure out which ones we want to use, and then finish off the store.opensourceecology.org migration script CHG steps
...
I still have 1 remaining TODO item on the hetzner3 backups, which was to wait some weeks and then verify that the backups cron/lifecycle rules are working
well, it's been >11 weeks since I setup backups on hetzner3 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Sun_Sep_22.2C_2024
here's what we see in the bucket

root@hetzner3 ~ # sudo rclone lsl b2:ose-server-backups | grep hetzner3
2258493547 2024-12-10 07:29:09.905000000 daily_hetzner3_20241210_072828.tar.gpg
2266009707 2024-12-11 07:58:32.491000000 daily_hetzner3_20241211_075750.tar.gpg
2272696427 2024-12-12 07:25:19.985000000 daily_hetzner3_20241212_072443.tar.gpg
1782579309 2024-10-01 08:05:13.312000000 monthly_hetzner3_20241001_080447.tar.gpg
1986529389 2024-11-01 07:37:07.030000000 monthly_hetzner3_20241101_073631.tar.gpg
2195302509 2024-12-01 07:45:57.990000000 monthly_hetzner3_20241201_074518.tar.gpg
2104289388 2024-11-18 07:23:52.227000000 weekly_hetzner3_20241118_072314.tar.gpg
2153154668 2024-11-25 07:50:49.139000000 weekly_hetzner3_20241125_075009.tar.gpg
2202644588 2024-12-02 08:05:51.395000000 weekly_hetzner3_20241202_080510.tar.gpg
2251448428 2024-12-09 07:31:51.872000000 weekly_hetzner3_20241209_073110.tar.gpg
root@hetzner3 ~ #

so we've got 3 monthlies already. that's great
we have the past 3 daily backups and nothing more. so that suggests that recent backups are working fine, and the lifecycle rules are too; great
and we have 4 weeklies, which sounds correct; all looks great!
...
another item I had pending on my TODO list was to verify munin after some time to ensure that the charts were getting populated with data
obviously this server is basically idle, but any lines at all are fine
apache looks good
1. apache logs were actually empy on hetzner2 for some reason, and there was only two charts: apache processes and apache volume (both empty)
2. on hetzner3, we have 3 charts: apache access (accesses per second, mostly zero with a couple spikes ), apache processes (it shows a solid 48-49 idle servers and a solid 100 "free slots" over the past few months), and apache volume (mostly zero with occasional spikes to ~2k-18k bytes per second)
we have some data for nginx, but not as many charts as we used to have
1. on hetzner2, it was called "webserver" whereas on hetzner3 it's called "nginx" for some reason
2. on hetzner2, we had 4 charts: nginx requests, nginx requests, nginx status, ngingx status
3. oh wait, no, that's the two charts duplicated; it's the same data.
4. on hetzner3 we also have these 2 chats. They're near zero, but both have spikes up to ~0.2-1.2 requests per second and 1-9 connections per second
our varnish charts on hetzner3 are also working
1. we have 9 charts on hetzner2 and 8 charts on hetzner3
2. looks like we're missing the "uptime" chart on hetzner3. I've read this can be useful just to monitor if varnish has some issue that causes it to restart its child processes, so it would probably be good to add that one, if trivial
3. all the other charts have data except the "misbehavoiur" chart is all zeros. I'm guessing that's possibly because there are no misbehaviours. Maybe.
4. curiously, this section on hetzner2 used to be called 'varnish4' whereas on hetzner3 it's called "webserver"
I did notice that all of the mysql charts are empty on hetzner3, so we should probably investigate that too
the "process info" charts are also all empty, so we should check on that
well, I confirmed that the varnish_uptime plugin is in-place

root@hetzner3 /etc/munin/plugins # ls -lah | grep -i varnish
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_backend_traffic -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_bad -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_expunge -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_hit_rate -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_memory_usage -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_objects -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_request_rate -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_threads -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_transfer_rate -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_uptime -> /usr/share/munin/plugins/varnish5_
root@hetzner3 /etc/munin/plugins #

for comparison, here's what we have on hetzner2

[root@opensourceecology plugins]# ls -lah | grep -i varnish
-rwxr-xr-x 1 root root  26K Mar  3  2018 varnish4_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_backend_traffic -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_backend_traffic.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_bad -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_bad.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_expunge -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_expunge.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_hit_rate -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_hit_rate.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_memory_usage -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_memory_usage.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_objects -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_objects.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_request_rate -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_request_rate.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_threads -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_threads.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_transfer_rates -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_transfer_rates.bak -> /usr/share/munin/plugins/varnish_
lrwxrwxrwx 1 root root    9 Mar  3  2018 varnish_uptime -> varnish4_
lrwxrwxrwx 1 root root   33 Mar  3  2018 varnish_uptime.bak -> /usr/share/munin/plugins/varnish_
[root@opensourceecology plugins]#

oh, it looks like the new version that we have on hetzner3 has separated "varnish_uptime" into two distinct uptimes. my guess is that one is the parent and one is the child process
1. hetzner2

[root@opensourceecology plugins]# grep -A3 'uptime' varnish_uptime
		'uptime' => {
				'title' => 'Varnish uptime',
				'vlabel' => 'days',
				'scale' => 'no',
				'values' => {
						'uptime' => {
								'type' => 'GAUGE',
								'cdef' => 'uptime,86400,/'
						}
				}
		},
[root@opensourceecology plugins]#

1. hetzner3

root@hetzner3 /etc/munin/plugins # grep -A3 uptime varnish_uptime 
		'main_uptime' => {
				'type' => 'MAIN',
				'title' => 'Varnish Child uptime',
				'vlabel' => 'days',
				'scale' => 'no',
				'values' => {
						'uptime' => {
								'type' => 'GAUGE',
								'cdef' => 'uptime,86400,/'
						},
				}
		},
		'mgt_uptime' => {
				'type' => 'MGT',
				'title' => 'Varnish Management uptime',
				'vlabel' => 'days',
				'scale' => 'no',
				'values' => {
						'uptime' => {
								'type' => 'GAUGE',
								'cdef' => 'uptime,86400,/'
						},
				}
		},
You have new mail in /var/mail/root
root@hetzner3 /etc/munin/plugins #

I updated the symlinks to include these two charts on hetzner3's muni plugins dir

root@hetzner3 /etc/munin/plugins # ls -lah varnish_uptime 
lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_uptime -> /usr/share/munin/plugins/varnish5_
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # rm -f varnish_uptime
root@hetzner3 /etc/munin/plugins # ln -s /usr/share/munin/plugins/varnish5_ varnish_main_uptime
root@hetzner3 /etc/munin/plugins # ln -s /usr/share/munin/plugins/varnish5_ varnish_mgt_uptime
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # ls -lah | grep -i varnish_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_backend_traffic -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_bad -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_expunge -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_hit_rate -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Dec 13 00:03 varnish_main_uptime -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_memory_usage -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Dec 13 00:03 varnish_mgt_uptime -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_objects -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_request_rate -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_threads -> /usr/share/munin/plugins/varnish5_
lrwxrwxrwx 1 root root   34 Sep 25 01:47 varnish_transfer_rate -> /usr/share/munin/plugins/varnish5_
root@hetzner3 /etc/munin/plugins # 

root@hetzner3 /etc/munin/plugins # service munin-node restart
root@hetzner3 /etc/munin/plugins #

a manual test run looks good

root@hetzner3 /etc/munin/plugins # munin-run varnish_main_uptime
uptime.value 6635471
root@hetzner3 /etc/munin/plugins # munin-run varnish_mgt_uptime
uptime.value 6635477
root@hetzner3 /etc/munin/plugins #

I refreshed the munin WUI and, yep, I now see two additional charts. One's named "Varnish Child Uptime" and the other "Varnish Management Uptime". Perfect.
varnish was the lowest hanging fruit, but the most important missing chart is mysql. Well, all our mysql charts are missing. what gives?
first, our plugins dirs look very different on both servers
1. hetzner2

[root@opensourceecology plugins]# ls -lah | grep -i mysql
lrwxrwxrwx 1 root root   31 Oct  8  2019 bin_relay_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 commands -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 files_tables -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_bpool -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_bpool_act -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_insert_buf -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_io -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_io_pend -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_rows -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_semaphores -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 innodb_tnx -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 myisam_indexes -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 mysql_ -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   38 Oct  8  2019 mysql_queries -> /usr/share/munin/plugins/mysql_queries
lrwxrwxrwx 1 root root   31 Oct  8  2019 network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   28 Oct  8  2019 ps_mysqld -> /usr/share/munin/plugins/ps_
lrwxrwxrwx 1 root root   31 Oct  8  2019 qcache -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 qcache_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 replication -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 select_types -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 slow -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 sorts -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 table_locks -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   31 Oct  8  2019 tmp_tables -> /usr/share/munin/plugins/mysql_
[root@opensourceecology plugins]#

1. hetzner3

root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql
lrwxrwxrwx 1 root root   31 Sep 25 01:47 mysql_ -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root   36 Sep 25 01:47 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes
lrwxrwxrwx 1 root root   37 Sep 25 01:47 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb
lrwxrwxrwx 1 root root   42 Sep 25 01:47 mysql_isam_space_ -> /usr/share/munin/plugins/mysql_isam_space_
lrwxrwxrwx 1 root root   38 Sep 25 01:47 mysql_queries -> /usr/share/munin/plugins/mysql_queries
lrwxrwxrwx 1 root root   42 Sep 25 01:47 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries
lrwxrwxrwx 1 root root   38 Sep 25 01:47 mysql_threads -> /usr/share/munin/plugins/mysql_threads
root@hetzner3 /etc/munin/plugins #

there's a lot of different files here, but let's just test the "mysql_queries" one
1. hetzner2

[root@opensourceecology plugins]# munin-run mysql_queries
delete.value 17433
insert.value 20779
replace.value 206033
select.value 94566392
update.value 39021
cache_hits.value 0
[root@opensourceecology plugins]#

1. hetzner3

root@hetzner3 /etc/munin/plugins # munin-run mysql_queries
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'munin'@'localhost' (using password: NO)'
You have new mail in /var/mail/root
root@hetzner3 /etc/munin/plugins #

ok, it's an auth error
if I check the munin configs, then it's pretty obvious. I have a password defined for hetzner2 and nothing for hetzner3. I imagine a password might not be necessary if we allow passwordless auth from localhost or something, but I guess I never set that up?
1. hetzner2

[root@opensourceecology plugins]# cat ../plugin-conf.d/zzz-ose 
# ose-specific configs go here per this doc
#  * http://guide.munin-monitoring.org/en/latest/plugin/use.html#configuring

[nginx_wiki.opensourceecology.org_*]
env.url https://wiki.opensourceecology.org/nginx_status
env.graph_title graph title
env.graph_info graph info goes here

[nginx_www.opensourceecology.org_*]
env.url https://www.opensourceecology.org/nginx_status

[mysql*]
user root
group wheel
env.mysqlopts -u munin_user -pREDACTED

[multips_memory]
env.names varnishd mysqld httpd varnishlog systemd-journal rsyslogd b2 nginx munin munin-node ssh sshd openvpn tuned ossec-analysisd bash vim screen tail gpg gpg2 polkitd tuned
[root@opensourceecology plugins]#

1. hetzner3

root@hetzner3 /etc/munin/plugins # cat ../plugin-conf.d/zzz-myconf 
# Ansible managed

################################################################################
# File:    zzz-myconf
# Version: 0.1
# Purpose: Munin custom config
#          we set custom munin configs in 'zzz-myconf' per this doc
#           * http://guide.munin-monitoring.org/en/latest/plugin/use.html#configuring
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-14
# Updated: 2024-09-14
################################################################################

[nginx_*]
env.url http://127.0.0.1/nginx_status

[apache_*]
env.url   http://127.0.0.1:%d/server-status?auto
env.ports 8000

[mysql*]
env.mysqlopts -u munin
env.mysqluser munin

[multips_memory]
env.names mysqld apache2 cache-main ossec-analysisd wazuh-db ossec-syscheckd munin-node munin-html munin-update nginx ssh sshd pickup journalctl trivial-rewrite systemd-journal varnishd wazuh-modulesd unattended-upgrades bash sudo tcpdump varnishlog ossec-remoted systemd-logind su dhclient ossec-authd ossec-execd ossec-logcollector ossec-maild ossec-monitord screen (sd-pam) systemd cleanup qmgr tlsmgr rewrite bounce defer trace verify flush proxymap proxywrite smtp relay showq error retry discard local virtual lmtp anvil scache submission grep tail rsyslogd systemd-udevd dbus-daemon gpg gpg2 cleanMega backup.sh rclone b2 chown chmod tar rm mv python agetty sh qemu-ga gpg-agent awstats.pl openvpn tuned vim polkitd

[varnish*]
user root
env.varnishstat /usr/bin/varnishstat

[proc]
env.procname mysqld apache2
root@hetzner3 /etc/munin/plugins #

checking my logs on the wiki, it looks like I first

Wed Dec 11, 2024

Catarina responded to my mail from yesterday affirming that I should reset the unused store.opensourceecology.org wordpress site to some free wordpress theme for now, and use the license that would otherwise be used for it for www.opensourceecology.org instead
I updated my /etc/hosts to point to the hetzner3 server for store.opensourceecology.org
I checked http://store.opensourceecology.org/ in my web browser. it still just returns the generic wordpress critical error message
I was able to login to the wp admin wui on the hetzner3 store site
I changed the theme from 'oshine' to 'twenty seventeen'
now when I load the store site's frontpage, the error has disappeared, but it still says "oshine" at the top and the body is lots of broken shorcodes
if I go to Settings -> Reading, then I can see that the "Homepage" is set to "Home v37"
I changed the store hoepage to "Sample Page"
that's the best page I found. It still has broken tatsu shortcodes in the footer, but at last the word "oshine" doesn't appear anywhere.
this site is temporary, and these actions are going to get overwritten at the time we actually rsync from live to hetzner3 during the migration, so I need a place to record each of these little changes as part of the migration playbook
I created a wiki CHG "ticket" for the migration of store.opensourceecology.org from hetzner2 to hetzner3 https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_store_to_hetzner3
I'm using wp-cli to more easily get the list of themes & plugins versions
1. hetzner2

[root@opensourceecology ~]# sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ plugin list
...
+------------------------------------------------+----------+--------+---------+
| name                                           | status   | update | version |
+------------------------------------------------+----------+--------+---------+
| akismet                                        | inactive | none   | 4.1.1   |
| be-gdpr                                        | active   | none   | 1.1.2   |
| be-portfolio-post                              | active   | none   | 1.1     |
| classic-editor                                 | inactive | none   | 1.4     |
| colorhub                                       | active   | none   | 1.0.5   |
| contact-form-7                                 | active   | none   | 5.1.1   |
| force-strong-passwords                         | active   | none   | 1.8.0   |
| google-authenticator                           | active   | none   | 0.48    |
| google-authenticator-encourage-user-activation | active   | none   | 0.2     |
| hello                                          | inactive | none   | 1.7.1   |
| masterslider                                   | active   | none   | 3.2.7   |
| meta-box                                       | active   | none   | 4.17.3  |
| meta-box-conditional-logic                     | active   | none   | 1.6.4   |
| meta-box-show-hide                             | active   | none   | 1.1.0   |
| meta-box-tabs                                  | active   | none   | 1.1.1   |
| oshine-core                                    | active   | none   | 1.3.7   |
| oshine-modules                                 | active   | none   | 2.2.9   |
| redux-vendor-support                           | active   | none   | 1.0.1   |
| rename-wp-login                                | active   | none   | 2.5.5   |
| revslider                                      | active   | none   | 5.4.8.3 |
| ssl-insecure-content-fixer                     | active   | none   | 2.7.2   |
| tatsu                                          | active   | none   | 2.9.3.3 |
| typehub                                        | active   | none   | 1.4.3   |
| vcaching                                       | active   | none   | 1.6.9   |
| woocommerce                                    | active   | none   | 3.5.7   |
| coingate-for-woocommerce                       | active   | none   | 1.2.2   |
+------------------------------------------------+----------+--------+---------+
[root@opensourceecology ~]#

hetzner3

+---------------+----------+--------+---------+---------------+---------------+
| name          | status   | update | version | update_versio | auto_update   |
|               |          |        |         | n             |               |
+---------------+----------+--------+---------+---------------+---------------+
| akismet       | inactive | none   | 5.3.3   |               | off           |
| classic-edito | inactive | none   | 1.6.5   |               | off           |
| r             |          |        |         |               |               |
| contact-form- | active   | none   | 5.9.8   |               | off           |
| 7             |          |        |         |               |               |
| google-authen | active   | none   | 0.2     |               | off           |
| ticator-encou |          |        |         |               |               |
| rage-user-act |          |        |         |               |               |
| ivation       |          |        |         |               |               |
| google-authen | active   | none   | 0.54    |               | off           |
| ticator       |          |        |         |               |               |
| hello         | inactive | none   | 1.7.1   |               | off           |
| meta-box      | active   | none   | 5.10.2  |               | off           |
| ssl-insecure- | active   | none   | 2.7.2   |               | off           |
| content-fixer |          |        |         |               |               |
| vcaching      | active   | none   | 1.8.3   |               | off           |
| woocommerce   | active   | none   | 9.3.3   |               | off           |
| coingate-for- | inactive | none   | 2.1.1   |               | off           |
| woocommerce   |          |        |         |               |               |
+---------------+----------+--------+---------+---------------+---------------+
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ plugin list
<pre>
# here's what I got so far
<pre>

update plugin 'akismet' from v4.1.1 to v5.3.3
uninstall plugin 'be-gdpr'
uninstall plugin 'be-portfolio-post'
update plugin 'classic-editor' from v1.4 to v1.6.5
uninstall plugin 'colorhub'
update plugin 'contact-form-7' from v5.1.1 to v5.9.8
uninstall plugin 'force-strong-passwords'
update plugin 'google-authenticator' from v0.48 to 0.54
uninstall plugin 'masterslider'
update plugin 'meta-box' from v4.17.3 to v5.10.2
uninstall plugin 'meta-box-conditional-logic'
uninstall plugin 'meta-box-show-hide'
uninstall plugin 'meta-box-tabs'
uninstall plugin 'oshine-core'
uninstall plugin 'oshine-modules'
uninstall plugin 'redux-vendor-support'
uninstall plugin 'rename-wp-login'
uninstall plugin 'revslider'
uninstall plugin 'tatsu'
uninstall plugin 'typehub'
update plugin 'vaching' from v1.6.9 to v1.8.3
update plugin 'woocommerce' from v3.5.7 to v9.3.3
update plugin 'coingate-for-woocommerce' from v1.2.2 to v2.1.1

but I think I'm going to have to add some other plugins. For example we need something to replace the now-defunct 'rename-wp-login' and 'forece-strong-passwords' plugins
I also got the theme info on hetzner2 & hetzner3 to diff and add to the ticket
1. hetzner2

[root@opensourceecology ~]# sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ theme list
...
+-----------------+----------+--------+---------+
| name            | status   | update | version |
+-----------------+----------+--------+---------+
| oshin           | active   | none   | 6.6.4.4 |
| storefront      | inactive | none   | 2.4.5   |
| twentyeleven    | inactive | none   | 3.2     |
| twentyfifteen   | inactive | none   | 2.4     |
| twentyfourteen  | inactive | none   | 2.6     |
| twentynineteen  | inactive | none   | 1.3     |
| twentyseventeen | inactive | none   | 2.1     |
| twentysixteen   | inactive | none   | 1.9     |
| twentyten       | inactive | none   | 2.8     |
| twentythirteen  | inactive | none   | 2.8     |
| twentytwelve    | inactive | none   | 2.9     |
+-----------------+----------+--------+---------+
[root@opensourceecology ~]#

1. hetzner3

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ theme list
...
+---------------+----------+--------+---------+---------------+---------------+
| name          | status   | update | version | update_versio | auto_update   |
|               |          |        |         | n             |               |
+---------------+----------+--------+---------+---------------+---------------+
| oshin         | inactive | none   | 7.2.1   |               | off           |
| storefront    | inactive | none   | 4.6.0   |               | off           |
| twentyeleven  | inactive | none   | 4.7     |               | off           |
| twentyfifteen | inactive | none   | 3.8     |               | off           |
| twentyfourtee | inactive | none   | 4.0     |               | off           |
| n             |          |        |         |               |               |
| twentyninetee | inactive | none   | 2.9     |               | off           |
| n             |          |        |         |               |               |
| twentysevente | active   | none   | 3.7     |               | off           |
| en            |          |        |         |               |               |
| twentysixteen | inactive | none   | 3.3     |               | off           |
| twentyten     | inactive | none   | 4.2     |               | off           |
| twentythirtee | inactive | none   | 4.2     |               | off           |
| n             |          |        |         |               |               |
| twentytwelve  | inactive | none   | 4.3     |               | off           |
+---------------+----------+--------+---------+---------------+---------------+
You have new mail in /var/mail/root
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

I then spent some time updating the CHG script
I got stuck a bit on the rsync
first I tried pulling on the hetzner3 from hetzner2, but it didn't like that we have to auth with a password as sudo on hetzner2

root@hetzner3 ~ # rsync -avvv --progress --rsync-path="sudo rsync" -e "ssh -p 32415" maltfield@138.201.84.223:32415${backupDir_hetzner2}/current/* ${backupDir_hetzner3}/current/
opening connection using: ssh -p 32415 -l maltfield 138.201.84.223 "sudo rsync" --server --sender -vvvlogDtpre.iLsfxCIvu . "32415/var/tmp/backups_for_migration_to_hetzner3/store.opensourceecology.org_20241212/current/*"  (12 args)
sudo: no tty present and no askpass program specified
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [Receiver=3.2.7]
[Receiver] _exit_cleanup(code=12, file=io.c, line=231): about to call exit(12)
root@hetzner3 ~ #

instead I decided to push on hetzner2 to hetzner3
first I updated the hetzner2:/etc/hosts file to hard-code the IP address for "hetzner3", so this will go smother in the future
ok, this worked

[maltfield@opensourceecology ~]$ rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
sending incremental file list
mysqldump_store.opensourceecology.org.20241211.sql.bz2
      1,406,610 100%  109.18MB/s    0:00:00 (xfr#1, to-chk=1/2)
store.opensourceecology.org_files.20241211.tar.gz
    184,205,865 100%   92.26MB/s    0:00:01 (xfr#2, to-chk=0/2)

sent 185,658,021 bytes  received 54 bytes  53,045,164.29 bytes/sec
total size is 185,612,475  speedup is 1.00
[maltfield@opensourceecology ~]$

Tue Dec 10, 2024

last week we got an email from the let's encrypt expiry bot saying that our cert is going to expire soon
we don't normally get these, so it stood-out

Hello,

Your certificate (or certificates) for the names listed below will expire in 19 days (on 2024-12-24). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors.

We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details.

awstats.openbuildinginstitute.org
awstats.opensourceecology.org
fef.opensourceecology.org
forum.opensourceecology.org
microfactory.opensourceecology.org
munin.opensourceecology.org
openbuildinginstitute.org
opensourceecology.org
oswh.opensourceecology.org
phplist.opensourceecology.org
seedhome.openbuildinginstitute.org
staging.opensourceecology.org
store.opensourceecology.org
wiki.opensourceecology.org
www.openbuildinginstitute.org
www.opensourceecology.org

For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message.

For any questions or support, please visit: https://community.letsencrypt.org/ Unfortunately, we can't provide support by email.

To learn more about the latest technical and organizational updates from Let's Encrypt, sign up for our newsletter: https://letsencrypt.org/opt-in/

I checked the website in my browser and confirmed that I see a cert that says it's going to expire soon

Not Before Wed, 25 Sep 2024 17:04:09 GMT
Not After Tue, 24 Dec 2024 17:04:08 GMT

It looks like here's the cron file for the cert renewal

[root@opensourceecology log]# cat /etc/cron.d/letsencrypt 
# once a month, update our letsencrypt cert
20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log
[root@opensourceecology log]#

the latest log entry isn't dated, but the last time it ran, it appears to have decided that Dec 24 was too far away

The following certificates are not due for renewal yet:
  /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem expires on 2024-12-24 (skipped)
  /etc/letsencrypt/live/opensourceecology.org/fullchain.pem expires on 2024-12-24 (skipped)
No renewals were attempted.

It looks like the last time cron mentions it being executed was in 2024-11-13, which makes sense

[root@opensourceecology log]# grep letsencrypt cron*
[root@opensourceecology log]# zgrep letsencrypt cron*
cron-20241113.gz:Nov 13 04:20:01 opensourceecology CROND[1103]: (root) CMD (/root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log)
[root@opensourceecology log]#

oh, checking the cron again, it is set to run only on the 13th of every month
I checked our new server, which has this (installed by debian)

root@hetzner3 ~ # cat /etc/cron.d/certbot 
# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
#
# Important Note!  This cronjob will NOT be executed if you are
# running systemd as your init system.  If you are running systemd,
# the cronjob.timer function takes precedence over this cronjob.  For
# more details, see the systemd.timer manpage, or use systemctl show
# certbot.timer.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew --no-random-sleep-on-renew
root@hetzner3 ~ #

wow, so certbot says that let's encrypt says that we should check twice per day. then once per month seems ridiculous
I went ahead and changed it to do once per day

[root@opensourceecology cron.d]# vim letsencrypt 
[root@opensourceecology cron.d]# 

[root@opensourceecology cron.d]# cat letsencrypt 
# once a month, update our letsencrypt cert
20 4 * * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log
[root@opensourceecology cron.d]#

and I gave it a manual run

[root@opensourceecology cron.d]# /root/bin/letsencrypt/renew.sh
...
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Renewing an existing certificate for fef.opensourceecology.org and 11 more domains
Performing the following challenges:
http-01 challenge for awstats.opensourceecology.org
http-01 challenge for fef.opensourceecology.org
http-01 challenge for forum.opensourceecology.org
http-01 challenge for microfactory.opensourceecology.org
http-01 challenge for munin.opensourceecology.org
http-01 challenge for opensourceecology.org
http-01 challenge for oswh.opensourceecology.org
http-01 challenge for phplist.opensourceecology.org
http-01 challenge for staging.opensourceecology.org
http-01 challenge for store.opensourceecology.org
http-01 challenge for wiki.opensourceecology.org
http-01 challenge for www.opensourceecology.org
Using the webroot path /var/www/html/staging.opensourceecology.org/htdocs for all unmatched domains.
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/opensourceecology.org/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (success)
  /etc/letsencrypt/live/opensourceecology.org/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Redirecting to /bin/systemctl reload nginx.service
[root@opensourceecology cron.d]#

I confirmed that the cert has been updated

user@disp2766:~$ echo -n | openssl s_client -showcerts -connect opensourceecology.org:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > mycert.pem
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R10
verify return:1
depth=0 CN = fef.opensourceecology.org
verify return:1
DONE
user@disp2766:~$ openssl x509 -text -in mycert.pem | less
user@disp2766:~$ openssl x509 -text -in mycert.pem | grep 'Not '
			Not Before: Dec 10 23:19:37 2024 GMT
			Not After : Mar 10 23:19:36 2025 GMT
user@disp2766:~$

so it looks like nothing was broken here, we just hadn't previously been getting these alerts – and this is the first time we've gotten this alert since I've setup the google group to forward admin emails to members of the google group (incuding marcin and I)
I wrote Marcin to let him know the issue is not an issue, but I've updated the cron anyway
...
returning to the work of the hetzner3 project on my last work day (2024-10-07), I checked up on hetzner2's use of the oshine plugin, which I thought was in-use in the following sites:
1. obi
2. osemain
3. microfactory
4. store
I confirmed that we are using the 'oshine' theme on obi, microfactory, and store, but osemain is using Enegmatic
iirc, Catarina asked me to switch osemain to oshine.
most importantly, we only have two licenses for oshine.
1. store was never setup
2. microfactory's last blog post was for an event in Feb 2019
3. obi's last workshop listing is from 2014, so that's even 10 years old
I sent an email to Catarina and Marcin asking how they want to allocate the license they *do* have with these sites on the new server

Hey Catarina & Marcin,

I think we don't have enough theme licenses for our wordpress sites.

When we last visited this in August, I downloaded:

 [a] two licenses of oshine

 [b] one license of enigmatic

Unfortunately, you have three sites using oshine:

 [1] openbuildinginstitute.org

 [2] microfactory.opensourceecology.org

 [3] store.opensourceecology.org

I think you also mentioned that you might want to switch www.opensourceecology.org to Oshine? Well we definitely don't have a license for that.

I tried to find some way to copy the license keys from hetzner2 (our old/live prod server), but I couldn't find a way to view the currently-used license keys for each site. And when I install the latest version of the Oshine theme on the new servers, it requests a license key for each site.

If you'd like, I can just setup:

 * openbuildinginstitute.org with oshine #1

 * microfactory.opensourceecology.org with oshine #2

 * www.opensourceecology.org with enigmatic

 * change store.opensourceecology.org to use some free theme like twentytwentyfour

Or you could buy more licenses and I'll set up the sites with those.

Please let me know how you want to allocate your wordpress licenses for your sites.


Thank you,

ah, shortly after I sent that email, I found a third oshine license in my files
so the only issue is if we still want to use oshine for www.opensourceecology.org – then we'd need to buy a new license or take the one away from store.opensourceecology.org

Update: sorry, I found the third oshine license:

1. Purchased 2016 by "Catarina Mota"

2. Purchased 2018 by "Open Source Ecology"

3. Purchased 2019 by "Catarina Mota"

So I think the only issue is that we'd have to either buy a new license for www.opensourceecology.org, if that's a change you'd like to make.

Or use the license that's currently in-use by store.opensourceecology.org for www.opensourceecology.org and set store.opensourceecology.org to some free theme.

Please let me know how I should proceed in allocating these oshine licenses.

...
anyway, until we hear back about how to handle these oshine licenses, I think we're basically blocked from store.opensourceecology.org. I don't want to associate one of our precious license on the newly installed server's wordpress config if we're not going to use it.
refresher: here's our site migration order list

1. forum.opensourceecology.org
2. store.opensourceecology.orgc
3. microfactory.opensourceecology.org
4. fef.opensourceecology.org
5. oswh.opensourceecology.org
6. seedhome.openbuildinginstitute.org
7. www.openbuildinginstitute.org
8. www.opensourceecology.org
9. phplist.opensourceecology.org
10. wiki.opensourceecology.org

well next is microfatory, which is yet-another oshine ambiguity for now
after that we have fef
I logged into the fef wp admin wui dashboard https://fef.opensourceecology.org/
1. I confirmed that fef is using some theme called "Simple Photo Responsive"

Mon Oct 07, 2024

I installed the latest version of the 'oshine' theme to the store.opensourceecology.org wordpress site

rsync -av --progress /var/tmp/wordpress/themes/oshin /var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do

	wp_docroot="$(dirname "${wordpress_site}")"
	vhost_dir="$(dirname "${wp_docroot}")"

	chown -R not-apache:www-data "${vhost_dir}"
	find "${vhost_dir}" -type d -exec chmod 0050 {} \;
	find "${vhost_dir}" -type f -exec chmod 0040 {} \;

	chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
	chmod 0040 "${vhost_dir}/wp-config.php"

	[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
	find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;

	[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
	find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;

done

ok, after that loading it in the browser still yeilds a blank page, but it's just because it's cached https://store.opensourceecology.org/
if I just append a bullshit GET variable on the end, then it loads https://store.opensourceecology.org/?nocache=2
the site is finally loading, but it's all fucked
I'm not seeing any 403 errors in the network tab of firefox on-load, so I'm pretty sure the issue is just missing plugins
for example, there's a bunch of shortcodes being displayed raw, like this one at the top

[tatsu_section bg_color= “rgba(29,29,29,1)” bg_image= “http://brandexponents.com/oshine-lite/v37/wp-content/uploads/sites/44/2018/02/home-hero.jpeg” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= “center center” bg_size= “cover” bg_animation= “none” padding= ‘{“d”:”200px 0% 200px 0% “}’ margin= “0px 0px 0px 0px” border= “0px 0px px 0px” border_color= “” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “1” overlay_color= “rgba(0,0,0,0.1)” full_screen= “1” section_id= “” section_class= “” section_title= “” offset_section= “” offset_value= “0” full_screen_header_scheme= “background–dark” hide_in= “0” bg_stretch= “1” key= “REDACTED”]

ok, if I login to the WUI and go to appearance -> themes, now it does show the oshine theme https://store.opensourceecology.org/wp-admin/themes.php
I deactivated it and reactivated it, and I got a message at the top

This theme requires the following plugins: BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Safe SVG, Slider Revolution and WPForms Lite. Begin installing plugins | Dismiss this notice

If I click to customize the theme, it has a tab "Install Plugins"
If I click the "Install Plugins" tab, it yells at me with big red text

Please provide a valid purchase code of the theme in order to install plugins and import demo

on another DispVM, I logged into the hetzner2 store.opensourceecology.org
clicked appearance -> themes -> oshin -> customize
1. no, that didn't work
clicked "Oshine Options" in the left-hand navbar ->
1. no, I went through all the settings and couldn't find it there either :(
I mean, I have the keys already downloaded, but I'd like to keep them consistent. We have two keys and I don't know which was used for store.opensourceecology.org. This is dumb. Why do they make it so hard to find?
I tried pulling it out of the DB, but I didn't find anything obvious in the options table named "*oshine*"

MariaDB [store_db]> select * from wp_options where option_name like '%oshine%' limit 100;
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
| option_id | option_name                     | option_value                                                                                      | autoload |
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
|       347 | external_updates-oshine-core    | O:8:"stdClass":3:{s:9:"lastCheck";i:1728339766;s:14:"checkedVersion";s:5:"1.3.7";s:6:"update";N;} | no       |
|       349 | external_updates-oshine-modules | O:8:"stdClass":3:{s:9:"lastCheck";i:1728339766;s:14:"checkedVersion";s:5:"2.2.9";s:6:"update";N;} | no       |
|       352 | oshine_redux_to_colorhub        | 1                                                                                                 | yes      |
|       355 | oshine_redux_to_typehub         | 1                                                                                                 | yes      |
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
4 rows in set (0.00 sec)

MariaDB [store_db]>

I realized it's probably easier to just search the mysqldump file
1. got it!

root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # grep -ir 'purchase' mysqldump.20240926_072001b.sql | grep -ir code | less
...
193,'be_themes_purchase_data','a:2:{s:8:\"last_tab\";s:0:\"\";s:19:\"theme_purchase_code\";s:36:\"REDACTED\";}','yes'),(194,'be_themes_purchase_data-transients','a:2:{s:14:\"changed_values\";a:0:{}s:9:\"last_save\";i:1471011372;}','yes')

actually, that still doesn't tell me which server it is
I think that's the wrong site's db, because I see nothing when I query just the store wordpress db

MariaDB [store_db]> select * from wp_options where option_name like '%be%' and option_value like '%theme_purchase_code%' limit 100;
Empty set (0.00 sec)

MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase_code%' limit 100;
Empty set (0.01 sec)

MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase_data%' limit 100;
Empty set (0.01 sec)

MariaDB [store_db]> select * from wp_options where option_value like '%themes_purchase_data%' limit 100;
Empty set (0.01 sec)

MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase%' limit 100;
Empty set (0.01 sec)

MariaDB [store_db]>

I exported the data from the hetzner2 store theme too, but it wasn't there

user@disp928:~/Downloads$ du -sh *
28K	redux_options_be_themes_data_backup_07-10-2024.json
user@disp928:~/Downloads$

fuck it, I'm just going to use these alphabetically
which sites use this?

[root@opensourceecology hetzner3]# nice find /var/www/html -type d -iname oshin
/var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/themes/oshin
/var/www/html/d3d.opensourceecology.org/htdocs/wp-content/themes/oshine_6.5/Oshine Buyers Package 6.5/oshin
/var/www/html/d3d.opensourceecology.org/htdocs/wp-content/themes/oshin
/var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/themes/oshin
/var/www/html/staging.openbuildinginstitute.org/htdocs/wp-content/themes/oshin
/var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/oshin
/var/www/html/3dp.opensourceecology.org/htdocs/wp-content/themes/oshin
[root@opensourceecology hetzner3]#

d3d and 3dp are both broken with cert errors right now
right, my notes say these were two sites that marcin abandoned domain naames for. we eventually built microfactory.opensourceecology.org instead
ugh, power went out

Sun Oct 06, 2024

I checked on the status of the inventory job of our 'deleteMeIn2020' galcier vault; looks like it's still unavailable. Guess I'll give it a week or so before trying again

user@disp8678:~$  aws configure set aws_access_key_id 'REDACTED"
user@disp8678:~$  

user@disp8678:~$  aws configure set aws_secret_access_key 'REDACTED'
user@disp8678:~$ 

user@disp8678:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS
user@disp8678:~$

...

I returned to work on fixing the vhost config to permit traffic to wp-config.php temporarily, but I kept getting 429 errors from wordpress.org
This has been a frustrating, recurring issue for many months. I finally filed a bug report https://meta.trac.wordpress.org/ticket/7792#ticket

Title: Too Many "429 Too Many Requests" Errors (Nginx Misconfiguration causing False-Positives)

Since the past ~6 months, I have been frequently unable to access content on wordpress.org

If I'm lucky, then when I'm browsing wordpress documentation pages, I'm able to load the main html file with the content, but the website is horribly mis-rendered because many dependent assets don't load (eg css files, images, javascript, etc) due to "429 Too Many Requests" errors.

If I'm unlucky, even the main page doesn't load load at all -- due to "429 Too Many Requests".

Usually, I start-off being able to load one or more pages, but as I click around the website trying to find the page that I need, I eventually get this error.

I am not a bot. I am a human. I'm just trying to load reference documentation as I develop a wordpress plugin. This has been extremely frustrating, and forced me to third party websites and to "guess" php functions, attributes, and return values as I'm developing, reducing my productivity.

Since the Snowden revelations of 2013, it's become clear that many at-risk users should not be using the Internet without using privacy-protections like Tor. For security and privacy reasons, I do not access the internet without passing my traffic through Tor or a VPN. To prevent discrimination against at-risk folks, it's important that WordPress servers do not block traffic from shared networks, such as VPNs or Tor exit nodes.

It appears that nginx's settings are too strict, and lots of good users are getting caught in the dragnet.

Whatever the current nginx config is, please double it to fix these false-positives.

alright, I updated the apache config and pushed it with ansible

user@personal:~/sandbox_local/ansible/hetzner3$ git diff
diff --git a/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2 b/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
index c0575a3..c413c74 100644
--- a/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
+++ b/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
@@ -2,12 +2,12 @@
 
 ################################################################################
 # File:    security.virtualhost.include
-# Version: 0.2
+# Version: 0.3
 # Purpose: File includes some common security-hardening that's intended to be
 #          Include()d into other vhost files' <VirtualHost> blocks
 # Author:  Michael Altfield <michael@michaelaltfield.net>
 # Created: 2024-09-14
-# Updated: 2024-09-24
+# Updated: 2024-10-06
 ################################################################################
 
        # don't execute any php files inside uploads directories
@@ -56,7 +56,10 @@
 
    # block access to 'wp-login.php' from brute-forcers;
        # see wp plugin 'rename-wp-login'
-   <LocationMatch ".*wp-login.php">
-               Require all denied
-   </LocationMatch>
+       # TODO: 2024-10: we need to re-enable this after we find a replacement for the
+       #                (now-deprecated) 'rename-wp-login' wordpress plugin
+       #               * https://wordpress.org/plugins/rename-wp-login/
+#   <LocationMatch ".*wp-login.php">
+#              Require all denied
+#   </LocationMatch>
 
user@personal:~/sandbox_local/ansible/hetzner3$

now I'm able to load the login page, but when I do, I still get 403 errors on a few of the dependent requests
1. https://store.opensourceecology.org/wp-admin/images/wordpress-logo.svg?ver=20131107
2. https://store.opensourceecology.org/wp-includes/images/w-logo-blue-white-bg.png
the mod_security log shows the 403 response for these images, but I'm not sure why it's happening

root@hetzner3 /var/log/apache2 # cat modsec_audit.log
...
--69ede32a-H--
Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-admin/images/wordpress-logo.svg
Stopwatch: 1728256656835561 1333 (- - -)
Stopwatch2: 1728256656835561 1333; combined=33, p1=31, p2=0, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

--69ede32a-Z--
...
--6a3d866d-H--
Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png
Stopwatch: 1728256938397718 1105 (- - -)
Stopwatch2: 1728256938397718 1105; combined=27, p1=23, p2=0, p3=0, p4=0, p5=3, sr=0, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

--6a3d866d-Z--

oh, I wonder if this is it

user@personal:~/sandbox_local/ansible/hetzner3$ grep -irC4 'require all' | less
...
roles/maltfield.apache/templates/security.virtualhost.include.j2-       <LocationMatch "/images/">
roles/maltfield.apache/templates/security.virtualhost.include.j2-               SetHandler !
roles/maltfield.apache/templates/security.virtualhost.include.j2:               Require all denied
roles/maltfield.apache/templates/security.virtualhost.include.j2-       </LocationMatch>

so one thing we changed from hetzner2 was the logic for preventing php scripts from being executed inside user-uploadable directories.
in hetzner2, we used mod_php, so this was done with 'php_flag engine off' -- but that doesn't work with php-fpm

        # don't execute any php files inside the uploads directory
        <LocationMatch "/wp-content/uploads/">
                php_flag engine off
        </LocationMatch>
        <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
                Order Deny,Allow
                Deny from All
        </LocationMatch>

        # block dot files, such as svn files from checking out wp core
        <LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
                Deny From All
        </LocationMatch>

        # block access to 'wp-login.php' from brute-forcers; see wp plugin 'rename-wp-login'
        <LocationMatch .*wp-login.php>
                Deny From All
        </LocationMatch>

I think the idea of "Require all denied" was that it would capture anything that wasn't already sent-off to the php-fpm server proxy. Basically because that should be captured first, and then if apache still sees it, we should deny access to it
this assumes that the filename ends with php (or our more complex regex above), but that logic doesn't translate for the whole /images/ directory
alright, I made these changes, which fixed it. basically we want to "SetHandler !" on everything, but only "Require all denied" for the .php files
1. Note: I wasn't able to figure out what "SetHandler !" does. Bing/ddg returns no results. And Google just ignores any queries with an exclamation mark in them. It's literally not possible to search-for. But I did find lots of results asking about how to use SetHandler in Apache to point to php-fpm, so my best-guess is that this sets the Handler to 'null' or something, which would overwrite any previous setting that told it to send it to some other cgi proxy or something

        # don't execute any php files inside uploads directories
        <LocationMatch "/wp-content/uploads/">
                SetHandler !
-               Require all denied
        </LocationMatch>
        <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
                Require all denied
@@ -21,7 +20,6 @@
 
        <LocationMatch "/uploadimages/">
                SetHandler !
-               Require all denied
        </LocationMatch>
        <LocationMatch "/uploadimages/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
                Require all denied
@@ -29,7 +27,6 @@
 
        <LocationMatch "/images/">
                SetHandler !
-               Require all denied
        </LocationMatch>
        <LocationMatch "/images/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
                Require all denied
@@ -38,7 +35,6 @@
        # don't execute php files in W3 Total Cache's tmp dir
        <LocationMatch "/wp-content/cache/">
                SetHandler !
-               Require all denied
        </LocationMatch>
        <LocationMatch "/wp-content/cache/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
                Require all denied
@@ -46,17 +42,22 @@
 
        # block dot (hidden) files
        <LocationMatch "/\.(?!well\-known)">
+               SetHandler !
                Require all denied
        </LocationMatch>
 
        # block config files
        <LocationMatch "config.php">
+               SetHandler !
                Require all denied
        </LocationMatch>
 
    # block access to 'wp-login.php' from brute-forcers;
        # see wp plugin 'rename-wp-login'
-   <LocationMatch ".*wp-login.php">
-               Require all denied
-   </LocationMatch>

cool, I was able to login to store.opensourceecology.org on hetzner3 with my old creds now
the dashboard is littered with alerts:

Action Scheduler: 3 past-due actions found; something may be wrong. Read documentation »

WooCommerce database update required

WooCommerce has been updated! To keep things running smoothly, we have to update your database to the newest version. The database update process runs in the background and may take a little while, so please be patient. Advanced users can alternatively update via WP CLI.

Update WooCommerce Database Learn more about updates

 Geolocation has not been configured.

You must enter a valid license key on the MaxMind integration settings page in order to use the geolocation service. If you do not need geolocation for shipping or taxes, you should change the default customer location on the general settings page.

The plugin be-gdpr/be-gdpr.php has been deactivated due to an error: Plugin file does not exist.

The plugin be-portfolio-post/be-portfolio-post.php has been deactivated due to an error: Plugin file does not exist.

The plugin coingate-for-woocommerce/coingate.php has been deactivated due to an error: Plugin file does not exist.

The plugin colorhub/colorhub.php has been deactivated due to an error: Plugin file does not exist.

The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist.

The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist.

The plugin meta-box-conditional-logic/meta-box-conditional-logic.php has been deactivated due to an error: Plugin file does not exist.

The plugin meta-box-show-hide/meta-box-show-hide.php has been deactivated due to an error: Plugin file does not exist.

The plugin meta-box-tabs/meta-box-tabs.php has been deactivated due to an error: Plugin file does not exist.

The plugin oshine-core/oshine-core.php has been deactivated due to an error: Plugin file does not exist.

The plugin oshine-modules/oshine-modules.php has been deactivated due to an error: Plugin file does not exist.

The plugin redux-vendor-support/redux-vendor-support.php has been deactivated due to an error: Plugin file does not exist.

The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist.

The plugin revslider/revslider.php has been deactivated due to an error: Plugin file does not exist.

The plugin tatsu/tatsu.php has been deactivated due to an error: Plugin file does not exist.

The plugin typehub/typehub.php has been deactivated due to an error: Plugin file does not exist.

I kicked-off the woocommerce db upgrade
ugh, akismet isn't activated. we have 2,213 comments in the queue
if I click on 'themes' in the wui, then I get a notice at the top

The active theme is broken. Reverting to the default theme.

it says that 'oshine' is the active theme
allright, I downloaded these files before

user@ose:~/tmp/hetzner3$ ls
13757819-enigmatic-responsive-multipurpose-wp-theme-license.txt
28755060-oshine-creative-multipurpose-wordpress-theme-license.txt
47932235-oshine-creative-multipurpose-wordpress-theme-license.txt
52287820-oshine-creative-multipurpose-wordpress-theme-license.txt
backup-restore-test
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
user@ose:~/tmp/hetzner3$

unfortunately, these are paid themes, and I have to coordinate with catarina to get an OTP every time I login, so I can't 3TOFU these :( I'll just have to 1TOFU it
apparently these two oshine themes have identical contents but different names

user@ose:~/tmp/hetzner3$ sha256sum *.zip
ed0628d0e57bb4e44b1af24eb235c6c384433c9ca94806c11b881e16f7f2b74a  themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789  themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789  themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
user@ose:~/tmp/hetzner3$

I rsync'd these files up to hetzner3

user@ose:~/tmp/hetzner3$ rsync -av --progress themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip hetzner3:
Enter passphrase for key '/home/user/.ssh/id_rsa': 
Enter passphrase for key '/home/user/.ssh/id_rsa': 
sending incremental file list
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
     10,582,975 100%  318.17kB/s    0:00:32 (xfr#1, to-chk=0/1)

sent 10,585,730 bytes  received 35 bytes  201,633.62 bytes/sec
total size is 10,582,975  speedup is 1.00
user@ose:~/tmp/hetzner3$ 

user@ose:~/tmp/hetzner3$ rsync -av --progress themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip hetzner3:
Enter passphrase for key '/home/user/.ssh/id_rsa': 
sending incremental file list
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
     11,394,173 100%  996.24kB/s    0:00:11 (xfr#1, to-chk=0/1)

sent 11,397,129 bytes  received 35 bytes  303,924.37 bytes/sec
total size is 11,394,173  speedup is 1.00
user@ose:~/tmp/hetzner3$

I copied them over to our other dir with all the themes

root@hetzner3 /var/tmp/wordpress/themes # ls /home/maltfield/*.zip
/home/maltfield/themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
/home/maltfield/themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # rsync -av --progress /home/maltfield/*.zip .
sending incremental file list
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
     10.582.975 100%  670,76MB/s    0:00:00 (xfr#1, to-chk=1/2)
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
     11.394.173 100%  329,28MB/s    0:00:00 (xfr#2, to-chk=0/2)

sent 21.982.821 bytes  received 54 bytes  43.965.750,00 bytes/sec
total size is 21.977.148  speedup is 1,00 
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # shred -u /home/maltfield/*.zip
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # chown root:root themeforest-*
root@hetzner3 /var/tmp/wordpress/themes # chmod 0400 themeforest-*
root@hetzner3 /var/tmp/wordpress/themes #

Fri Oct 04, 2024

Marcin gave me the go-ahead to delete the 'deleteMeIn2020' vault from our AWS Glacier account

1. Yes, delete the vault.

2. Thanks, good insights - i'll look into those more closely to see what
would fit best.

MJ

ah ffs, I logged into the amazon WUI, but when I clicked "delete" on the vault, it gave me an error saying I have to delete all the objects in the vault first

This vault is not empty
Vaults can be deleted only if there are no archives in the vault as of the last inventory it computed and there have been no writes to the vault since the last inventory. To delete all archives in the vault, use the REST API, the AWS SDK for Java, the AWS SDK for .NET or the AWS CLI.

apparently this can only be done in the CLI via the API!?! It links to this https://docs.aws.amazon.com/console/glacier/using-aws-sdk
we do have some 'glacier.py' script on our old server, but it complains about missing module(s)

[root@opensourceecology backups]# ls
backup.old.20180115.sh    backup.settings.20221028  glacierRestore        sync
backupReport.sh           backup.sh                 glacierTest.py        sync.old
backupReport.sh.20221028  backup.sh.20221028        ose-backups-cron.key
backup.settings           cleanLocal.pl             README.txt
[root@opensourceecology backups]#

[root@opensourceecology backups]# glacier.py 
Traceback (most recent call last):
  File "/root/bin/glacier.py", line 36, in <module>
    import boto.glacier
ImportError: No module named boto.glacier
[root@opensourceecology backups]#

allright, at least debian has the cli in its repos

user@disp3919:~/Downloads$ apt-cache search awscli
awscli - Unified command line interface to Amazon Web Services
user@disp3919:~/Downloads$

user@disp3919:~/Downloads$ sudo apt-get install awscli
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
awscli is already the newest version (2.9.19-1).
The following packages were automatically installed and are no longer required:
  librnp0 libwpe-1.0-1 libwpebackend-fdo-1.0-1 linux-image-6.1.0-10-amd64
  linux-image-6.1.0-11-amd64 linux-image-6.1.0-13-amd64
  linux-image-6.1.0-17-amd64 linux-image-6.1.0-18-amd64
  linux-image-6.1.0-20-amd64 linux-image-6.1.0-21-amd64
  linux-image-6.1.0-22-amd64
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
user@disp3919:~/Downloads$ aws

I was able to auth with some creds I found on hetzner2:/root/backups/glacierTest.py

user@disp3919:~/Downloads$  aws configure set aws_access_key_id 'REDACTED'
user@disp3919:~/Downloads$  aws configure set aws_secret_access_key 'REDACTED'
user@disp3919:~/Downloads$ aws sts get-caller-identity
{
    "UserId": "REDACTED",
    "Account": "REDACTED",
    "Arn": "arn:aws:iam::REDACTED:user/backup-cron"
}
user@disp3919:~/Downloads$

apparently we now have to create an inventory and then iterate though that inventory to delete all of the objects that it lists https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5
creating an inventory can take hours or days; let's initiate it now

user@disp3919:~/Downloads$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
    "location": "/099400651767/vaults/deleteMeIn2020/jobs/ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS",
    "jobId": "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS"
}
user@disp3919:~/Downloads$

I guess now we wait a few days for the job to complete before we can download it, parse it, and then delete all of the objects it identifies per https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5

user@disp3919:~/Downloads$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json

An error occurred (InvalidParameterValueException) when calling the GetJobOutput operation: The job is not currently available for download: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS
user@disp3919:~/Downloads$

...

after much debugging, I figured out why store.opensourceecology.org gives different results for a `curl` coming from my laptop vs the server
I found that the `curl` from my laptop was making it to nginx -> varnish -> apache
but the logs were mysteriously absent for varnish & apache when I did the curl from the machine itself
I even did a tcpdump, but I only saw a tiny blip of traffic when doing the command locally
here's why: the server returns an http -> https redirect to store.opensourceecology.org. When the *server*'s curl command gets that, it does a public DNS lookup and then sends the query to hetzner2!
I updated the /etc/hosts file to prevent this

root@hetzner3 ~ # cd /etc
root@hetzner3 /etc # 

root@hetzner3 /etc # vim hosts
root@hetzner3 /etc # 

root@hetzner3 /etc # diff hosts.20241004 hosts
2a3,13
> 127.0.0.1 forum.opensourceecology.org
> 127.0.0.1 store.opensourceecology.org
> 127.0.0.1 microfactory.opensourceecology.org
> 127.0.0.1 fef.opensourceecology.org
> 127.0.0.1 oswh.opensourceecology.org
> 127.0.0.1 seedhome.openbuildinginstitute.org
> 127.0.0.1 www.openbuildinginstitute.org
> 127.0.0.1 www.opensourceecology.org
> 127.0.0.1 phplist.opensourceecology.org
> 127.0.0.1 wiki.opensourceecology.org
>
3a15
>
root@hetzner3 /etc #

ok, now it's stuck in an infinite redirect. It just keeps going back-and-forth adding and removing the slash at the end

maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local5
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:23:38 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://opensourceecology.org
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:22:29 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://www.opensourceecology.org/
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:23:39 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://opensourceecology.org
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
...
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:22:29 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://www.opensourceecology.org/
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:23:39 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://opensourceecology.org
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

curl: (47) Maximum (50) redirects followed
maltfield@hetzner3:~$

ok, so it looks like it's getting picked-up by the default site

root@hetzner3 /etc/nginx # grep -ir 301 *
nginx.conf:                     return 301 https://$host$request_uri;
nginx.conf.1282157.2024-09-28@23:10:52~:                        return 301 https://$host$request_uri;
sites-enabled/00-default.conf:  return 301 https://opensourceecology.org;
root@hetzner3 /etc/nginx #

ah shit, yeah, nginx isn't even listening on 127.0.0.1 lol

root@hetzner3 /etc/nginx # less sites-enabled/store.opensourceecology.org.conf 
# Ansible managed

################################################################################
# File:    store.opensourceecology.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
#          protection, and passing to varnish cache (varnish then passes to
#          apache)
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
# Updated: 2024-09-14
################################################################################

server {

        access_log /var/log/nginx/store.opensourceecology.org/access.log main;
        error_log /var/log/nginx/store.opensourceecology.org/error.log;

   include conf.d/secure.include;
   include conf.d/https.opensourceecology.org.include;

   listen 144.76.164.201:443;
   listen [2a01:4f8:200:40d7::2]:443;

   server_name store.opensourceecology.org;

        #############
        # SITE_DOWN #
        #############
        # uncomment this block && restart nginx prior to apache work to display the
        # "SITE DOWN" webpage for our clients

#       root /var/www/html/SITE_DOWN/htdocs/;
#   index index.html index.htm;
#
#       # force all requests to load exactly this page
#       location / {
#               try_files $uri /index.html;
#       }

        ###################
        # SEND TO VARNISH #
        ###################

   location / {
      proxy_pass http://127.0.0.1:6081;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header X-Forwarded-Port 443;
      proxy_set_header Host $host;
   }

}

well it is, but this server block is not

root@hetzner3 /etc/nginx # netstat -plan | grep -i 443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      3914728/nginx: mast 
tcp        0      0 144.76.164.201:4443     0.0.0.0:*               LISTEN      3914728/nginx: mast 
tcp       25      0 144.76.164.201:51710    104.21.40.220:443       CLOSE_WAIT  15751/wazuh-modules 
tcp        0      0 127.0.0.1:80            127.0.0.1:54436         TIME_WAIT   -                   
tcp        0      0 127.0.0.1:54432         127.0.0.1:80            TIME_WAIT   -                   
tcp6       0      0 :::443                  :::*                    LISTEN      3914728/nginx: mast 
tcp6       0      0 2a01:4f8:200:40d7::4443 :::*                    LISTEN      3914728/nginx: mast 
tcp6      25      0 2a01:4f8:200:40d7:49016 2606:4700:3033::ac4:443 CLOSE_WAIT  15751/wazuh-modules 
You have new mail in /var/mail/root
root@hetzner3 /etc/nginx #

yeah, default is -- which is why it's picking it up instead

root@hetzner3 /etc/nginx # grep -ir listen
sites-available/forum.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS
sites-available/forum.opensourceecology.org.conf:   listen 144.76.164.201:443;
sites-available/forum.opensourceecology.org.conf:   listen [2a01:4f8:200:40d7::2]:443;
sites-available/default:        listen 80 default_server;
sites-available/default:        listen [::]:80 default_server;
sites-available/default:        # listen 443 ssl default_server;
sites-available/default:        # listen [::]:443 ssl default_server;
sites-available/default:#       listen 80;
sites-available/default:#       listen [::]:80;
sites-available/store.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS
sites-available/store.opensourceecology.org.conf:   listen 144.76.164.201:443;
sites-available/store.opensourceecology.org.conf:   listen [2a01:4f8:200:40d7::2]:443;
nginx.conf.1282157.2024-09-28@23:10:52~:                listen 80;
nginx.conf.1282157.2024-09-28@23:10:52~:                listen [::]:80;
nginx.conf:             listen 80;
nginx.conf:             listen [::]:80;
nginx.conf.85740.2024-09-24@04:17:16~:#         listen     localhost:110;
nginx.conf.85740.2024-09-24@04:17:16~:#         listen     localhost:143;
sites-enabled/00-default.conf:  listen 443;
sites-enabled/00-default.conf:  listen [::]:443;
sites-enabled/awstats.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS
sites-enabled/awstats.opensourceecology.org.conf:   listen 144.76.164.201:443;
sites-enabled/awstats.opensourceecology.org.conf:   listen [2a01:4f8:200:40d7::2]:443;
sites-enabled/awstats.opensourceecology.org.conf:   listen 144.76.164.201:4443;
sites-enabled/awstats.opensourceecology.org.conf:   listen [2a01:4f8:200:40d7::2]:4443;
sites-enabled/munin.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS
sites-enabled/munin.opensourceecology.org.conf:   listen 144.76.164.201:443;
sites-enabled/munin.opensourceecology.org.conf:   listen [2a01:4f8:200:40d7::2]:443;
sites-enabled/munin.opensourceecology.org.conf:   listen 144.76.164.201:4443;
sites-enabled/munin.opensourceecology.org.conf:   listen [2a01:4f8:200:40d7::2]:4443;
root@hetzner3 /etc/nginx #

this is actually the same as our hetzner2 config
I updated the nginx config in ansible and pushed it out again

diff --git a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
index f4b62cd..f750651 100644
--- a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
@@ -2,13 +2,13 @@
 
 ################################################################################
 # File:    store.opensourceecology.org.conf
-# Version: 0.2
+# Version: 0.3
 # Purpose: Internet-listening web server for truncating https, basic DOS
 #          protection, and passing to varnish cache (varnish then passes to
 #          apache)
 # Author:  Michael Altfield <michael@michaelaltfield.net>
 # Created: 2019-04-09
-# Updated: 2024-09-14
+# Updated: 2024-10-04
 ################################################################################
 
 server {
@@ -19,6 +19,8 @@ server {
    include conf.d/secure.include;
    include conf.d/https.opensourceecology.org.include;
 
+   listen 127.0.0.1:443;
+   listen [::1]:443;
    listen {{ ansible_default_ipv4.address }}:443;
    listen [{{ ansible_default_ipv6.address }}]:443;
 
user@personal:~/sandbox_local/ansible/hetzner3$ 
diff --git a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
index f4b62cd..f750651 100644
--- a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
@@ -2,13 +2,13 @@
 
 ################################################################################
 # File:    store.opensourceecology.org.conf
-# Version: 0.2
+# Version: 0.3
 # Purpose: Internet-listening web server for truncating https, basic DOS
 #          protection, and passing to varnish cache (varnish then passes to
 #          apache)
 # Author:  Michael Altfield <michael@michaelaltfield.net>
 # Created: 2019-04-09
-# Updated: 2024-09-14
+# Updated: 2024-10-04
 ################################################################################
 
 server {
@@ -19,6 +19,8 @@ server {
    include conf.d/secure.include;
    include conf.d/https.opensourceecology.org.include;
 
+   listen 127.0.0.1:443;
+   listen [::1]:443;
    listen {{ ansible_default_ipv4.address }}:443;
    listen [{{ ansible_default_ipv6.address }}]:443;
 
user@personal:~/sandbox_local/ansible/hetzner3$

and, well, the good/bad news is that now the curl from the local machine is as equally broken as the curl from my laptop

maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local6
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:46:30 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: https://store.opensourceecology.org/?nocache=local6
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 89
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 05 Oct 2024 03:46:30 GMT
Content-Type: text/html
Content-Length: 5
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT
ETag: "5-6239f651921da"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
Pragma: public
Cache-Control: public, max-age=300
X-Varnish: 98500
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

true
maltfield@hetzner3:~$

the 'true' is obviously coming from 'index.html', so my first thought was just to get rid of that file

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # rm index.html
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

but now we're just back to the empty output (again)

maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local7
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 05 Oct 2024 03:49:48 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: https://store.opensourceecology.org/?nocache=local7
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 94
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 200 OK
Server: nginx
Date: Sat, 05 Oct 2024 03:49:49 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2796>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 97
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

maltfield@hetzner3:~$

varnish logs look fine; it basically just calls the backend
apahce logs indicate that it did figure out which file to server with php

==> store.opensourceecology.org/error.log <==
[Sat Oct 05 03:56:05.631466 2024] [authz_core:debug] [pid 3909393:tid 3909439] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
[Sat Oct 05 03:56:05.631539 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(123): [client 127.0.0.1:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Sat Oct 05 03:56:05.631557 2024] [proxy:debug] [pid 3909393:tid 3909439] mod_proxy.c(1465): [client 127.0.0.1:0] AH01143: Running scheme fcgi handler (attempt 0)
[Sat Oct 05 03:56:05.631571 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(1078): [client 127.0.0.1:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
[Sat Oct 05 03:56:05.631584 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(1087): [client 127.0.0.1:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Sat Oct 05 03:56:05.631597 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80)
[Sat Oct 05 03:56:05.631610 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3242): [client 127.0.0.1:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
[Sat Oct 05 03:56:05.631624 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3309): [client 127.0.0.1:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
[Sat Oct 05 03:56:05.631638 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3450): [client 127.0.0.1:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
[Sat Oct 05 03:56:05.631673 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)
[Sat Oct 05 03:56:06.720816 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80)

==> store.opensourceecology.org/access.log <==
127.0.0.1 - - [05/Oct/2024:03:56:05 +0000] "GET /index.php?nocache=local10 HTTP/1.1" 301 436 "-" "curl/7.88.1"

==> store.opensourceecology.org/error.log <==
[Sat Oct 05 03:56:06.725670 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
[Sat Oct 05 03:56:06.725738 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
[Sat Oct 05 03:56:06.725854 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
[Sat Oct 05 03:56:06.725886 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(123): [client 127.0.0.1:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Sat Oct 05 03:56:06.725895 2024] [proxy:debug] [pid 3909393:tid 3909441] mod_proxy.c(1465): [client 127.0.0.1:0] AH01143: Running scheme fcgi handler (attempt 0)
[Sat Oct 05 03:56:06.725901 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(1078): [client 127.0.0.1:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
[Sat Oct 05 03:56:06.725928 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(1087): [client 127.0.0.1:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Sat Oct 05 03:56:06.725935 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80)
[Sat Oct 05 03:56:06.725941 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3242): [client 127.0.0.1:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
[Sat Oct 05 03:56:06.725950 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3309): [client 127.0.0.1:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
[Sat Oct 05 03:56:06.725959 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3450): [client 127.0.0.1:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
[Sat Oct 05 03:56:06.726002 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)
[Sat Oct 05 03:56:07.778759 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80)

==> store.opensourceecology.org/access.log <==
127.0.0.1 - - [05/Oct/2024:03:56:06 +0000] "GET /?nocache=local10 HTTP/1.1" 200 586 "-" "curl/7.88.1"

this suggests that it might do this if the theme dir is empty? that would likely apply in our case https://serverfault.com/a/766146
oh, it *does* load if I try '/wp-admin/'

maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/wp-admin/
...
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 05 Oct 2024 04:24:26 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 1516
Connection: keep-alive
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98551
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

<!DOCTYPE html>
<html lang="en-US">
<head>
        <meta name="viewport" content="width=device-width" />
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <meta name="robots" content="noindex,nofollow" />
        <title>WordPress › Update</title>
        <link rel='stylesheet' id='dashicons-css' href='https://store.opensourceecology.org/wp-includes/css/dashicons.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='https://store.opensourceecology.org/wp-includes/css/buttons.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='https://store.opensourceecology.org/wp-admin/css/forms.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='https://store.opensourceecology.org/wp-admin/css/l10n.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='https://store.opensourceecology.org/wp-admin/css/install.min.css?ver=6.6.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui">
<p id="logo"><a href="https://wordpress.org/">WordPress</a></p>

        <h1>Database Update Required</h1>
<p>WordPress has been updated! Next and final step is to update your database to the newest version.</p>
<p>The database update process may take a little while, so please be patient.</p>
<p class="step"><a class="button button-large button-primary" href="upgrade.php?step=1&backto=%2Fwp-admin%2F">Update WordPress Database</a></p>
                        </body>
</html>
maltfield@hetzner3:~$

I loaded that in the web browser, and it told me a wordpress database update was needed. I just pressed the button -- it didn't even prompt me to auth

Your WordPress database has been successfully updated!

I clicked "Continue"
that redirected me here, and I immediately got '403 forbidden' https://store.opensourceecology.org/wp-login.php?redirect_to=https%3A%2F%2Fstore.opensourceecology.org%2Fwp-admin%2F&reauth=1
that would be because block access to 'wp-login.php' since we were using a plugin to rename it; we'll have to temp disable that until we replace that (now deprecated) plugin

Thr Oct 03, 2024

I sent an invoice (AS-0106) to OSE for 67 hours in Sep 2024

...

continuing to debug store.opensourceecology.org, I see that it's redirecting from '/index.php' to '/'

user@disp3919:~$ curl -i https://store.opensourceecology.org/index.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 04 Oct 2024 04:47:39 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: https://store.opensourceecology.org/
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 131132 98385
Age: 88
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

user@disp3919:~$

if I try to hit 'index.html', I don't get a redirect -- I just get a 404

user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html
HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 04 Oct 2024 04:48:25 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/"
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 131134
Age: 0
Via: 1.1 varnish (Varnish/7.1)

user@disp3919:~$

if I create it

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cp is_hetzner3 index.html
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # ls -lah is_hetzner3 index.html 
----r----- 1 root       root     5 Oct  4 04:49 index.html
----r----- 1 not-apache www-data 5 Sep 27 04:44 is_hetzner3
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown not-apache:www-data index.html 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

then it works

user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Oct 2024 04:49:51 GMT
Content-Type: text/html
Content-Length: 5
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT
ETag: "5-6239f651921da"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
Pragma: public
Cache-Control: public, max-age=300
X-Varnish: 98387
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

true
user@disp3919:~$

hmm, so apache is just refusing to serve 'index.php' files. -- what about 'something.php'?

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # echo "<?php echo 'it works'; ?>" > something.php
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown root:www-data something.php 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chmod 0040 something.php 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

it works, so it's something specific to 'index.php'

user@disp3919:~$ curl -i https://store.opensourceecology.org/something.php
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Oct 2024 04:52:35 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98390
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

it worksuser@disp3919:~$

I'd think it's an issue with the DirectoryIndex, but this looks good

root@hetzner3 /etc/apache2 # grep -ir 'index.php' *
conf-available/wordpress.directory.include:#            RewriteRule . /index.php [L]
mods-available/dir.conf:DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # ls -lah mods-enabled/dir.conf 
lrwxrwxrwx 1 root root 26 Sep 25 01:24 mods-enabled/dir.conf -> ../mods-available/dir.conf
root@hetzner3 /etc/apache2 #

I checked the old server, but I didn't see anything that we're missing in the new server

[root@opensourceecology httpd]# grep -ir 'index.php' conf
[root@opensourceecology httpd]# grep -ir 'index.php' conf.d
conf.d/php.conf:# Add index.php to the list of files that will be served as directory
conf.d/php.conf:DirectoryIndex index.php
conf.d/00-wiki.opensourceecology.org.conf:	Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php
conf.d/mod_evasive.conf:    #   http://security.lss.hr/index.php?page=details&ID=LSS-2005-01-01
[root@opensourceecology httpd]# grep -ir 'index.php' conf.modules.d/
[root@opensourceecology httpd]# grep -ir 'index.php' modsecurity.d/
[root@opensourceecology httpd]#

I changed wp-config.php to have WP_DEBUG set to 'true', but it didn't print anything extra. It seems like the error is occurring before wordpress
I set LogLevel of apache.conf to 'debug', and this popped-up

==> forum.opensourceecology.org/access.log <==
127.0.0.1 - - [04/Oct/2024:05:10:02 +0000] "GET /server-status?auto HTTP/1.1" 200 1202 "-" "munin/2.0.73 (libwww-perl/6.68)"

==> forum.opensourceecology.org/error.log <==
[Fri Oct 04 05:10:03.564975 2024] [authz_core:debug] [pid 3581402:tid 3581414] mod_authz_core.c(815): [client 127.0.0.1:32934] AH01626: authorization result of Require all denied: denied

well, that's an unrelated issue with munin, but it seems that the requests to '/server-status' are getting sent to the wrong vhost (forum.opensourceecology.org) and also denied access
here's the actual output when I do the curl
1. first, it outputs this immediately, then it pauses for maybe 10 seconds

==> store.opensourceecology.org/error.log <==
[Fri Oct 04 05:11:53.426292 2024] [authz_core:debug] [pid 3581402:tid 3581422] mod_authz_core.c(733): [client 81.17.16.91:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
[Fri Oct 04 05:11:53.426458 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(123): [client 81.17.16.91:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Fri Oct 04 05:11:53.426496 2024] [proxy:debug] [pid 3581402:tid 3581422] mod_proxy.c(1465): [client 81.17.16.91:0] AH01143: Running scheme fcgi handler (attempt 0)
[Fri Oct 04 05:11:53.426517 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1078): [client 81.17.16.91:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
[Fri Oct 04 05:11:53.426535 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1087): [client 81.17.16.91:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Fri Oct 04 05:11:53.426586 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80)
[Fri Oct 04 05:11:53.426612 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3242): [client 81.17.16.91:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
[Fri Oct 04 05:11:53.426658 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3309): [client 81.17.16.91:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
[Fri Oct 04 05:11:53.426718 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3450): [client 81.17.16.91:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
[Fri Oct 04 05:11:53.426793 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)

1. after maybe 10 seconds, it outputs this

[Fri Oct 04 05:12:03.646185 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80)

==> store.opensourceecology.org/access.log <==
81.17.16.91 - - [04/Oct/2024:05:11:53 +0000] "GET /index.php?nocache=6 HTTP/1.1" 301 430 "-" "curl/7.88.1"

so it sounds like maybe this is an issue with the php-fpm config?
I tried to hit apache through the cli on the server itself and, oh, I get the payload as desired

root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
<script>
        //jQuery(document).ready(function(){
                        // });
</script>
</body>
</html>root@hetzne
You have new mail in /var/mail/root

I loosened the error reporting settings on php.ini and I got it to spit this out when I curl from my laptop

--2d1eeb03-H--
Apache-Error: [file "mod_authz_core.c"] [line 733] [level 7] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 123] [level 7] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "mod_proxy.c"] [line 1465] [level 7] AH01143: Running scheme fcgi handler (attempt 0)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1078] [level 7] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1087] [level 7] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "proxy_util.c"] [line 3242] [level 7] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
Apache-Error: [file "proxy_util.c"] [line 3309] [level 7] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
Apache-Error: [file "proxy_util.c"] [line 3450] [level 7] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 911] [level 3] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function ini_set() in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php:590\\nStack trace:\\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(82): wp_debug_mode()\\n#1 /var/www/html/store.opensourceecology.org/wp-config.php(105): require_once('...')\\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-blog-header.php(13): require_once('...')\\n#4 /var/www/html/store.opensourceecology.org/htdocs/index.php(17): require('...')\\n#5 {main}\\n  thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php on line 590'
Apache-Handler: proxy:unix:/run/php/php8.2-fpm.sock|fcgi://localhost
Stopwatch: 1728019471413382 53626 (- - -)
Stopwatch2: 1728019471413382 53626; combined=41, p1=21, p2=18, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

yeah, so this is the wordpress bug that I submitted a PR for last month
after a brief dialog with the wordpress devs, a workaround until they merge is to define a fake ini_set() function in wp-config.php
1. from personal experience, I found it's best to wrap this in a conditional to make sure the function doesn't exist yet

root@hetzner3 /var/www/html/store.opensourceecology.org # cp wp-config.php wp-config.php.20241003
root@hetzner3 /var/www/html/store.opensourceecology.org # 

root@hetzner3 /var/www/html/store.opensourceecology.org # vim wp-config.php
root@hetzner3 /var/www/html/store.opensourceecology.org # 

root@hetzner3 /var/www/html/store.opensourceecology.org # diff wp-config.php.20241003 wp-config.php
1a2,9
> 
> # fix wordpress bug https://core.trac.wordpress.org/ticket/48693
> if( ! function_exists('ini_set') ){
>       function ini_set(){
>               return;
>       }
> }
> 
root@hetzner3 /var/www/html/store.opensourceecology.org #

after that, I'm back to getting blank pages on my curl on my laptop. it's flapping?
alright, let me see if I can harden php back up again, but with errors actually logging. I'll update something.php to write to the error log

root@hetzner3 /var/www/html/store.opensourceecology.org # cd htdocs/
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # vim something.php 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cat something.php 
<?php
error_log( "executing something.php" );
echo 'it works';
?>
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

ok, it's visible

[Fri Oct 04 05:47:06.734936 2024] [proxy_fcgi:error] [pid 3581402:tid 3581452] [client 127.0.0.1:36330] AH01071: Got error 'PHP message: executing something.php'

first I reduced the apache logs down to 'warn' again. looks good


[Fri Oct 04 05:48:39.732669 2024] [proxy_fcgi:error] [pid 3591906:tid 3591913] [client 127.0.0.1:41102] AH01071: Got error 'PHP message: executing something.php'

==> store.opensourceecology.org/access.log <==
127.0.0.1 - - [04/Oct/2024:05:48:39 +0000] "GET /something.php HTTP/1.1" 200 302 "-" "curl/7.88.1"

I obliterated my manual changes by pushing ansible's apache & php roles
cool, I confirmed that both curl on my laptop and on the server produce the logs after restarting both apache2 & php8.2-fpm
for some reason I still get an 'true' on my laptop

user@disp3919:~$ 3919:~$ curl -iL https://store.opensourceecolindex.php?nocache=19
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 04 Oct 2024 05:56:15 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: https://store.opensourceecology.org/?nocache=19
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98469
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Oct 2024 05:56:16 GMT
Content-Type: text/html
Content-Length: 5
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT
ETag: "5-6239f651921da"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
Pragma: public
Cache-Control: public, max-age=300
X-Varnish: 131203
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

true
user@disp3919:~$

but I get the actual html on the local machine

root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
/* ]]> */
</script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-content/themes/oshin/js/script.js?ver=5.0'></script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-includes/js/wp-embed.min.js?ver=5.1.1'></script>
<!-- Option Panel Custom JavaScript -->
<script>
        //jQuery(document).ready(function(){
                        // });
</script>
</body>
</html>root@hetzner3 ~ #

Wed Oct 02, 2024

Marcin sent me a few emails in the past months asking about OSE's use of Amazon Glacier
Today he sent a message saying that he got charged $1.03, and isn't sure why

Michael,

I'm getting charged $1.03 for Glacier. Can we cancel that?

Marcin

It took me a while to auth
1. first I tried to login with my 'maltfield' aws user, but aws rejected my creds (stored in my personal keepass)
2. eventually I realized I had to click "Sign in using root user email" -- and then I could auth using the creds stored in the shared keepass
after logging-in, I went to the "Billing and Cost Management" app https://us-east-1.console.aws.amazon.com/costmanagement/home?region=us-west-2#/home
on this page, there was a link that said "Last month's total cost: $1.03". Yep, that's all accounted-for. I clicked it.
the next page showed a joke of a chart with one bar on a bar graph that said "$1.03". And the bar was labeled "Total Cost"
I had to click on the dropdown menu for "Dimension" and set it to "Service" -- then it listed 4 items
1. Glacier - $1.03
2. S3 - $0.00
3. Tax $0.00
So I switched over to the "Glacier" app https://us-east-1.console.aws.amazon.com/glacier/home?region=us-east-1
1. Curiously, it listed 0 vaults
2. but there was a note at the top saying we should use S3 for glaicer, so I clicked over to the "S3" app

We recommend that you use Glacier storage classes in Amazon S3 for archival storage

here I saw one bucket called "oseserverbackups" in "US West (Oregon) us-west-2"
the bucket had one 34.0 byte file in it called "test.txt". That's it!
1. this file was created July 6, 2018, 19:18:03 (UTC-05:00)
2. I downloaded it; it has one line of text

some file destined for s3 this is

1. I deleted the 'test.txt' file object from the s3 bucket
I then deleted the (now empty) 'oseserverbackups' bucket
unconvinced that that was the issue, I went back to the "glacier" app. This time I cycled through a few of the regions until I got to "us-west-2" -- this time I showed one vault named "deleteMeIn2020"
I clicked on it, and it said
1. this vault was created March 29, 2018, 16:36:06 (UTC-05:00)
2. this vault was last inventoried August 1, 2018, 02:41:31 (UTC-05:00)
3. this vault is 285.3 GB (as of last inventory)
well, it's after 2020. So I think we should delete it.
I sent an email to Marcin asking for a confirmation before I delete it

Hey Marcin,

You have a 285.3 GB vault in Amazon Glacier's us-west-2 region.

I logged-into your AWS account today and did some digging. I found this vault 285.3 GB vault named 'deleteMeIn2020'. I created this vault in 2018 Q1. It contains a final backup of files from hetzner1. I created it as part of the hetzner2 migration project, thinking that we should delete it in 2020 if we never needed to restore anything from it for 2 years.

 * https://wiki.opensourceecology.org/wiki/CHG-2018-07-06_hetzner1_deprecation

 * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q1#Sat_Mar_31.2C_2018

Well, 2020 came and past. Four more years passed. I think you can safely delete the 'deleteMeIn2020' vault.

By the way, I also deleted a 53-byte test file from an S3 bucket named 'test.txt' in a bucket in s3 called 'oseserverbackups' in us-west-2. It was the only file in the bucket. I deleted the file and the empty bucket.

Would you like me to proceed with deleting the 285.3 GB 'deleteMeIn2020' glacier bucket from your AWS account?


Thank you,

Michael Altfield
Senior Technology Advisor
PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B

Open Source Ecology
www.opensourceecology.org

meanwhile, I tried to figure out why I couldn't login as 'maltfield', and I realized that, ffs, we don't have IAM setup for our account?? Maybe Marcin deleted it when trying to elimiate costs? IAM is free, though..
ok, I found my 'maltfield' user under "Security Credentials" -> "Access Management" -> "Users"
it says my last console sign-in was 424 days ago
I went to my user's settings, selected the MFA token, and selected "Resync" -- then entered two consecutive OTPs
I tried to login, and this time it let me in. Well that was annoying.
I opened cloudtrail and reviewed the latest account events https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?ReadOnly=false
1. the most recent event was the 'root' user resyncing the MFA token of the 'matlfield' token
2. before that we have two ConsoleLogin for today
3. before that 'mjakubowski' user has a MakePayment event (and some other payment related events) on Sep 19
4. before that we have a bunch of login & mfa-related entries for Marcin's user on Sep 06, 14, 17, and 19.
5. and that's where the log ends; looks like we just get 90 days of logs for free.

...

hetzner responded to my support inquery about how they handle failed disks

Dear Mr Altfield

Unfortunately it's an unmanaged root server monitoring is your responsibility I'm afraid. 

If you have a problem please open a ticket in your robot account. 

Please click on "Servers" from the menu on the left and then select the corresponding server. Under the "Support" tab, you can choose "Hard drive is broken". Please follow the instructions.

https://docs.hetzner.com/robot/dedicated-server/troubleshooting/serial-numbers-and-information-on-defective-hard-drives/

Our DC is 24/7 available and we exchange broken hardware as soon as possible for free.

Hetzner clients can use the Server Monitoring System to monitor their servers and have an email sent to them when the status of one of the monitored services changes: 

https://docs.hetzner.com/robot/dedicated-server/security/system-monitor/

https://docs.hetzner.com/robot/dedicated-server/raid/software-raid/#email-notification-when-a-drive-in-a-software-raid-fails

Please use hetzner-status:

https://www.hetzner-status.de/en.html

This web page publishes announcements and current fault reports from our datacenters. Would you like to receive email notification of fault reports? Log on as exclusive Hetzner client in your administrations interface. 

If you have any questions please do not hesitate to contact us. 

Kind regards

Jan Kolb

Sales

Hetzner Online GmbH
Sigmundstrasse 135
90431 Nürnberg
Tel: +49 911 234 226-927
Fax: +49 9831 505-3
sales@hetzner.com
www.hetzner.com

Register Court: Registergericht Ansbach, HRB 6089
CEO: Martin Hetzner, Stephan Konvickova, Günther Müller

For the purposes of this communication, we may save some 
of your personal data. For information on our data privacy 
policy, please see: www.hetzner.com/datenschutzhinweis

09/29/2024 21:23 - marcin@opensourceecology.org michael@opensourceecology.org wrote:

>
>
> Hi Hetzner,
>
> Can you please tell us more about the process of disk failure on our new dedicated
> server plan (Server Auction #2443019)?
>
> Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk?
> Or do we have to pay a fee? If so, how much?
>
> And does Hetzner have some system in-place that monitors the hardware for disk
> failure? Or do we have to monitor this in software and alert Hetnzer that a disk
> is failing? If Hetzner does monitor for disk failure, how does it do it?
>
>
> Thank you,
>
> Michael Altfield
> Senior Technology Advisor
> PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B
>
> Open Source Ecology
> www.opensourceecology.org
>

the docs linked-to actually don't mention mdadm, which I setup earlier to monitor and send us email alerts on our disks
instead, hetzner mentions `smartctl`, which is included in the debian package `smartmontools` -- which wasn't even installed!

root@hetzner3 /etc/mdadm # sudo apt-get install smartmontools
...
root@hetzner3 /etc/mdadm # 

root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme1n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

root@hetzner3 /etc/mdadm #

we can get more information with the `-A` argument

root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        36 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    142.729.615 [73,0 TB]
Data Units Written:                 20.452.874 [10,4 TB]
Host Read Commands:                 6.862.184.005
Host Write Commands:                876.931.661
Controller Busy Time:               15.948
Power Cycles:                       28
Power On Hours:                     16.350
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      159
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               36 Celsius
Temperature Sensor 2:               45 Celsius

root@hetzner3 /etc/mdadm # 

root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme1n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    130.064.348 [66,5 TB]
Data Units Written:                 24.932.683 [12,7 TB]
Host Read Commands:                 1.276.781.490
Host Write Commands:                879.017.438
Controller Busy Time:               14.879
Power Cycles:                       23
Power On Hours:                     14.678
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      149
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               34 Celsius
Temperature Sensor 2:               37 Celsius

root@hetzner3 /etc/mdadm #

oh nvm, their third link describes mdadm alerts for monitoring our software raid
they also said to check /etc/default/mdadm, which I didn't do before

root@hetzner3 /etc/mdadm # cat /etc/default/mdadm 
# mdadm Debian configuration
#
# You can run 'dpkg-reconfigure mdadm' to modify the values in this file, if
# you want. You can also change the values here and changes will be preserved.
# Do note that only the values are preserved; the rest of the file is
# rewritten.
#

# AUTOCHECK:
#   should mdadm run periodic redundancy checks over your arrays? See
#   /etc/cron.d/mdadm.
AUTOCHECK=true

# AUTOSCAN:
#   should mdadm check once a day for degraded arrays? See
#   /etc/cron.daily/mdadm.
AUTOSCAN=true

# START_DAEMON:
#   should mdadm start the MD monitoring daemon during boot?
START_DAEMON=true

# DAEMON_OPTIONS:
#   additional options to pass to the daemon.
DAEMON_OPTIONS="--syslog"

# VERBOSE:
#   if this variable is set to true, mdadm will be a little more verbose e.g.
#   when creating the initramfs.
VERBOSE=false
root@hetzner3 /etc/mdadm #

note that "AUTOCHECK" is enabled -- so we're all good here.

...

ok, back to updating wordpress.
first, I'm just going to unzip all these (now TOFU-verified) .zip files and make sure there's no zipbombs

root@hetzner3 ~ # cd /var/tmp/wordpress/themes/
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # ls
bouquet.1.2.5.zip          sketch.1.2.4.zip      twentyfifteen.3.8.zip   twentyseventeen.3.7.zip  twentythirteen.4.2.zip
gk-portfolio.1.5.3.zip     storefront.4.6.0.zip  twentyfourteen.4.0.zip  twentysixteen.3.3.zip    twentytwelve.4.3.zip
portfolio-press.2.8.0.zip  twentyeleven.4.7.zip  twentynineteen.2.9.zip  twentyten.4.2.zip
root@hetzner3 /var/tmp/wordpress/themes #

root@hetzner3 /var/tmp/wordpress/themes # for file in $(ls *.zip); do unzip $file; done
...
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # ls
bouquet                 portfolio-press.2.8.0.zip  twentyeleven           twentyfourteen.4.0.zip   twentysixteen          twentythirteen.4.2.zip
bouquet.1.2.5.zip       sketch                     twentyeleven.4.7.zip   twentynineteen           twentysixteen.3.3.zip  twentytwelve
gk-portfolio            sketch.1.2.4.zip           twentyfifteen          twentynineteen.2.9.zip   twentyten              twentytwelve.4.3.zip
gk-portfolio.1.5.3.zip  storefront                 twentyfifteen.3.8.zip  twentyseventeen          twentyten.4.2.zip
portfolio-press         storefront.4.6.0.zip       twentyfourteen         twentyseventeen.3.7.zip  twentythirteen
root@hetzner3 /var/tmp/wordpress/themes #

root@hetzner3 /var/tmp/wordpress/themes # cd ../plugins/
root@hetzner3 /var/tmp/wordpress/plugins #

root@hetzner3 /var/tmp/wordpress/plugins # for file in $(ls *.zip); do unzip $file; done
...
root@hetzner3 /var/tmp/wordpress/plugins #

root@hetzner3 /var/tmp/wordpress/plugins # ls
akismet                                                 jetpack                               vcaching
akismet.5.3.3.zip                                       jetpack.13.8.1.zip                    vcaching.1.8.3.zip
black-studio-tinymce-widget                             meta-box                              w3-total-cache
black-studio-tinymce-widget.2.7.3.zip                   meta-box.5.10.2.zip                   w3-total-cache.2.7.6.zip
chartbeat                                               ml-slider                             wonderm00ns-simple-facebook-open-graph-tags
chartbeat.2.0.7.zip                                     ml-slider.3.91.0.zip                  wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip
classic-editor                                          open-in-new-window-plugin             woocommerce
classic-editor.1.6.5.zip                                open-in-new-window-plugin.3.0.zip     woocommerce.9.3.3.zip
coingate-for-woocommerce                                post-types-order                      wordpress-importer
coingate-for-woocommerce.2.1.1.zip                      post-types-order.2.2.6.zip            wordpress-importer.0.8.2.zip
contact-form-7                                          revision-control                      wordpress-seo
contact-form-7.5.9.8.zip                                revision-control.2.3.2.zip            wordpress-seo.23.5.zip
duplicate-page                                          shareaholic                           wpautop-control
duplicate-page.4.5.zip                                  shareaholic.9.7.12.zip                wpautop-control.1.6.zip
duplicate-post                                          share-on-diaspora                     wp-memory-usage
duplicate-post.4.5.zip                                  share-on-diaspora.0.7.9.zip           wp-memory-usage.1.2.10.zip
google-authenticator                                    shariff                               wp-optimize
google-authenticator.0.54.zip                           shariff.4.6.14.zip                    wp-optimize.3.6.0.zip
google-authenticator-encourage-user-activation          ssl-insecure-content-fixer            wp-smushit
google-authenticator-encourage-user-activation.0.2.zip  ssl-insecure-content-fixer.2.7.2.zip  wp-smushit.3.16.6.zip
insert-headers-and-footers                              varnish-http-purge                    wp-super-cache
insert-headers-and-footers.2.2.2.zip                    varnish-http-purge.5.2.2.zip          wp-super-cache.1.12.4.zip
root@hetzner3 /var/tmp/wordpress/plugins #

ok, that looks good. now let's see if we can script copying-over these themes as-needed
1. and, to err on the side of caution, I'm going to intentionally delete any theme or plugin dir, even if we don't have one to replace it.

wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	
	echo "${theme}"
	rm -rf ${theme_path};
	rsync -av --progress "/var/tmp/wordpress/themes/${theme}/" "${theme_path}/"
done

after execution, looks like it worked

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 68K
d---r-x--- 16 not-apache www-data 4,0K Oct  3 04:02 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
----r-----  1 not-apache www-data   28 Jun  5  2014 index.php
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 oshin
drwxr-xr-x  5 root       root     4,0K May 16 08:29 storefront
drwxr-xr-x  7 root       root     4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x  7 root       root     4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x  5 root       root     4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x  4 root       root     4,0K Jul 15 17:17 twentyten
drwxr-xr-x  8 root       root     4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:17 twentytwelve
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentyfour
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentythree
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

oh, wait, no. it created some silly empty dirs when it didn't have a source to copy-from

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/oshin/
total 8,0K
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 .
d---r-x--- 16 not-apache www-data 4,0K Oct  3 04:02 ..
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

let's wrap that in a condition. and also disable verbose & progress on rsync, so we can see the whole output

for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	source_path="/var/tmp/wordpress/themes/${theme}"
	
	echo "${theme}"
	rm -rf ${theme_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${theme_path}/"
	fi
done

here's the execution; that's better

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
        theme=$(basename "${theme_path}")
        source_path="/var/tmp/wordpress/themes/${theme}"
        
        echo "${theme}"
        rm -rf ${theme_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${theme_path}/"
        fi
done
twentytwelve
twentysixteen
storefront
twentyseventeen
twentyfourteen
twentyeleven
twentytwentythree
oshin
twentytwentyfour
twentythirteen
twentyten
twentyfifteen
twentynineteen
twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 52K
d---r-x--- 12 not-apache www-data 4,0K Oct  3 04:04 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
----r-----  1 not-apache www-data   28 Jun  5  2014 index.php
drwxr-xr-x  5 root       root     4,0K May 16 08:29 storefront
drwxr-xr-x  7 root       root     4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x  7 root       root     4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x  5 root       root     4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x  4 root       root     4,0K Jul 15 17:17 twentyten
drwxr-xr-x  8 root       root     4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:17 twentytwelve
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

now let's do the plugins with this

wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
	plugin=$(basename "${plugin_path}")
	source_path="/var/tmp/wordpress/plugins/${plugin}"
	
	echo "${plugin}"
	rm -rf ${plugin_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${plugin_path}/"
	fi
done

I actually messed this up, and I had to restore the original plugins dir from the backup; easy enough

rsync -av --progress /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/

alright, here's the run

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
        plugin=$(basename "${plugin_path}")
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        echo "${plugin}"
        rm -rf ${plugin_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done
meta-box-show-hide
classic-editor
be-portfolio-post
colorhub
ssl-insecure-content-fixer
oshine-core
tatsu
revslider
redux-vendor-support
akismet
rename-wp-login
meta-box-tabs
google-authenticator
coingate-for-woocommerce
be-gdpr
google-authenticator-encourage-user-activation
typehub
meta-box
woocommerce
meta-box-conditional-logic
contact-form-7
vcaching
force-strong-passwords
masterslider
oshine-modules
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah plugins/
total 56K
d---r-x--- 12       1012       48 4,0K Oct  3 04:09 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
drwxr-xr-x  4 root       root     4,0K Jul 10 22:16 akismet
drwxr-xr-x  3 root       root     4,0K Sep 27 21:51 classic-editor
drwxr-xr-x  8 root       root     4,0K Nov 21  2022 coingate-for-woocommerce
drwxr-xr-x  7 root       root     4,0K Jul 25 08:28 contact-form-7
drwxr-xr-x  3 root       root     4,0K Jul  4  2022 google-authenticator
drwxr-xr-x  4 root       root     4,0K Apr 23  2021 google-authenticator-encourage-user-activation
----r-----  1       1012       48 2,3K Apr  9  2019 hello.php
----r-----  1       1012       48   28 Apr  9  2019 index.php
drwxr-xr-x  8 root       root     4,0K Sep 27 07:22 meta-box
drwxr-xr-x  8 root       root     4,0K Mar 17  2024 ssl-insecure-content-fixer
drwxr-xr-x  4 root       root     4,0K Oct 21  2019 vcaching
drwxr-xr-x 13 root       root     4,0K Sep 25 13:56 woocommerce
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

with that, I tried wp-cli again, but it gave us an empty plugin list?

wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
+------+--------+--------+---------+----------------+-------------+
| name | status | update | version | update_version | auto_update |
+------+--------+--------+---------+----------------+-------------+
+------+--------+--------+---------+----------------+-------------+
wp@hetzner3:~$

oh shoot, I forgot to update permissions. I'll do that now

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do

	wp_docroot="$(dirname "${wordpress_site}")"
	vhost_dir="$(dirname "${wp_docroot}")"

	chown -R not-apache:www-data "${vhost_dir}"
	find "${vhost_dir}" -type d -exec chmod 0050 {} \;
	find "${vhost_dir}" -type f -exec chmod 0040 {} \;

	chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
	chmod 0040 "${vhost_dir}/wp-config.php"

	[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
	find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;

	[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
	find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;

done

ok, then I retry wp-cli; it works!

wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+------------------------------------------------+----------+--------+---------+----------------+-------------+
| name                                           | status   | update | version | update_version | auto_update |
+------------------------------------------------+----------+--------+---------+----------------+-------------+
| akismet                                        | inactive | none   | 5.3.3   |                | off         |
| classic-editor                                 | inactive | none   | 1.6.5   |                | off         |
| contact-form-7                                 | active   | none   | 5.9.8   |                | off         |
| google-authenticator-encourage-user-activation | active   | none   | 0.2     |                | off         |
| google-authenticator                           | active   | none   | 0.54    |                | off         |
| hello                                          | inactive | none   | 1.7.1   |                | off         |
| meta-box                                       | active   | none   | 5.10.2  |                | off         |
| ssl-insecure-content-fixer                     | active   | none   | 2.7.2   |                | off         |
| vcaching                                       | active   | none   | 1.8.3   |                | off         |
| woocommerce                                    | active   | none   | 9.3.3   |                | off         |
| coingate-for-woocommerce                       | inactive | none   | 2.1.1   |                | off         |
+------------------------------------------------+----------+--------+---------+----------------+-------------+
wp@hetzner3:~$

unfortunately, I get a blank page when I try to load store.opensourceecology.org in my web browser
nginx is fine, but the varnish logs show that apache is returning a 403

[Thu Oct 03 04:19:37.076411 2024] [authz_core:error] [pid 3116759:tid 3116768] [client 81.17.16.77:0] AH01630: client denied by server configuration: 
/var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png, referer: https://store.opensourceecology.org/

==> modsec_audit.log <==
--fd8c6d25-A--
[03/Oct/2024:04:19:37.076625 +0000] Zv4bWZVyO5GHCka9cecUKwAAAEE 127.0.0.1 40720 127.0.0.1 8000
--fd8c6d25-B--
GET /wp-includes/images/w-logo-blue-white-bg.png HTTP/1.1
X-Real-IP: 81.17.16.77
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: store.opensourceecology.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36
Accept: image/avif,image/webp,*/*
Accept-Language: en-US,en;q=0.5
Referer: https://store.opensourceecology.org/
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
Sec-GPC: 1
Pragma: no-cache
Accept-Encoding: gzip
hash: #store.opensourceecology.org
X-Varnish: 98343

--fd8c6d25-F--
HTTP/1.1 403 Forbidden
X-Frame-Options: SAMEORIGIN
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1

--fd8c6d25-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

--fd8c6d25-H--
Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png
Stopwatch: 1727929177076046 856 (- - -)
Stopwatch2: 1727929177076046 856; combined=26, p1=24, p2=0, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

--fd8c6d25-Z--

Maltfield Log/2024 Q4

Contents

See Also

Fri Dec 13, 2024

Thr Dec 12, 2024

Wed Dec 11, 2024

Tue Dec 10, 2024

Mon Oct 07, 2024

Sun Oct 06, 2024

Fri Oct 04, 2024

Thr Oct 03, 2024

Wed Oct 02, 2024

Navigation menu

Maltfield Log/2024 Q4

See Also

Fri Dec 13, 2024

Thr Dec 12, 2024

Wed Dec 11, 2024

Tue Dec 10, 2024

Mon Oct 07, 2024

Sun Oct 06, 2024

Fri Oct 04, 2024

Thr Oct 03, 2024

Wed Oct 02, 2024

Navigation menu

Search