Revision as of 22:33, 12 February 2025

My work log from the first quarter of the year 2025. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

Thr Feb 06, 2025

Here's TOFU 3/3 (ISP, exit in Ecuador)

Ecuador
2025-02-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. 
INFO: Determining Latest Version of Wordpress Themes 
	. . . . . 


https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
							  -=O=-                                    #  # ## 
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
							-=O=-                                 #   #  #  #  
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/astra.4.8.11.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/generatepress.3.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/oceanwp.4.0.5.zip
######################################################################### 100.0%
2025-02-06
5.9M	astra.4.8.11.zip
1.1M	generatepress.3.5.1.zip
8.0M	hestia.3.2.8.zip
7.1M	neve.4.0.1.zip
6.0M	oceanwp.4.0.5.zip
21M	phplist-3.6.15.tgz
29M	phplist-3.6.15.zip
32K	plugin.json
4.6M	v3.6.15.tar.gz
5.0M	v3.6.15.zip
1.4M	woocommerce-gateway-stripe.9.1.1.zip
28M	wordpress-6.7.1.zip
0e106338e48c9d4d023a46a368f0069b807ce2118a9bf51ddf04c76070867aba  astra.4.8.11.zip
a5e1d7478cce21b7bbef511bbea44156f59acd40cc7e51ad469c403013ba29f5  generatepress.3.5.1.zip
241b8c804ed1af72b1c9aa52f603730a52ebf7850383ac2e4d9dd163f6cfc3ca  hestia.3.2.8.zip
2b51e758d61b9d78ebd57d2afd9a967335bcb6866c5bff5a0d7157eecb6ec8cb  neve.4.0.1.zip
d1392dbe5f729178c968f5551855a412b41866e876dcf2a79ffd4e07298c4d13  oceanwp.4.0.5.zip
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15.tgz
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15.zip
69708af323f3bad6654dd7b52bd385df4f0fa600d79261b608626ce7109136ef  plugin.json
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15.tar.gz
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15.zip
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@disp4288:/tmp/tmp.aqIzX10FIC$

meld diff of these 3 TOFUs had two issues
1. hestia changed from v3.2.7 to v3.2.8
2. neve changed from v4.0.0 to v4.0.1
well that sucks; neve was the most important reason we were doing this.
so let's do this again. And this time I'm adding some translation plugins.
1. I recently did a ton of research (for my personal sites) for the best free wordpress i18nl plugins, and narrowed it down to 4, which I need to play-with
2. so I'm mostly doing this TOFU for my own benefit, but I'll do the comparison for free and I'll probably install the winner (but not activate it) on all of OSE's wordpress sites, in-case they want to have posts or pages in >1 language in the future (useful when doing collaboration with German, Italian, Spanish, etc-speaking orgs)
the most important result of this last 3TOFU is that we now finally have our phpList releases
unfortunately, there's 4 files and they all differ

user@ose:~/tmp/hetzner3/3tofu3$ sha256sum */*.zip
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15-zip/phplist-3.6.15.zip
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15-zip/v3.6.15.zip
user@ose:~/tmp/hetzner3/3tofu3$ sha256sum */*.tar.gz
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15-tar-gz/v3.6.15.tar.gz
user@ose:~/tmp/hetzner3/3tofu3$ sha256sum */*.tgz
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15-tgz/phplist-3.6.15.tgz
user@ose:~/tmp/hetzner3/3tofu3$

this is somewhat expected, especially from the compression algorithms of the archives being totally distinct
I confirmed that he actual contents of phplist-3.6.15.zip and phplist-3.6.15.tgz are identical

user@ose:~/tmp/hetzner3/3tofu3$ diff -r phplist-3.6.15-zip/phplist-3.6.15 phplist-3.6.15-tgz/phplist-3.6.15
user@ose:~/tmp/hetzner3/3tofu3$

and I also confirmed that the contents of the other set of files (I think these ones came from github -- as opposed to sourceforge) were also identical

user@ose:~/tmp/hetzner3/3tofu3$ diff -r v3.6.15-zip/phplist3-3.6.15/ v3.6.15-tar-gz/phplist3-3.6.15/
user@ose:~/tmp/hetzner3/3tofu3$

but it looks like there's lots of differences between the ones named 'phplist-3.6.15' and the ones named 'v3.6.15'

user@ose:~/tmp/hetzner3/3tofu3$ diff -r phplist-3.6.15-zip/phplist-3.6.15 v3.6.15-zip/phplist3-3.6.15/
Only in v3.6.15-zip/phplist3-3.6.15/bin: fake-sendmail.sh
Only in v3.6.15-zip/phplist3-3.6.15/bin: imgur-uploader.sh
Only in v3.6.15-zip/phplist3-3.6.15/bin: start-selenium
Only in v3.6.15-zip/phplist3-3.6.15/: composer.json
Only in v3.6.15-zip/phplist3-3.6.15/: composer.lock
Only in v3.6.15-zip/phplist3-3.6.15/: default.behat.yml
Only in v3.6.15-zip/phplist3-3.6.15/: .dotgitlab-ci.yml
Only in v3.6.15-zip/phplist3-3.6.15/: .github
Only in v3.6.15-zip/phplist3-3.6.15/: .gitignore
Only in v3.6.15-zip/phplist3-3.6.15/: .gitmodules
Only in v3.6.15-zip/phplist3-3.6.15/: .gitsvnextmodules
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin: help
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin: info
diff -r phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/init.php v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin/init.php
11c11,44
< define("VERSION","3.6.15");
---
> //## remove on rollout ###
> if (is_file(dirname(__FILE__).'/../../../VERSION')) {
>     $fd = fopen(dirname(__FILE__).'/../../../VERSION', 'r');
>     while ($line = fscanf($fd, '%[a-zA-Z0-9,. ]=%[a-zA-Z0-9,. ]')) {
>         list($key, $val) = $line;
>         if ($key == 'VERSION') {
>             $version = $val;
>         }
>     }
>     fclose($fd);
> } else {
>     $version = 'dev';
> }
> 
> if (!defined('VERSION')) {
>     if (!ini_get('open_basedir') && is_dir(dirname(__FILE__).'/../../../.git')) {
>         define('VERSION', $version.'-dev');
>         define('DEVVERSION', true);
>     } else {
>         define('VERSION', $version);
>         define('DEVVERSION', false);
>     }
> } else {
>     define('DEVVERSION', false);
> }
> 
> if (empty($GLOBALS['commandline']) && isset($GLOBALS['developer_email']) && $_SERVER['HTTP_HOST'] != 'dev.phplist.com' && !empty($GLOBALS['show_dev_errors'])) {
>     error_reporting(E_ALL);
>     ini_set('display_errors', 1);
>     foreach ($_REQUEST as $key => $val) {
>         unset($$key);
>     }
> }
> //## end remove on rollout ###
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: campaignslicer.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CaptchaPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CaptchaPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CKEditorPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CKEditorPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: Common
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CommonPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CommonPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: COPYING.txt
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: dateplaceholder.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: disposablemailblock.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: domainthrottlemap.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: embedremoteimages.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: inviteplugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: SegmentPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: SegmentPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: subjectLinePlaceholdersPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: UpdaterPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: UpdaterPlugin.php
diff -r phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/structure.php v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin/structure.php
16c16
<     define('STRUCTUREVERSION',"3.6.15");
---
>     define('STRUCTUREVERSION', 'dev');
Only in v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin: tests
Only in v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin/ui: default
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/ui: phplist-ui-bootlist
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists: base
diff -r phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/config/config.php v3.6.15-zip/phplist3-3.6.15/public_html/lists/config/config.php
35,38c35
< define('PHPMAILERHOST', 'localhost');
< define('PHPMAILERPORT',2500);
< define('PHPMAILER_SECURE',false);
< 
---
> define('PHPMAILERHOST', '');
44c41
< define('TEST', 0);
---
> define('TEST', 1);
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists: texts
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists: updater
Only in v3.6.15-zip/phplist3-3.6.15/: scripts
Only in v3.6.15-zip/phplist3-3.6.15/: TESTING.md
Only in v3.6.15-zip/phplist3-3.6.15/: tests
Only in v3.6.15-zip/phplist3-3.6.15/: Vagrantfile
user@ose:~/tmp/hetzner3/3tofu3$

I asked the community which release we should use. I'm thinking sourceforge, but I want to be sure
1. at first I'd think "don't use the one that has 'TEST' set to '1'), which is the sourceforge one
2. but then there's the code block that says "remove on rollout", which also is in the sourceforge one.
3. so basically there's a couple signs that say I should use one instead of the other and other signs that say I should use the other instead of one
I'll wait to hear back from the phpList team before proceeding
anyway, we do have all the files we need. 3TOFU is done for phpList
...
here's our new 3TOFU script

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --location --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='polylang translatepress-multilingual google-language-translator gtranslate'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

themes='hestia neve'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
du -sh *
sha256sum *

and here's TOFU 1/3 (Tor, exit in Poland)

Congratulations. This browser is configured to use Tor.
2025-02-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . . . 
INFO: Determining Latest Version of Wordpress Themes 
	. . 


https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/polylang.3.6.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/translatepress-multilingual.2.9.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-language-translator.6.0.20.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/gtranslate.3.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.1.zip
######################################################################### 100.0%
2025-02-06
296K	google-language-translator.6.0.20.zip
700K	gtranslate.3.0.7.zip
8.0M	hestia.3.2.8.zip
7.1M	neve.4.0.1.zip
64K	plugin.json
592K	polylang.3.6.6.zip
2.1M	translatepress-multilingual.2.9.4.zip
28M	wordpress-6.7.1.zip
01246c9c90f1373ee83d8e5884e5abba264c62e7301d99a468fd8eb95144be05  google-language-translator.6.0.20.zip
171eb362801ea28e74b302cd0fee472dbd5025a89f6c1634b5ca1029362a678c  gtranslate.3.0.7.zip
241b8c804ed1af72b1c9aa52f603730a52ebf7850383ac2e4d9dd163f6cfc3ca  hestia.3.2.8.zip
2b51e758d61b9d78ebd57d2afd9a967335bcb6866c5bff5a0d7157eecb6ec8cb  neve.4.0.1.zip
4a87243dd3afb7c15bc8e1c51df6f63604996110691caa0d80d2119ccc59057a  plugin.json
3a4a7f8872d16cb3538e948b4f85acee7d823c04fe8c820259ef6e2735093bbb  polylang.3.6.6.zip
16f2e24d14af341c6208fb9cc08d0c4db33bf813a5b1668930e30c6179a1111c  translatepress-multilingual.2.9.4.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@host:/tmp/user/1000/tmp.JC5exXrKLS$

...
I was supposed to have a meeting with Marcin & Catarina today, but they asked to reschedule
so I'm blocked on phplist, obi, and osemain
let's continue with the wiki!
ok, here's the last error we got (when trying to do the second upgrade)

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Cite extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(282): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Cite/extension.json: filemtime(): stat failed for /Cite/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,077s
user    0m0,002s
sys     0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

I fixed this with

# don't yet load Cite (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Cite\(.*\)%\1#wfLoadExtension\2Cite\3%' ${vhostDir}/LocalSettings.php

subsequent upgrade attempts resulted in an error with the interwiki extension

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Interwiki extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(289): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Interwiki/extension.json: filemtime(): stat failed for /Interwiki/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,052s
user    0m0,008s
sys     0m0,000s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

fixed with

# don't yet load Interwiki (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Interwiki\(.*\)%\1#wfLoadExtension\2Interwiki\3%' ${vhostDir}/LocalSettings.php

then it complains about Gadgets

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Gadgets extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(308): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Gadgets/extension.json: filemtime(): stat failed for /Gadgets/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,051s
user    0m0,004s
sys     0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

fixed with

# don't yet load Gadgets (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Gadgets\(.*\)%\1#wfLoadExtension\2Gadgets\3%' ${vhostDir}/LocalSettings.php

then ReplaceText

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The ReplaceText extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(311): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /ReplaceText/extension.json: filemtime(): stat failed for /ReplaceText/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,053s
user    0m0,002s
sys     0m0,007s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

fixed with

# don't yet load ReplaceText (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)ReplaceText\(.*\)%\1#wfLoadExtension\2ReplaceText\3%' ${vhostDir}/LocalSettings.php

then RenameUser

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Renameuser extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(314): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Renameuser/extension.json: filemtime(): stat failed for /Renameuser/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,052s
user    0m0,004s
sys     0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

fixed with

# don't yet load Renameuser (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Renameuser\(.*\)%\1#wfLoadExtension\2Renameuser\3%' ${vhostDir}/LocalSettings.php

tried again, and it complained about Nuke

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Nuke extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(326): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Nuke/extension.json: filemtime(): stat failed for /Nuke/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,053s
user    0m0,003s
sys     0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

this seems a bit excessive; these dirs are in-place

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # ls -lah ${vhostDir}/htdocs/extensions
total 144K
d---r-x--- 35 not-apache www-data 4,0K Dec 29 18:14 .
d---r-x--- 14 not-apache www-data 4,0K Feb  2 00:39 ..
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 AbuseFilter
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 CategoryTree
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 Cite
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 CiteThisPage
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 CodeEditor
d---r-x--- 13 not-apache www-data 4,0K Dec 29 18:14 ConfirmEdit
d---r-x---  9 not-apache www-data 4,0K Dec 29 18:14 DiscussionTools
d---r-x--- 10 not-apache www-data 4,0K Dec 29 18:14 Echo
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 Gadgets
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 ImageMap
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 InputBox
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 Interwiki
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 Linter
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 LoginNotify
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 Math
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 MultimediaViewer
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 Nuke
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 OATHAuth
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 PageImages
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 ParserFunctions
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 PdfHandler
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 Poem
----r-----  1 not-apache www-data 1,1K Dec  5 15:41 README
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 ReplaceText
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 Scribunto
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 SecureLinkFixer
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 SpamBlacklist
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 SyntaxHighlight_GeSHi
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 TemplateData
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 TextExtracts
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 Thanks
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 TitleBlacklist
d---r-x--- 12 not-apache www-data 4,0K Dec 29 18:14 VisualEditor
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 WikiEditor
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

I thought maybe there was another issue with the path to the 'extensions' dir itself, but that looks like I put that in-place already

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep wgExtensionsDirectory ${vhostDir}/LocalSettings.php
$wgExtensionsDirectory = "$IP/extensions";
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

ah, crap, I just realized the var name according to the wiki is wgExtensionDirectory, not wgExtensionsDirectory https://www.mediawiki.org/wiki/Manual:$wgExtensionDirectory
I updated this, and now I get a new error

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
PHP Fatal error:  Uncaught FatalError: $wgBaseDirectory must not be modified in settings files! Use the MW_INSTALL_PATH environment variable to override the installation root directory. in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php:274
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once()
#1 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 274

real    0m0,053s
user    0m0,008s
sys     0m0,000s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

ok, I don't understand this error; we are *not* using that var

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep wgBaseDirectory ${vhostDir}/LocalSettings.php
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

here's where the error came-from

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep -C4 wgBaseDirectory ${vhostDir}/htdocs/includes/Setup.php 
 *     from executing.
 *
 * This file does:
 * - run-time environment checks,
 * - define MW_INSTALL_PATH, $IP, and $wgBaseDirectory,
 * - load autoloaders, constants, default settings, and global functions,
 * - load the site configuration (e.g. LocalSettings.php),
 * - load the enabled extensions (via ExtensionRegistry),
 * - trivial expansion of site configuration defaults and shortcuts
--
if ( defined( 'MW_AUTOLOAD_TEST_CLASSES' ) ) {
        require_once __DIR__ . '/../tests/common/TestsAutoLoader.php';
}

if ( $wgBaseDirectory !== MW_INSTALL_PATH ) {
        throw new FatalError(
                '$wgBaseDirectory must not be modified in settings files! ' .
                'Use the MW_INSTALL_PATH environment variable to override the installation root directory.'
        );
}

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

we didn't modify that file; it's identical to the file as included in the release of mediawiki-1.43.0

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # diff ${vhostDir}/htdocs/includes/Setup.php /var/tmp/mediawiki/mediawiki-1.43.0/includes/Setup.php 
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

ok, looks like we do try to set the "IP" dir

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep -C4 MW_INSTALL_PATH ${vhostDir}/LocalSettings.php
# http://www.mediawiki.org/wiki/Manual:Configuration_settings

# If you customize your file layout, set $IP to the directory that contains
# the other MediaWiki files. It will be used as a base to locate files.
if( defined( 'MW_INSTALL_PATH' ) ) {
        $IP = MW_INSTALL_PATH;
} else {
        $IP = dirname( __FILE__ ) . "/htdocs" ;
}

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

the docs on the IP var suggest that you used to be able to update it, but that won't work since Mediawiki >=v1.18
let's see what happens if we comment-out all these lines. if it still has issues, I guess we should set 'MW_INSTALL_PATH' to `dirname( __FILE__ ) . "/htdocs"` instead
well, after commenting them out, I *still* have an error complaining about this setting

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
PHP Fatal error:  Uncaught FatalError: $wgBaseDirectory must not be modified in settings files! Use the MW_INSTALL_PATH environment variable to override the installation root directory. in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php:274
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once()
#1 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 274

real    0m0,055s
user    0m0,004s
sys     0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

looking at the error above it, I wonder if it's possible that the issue is coming from here

root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep 'DefaultSettings.php' LocalSettings.php 
# See includes/DefaultSettings.php for all configurable settings
require_once( "$IP/includes/DefaultSettings.php" );
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

nah, looks like this file is present both in our docroot *and* in the release dir. And they're identical

root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/includes/DefaultSettings.php 
htdocs/includes/DefaultSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /var/tmp/mediawiki/mediawiki-1.43.0/includes/DefaultSettings.php
/var/tmp/mediawiki/mediawiki-1.43.0/includes/DefaultSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org # diff htdocs/includes/DefaultSettings.php /var/tmp/mediawiki/mediawiki-1.43.0/includes/DefaultSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

ok, yeah, the only thing in this file is the error about using this file

root@hetzner3 /var/www/html/wiki.opensourceecology.org # cat htdocs/includes/DefaultSettings.php 
<?php
/**
 * THIS IS A DEPRECATED STUB FILE!
 *
 * Default settings are now defined in the MainConfigSchema class.
 *
 * To get default values for configuration variables, use MainConfigSchema::listDefaultValues()
 * or MainConfigSchema::getDefaultValue().
 *
 * @file
 * @deprecated since 1.39
 */

use MediaWiki\MainConfigSchema;

if ( function_exists( 'wfDeprecatedMsg' ) ) {
        wfDeprecatedMsg(
                'DefaultSettings.php is deprecated and will be removed. '
                . 'Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead.',
                '1.39'
        );
}

// Extract the defaults into the current scope
foreach ( MainConfigSchema::listDefaultValues( 'wg' ) as $defaultSettingsVar => $defaultSettingsValue ) {
        $$defaultSettingsVar = $defaultSettingsValue;
}

unset( $defaultSettingsVar );
unset( $defaultSettingsValue );
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

oh, actually there is some stuff at the end
yeah, that *was* it. If I comment-out that line, then it continues

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
MediaWiki 1.43.0 Updater

Your composer.lock file is up to date with current dependencies!
Going to run database updates for osewiki_db-wiki_
Depending on the size of your database this may take a while!
Abort with control-c in the next five seconds (skip this countdown with --quick) ...0
Can not upgrade from versions older than 1.35, please upgrade to that version or later first.

real    0m5,144s
user    0m0,004s
sys     0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

but, crap, this is the same problem we had before. It claims we didn't upgrade yet. Even though we did!
maybe the update failed somewhere in the middle because of this same error about DefaultSettings.php
I updated the wiki CHG to store all the output from the upgrade runs to log files in the $chg_dir so we can review it after it runs
I'm also pretty annoyed about these huge permissions commands being stored in all the CHG files separately; I'm just going to create a script in /usr/local/bin/ for this
ok, let's restore to our snapshot and try the whole process again
ugh, for some reason I got a new error this time

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat $chg_dir/update-to-v1.35.log 
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Fatal error:  Uncaught Exception: Unable to open file /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha/extension.json: filemtime(): stat failed for /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php:177
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(71): ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(257): wfLoadExtensions()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(143): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(91): require_once('...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#6 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php on line 177
PHP Fatal error:  Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63
Stack trace:
#0 [internal function]: MediaWiki\HeaderCallback::callback()
#1 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

curiously, the path to the ReCaptcha plugin is present; just the ReCaptcha dir is missing
1. I hate ReCaptcha, and I noticed there's a dir for hCaptcha instead

root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha
ls: cannot access '/var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha': No such file or directory
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit
total 140K
d---r-x--- 13 not-apache www-data 4,0K Dec 26 23:53 .
d---r-x--- 30 not-apache www-data 4,0K Dec 26 23:53 ..
----r-----  1 not-apache www-data 3,1K Jun  7  2019 AUTHORS.txt
----r-----  1 not-apache www-data  510 Sep 25  2020 blacklist
----r-----  1 not-apache www-data 9,8K Sep 24  2020 captcha-old.py
----r-----  1 not-apache www-data  11K Sep 24  2020 captcha.py
----r-----  1 not-apache www-data  135 Sep 25  2020 CODE_OF_CONDUCT.md
----r-----  1 not-apache www-data  439 Sep 25  2020 composer.json
----r-----  1 not-apache www-data 2,7K Sep 25  2020 ConfirmEdit.alias.php
----r-----  1 not-apache www-data  18K Dec 12  2018 COPYING
----r-----  1 not-apache www-data 4,4K Sep 25  2020 extension.json
d---r-x---  5 not-apache www-data 4,0K Dec 26 23:53 FancyCaptcha
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 hCaptcha
d---r-x---  3 not-apache www-data  12K Dec 26 23:53 i18n
d---r-x---  5 not-apache www-data 4,0K Dec 26 23:53 includes
d---r-x---  2 not-apache www-data 4,0K Dec 26 23:53 maintenance
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 MathCaptcha
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 QuestyCaptcha
----r-----  1 not-apache www-data 5,1K Sep 25  2020 README.md
d---r-x---  5 not-apache www-data 4,0K Dec 26 23:53 ReCaptchaNoCaptcha
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 resources
d---r-x---  3 not-apache www-data 4,0K Dec 26 23:53 SimpleCaptcha
d---r-x---  3 not-apache www-data 4,0K Dec 26 23:53 tests
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

whatever, I'm going to disable this pesky extension again during our upgrades
ok, the upgrade appears to have exited with an error 27 minutes-in

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.35.log

real    27m59,071s
user    0m0,008s
sys     0m0,000s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

here's the error log

root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail -n40 /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.35.log
Completed migration, inserted 1 row(s) with 0 new actor(s), 0 error(s)
errors were encountered.
Modifying rev_text_id field of table revision ...done.
Modifying table site_stats ...done.
Populating ar_rev_id.
Populating ar_rev_id...
MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1217): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1528): MessageCache->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1167): WANObjectCache->getWithSetCallback()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1163): BagOStuff->getWithSetCallback()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1106): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1016): MessageCache->getMsgFromNamespace()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(927): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(661): Message->text()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(213): Title::newMainPage()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(118): PopulateArchiveRevId::makeDummyRevisionRow()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(63): PopulateArchiveRevId::checkMysqlAutoIncrementBug()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(45): PopulateArchiveRevId->doDBUpdates()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1377): LoggedUpdateMaintenance->execute()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(512): DatabaseUpdater->populateArchiveRevId()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(475): DatabaseUpdater->runUpdates()
#29 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(181): DatabaseUpdater->doUpdates()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(107): UpdateMediaWiki->execute()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

I believe this is why we had the second upgrade command; let's try that and then maybe try the first command again

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php" &> ${chg_dir}/populateContentTables-to-v1.35.log

real    0m13,673s
user    0m0,004s
sys     0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

that only took 13 seconds; let's do the first command again

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.35b.log

real    1m52,641s
user    0m0,004s
sys     0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

cool, on double-tap it finished in less than 2 minutes, and this time it ended with a "Done" message instead of an error

root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.35b.log
el_id 30000 - 40000 of 76644
el_id 40000 - 50000 of 76644
el_id 50000 - 60000 of 76644
el_id 60000 - 70000 of 76644
el_id 70000 - 76644 of 76644
Done, 689 rows updated, 2 deleted.
Set the local repo temp zone container to be private.
Purging caches...done.

Done in 1 min 48 s.
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

now I went to the version page, and I confirmed that it lists the MediaWiki version as "1.35.0". Perfect! https://wiki.opensourceecology.org/wiki/Special:Version

Installed software
Product 	Version
MediaWiki 	1.35.0
PHP 	8.2.26 (fpm-fcgi)
MariaDB 	10.11.6-MariaDB-0+deb12u1
ICU 	72.1

Sun Feb 02, 2025

Here's TOFU 2/3 (VPN, exit in US)

United States
2025-02-03
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. 
INFO: Determining Latest Version of Wordpress Themes 
	. . . . . 


https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
							   -=O=-                                #  #  # #  
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
						  -=O=-                             #    #   #   #     
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/astra.4.8.11.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/generatepress.3.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/oceanwp.4.0.5.zip
######################################################################### 100.0%
2025-02-03
5.9M	astra.4.8.11.zip
1.1M	generatepress.3.5.1.zip
8.0M	hestia.3.2.7.zip
7.1M	neve.4.0.0.zip
6.0M	oceanwp.4.0.5.zip
21M	phplist-3.6.15.tgz
29M	phplist-3.6.15.zip
32K	plugin.json
4.6M	v3.6.15.tar.gz
5.0M	v3.6.15.zip
1.4M	woocommerce-gateway-stripe.9.1.1.zip
28M	wordpress-6.7.1.zip
0e106338e48c9d4d023a46a368f0069b807ce2118a9bf51ddf04c76070867aba  astra.4.8.11.zip
a5e1d7478cce21b7bbef511bbea44156f59acd40cc7e51ad469c403013ba29f5  generatepress.3.5.1.zip
cf89c4f50301f2d030ae3662d4c8b4359445b8ee31fcdd5f67cb0c2c1c50adf8  hestia.3.2.7.zip
cb4cee1ad2cdd020729884d3bd97ecabd16194f3388e301bc9164c3f980d10a3  neve.4.0.0.zip
d1392dbe5f729178c968f5551855a412b41866e876dcf2a79ffd4e07298c4d13  oceanwp.4.0.5.zip
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15.tgz
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15.zip
c0a5eabed1f1317a58dcaeff254865764c071eb03aa670c0084b3dc6767de489  plugin.json
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15.tar.gz
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15.zip
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@disp2327:/tmp/tmp.0MQDfRxGyc$

Sat Feb 01, 2025

Catarina responded to my email.
It sounds like she doesn't want to pay someone to convert the existing osemain to a new theme because she wants to do a complete relaunch of the website

Sorry I didn't explain myself. What I meant is that regardless of any
issues with the migration, we were planning to re-do OSE's main website
because: 1) it's outdated and needs a refresh, and 2) we're about to launch
a new phase of the project and need a website that reflects that. There is
no point in hiring someone to recreate the old site when we're going to
create a new one anyway. We just need the old one to be online while we
work on the new one (I guess?)

OBI's website is outdated too (in terms of content) and a new website is in
the cards, but I just don't know if we want to tackle that right now or
just let the old one in place for a while longer. Regardless of timing, I
agree that the page-builder thing is not the way to go and we would want to
switch to what you suggest (native wordpress galleries and a popular theme)
when we update the website. The question is just whether to do that now or
later.

I'll take a look then. Can you send me the instructions? My main computer
runs linux, but I've been having trouble with the browser on that one. My
laptop runs windows 11 (🤢).

On Sat, Feb 1, 2025 at 10:08 AM Michael Altfield <REDACTED>
wrote:

>  > 1) If we decide to create a new website for OBI, what is the best
>  > way to access the visual assets (images, animations, graphics)
>  > from the old site?
>  > I think we'd want to re-use quite a few.
>
> I'm not an expert at wordpress design, but I think what you've built
> with "portfolios" and the "be-page-builder" is non-trivial to replace. I
> was hoping (since you built it) you would know better after looking at
> it -- maybe it's just some quirk of oshine that you know how to quickly
> fix? I think you should take a look at it.
>
> The OSE site would be pretty cheap to be re-built with a more popular
> theme (I recommend Neve), but I think it would be quite a large
> undertaking for the OBI website.
>
> Migrating the images is fine, but the interactive tabs is the type of
> complexity that I'd avoid. But if you just want to replace all that with
> native wordpress galleries, that's not too hard.
>
> What I'm suggesting for the "osemain" (www.opensourceecology.org)
> website is that we don't create a "new" site -- but we just migrate the
> site over to our new server, switch to a popular, flexible, lightweight,
> and freemium theme (I recommend Neve), and then hire some freelancer to
> hack the theme config and content until it looks more-or-less identical
> to the old site, but using only wordpress-built-in Gutenberg blocks --
> so we don't have this issue in the future.
>
>  > 2) If we decide that it doesn't make sense to create a new website
>  > for OBI at this moment, would it be possible to turn the existing
>  > one into a static site (like you did for FeF)? If so, how much
>  > time would that take you?
>
> This is pretty trivial. It depends a lot on the website, but it took me
> a few hours for fef and half a day for oswh.
>
> I can definitely do it for OBI, but then you can't edit the content
> anymore. Of course, as with the other sites that I'm converting to
> static-site, I'll make a full backup of the db and files, so you'd be
> able to use these backups to create a new DB and wordpress site to
> create the website again in wordpress. This restored site would have the
> same rendering issues I'm describing, but all the content will be
> present and you could then try to fix the rendering issues within oshine
> or switch to another theme and rebuild it, as I'm suggesting for osemain.
>
> But I think it would be best if you just access the site to know what
> I'm talking about. Are you running linux? Do you have firefox installed?
>
>
> Cheers,
>
> Michael Altfield
> https://www.michaelaltfield.net
> PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41
>
> Note: If you cannot reach me via email, please check to see if I have
> changed my email address by visiting my website at
> https://email.michaelaltfield.net
>
> On 2/1/25 10:43, Catarina Mota wrote:
>> Thank you Michael. I was planning on re-doing OSE's site very soon, but
> was
>> hoping OBI's would hang in there a while longer.
>>
>> A couple questions:
>>
>> 1) If we decide to create a new website for OBI, what is the best way to
>> access the visual assets (images, animations, graphics) from the old
> site?
>> I think we'd want to re-use quite a few.
>>
>> 2) If we decide that it doesn't make sense to create a new website for
> OBI
>> at this moment, would it be possible to turn the existing one into a
> static
>> site (like you did for FeF)? If so, how much time would that take you?
>>
>> Thank you,
>> Catarina
>>
>> On Fri, Jan 31, 2025 at 4:44 PM Michael Altfield <REDACTED>
>> wrote:
>>
>>> Hey Catarina,
>>>
>>> I just finished migrating a snapshot of OBI to hetzner3. Can you please
>>> review it?
>>>
>>> I'm sorry to say that OBI didn't handle the upgrades very well. About
>>> half of the content is missing. I was able to fix some of the issues by
>>> changing the web server config, but it appears that most of the issues
>>> are theme- or page-builder-related, and I'll need to defer to you as the
>>> SME.
>>>
>>> Can you please tell me:
>>>
>>> 1. What OS does your computer use?
>>> 2. What web browser do you use?
>>>
>>> And I can send instructions for you to view/edit the OBI site on
> hetzner3.
>>>
>>>
>>> Thank you,
>>>
>>> Michael Altfield
>>> https://www.michaelaltfield.net
>>> PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41
>>>
>>> Note: If you cannot reach me via email, please check to see if I have
>>> changed my email address by visiting my website at
>>> https://email.michaelaltfield.net

before I send her instructions for accessing the websites on hetzner3, I created some files named 'is_hetzner3' at the docroot of each site, so we can confirm it easily

echo "true" > /var/www/html/fef.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/www.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/microfactory.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/forum.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/store.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/wiki.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/www.openbuildinginstitute.org/htdocs/is_hetzner3
echo "true" > /var/www/html/seedhome.openbuildinginstitute.org/htdocs/is_hetzner3
echo "true" > /var/www/html/oswh.opensourceecology.org/htdocs/is_hetzner3

I sent an email describing how to access the new sites

So the way your computer knows to connect to the old server (hetzner2) vs the new server (hetzner3) is via DNS.

You can override your DNS settings locally on your computer by editing the /etc/hosts file.

Add these lines to your /etc/hosts file

144.76.164.201 forum.opensourceecology.org
144.76.164.201 store.opensourceecology.orgc
144.76.164.201 microfactory.opensourceecology.org
144.76.164.201 seedhome.openbuildinginstitute.org
144.76.164.201 fef.opensourceecology.org
144.76.164.201 oswh.opensourceecology.org
144.76.164.201 www.openbuildinginstitute.org
144.76.164.201 openbuildinginstitute.org
144.76.164.201 www.opensourceecology.org
144.76.164.201 opensourceecology.org
144.76.164.201 phplist.opensourceecology.org
144.76.164.201 wiki.opensourceecology.org

You can edit the /etc/hosts file with nano, vim, or your fav editor:

  sudo nano /etc/hosts
  sudo vim /etc/hosts

Windows also has a hosts file, but it's instead located at 

  C:\Windows\System32\Drivers\etc\hosts

 * https://www.wikihow.com/Edit-the-Hosts-File-on-Windows

After making that change, open firefox and tell it to clear its cache by going to about:networking -> DNS -> Clear DNS Cache

 * https://www.groovypost.com/howto/flush-dns-cache-on-windows-11/

Now when you visit any of the OSE sites listed in your hosts file, you'll view it on the new server (hetzner3).

To confirm that you're viewing the site on the new server, I've created a file called '/is_hetnzer3' with the contents "true" on every host. For example:

 * https://www.openbuildinginstitute.org/is_hetzner3

If ^ that page returns a 404 error, you're on hetzner2 (the old server).

If ^ that page returns a page that says "true", then you're on hetzner3 (the new server).

When you're finished, just delete those lines from your hosts file and clear your DNS cache again, and you'll be able to access the sites on hetzner2 again.

Note that these sites have been migrated as a snapshot. They will be deleted. Feel free to edit them however you want, but know that anything you change will be deleted.

Please review all of the webpages of obi and osemain. Login, make test changes, and let me know how you want to proceed.


Thank you,

Michael Altfield
https://www.michaelaltfield.net
PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41

Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net

...
at this point, I'm thinking we're probably going to make static sites of both bi and osemain and make those the "live" sites on hetzner3 when we migrate
1. ...but we'll *also* migrate the actual wordpress site. To avoid the name collision, we can just expose it on the Internet over a different IP address, which DNS doesn't point-to
2. either we can buy another IPv4 address (expensive, but cheaper than having a second server or keeping hetzner2 around) or we can try to see if Marcin can access it over IPv6
3. that will buy them some time, so they can hack away at a new site on hetzner3 as much as they want without the old site going down. it just means a content freeze until they finish with the new site
4. and when they do finally finish it, we just have to make a simple nginx config change to swap the old static site for the new site
Catarina wrote me back saying she'd review the sites and discuss with Marcin how they want to proceed
I'm hoping that we can have a call and I can explain my suggestion above after they've had time to review it, so they have a better understanding of how it's broken
anyway, I'm blocked on obi and osemain until they've reviewed it
...
I finished my logs and hours and sent a bill to OSE for 18 hours in the month of Jan 2025
...
next-up is phplist
I 3TOFU'd two distinct files for phpList

root@hetzner3 /var/tmp/phplist # sha256sum *
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
root@hetzner3 /var/tmp/phplist #

it's surprising that the tarball and the zip file have the same hash; that tells me the file extension is probably a lie
but I don't understand why the two versions (I think one came from sourceforge and one came from github) don't match
ok, apparently neither is a gzip tarball file

root@hetzner3 /var/tmp/phplist # tar -xzvf phplist-3.6.15.zip

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
root@hetzner3 /var/tmp/phplist # tar -xzvf phplist-3.6.15.tgz 

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
root@hetzner3 /var/tmp/phplist #

wtf, neither is a zip file either

root@hetzner3 /var/tmp/phplist # unzip phplist-3.6.15.tgz
Archive:  phplist-3.6.15.tgz
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of phplist-3.6.15.tgz or
		phplist-3.6.15.tgz.zip, and cannot find phplist-3.6.15.tgz.ZIP, period.
root@hetzner3 /var/tmp/phplist # unzip phplist-3.6.15.zip
Archive:  phplist-3.6.15.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of phplist-3.6.15.zip or
		phplist-3.6.15.zip.zip, and cannot find phplist-3.6.15.zip.ZIP, period.
root@hetzner3 /var/tmp/phplist #

ah, crap, they look empty

root@hetzner3 /var/tmp/phplist # du -sh *
4,0K    phplist-3.6.15.tgz
4,0K    phplist-3.6.15.zip
0       v3.6.15.tar.gz
0       v3.6.15.zip
root@hetzner3 /var/tmp/phplist #

yeah, two of the files are literally empty and the other two are just 301 redirect errors. crap

root@hetzner3 /var/tmp/phplist # for file in $(ls -1); do echo $file; cat $file; echo; done
phplist-3.6.15.tgz
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

phplist-3.6.15.zip
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

v3.6.15.tar.gz

v3.6.15.zip

root@hetzner3 /var/tmp/phplist #

yeah, so apparently I need to update the curl command in my 3TOFU script

user@disp6255:/tmp/tmp.t8DUYsEVAi$ CURL="/usr/bin/curl --retry 5 --retry-all-errors"
user@disp6255:/tmp/tmp.t8DUYsEVAi$ ${CURL} --progress-bar -O https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
user@disp6255:/tmp/tmp.t8DUYsEVAi$ ls
phplist-3.6.15.tgz
user@disp6255:/tmp/tmp.t8DUYsEVAi$ cat phplist-3.6.15.tgz 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp6255:/tmp/tmp.t8DUYsEVAi$

let's try this instead

CURL="/usr/bin/curl --retry 5 --retry-all-errors --location"
${CURL} --progress-bar -O https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz

ok, that worked

user@disp6255:/tmp/tmp.t8DUYsEVAi$ rm phplist-3.6.15.tgz 
user@disp6255:/tmp/tmp.t8DUYsEVAi$ ls
user@disp6255:/tmp/tmp.t8DUYsEVAi$ CURL="/usr/bin/curl --retry 5 --retry-all-errors --location"
${CURL} --progress-bar -O https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
user@disp6255:/tmp/tmp.t8DUYsEVAi$ du -sh *
21M	phplist-3.6.15.tgz
user@disp6255:/tmp/tmp.t8DUYsEVAi$

that also appears to fix the github files too
I updated the hetzner3:/usr/local/bin/wordpress_3tofu.sh file to include a `du -sh` and `--location` arg to CURL to prevent this in the future
alright, let's do another 3TOFU for phpList and also for some wordpress themes to pre-install for the replacement obi & osemain sites (might not be needed)
here's the 3TOFU script

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --location --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES="https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz"
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='woocommerce-gateway-stripe'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

themes='astra generatepress hestia neve oceanwp'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
du -sh *
sha256sum *

and here's TOFU 1/3 (Tor, exit in Poland)

Congratulations. This browser is configured to use Tor.
2025-02-01
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. 
INFO: Determining Latest Version of Wordpress Themes 
	. . . . . 


https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
				   #    #     #     #                -=O=-                     
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
# #                                                                      -=O=- 
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/astra.4.8.11.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/generatepress.3.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/oceanwp.4.0.5.zip
######################################################################### 100.0%
2025-02-01
5.9M	astra.4.8.11.zip
1.1M	generatepress.3.5.1.zip
8.0M	hestia.3.2.7.zip
7.1M	neve.4.0.0.zip
6.0M	oceanwp.4.0.5.zip
21M	phplist-3.6.15.tgz
29M	phplist-3.6.15.zip
32K	plugin.json
4.6M	v3.6.15.tar.gz
5.0M	v3.6.15.zip
1.4M	woocommerce-gateway-stripe.9.1.1.zip
28M	wordpress-6.7.1.zip
0e106338e48c9d4d023a46a368f0069b807ce2118a9bf51ddf04c76070867aba  astra.4.8.11.zip
a5e1d7478cce21b7bbef511bbea44156f59acd40cc7e51ad469c403013ba29f5  generatepress.3.5.1.zip
cf89c4f50301f2d030ae3662d4c8b4359445b8ee31fcdd5f67cb0c2c1c50adf8  hestia.3.2.7.zip
cb4cee1ad2cdd020729884d3bd97ecabd16194f3388e301bc9164c3f980d10a3  neve.4.0.0.zip
d1392dbe5f729178c968f5551855a412b41866e876dcf2a79ffd4e07298c4d13  oceanwp.4.0.5.zip
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15.tgz
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15.zip
88ee2ac6c1c04d4ebbae269d1117b6727a7152a6dd683aa444e4452333cd9c13  plugin.json
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15.tar.gz
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15.zip
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@host:/tmp/user/1000/tmp.WJ1lwTXhlC$

...
ok, back to the wiki
I'm not sure where we left-off. I don't think I was ever able to successfully do the double upgrade
previously I couldn't figure out the MediaWiki version without logging-in and dumping it from some special page
well, if I try to load the login page, get an error https://wiki.opensourceecology.org/index.php?title=Special:UserLogin

[Z56tEJjVN3m_4ExsymEpZwAAAAQ] 2025-02-01 23:24:01: Fatal exception of type "mysqli_sql_exception"

I'm just going to start with a fresh reinstall from the snapshot taken on 2024-12-28
first half of the first upgrade took 28 minutes

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php"
...
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}

real    28m16,791s
user    0m0,168s
sys     0m0,268s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

the next command finished in in 15 seconds

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php"
...
... archive processed up to revision id 302632 of 302632 (3178 rows in 1.3690750598907 seconds)
Done populating archive table. Processed 3178 rows in 1.3690841197968 seconds
Done. Processed 302790 rows in 14.684278964996 seconds

real    0m14,776s
user    0m0,009s
sys     0m0,039s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

loading the site still fails with the same error as before; let's continue with the second upgrade

[Z56-K4l3m_5hOvSLliKhkgAAAAM] 2025-02-02 00:37:00: Fatal exception of type "mysqli_sql_exception"

well my next documented step is to visit https://wiki.opensourceecology.org/wiki/Special:Version
1. but that just spits out the same error..
I rsynce'd in the next version's files, fixed permissions, and attempted an upgrade, but it failed

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Cite extension cannot be loaded. Check that all of its files are installed properly.

#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(282): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Cite/extension.json: filemtime(): stat failed for /Cite/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102

real    0m0,077s
user    0m0,002s
sys     0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

Fri Jan 31, 2025

yesterday we mostly migrated obi, but we got stuck on verification because one of the sliders on the frontpage was missing
I realized that second slider was buried under a "page builder" on hetzner2, and 'be-page-builder' is missing on hetzner3
yesterday I emailed brand exponents support asking where I can download the latest version of 'be-page-builder'. I have not received a response.
I confirmed that the files they gave me (via google drive) before, didn't include be-page-builder https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Tue_Dec_31.2C_2024
but later that day I had managed to get some error output that gave me the path it was using to 'download be-portfolio-post' https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
I tried to just sub the plugin slug for 'be-page-builder', and it worked! https://brandexponents.com/oshin-plugins/be-page-builder.zip

user@disp6255:~$ wget https://brandexponents.com/oshin-plugins/be-page-builder.zip
--2025-01-31 15:37:46--  https://brandexponents.com/oshin-plugins/be-page-builder.zip
Resolving brandexponents.com (brandexponents.com)... 188.114.96.12, 188.114.97.12, 2a06:98c1:3120::c, ...
Connecting to brandexponents.com (brandexponents.com)|188.114.96.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘be-page-builder.zip’

be-page-builder.zip     [      <=>           ]   1.61M  1.36MB/s    in 1.2s    

2025-01-31 15:37:48 (1.36 MB/s) - ‘be-page-builder.zip’ saved [1691575]

user@disp6255:~$ unzip be-page-builder.zip 
Archive:  be-page-builder.zip
   creating: be-page-builder/
...
user@disp6255:~$ grep -i version be-page-builder/be-page-builder.php 
Version: 4.6.1
user@disp6255:~$

well shit, that's the same version that we have installed on hetzner2. So no need to update or 3TOFU
I just copied the plugin dir that I had copied from hetzner2 into the dir of plugins on hetzner3 (so that future migrations won't delete the dir)s

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # rsync -av --progress var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/be-page-builder /var/tmp/wordpress/plugins/
...
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #

great. after I copied the plugin back to the wordpress site's 'plugins/' dir, activated it in the wp dashboard, cleared the varnish cache, and refreshed – the second slider re-appeared ☺
oh, but the pages aren't exact.
1. there's a white box around the second slider in the new site that says "OPEN BUILDING INSTITUTE" honestly it looks better on the new site than the old imho, but it is an important difference
2. also the content in the footer has less padding in the new site. not a big deal
there's also differences on /about-who-we-are/
1. curiously the profiles of everyone looks good *except* for Catarina and Marcin :/
2. oh, it looks like that's because Catarina & Marcin have some sort of "tabs" that the rest don't: bio, talks, press & publications, and education
if I try to edit this page on hetzner3, I can see all the tabs in the page builder. so that seems ok
if I open the js console, I don't see any obvious errors
1. here's the console output on-load on hetzner2

GET
https://f.vimeocdn.com/js/froogaloop2.min.js?ver=4.8.1
NS_BINDING_ABORTED

This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. about-who-we-are
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. node.js:416:1
A resource is blocked by OpaqueResponseBlocking, please check browser console for details. froogaloop2.min.js
JQMIGRATE: Migrate is installed, version 1.4.1 jquery-migrate.min.js:2:552
Google Analytics and Tag Manager is being shimmed by Firefox. See https://bugzilla.mozilla.org/show_bug.cgi?id=1713687 for details. sandbox eval code:1:9
GET
https://f.vimeocdn.com/js/froogaloop2.min.js?ver=4.8.1
NS_BINDING_ABORTED

A resource is blocked by OpaqueResponseBlocking, please check browser console for details. froogaloop2.min.js
Loading failed for the <script> with source “https://f.vimeocdn.com/js/froogaloop2.min.js?ver=4.8.1%E2%80%9D. about-who-we-are:2032:93
Google Maps JavaScript API has been loaded directly without loading=async. This can result in suboptimal performance. For best-practice loading patterns please see https://goo.gle/js-api-loading js:278:278
unreachable code after return statement be-modules-plugin.js:35:12433
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://maps.googleapis.com/maps/api/mapsjs/gen_204?csp_test=true. (Reason: CORS request did not succeed). Status code: (null).
downloadable font: OS/2: Bad sTypoLineGap, setting it to 0: -32 (font-family: "icomoon" style:normal weight:400 stretch:100 src index:1) source: https://www.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i
downloadable font: Glyph bbox was incorrect (glyph ids 7 9 10 13 14 15 19 20 21 23 24 26 30 33 44 52 57 58 59 60 63 72 77 80 83 84 87 88 90 91 93 94 95 99 106 114 121 129 142 143 146 147 149 150 151 152 156 159 160 161 163 166 167 168 169 170 171 172 176 177 182 183 184 185 191 197 207 208 209 212 214 215 216 218 220 222 226 227 228 236 237 239 240 241 242 243 244 245 246 247 248 250 252 254 256 257 258 259 260 262 263 264 266 269 271 273 276 278 279 281 283 284 285 286 287 288 299 302 305 309 310 316 317 318 321 322 326 327 328 329 330 335 336 339 341 342 343 344 345 347 348 351 355 357 361 363 367 369 370 372 375 376 377 378 382 383 385 386 388 389 390 391 393 394 407 408 411 413 414 416 417 418 419 420 421 422 423 428 430 432 438 443 445 449 451 458 464 465 468 470 474 475 477 478 482 483 484 485 486 488 489 490 491 494 497 498 500 501 503 504 506 507 509 510 513 514 515 517 519 525 526 527 528 529 530 531 532 534 535 536 537 538 541 544 547 550 552 553 554 561 564 573 577 578 579 580 583 588 591 593 598 599 601 611 612 624 627 628 631 632 648 656 661 662 680 704 706 718 720 721 729 731 738 743 744 746 747 748 749 750 755 770 771 775 790 814 820 826 841 844) (font-family: "icomoon" style:normal weight:400 stretch:100 src index:1) source: https://www.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i
downloadable font: Glyph bbox was incorrect (glyph ids 91 223) (font-family: "Economica" style:normal weight:700 stretch:100 src index:0) source: https://fonts.gstatic.com/s/economica/v15/Qw3aZQZaHCLgIWa29ZBTjecUDXx4.woff2
Google Maps JavaScript API warning: NoApiKeys https://developers.google.com/maps/documentation/javascript/error-messages#no-api-keys util.js:81:65

1. and here's it on hetzner3

Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://fonts.googleapis.com/css?family=Economica%3A400%2C700%7COpen+Sans%3A400%7CMontserrat%3A400%2C700%7CCrimson+Text%3A400italic%7CRaleway%3A400%2C600&subset=latin&ver=1738275627 css
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.google-analytics.com/analytics.js about-who-we-are:26:67
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://i.creativecommons.org/l/by-sa/4.0/88x31.png 88x31.png
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps/api/js?ver=6.7.1 js
This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. about-who-we-are
JQMIGRATE: Migrate is installed, version 3.4.1 jquery-migrate.min.js:2:981
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. modernizr.js:4:3947
unreachable code after return statement be-modules-plugin.js:35:12433
Google Maps JavaScript API has been loaded directly without loading=async. This can result in suboptimal performance. For best-practice loading patterns please see https://goo.gle/js-api-loading js:278:278
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps/api/mapsjs/gen_204?csp_test=true js:303:207
downloadable font: Glyph bbox was incorrect (glyph ids 91 223) (font-family: "Economica" style:normal weight:700 stretch:100 src index:0) source: https://fonts.gstatic.com/s/economica/v15/Qw3aZQZaHCLgIWa29ZBTjecUDXx4.woff2
downloadable font: OS/2: Bad sTypoLineGap, setting it to 0: -32 (font-family: "icomoon" style:normal weight:400 stretch:100 src index:1) source: https://www.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i
Some cookies are misusing the recommended “SameSite“ attribute 4
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.google-analytics.com/j/collect?v=1&_v=j101&a=847737016&t=pageview&_s=1&dl=https%3A%2F%2Fwww.openbuildinginstitute.org%2Fabout-who-we-are%2F%3Fnocache%3D3&ul=en-us&de=UTF-8&dt=About%3A%20Who%20We%20Are%20%E2%80%93%20Open%20Building%20Institute&sd=24-bit&sr=1920x1200&vp=1289x510&je=0&_u=AACAAEABAAAAACAAI~&jid=383980088&gjid=939186075&cid=1363777290.1738276114&tid=UA-78141409-1&_gid=1042838251.1738276114&_r=1&_slc=1&z=931548647 analytics.js:36:32
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/gtag/js?id=G-2XNSR98CNW&cx=c&_slc=1 analytics.js:24:55
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtm.init_consent&eid=1&tr=1ogtdma&ti=2ogtdma&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtm.init&eid=2&tr=1ogtgasend.1ogtreferralexclusion.1ogtsessiontimeout.1ogt1pdatav2.1ccdgafirst.1setproductsettings.1ccdgaregscope.1ccdconversionmarking.1ccdautoredact.1ccdgalast&ti=2ogtgasend.2ogtreferralexclusion.2ogtsessiontimeout.2ogt1pdatav2.2ccdgafirst.2setproductsettings.2ccdgaregscope.2ccdconversionmarking.2ccdautoredact.2ccdgalast&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtm.js&eid=3&tr=1gct&ti=1gct&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtag.config&eid=4&u=AAAAAAAI&epr=1G.3G&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=*&eid=5&u=AAAAAAAIAAAAAIA&ut=Ag&h=Ag&epr=1G.2G&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps-api-v3/api/js/59/7/common.js js:167:267
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps-api-v3/api/js/59/7/util.js js:167:267
Google Maps JavaScript API warning: NoApiKeys https://developers.google.com/maps/documentation/javascript/error-messages#no-api-keys util.js:81:65
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://region1.google-analytics.com/g/collect?v=2&tid=G-2XNSR98CNW&gtm=45je51u0v9124516828za200&_p=1738357726052&gcd=13l3l3l3l2l1&npa=0&dma=0&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&ul=en-us&sr=1920x1200&cid=1363777290.1738276114&ir=1&frm=0&pscdl=noapi&_eu=EBAI&_s=1&dl=https%3A%2F%2Fwww.openbuildinginstitute.org%2Fabout-who-we-are%2F%3Fnocache%3D3&dt=About%3A%20Who%20We%20Are%20%E2%80%93%20Open%20Building%20Institute&sid=1738356502&sct=3&seg=1&en=page_view&_ee=1&tfd=12191 js:169:286

I don't see anything obvious. I'm going to keep going.
on '/supporters/' I see there's a new header that says SUPPORTERS and has breadcrumbs at the top (just below the navbar)
1. this is a thing on all the pages, it seems. maybe a change to oshine theme, idk
the '/library/' page has a broken shortcode

[do_widget id=nav_menu-2]

looks like this is supposed to be from a plugin named 'amr-shortcode-any-widget'

[root@opensourceecology www.openbuildinginstitute.org]# grep -irl 'do_widget' *
htdocs/wp-content/plugins/amr-shortcode-any-widget/amr-utilities.php
htdocs/wp-content/plugins/amr-shortcode-any-widget/readme.txt
htdocs/wp-content/plugins/amr-shortcode-any-widget/amr-admin-form-html.php
htdocs/wp-content/plugins/amr-shortcode-any-widget/amr-shortcode-any-widget.php
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget-sr_RS.po
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget-sr_RS.mo
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget.pot
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget-id_ID.po
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr_shortcode_any_widget_id_ID.mo
[root@opensourceecology www.openbuildinginstitute.org]#

yeah, we don't have a version of this plugin because it was closed by wordpress due to a "security issue" https://wordpress.org/plugins/amr-shortcode-any-widget/

This plugin has been closed as of January 19, 2023 and is not available for download. Reason: Security Issue.

this other plugin claims to offer a 'do_wiget' shortcode, but it only has 400 installs and it hasn't been updated for 4 years https://wordpress.org/plugins/shortcodely/
looks like the security issue was a 6.4/10 XSS vuln = CVE-2022-4458 https://www.wordfence.com/threat-intel/vulnerabilities/wordpress-plugins/amr-shortcode-any-widget/amr-shortcode-any-widget-40-authenticated-contributor-stored-cross-site-scripting-via-shortcode
curiously, I found a repo that claims to be a mirror on github, but it was last updated 3 months ago https://github.com/common-repository/amr-shortcode-any-widget
1. oh, it was created 3 months ago. it was probably just copied from the old (vulnerable?) codebase
there's more problems on '/structures/'
1. I can't change between tabs, and the images aren't showing
2. the div class says "portfolio", so I assume this is "be-portfolio-post', which we do have installed.
3. oh, each of these tabs actually cooresponds to an entry in the wp dashboard's "portfoilo" section https://www.openbuildinginstitute.org/wp-admin/edit.php?post_type=portfolio
4. huh, actually, some of these are missing
5. oh, it looks like the "checkboxes" that you choose from on the actual page are found in the Portfolio -> Categories slugs https://www.openbuildinginstitute.org/wp-admin/edit-tags.php?taxonomy=portfolio_categories&post_type=portfolio
6. well shit, if I view the category it works fine https://www.openbuildinginstitute.org/portfolio_categories/roof/
7. so it appears to just be the tab selection thing that wraps these portfolios?
this page has issues too https://www.openbuildinginstitute.org/library-utilities-appliances/
1. the HYRDOPONICS CONTROL PANEL text is missing on the gold label
lots of the same issues exist on the other "libraries" pages
on the '/buildings/' page there's issues
1. none of the sliders show up
on the '/use/' page
1. the iframe of the 3d studio doesn't load at all
2. js console gives this relevant message

Invalid X-Frame-Options header was found when loading “https://www.openbuildinginstitute.org/wp-content/3Dmodels/example_studio/%E2%80%9D: “invalid” is not a valid directive.example_studio

1. in the networking tab, I see a failed load of 'https://www.openbuildinginstitute.org/wp-content/3Dmodels/example_studio/' due to NS_ERROR_XFO_VIOLATION. I guess XFO = X-Frame-Options
2. the response headers for that request include both 'SAMEORIGIN' and 'deny'

X-Frame-Options: SAMEORIGIN
X-Frame-Options: deny

1. here's what mozilla has to say https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

If you specify DENY, not only will the browser attempt to load the page in a frame fail when loaded from other sites, attempts to do so will fail when loaded from the same site. On the other hand, if you specify SAMEORIGIN, you can still use the page in a frame as long as the site including it in a frame is the same as the one serving the page.

1. so it sounds like we want 'sameorigin'
2. well, shit, it looks like we've setup nginx to set it to 'sameorigin' and apache to 'deny'

user@personal:~/sandbox_local/ansible/hetzner3$ grep -ir 'x-frame-options' *
roles/maltfield.nginx/templates/nginx.conf.j2:	add_header X-Frame-Options "SAMEORIGIN";
roles/maltfield.apache/templates/security.conf.j2:#Header set X-Frame-Options: "sameorigin"
roles/maltfield.apache/templates/security.conf.j2:Header always append X-Frame-Options "SAMEORIGIN"
roles/maltfield.apache/templates/security.conf.j2:Header set X-Frame-Options "deny"
user@personal:~/sandbox_local/ansible/hetzner3$

1. it looks like our old server never used 'deny'

[root@opensourceecology ~]# grep -ir 'x-frame-options' /etc/nginx
/etc/nginx/nginx.conf:   add_header X-Frame-Options "SAMEORIGIN";
/etc/nginx/nginx.conf.20241230:   add_header X-Frame-Options "SAMEORIGIN";
[root@opensourceecology ~]# grep -ir 'x-frame-options' /etc/apache2
grep: /etc/apache2: No such file or directory
[root@opensourceecology ~]# grep -ir 'x-frame-options' /etc/httpd
/etc/httpd/conf/httpd.20170826.bak:Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf.20170720.bak:Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf:#Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf.20170901:#Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf.20170811.bak:Header always append X-Frame-Options SAMEORIGIN
[root@opensourceecology ~]#

1. since I know Marcin loves to use iframes, I'm going to make this 'sameorigin' the default for all our configs https://github.com/OpenSourceEcology/ansible/commit/5efbec50f239817be54764fcada4dafeade7a034
2. I made that change in ansible, pushed, restarted nginx/apache, cleared varnish, and refreshed. now the 3d model of the studio loads in the iframe fine ☺
loads of stuff is broken on '/workshops-events/'
1. I don't see any obvious errors
well, I think that's about all I can do for OBI
I want Catarina to look at it and see how she feels; maybe she just needs to do some simple change to fix them? Or maybe this is a disaster

Hey Catarina,

I just finished migrating a snapshot of OBI to hetzner3. Can you please review it?

I'm sorry to say that OBI didn't handle the upgrades very well. About half of the content is missing. I was able to fix some of the issues by changing the web server config, but it appears that most of the issues are theme- or page-builder-related, and I'll need to defer to you as the SME.

Can you please tell me:

1. What OS does your computer use?
2. What web browser do you use?

And I can send instructions for you to view/edit the OBI site on hetzner3.


Thank you,

...
I'm moving onto osemain
I migrated a snapshot of www.opensourceecology.org to hetnzer3 (surprisingly the first time) following the same guide as with the other sites
I updated ansible to use this vhost (and make it the default vhost for the server) and pushed it out https://github.com/OpenSourceEcology/ansible/commit/674a16babf2f250f80c7acdd06f1ac4d7a1888c7
the website actually looks ok on first load, except the social media links in the footer fail to load as expected (plugin no longer available) and instead just says '

[smbtoolbar]'

but after I login to the wp dashboard wui, I just get a crit error

There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.

Learn more about troubleshooting WordPress.

apache error log indicates it's the theme; damn

[Fri Jan 31 23:39:03.032865 2025] [proxy_fcgi:error] [pid 266776:tid 266776] [client 193.32.127.139:0] AH01071: Got error 'PHP message: PHP Deprecated:  Creation of dynamic property MO_Framework::$slider_manager is deprecated in /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes/enigmatic/framework/framework.php on line 292; PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function create_function() in /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes/enigmatic/framework/option-tree/includes/ot-functions-admin.php:69\nStack trace:\n#0 [internal function]: ot_register_theme_options_page()\n#1 /var/www/html/www.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()\n#2 /var/www/html/www.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()\n#3 /var/www/html/www.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()\n#4 /var/www/html/www.opensourceecology.org/htdocs/wp-settings.php(704): do_action()\n#5 /var/www/html/www.opensourceecology.org/wp-config.php(126): require_once('...')\n#6 /var/www/html/www.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\n#7 /var/www/html/www.opensourceecology.org/htdocs/wp-admin/admin.php(34): require_once('...')\n#8 /var/www/html/www.opensourceecology.org/htdo...'

allriht, it looks like there used to be a built-in function called 'create_function()' in php, but it was removed in PHP 8.0 https://www.php.net/manual/en/function.create-function.php
hetzner2 had php v5.6 and hetzner3 has v8.2, so that tracks
looks like we used to run theme v3.5 and now we run theme v3.6
1. hetzner2

[root@opensourceecology current]# sudo -u wp -i wp --path="/var/www/html/www.opensourceecology.org/htdocs/" theme list
...
+------------------------+----------+-----------+---------+
| name                   | status   | update    | version |
+------------------------+----------+-----------+---------+
| enigmatic.20170714.bak | inactive | none      | 3.1     |
| enigmatic              | active   | none      | 3.5     |
| twentyeleven           | inactive | available | 2.7     |
| twentyfifteen          | inactive | available | 1.9     |
| twentyfourteen         | inactive | available | 2.1     |
| twentyseventeen        | inactive | available | 1.4     |
| twentysixteen          | inactive | available | 1.4     |
| twentyten              | inactive | available | 2.4     |
| twentythirteen         | inactive | available | 2.3     |
| twentytwelve           | inactive | available | 2.4     |
+------------------------+----------+-----------+---------+
[root@opensourceecology current]#

1. hetzner3

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.opensourceecology.org_20250131/current # sudo -u wp -i wp --path="${docrootDir}" theme list
...
+-----------------+----------+-----------+---------+----------------+-------------+
| name            | status   | update    | version | update_version | auto_update |
+-----------------+----------+-----------+---------+----------------+-------------+
| enigmatic       | active   | none      | 3.6     |                | off         |
| twentyeleven    | inactive | available | 4.8     | 3.3            | off         |
| twentyfifteen   | inactive | available | 3.9     | 2.5            | off         |
| twentyfourteen  | inactive | available | 4.1     | 2.7            | off         |
| twentyseventeen | inactive | available | 3.8     | 2.2            | off         |
| twentysixteen   | inactive | available | 3.4     | 2.0            | off         |
| twentyten       | inactive | available | 4.3     | 2.9            | off         |
| twentythirteen  | inactive | available | 4.3     | 2.9            | off         |
| twentytwelve    | inactive | available | 4.4     | 3.0            | off         |
+-----------------+----------+-----------+---------+----------------+-------------+
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.opensourceecology.org_20250131/current #

PHP v8.0 came out in 2020 and became EOL in 2023 https://en.wikipedia.org/wiki/PHP
geez..I wonder how long it's been since the latest version of this theme (v3.6) was released if it doesn't work on a version of PHP that was EOL 2 years ago?
the URL listed for the theme points here, but it doesn't have any info about the theme; it's just a "Lorem ipsum" demo of the theme

root@hetzner3 /var/www/html/www.opensourceecology.org # head htdocs/wp-content/themes/enigmatic/style.css 
/*
Theme Name: Enigmatic
Theme URI: https://www.livemeshthemes.com/enigmatic
Description: A clean, responsive, seo-optimized Corporate theme from LiveMesh. Follow us on <a href="http://twitter.com/live_mesh">Twitter</a> for updates
Version: 3.6
Author: <a href="http://themeforest.net/user/livemesh">LiveMesh</a>
Author URI: http://themeforest.net/user/livemesh
License: GNU General Public License version 3.0
License URI: http://www.gnu.org/licenses/gpl-3.0.html
Tags: one-column, two-columns, three-columns, left-sidebar, right-sidebar, woocommerce, ecommerce, fixed-width, theme-options, threaded-comments, translation-ready
root@hetzner3 /var/www/html/www.opensourceecology.org #

this is their profile on themeforest, which lists all of the themes they have for sale http://themeforest.net/user/livemesh
unfortunately, enigmatic is at the bottom of the list, and when you click it, it brings you to this page – which says "this item is no longer available" https://themeforest.net/item/enigmatic-responsive-multipurpose-wp-theme/3919108
I can't find any changelog in the theme dir that suggets when it was last updated
Catarina anticpated this, and she wanted to change to the oshine theme. I'm not a huge fan of using premium themes like this with their weird & bespoke page builders. it seems fragile and not future-proof
in my own research for eco-libre, I've been looking at Neve https://wordpress.org/themes/neve/
1. It's freemium. pro is $69-$259/yr
2. 300,000+ active installs
3. 1,023 5-star reviews
4. responsive
5. lots of starter sites https://themeisle.com/themes/neve/starter-sites/
6. I had good interactions with the devs to my bug on github https://github.com/Codeinwp/neve/issues/4250
7. in my trials, I couldn't install their "starter sties", and support wasn't very helpful https://wordpress.org/support/topic/how-to-download-starter-site/
another great theme is just twentytwentyone https://wordpress.org/themes/twentytwentyone/
but let's see if we just install oshine and switch to it
here's how I copied the oshine theme and all its required plugins

rsync -av --progress /var/tmp/wordpress/themes/oshin /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes/
rsync -av --progress /var/tmp/wordpress/plugins/be-portfolio-post /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box-conditional-logic /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box-show-hide /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box-tabs /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/oshine-core /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/oshine-modules /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/tatsu /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/

oh, crap, well, I can't change the theme because I can't login because the existing theme is broken. So I have to get rid of the old theme first

root@hetzner3 /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes # mv enigmatic enigmatic.die
root@hetzner3 /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes #

alright, now I can refresh the wp wui after I logged-in, and the error goes away. now it's prompting me about a db upgrade. I did it.
after switching to the oshine plugin, the website is pretty horribly broken.
1. I mean at least I can login
2. and the navbar at the top is there
3. and the content is mostly there
4. but the footer and lots of naked shortcodes are around the page#
I sent an email to Marcin about this

Hey Marcin,

I just finished migrating a snapshot of osemain to hetzner3. It's bad, and I don't know how you want to proceed.

I'm sorry to say that osemain didn't handle the upgrades very well.

Most of the website's content is surprisingly present, but the worst part is that you can't access the wp dashboard. After you login, you get a critical error.

The critical error is caused by your enigmatic theme trying to call a PHP function that was removed from PHP in 2020.

It looks like the version of your enigmatic theme that we downloaded from themeforest is v3.6. You have v3.5 installed on hetzner2. I checked the theme's websites. I couldn't find any datestampped changelog, but it appears that the theme is no longer for sale. I guess it hasn't been updated for many, many years.

* https://www.livemeshthemes.com/enigmatic
* http://themeforest.net/user/livemesh
* https://themeforest.net/item/enigmatic-responsive-multipurpose-wp-theme/3919108

Catarina mentioned wanting to switch from enigmatic to oshine. I went ahead and installed the latest version of oshine (and its many dependent plugins). After that, you're able to login. But the website becomes pretty horribly broken.

We should probably talk about what you want to do with this website as there's no easy solution.

Best case would be for you to hire someone to build you a new website with a better theme. Catarina suggested oshine. That would work, but I have a similar worry about the future of oshine. You might want to consider a more popular freemium theme, such as Neve, Astra, Hestia, OceanWP, TwentyTwentyOne, etc..

of course I could convert the website over to one of these themes, but I'm not an expert at it. So we could probably find someone cheaper (and faster) than me
quick search on upwork shows some folks who cost half as much as I'm charging, and specifically mention experience with some of these themes (including neve, which is at the top of my list)
1. for example Bhavesh J in India https://www.upwork.com/freelancers/~01e0c758d97b22e145
if I could figure out some way to make ose on hetzner2 read-only (I tried researching this before, and didn't find a solution I liked), then perhaps I can actually do the fork and just give a new admin user to a wordpress dev like this and tell them "make this website (with neve installed) look like this old website"
1. then, after they're done and it's verified and approved by me and marcin, we just change DNS over to the new site
2. also, one alt to "making the wordpress site read-only is to just create an actual static site of osemain with wget and just replaced the docroot of the wordpress site with it on hetzner2 while we do the migration.

Thr Jan 30, 2025

so yesterday I realized that my oswh static site backup had some 404 errors loading js depend assets
today I checked and confirmed that there's a lot of those missing
1. here's a list of the js files we have in the website's theme dir on hetzner2

[root@opensourceecology oswh.opensourceecology.org]# find htdocs/wp-content/themes/Eventor -iname '*\.js'
htdocs/wp-content/themes/Eventor/script/countdown/jquery.countdown.js
htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js
htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors-min.js
htdocs/wp-content/themes/Eventor/script/animate-colors/my-animate-colors.js
htdocs/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js
htdocs/wp-content/themes/Eventor/script/horizontal/my-horizontal.js
htdocs/wp-content/themes/Eventor/script/spiner/spin.min.js
htdocs/wp-content/themes/Eventor/script/common.js
htdocs/wp-content/themes/Eventor/script/datepicker/js/jquery-ui-1.9.2.custom.min.js
htdocs/wp-content/themes/Eventor/script/datepicker/js/jquery-1.7.2.min.js
htdocs/wp-content/themes/Eventor/script/datepicker/js/jquery-ui-1.8.20.custom.min.js
htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.min.js
htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.js
htdocs/wp-content/themes/Eventor/script/anythingslider/my-anythingslider.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/swfobject.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.min.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.video.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.video.min.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.fx.min.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.easing.1.2.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.fx.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/newjavascript.js
htdocs/wp-content/themes/Eventor/script/contact/contact.js
htdocs/wp-content/themes/Eventor/script/menu/supersubs.js
htdocs/wp-content/themes/Eventor/script/menu/superfish.js
htdocs/wp-content/themes/Eventor/script/imagesloaded/imagesloaded.js
htdocs/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js
htdocs/wp-content/themes/Eventor/script/mailchimp/js/jquery-1.4.2.min.js
htdocs/wp-content/themes/Eventor/script/mailchimp/js/mailing-list.js
htdocs/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js
htdocs/wp-content/themes/Eventor/script/jscolor/jscolor.js
htdocs/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js
htdocs/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/plugin.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/js/base64.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/js/jquery.livequery.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/js/popup.js
[root@opensourceecology oswh.opensourceecology.org]#

1. and here's the same results on hetzner3

root@hetzner3 /var/www/html/oswh.opensourceecology.org # find htdocs/wp-content/themes/Eventor -iname '*\.js'
htdocs/wp-content/themes/Eventor/script/contact/contact.js
htdocs/wp-content/themes/Eventor/script/jscolor/jscolor.js
htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.js
htdocs/wp-content/themes/Eventor/script/common.js
htdocs/wp-content/themes/Eventor/script/spiner/spin.min.js
htdocs/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js
htdocs/wp-content/themes/Eventor/script/menu/superfish.js
htdocs/wp-content/themes/Eventor/script/quickpager/quickpager.js
htdocs/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js
htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js
htdocs/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js
htdocs/wp-content/themes/Eventor/script/countdown/jquery.countdown.js
htdocs/wp-content/themes/Eventor/script/horizontal/jquery.js
root@hetzner3 /var/www/html/oswh.opensourceecology.org #

the files are missing too in the wget-produced source files on hetzner2 (before we did the mv to remove the version pinned get var from the filename)

[root@opensourceecology oswh.opensourceecology.org]# find wp-content/themes/Eventor -iname '*\.js*'
wp-content/themes/Eventor/script/countdown/jquery.countdown.js?ver=4.9.1
wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js?ver=4.9.1
wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
wp-content/themes/Eventor/script/spiner/spin.min.js?ver=4.9.1
wp-content/themes/Eventor/script/pirobox/js/pirobox.js?ver=4.9.1
wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js?ver=4.9.1
wp-content/themes/Eventor/script/common.js?ver=4.9.1
wp-content/themes/Eventor/script/contact/contact.js?ver=4.9.1
wp-content/themes/Eventor/script/menu/superfish.js?ver=4.9.1
wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js?ver=4.9.1
wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1
wp-content/themes/Eventor/script/jscolor/jscolor.js?ver=4.9.1
wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js?ver=4.9.1
wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js?ver=4.9.1
[root@opensourceecology oswh.opensourceecology.org]#

I'll add this to add those missing files too

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/oswh.opensourceecology.org_20250129/current # rsync -av --progress --include="*.js" --include="*/" --exclude="*" var/www/html/${vhost_name}/htdocs/wp-content/themes/Eventor/ ${vhostDir}/htdocs/wp-content/themes/Eventor/
...

sent 392.159 bytes  received 765 bytes  785.848,00 bytes/sec
total size is 618.066  speedup is 1,57
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/oswh.opensourceecology.org_20250129/current #

after running that and fixing the permissions (and reloading the page), the carossel is working again. Now the errors look more sane:

This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. oswh.opensourceecology.org
Loading mixed (insecure) display content “http://oswh.opensourceecology.org/wp-content/uploads/2013/02/DocHackFavicon.png” on a secure page FaviconLoader.sys.mjs:175:20
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. 2 oswh.opensourceecology.org
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://api.flattr.com/js/0.6/load.js?mode=auto oswh.opensourceecology.org:491:35
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
The resource at “https://api.flattr.com/js/0.6/load.js?mode=auto” was blocked because content blocking is enabled. oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org:491:35
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” oswh.opensourceecology.org
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” 2 oswh.opensourceecology.org

we can probably fix those last errors with some sed that replaces youtube and vimeo embeds from http to https

find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://www.youtube.com%https://www.youtube.com%g' '{}' \;

find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://player.vimeo.com%https://player.vimeo.com%g' '{}' \;

find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://oswh.opensourceecology.org%https://oswh.opensourceecology.org%g' '{}' \;
<pre>
# ok, now the site looks good. I think we can call oswh done, pending Marcin's review.
# ...
# Let's move-on to OBI
# I followed the migration guide (as was used by microfactory), but the last steps (to activate the new plugins) failed
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
		sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
Error: Error establishing a database connection.
Error: Error establishing a database connection.
Error: Error establishing a database connection.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #

loading it in the web browser failed with https cert error
here's the logs for when I copied them over last https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Wed_Sep_25.2C_2024
that says I had copied the entire dir, so it should include OBI's certs
yep, hetnzer3 says it has them, and they're still valid until 2025-03-11

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: openbuildinginstitute.org
	Serial Number: 37201823a671e8c6da8373cddd4efde6c6a
	Key Type: RSA
	Domains: www.openbuildinginstitute.org awstats.openbuildinginstitute.org openbuildinginstitute.org seedhome.openbuildinginstitute.org
	Expiry Date: 2025-03-11 08:00:42+00:00 (VALID: 39 days)
	Certificate Path: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
	Private Key Path: /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem
  Certificate Name: opensourceecology.org
	Serial Number: 3ec0988ac3af1baa0909ce9a9f4a6409c21
	Key Type: RSA
	Domains: fef.opensourceecology.org awstats.opensourceecology.org forum.opensourceecology.org microfactory.opensourceecology.org munin.opensourceecology.org opensourceecology.org oswh.opensourceecology.org phplist.opensourceecology.org staging.opensourceecology.org store.opensourceecology.org wiki.opensourceecology.org www.opensourceecology.org
	Expiry Date: 2025-03-11 08:00:47+00:00 (VALID: 39 days)
	Certificate Path: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
	Private Key Path: /etc/letsencrypt/live/opensourceecology.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #

ah, here's the error in the browser; looks like it's configured to use the wrong cert

Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for www.openbuildinginstitute.org. The certificate is only valid for the following names: awstats.opensourceecology.org, fef.opensourceecology.org, forum.opensourceecology.org, microfactory.opensourceecology.org, munin.opensourceecology.org, opensourceecology.org, oswh.opensourceecology.org, phplist.opensourceecology.org, staging.opensourceecology.org, store.opensourceecology.org, wiki.opensourceecology.org, www.opensourceecology.org
 
Error code: SSL_ERROR_BAD_CERT_DOMAIN

https truncates at nginx
here's the obi nginx config

root@hetzner3 /etc/nginx # ls sites-enabled/
00-default.conf                     fef.opensourceecology.org.conf.20241230  munin.opensourceecology.org.conf         store.opensourceecology.org.conf
awstats.opensourceecology.org.conf  forum.opensourceecology.org.conf         oswh.opensourceecology.org.conf          wiki.opensourceecology.org.conf
fef.opensourceecology.org.conf      microfactory.opensourceecology.org.conf  seedhome.openbuildinginstitute.org.conf
root@hetzner3 /etc/nginx #

uhh, it's not there. right, so it's probably just the catchall fallback vhost that's triggering. we need to push it out with ansible.
I uncommented this vhost and pushed it out with ansible https://github.com/OpenSourceEcology/ansible/commit/ec1ea4da19722268fdabd095ab7b7e2deee5246a
I had to manaully restart nginx & apache on the server (before that the stale config redirected us to the forums, which is currently the default catchall site – until we migrate osemain)
loading the site just returns an error

Error establishing a database connection

so that's the same error that we saw on wp-cli
I get the same error trying to login on the wui https://www.openbuildinginstitute.org/wp-login.php
I actually don't have issues logging into the db using the creds copied from wp-config.php

root@hetzner3 /var/www/html/www.openbuildinginstitute.org # mysql -u obi_user -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 950748
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
ERROR 1227 (42000): Access denied; you need (at least one of) the SHOW DATABASES privilege(s) for this operation
MariaDB [(none)]>

MariaDB [(none)]> use obi_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [obi_db]> show tables;
+-------------------------------+
| Tables_in_obi_db              |
+-------------------------------+
| wp_commentmeta                |
| wp_comments                   |
...
+-------------------------------+
21 rows in set (0,000 sec)

MariaDB [obi_db]>

I checked to see if there's any differences between users in this wp site vs others (maybe it's a host 127.0.0.1 vs localhost issue vs the old public ipv4 address)

root@hetzner3 ~ # source /root/backups/backup.settings
root@hetzner3 ~ # mysql -u root -p${mysqlPass}
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 951030
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> 

MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mysql]> 

MariaDB [mysql]> select Host, User from user; 
+-----------+------------------+
| Host      | User             |
+-----------+------------------+
...
| localhost | obi2_user        |
| localhost | obi3_user        |
| localhost | obi_staging_user |
| localhost | obi_user         |
...
| localhost | oswh_user        |
...
25 rows in set (0,009 sec)

MariaDB [mysql]>

curiously, there's four distinct obi users, but all looks good with it
ah, I found this in the wp-config.php

/** MySQL hostname */
#define('DB_HOST', 'localhost');
define('DB_HOST', 'localhost:/var/lib/mysql/mysql.sock');

I'm actually surprised that I'm not already doing this for others, but it looks like mariadb ignores 'skip-networking' (which I have set) and instead listens on 'localhost' if you set that https://github.com/OpenSourceEcology/ansible/blob/0dc853cd08ce504ae5ccf7f51948edf7a3cce7dd/hetzner3/roles/maltfield.mariadb/templates/50-server.cnf.j2#L37-L40
I added two idempotent sed commands to the migration CHG wiki article, which reverses the comments on these two lines

# and use 'DB_HOST' of 'localhost'
sed -i "s%^#define('DB_HOST', 'localhost');%define('DB_HOST', 'localhost')%" ${vhostDir}/wp-config.php | grep -i DB_HOST
sed -i "s%^define('DB_HOST', 'localhost\:\(.*\)%#define('DB_HOST', 'localhost\:\1%" ${vhostDir}/wp-config.php | grep -i DB_HOST

re-running the plugin activate commands fails with new errors

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # # ACTIVATE NEW PLUGINS

activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
		sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
PHP Parse error:  syntax error, unexpected identifier "define" in phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1335) : eval()'d code on line 52
PHP Parse error:  syntax error, unexpected identifier "define" in phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1335) : eval()'d code on line 52
PHP Parse error:  syntax error, unexpected identifier "define" in phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1335) : eval()'d code on line 52
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #

attempting to load the login page in the wui spits out a similar error to the apache logs

[Thu Jan 30 22:16:06.053681 2025] [proxy_fcgi:error] [pid 3058592:tid 3058592] [client 198.54.133.36:0] AH01071: Got error 'PHP message: PHP Parse error:  syntax error, unexpected identifier "define" in /var/www/html/www.openbuildinginstitute.org/wp-config.php on line 54'

oh, it's because I'm missing a semicolon; I fixed it manually and in the wiki sed commands
now activation works

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # 
# ACTIVATE NEW PLUGINS

activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
		sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'activitypub' activated.
Success: Activated 1 of 1 plugins.
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'aurora-heatmap' activated.
Success: Activated 1 of 1 plugins.
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'melapress-login-security' activated.
Success: Activated 1 of 1 plugins.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #

I was then able to login to the wui, but the css was broken
I was still able to press the "button" to update the db, but then it redirected me to a page with just this crit error

There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.

Learn more about troubleshooting WordPress.

apache errors complain about master slider again :(

[Thu Jan 30 22:21:27.120144 2025] [proxy_fcgi:error] [pid 3071134:tid 3071134] [client 198.54.133.36:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught TypeError: ftp_fput(): Argument #1 ($ftp) must be of type FTP\\Connection, null given in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php:212\nStack trace:\n#0 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php(212): ftp_fput()\n#1 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/masterslider/admin/includes/msp-admin-functions.php(141): WP_Filesystem_FTPext->put_contents()\n#2 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(141): msp_save_custom_styles()\n#3 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(116): Master_Slider_Admin->after_plugin_update()\n#4 [internal function]: Master_Slider_Admin->admin_init()\n#5 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-includes/class-wp-hook.php(324): call_us...', referer: https://www.openbuildinginstitute.org/wp-admin/upgrade.php?step=1&backto=%2Fwp-admin%2Fprofile.php

I logged into the wui on the hetzner2 site
it looks like both "slider revolution" and "master slider" are installed, so maybe we don't use it?
yeah, I clicked on "Master Slider" and it says "no slider found" on the list of sliders available; so let's just do as we did on microfactory and just delete it

# delete Master Slider plugin (PHP Fatal error:  Uncaught TypeError: ftp_fput)
rm -rf ${docrootDir}/wp-content/plugins/masterslider*

ok, after running that, I can load the wp admin dashboard in the wui
I then logged-out on both wuis and did a side-by-side comparison
it's looking decent, but there is a slider on the frontpage that's entirely missing on hetzner3
actually the frontpage has two sliders. the top one loads fine, but the one below it (which appears to be a revslider div) doesn't appear at all
I didn't see anything obvious in the js console
and I didn't see any errors in the apache logs
ok, in the wp dashboard wui, I entered the "revolution slider" settings page, and it lists two sliders
1. the main page one is fine
2. there's another named "Buildings Slider" that's the one at-issue
if I open the "buildings slider" in the wp dashboard, then I see this in the js console

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openbuildinginstitute.org/wp-content/uploads/revslider/Buildings/library_7.jpg. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 301.

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openbuildinginstitute.org/wp-content/uploads/revslider/Buildings/library_7.jpg. (Reason: CORS request did not succeed). Status code: (null).

Slider Revolution has successfully re-requested image to rectify above CORS error.

so I guess it's a CORS issue between 'openbuildinginstitute.org' and 'www.openbuildinginstitute.org'
I tried fixing it with this change to the nginx config, but it didn't fix the homepage or make that error go away (which only appears in the admin page when editing the slider, anwyay)

user@personal:~/sandbox_local/ansible/hetzner3$ git diff
...
diff --git a/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2
index b2b75ec..0e50caf 100644
--- a/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2
@@ -65,7 +65,14 @@ server {
	   proxy_set_header X-Forwarded-Proto https;
	   proxy_set_header X-Forwarded-Port 443;
	   proxy_set_header Host $host;
-   }
+
+               # handle cors whitelist for naked domain
+               proxy_hide_header Access-Control-Allow-Origin;
+               if ( $http_origin ~ "^https://(www.openbuildinginstitue.org|openbuildinginstitute.org)$" ) {
+                       add_header Access-Control-Allow-Origin $http_origin;
+               }
+
+       }
 
 }
 
user@personal:~/sandbox_local/ansible/hetzner3$

I wonder if maybe revslider free changed something to only allow one slider per page?
looks like we're changing 'revslider' from v5.4.7.4 to 6.7.25
here's the changelog page
https://www.sliderrevolution.com/documentation/changelog/
1. christ, that page doesn't even list versions that old. it starts with "Version 6.0.1 (5th July 2019)"
I tried editing the page, but it looks empty.
1. there's a lower section below the content of the page with lots of input fields
2. on the General Settings -> Slider Shortcode input field, it says "[rev_slider alias="Home Slider"]"
3. so ^ that explains the top slider, but I can't find anywhere the second slider is referenced
I tried editing the frontpage on hetzner2, and I found it
1. below the actual content block, there's another block named "Page Builder"
2. inside the Page Builder block there's several more blocks
3. inside the second major block is another few sub blocks, one of which is white and says "PLUGIN SHORTCODES"
4. if you hover over the text that says 'PLUGIN SHORTCODES', then several icons appear, one of which is a pencil
5. if you click the pencil next to PLUGIN SHORTCODES, then a new modal window opens
6. after waiting about 30 seconds, there's a new content box that appears with the contents of '[rev_slider alias="Buildings"]'
back on hetzner3, this whole "page builder" block is entirely absent
if I go to the plugins page, then I see a bunch of messages

This theme requires the following plugins: Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Meta Box Framework and Safe SVG. The following recommended plugin is currently inactive: WPForms Lite. Begin installing plugins | Activate installed plugin | Dismiss this notice 

The plugin amr-shortcode-any-widget/amr-shortcode-any-widget.php has been deactivated due to an error: Plugin file does not exist.

The plugin be-page-builder/be-page-builder.php has been deactivated due to an error: Plugin file does not exist.

The plugin be-themes-one-click-import/init.php has been deactivated due to an error: Plugin file does not exist.

The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist.

The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist.

The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist.

ah crap, looks like 'be-page-builder' is missing from our plugins dir

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # ls -lah /var/tmp/wordpress/plugins | grep -i be-
drwxr-xr-x  7 root      root       4,0K Oct  9  2023 be-gdpr
-rw-r--r--  1 maltfield maltfield  306K Jan  6 01:50 be-gdpr.zip
drwxr-xr-x  4 root      root       4,0K Oct  4  2021 be-portfolio-post
-rw-r--r--  1 maltfield maltfield   56K Jan  6 01:50 be-portfolio-post.zip
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #

here's the plugins installed on hetzner2

[root@opensourceecology ~]# sudo -u wp -i wp --path="/var/www/html/www.openbuildinginstitute.org/htdocs/" plugin list
...
+------------------------------------------------+----------+--------+---------+
| name                                           | status   | update | version |
+------------------------------------------------+----------+--------+---------+
| akismet                                        | inactive | none   | 4.0     |
| amr-shortcode-any-widget                       | active   | none   | 3.6     |
| be-themes-one-click-import                     | active   | none   | 1.6     |
| be-page-builder                                | active   | none   | 4.6.1   |
| be-portfolio-post                              | active   | none   | 1.1     |
| duplicate-page                                 | active   | none   | 2.3     |
| wonderm00ns-simple-facebook-open-graph-tags    | active   | none   | 2.1.2   |
| force-strong-passwords                         | active   | none   | 1.7     |
| google-authenticator                           | active   | none   | 0.48    |
| google-authenticator-encourage-user-activation | active   | none   | 0.2     |
| masterslider                                   | active   | none   | 2.29.0  |
| meta-box-conditional-logic                     | active   | none   | 1.0.8   |
| meta-box-show-hide                             | active   | none   | 0.2.1   |
| meta-box-tabs                                  | active   | none   | 0.1.5   |
| oa-open-graph-for-fb                           | inactive | none   | 1.0.2   |
| open-in-new-window-plugin                      | active   | none   | 2.4     |
| rename-wp-login                                | active   | none   | 2.5.5   |
| revslider                                      | active   | none   | 5.2.5   |
| ssl-insecure-content-fixer                     | active   | none   | 2.5.0   |
| vcaching                                       | active   | none   | 1.6.7   |
| wordpress-importer                             | active   | none   | 0.6.3   |
+------------------------------------------------+----------+--------+---------+
[root@opensourceecology ~]#

it's curious that we're missing many of the plugins that oshine says are required, yet we do require this one that doesn't say it's required
I did a quick search of the wiki, my notes, and the Brand Exponents website, but I couldn't find anything that mentioned 'be-page-builder'
I asked them in an email

Hello,

Can you please tell me where I can download the 'be-page-builder' plugin?

Thank you for providing the other plugins; that was very helpful in fixing one of our oshine-based websites.

Unfortunately, another site appears to require 'be-page-builder', which I cannot find on wordpress.org

 * https://wordpress.org/plugin/be-page-builder

Can you please tell us where we can download the latest version of 'be-page-builder'?

Tue Jan 29, 2025

today I finished the setup of microfactory and documenting its CHG process on the wiki https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_microfactory_to_hetzner3#Change_Steps
...
moving on, I replaced the failed migration of oswh into making it a static site (since, unlike microfactory, it uses a theme that appears to be broken on the latest version of wordpress, and it's no longer developed)
I followed the same plan as documented for fef here https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_fef
I had to fix some hard-coded 'fef' in ^ that wiki article
after it was done, the frontpage verification looked mostly ok, but the carousel of sponsors at the bottom were clearly broken (instead of being a dynamic carousel, they just all appear in a vertical column)
also, some of the images on /be-a-partner/ were broken (well, some are broken on hetnzer2 too, but now there's more broken on hetnzer3)
1. https://oswh.opensourceecology.org/wp-content/uploads/2013/02/Scratch-300x166.png
also the image on this blog post was missing https://oswh.opensourceecology.org/oshw-docjam-how-to-make-a-satellite-jam/
1. https://oswh.opensourceecology.org/wp-content/uploads/2013/04/544788_10151353594600753_63894451_n.jpg
I re-ran the wget, this time writing to a log. But I didn't see any errors
here's all the files that we have on hetzner3

root@hetzner3 /var/www/html/oswh.opensourceecology.org # find htdocs/wp-content/uploads/2013
htdocs/wp-content/uploads/2013
htdocs/wp-content/uploads/2013/02
htdocs/wp-content/uploads/2013/02/FileMarcinted-180x152.jpeg
htdocs/wp-content/uploads/2013/02/CM_sq-300x300.jpeg
htdocs/wp-content/uploads/2013/02/CM_sq-300x300-180x152.jpeg
htdocs/wp-content/uploads/2013/02/BecomeAPartner-175x120.png
htdocs/wp-content/uploads/2013/02/Scratch-175x120.png
htdocs/wp-content/uploads/2013/02/FileMarcinted.jpeg
htdocs/wp-content/uploads/2013/02/8-copia-180x152.jpg
htdocs/wp-content/uploads/2013/02/8-copia-275x257.jpg
htdocs/wp-content/uploads/2013/02/open-source-ecology-175x120.png
htdocs/wp-content/uploads/2013/02/FileMarcinted-275x252.jpeg
htdocs/wp-content/uploads/2013/02/logouishare-175x120.jpeg
htdocs/wp-content/uploads/2013/02/CM_sq-300x300-275x257.jpeg
htdocs/wp-content/uploads/2013/02/8-copia.jpg
htdocs/wp-content/uploads/2013/02/Copia-di-Scratch-175x120.png
htdocs/wp-content/uploads/2013/03
htdocs/wp-content/uploads/2013/03/ODE-175x120-175x120.png
htdocs/wp-content/uploads/2013/03/sarapis-logo-square-175x120.png
htdocs/wp-content/uploads/2013/03/kb2-175x120.jpg
htdocs/wp-content/uploads/2013/03/wi-175x120.jpg
htdocs/wp-content/uploads/2013/03/Alchematter-175x120.png
htdocs/wp-content/uploads/2013/03/Adafruit-175x120.png
htdocs/wp-content/uploads/2013/03/ITP2-175x120.png
htdocs/wp-content/uploads/2013/03/sparkfun-logo.bmp
htdocs/wp-content/uploads/2013/04
htdocs/wp-content/uploads/2013/04/icehubs-175x120.png
root@hetzner3 /var/www/html/oswh.opensourceecology.org #

and here's all the files from the latest wget (which I didn't see errors)

[root@opensourceecology wget]# find oswh.opensourceecology.org/wp-content/uploads/
oswh.opensourceecology.org/wp-content/uploads/
oswh.opensourceecology.org/wp-content/uploads/2013
oswh.opensourceecology.org/wp-content/uploads/2013/02
oswh.opensourceecology.org/wp-content/uploads/2013/02/FileMarcinted.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/BecomeAPartner-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/02/open-source-ecology-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/02/8-copia-180x152.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/02/CM_sq-300x300-275x257.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/CM_sq-300x300-180x152.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/FileMarcinted-275x252.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/FileMarcinted-180x152.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/8-copia.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/02/8-copia-275x257.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/02/Copia-di-Scratch-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/02/logouishare-175x120.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/CM_sq-300x300.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/Scratch-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03
oswh.opensourceecology.org/wp-content/uploads/2013/03/sparkfun-logo.bmp
oswh.opensourceecology.org/wp-content/uploads/2013/03/kb2-175x120.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/03/ITP2-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/ODE-175x120-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/Alchematter-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/sarapis-logo-square-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/Adafruit-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/wi-175x120.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/04
oswh.opensourceecology.org/wp-content/uploads/2013/04/icehubs-175x120.png
[root@opensourceecology wget]#

the number of actual files is way, way higher.

[root@opensourceecology wget]# find /var/www/html/oswh.opensourceecology.org/htdocs/wp-content/uploads/ -type f | wc -l
360
[root@opensourceecology wget]#

it's not obvious to me why wget omitted this
maybe it's because the href desintation link is distinct from the image src?
anyway, I think the easiest thing to do is to just extract from the tarball of files and put them in-place

# STEP 5: Add all uploaded files (fix missing from wget)
tar -xjvf ${backupFileName_files_hetzner2}
rsync -av --progress var/www/html/${vhost_name}/htdocs/wp-content/uploads/ ${vhostDir}/htdocs/wp-content/uploads/

ok, adding ^ that fixed the broken links on the partners and the blog page
it looks like the header photo is broken because it's trying to load 'opensourcewarehouse.org', not 'oswh.opensourceecology.org'
I could probably fix this by doing some database find-and-replace CHG before converting it into a static site, but I'm not sure if it's worth the time/effort/money
the carousal at the bottom of the page is still broken. looks like it's updated by some 'jcarousal' library. I confirmed that I can load that jcarousal javascript file on the new server; it's not missing https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1
js console on-load has a ton of errors

This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. index.html
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png headfooter-logonew.png
Mixed Content: Upgrading insecure display request ‘http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png’ to use ‘https’ headfooter-logonew.png
Loading mixed (insecure) display content “http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png” on a secure page FaviconLoader.sys.mjs:175:20
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1”. index.html:191:110
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1”. index.html:192:101
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1”. index.html:193:115
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1”. index.html:194:93
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1”. index.html:195:105
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1”. index.html:196:102
The resource from “https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1”. index.html:197:111
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1”. index.html:198:116
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1”. index.html:199:126
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1”. index.html:200:123
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1”. index.html:201:113
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1”. index.html:202:118
GET
http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png
NS_BINDING_ABORTED

The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1”. index.html:203:102
A resource is blocked by OpaqueResponseBlocking, please check browser console for details. DocHackFavicon.png
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1”. index.html:204:102
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png headfooter-logonew.png
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Uncaught ReferenceError: jQuery is not defined
	<anonymous> https://oswh.opensourceecology.org/index.html:325
index.html:325:13
Uncaught ReferenceError: jQuery is not defined
	<anonymous> https://oswh.opensourceecology.org/index.html:402
index.html:402:21
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” index.html
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Mixed Content: Upgrading insecure display request ‘http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png’ to use ‘https’ headfooter-logonew.png
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” index.html
The resource from “https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0”. index.html:644:85
The resource from “https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1”. index.html:645:81
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://api.flattr.com/js/0.6/load.js?mode=auto index.html:652:26
Uncaught ReferenceError: jQuery is not defined
	<anonymous> https://oswh.opensourceecology.org/index.html:715
index.html:715:5
Uncaught ReferenceError: jQuery is not defined
	<anonymous> https://oswh.opensourceecology.org/index.html:925
index.html:925:33
Loading failed for the <script> with source “https://api.flattr.com/js/0.6/load.js?mode=auto”. index.html:1:1

here's the same console output when loading the site on hetzner2

This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. oswh.opensourceecology.org
Mixed Content: Upgrading insecure display request ‘http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png’ to use ‘https’ headfooter-logonew.png
Loading mixed (insecure) display content “http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png” on a secure page FaviconLoader.sys.mjs:175:20
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. node.js:416:1
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. jquery-1.7.2.min.js:3:32466
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” 2 oswh.opensourceecology.org
Loading failed for the <script> with source “https://api.flattr.com/js/0.6/load.js?mode=auto”. oswh.opensourceecology.org:1:1
GET
http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png
NS_BINDING_ABORTED

A resource is blocked by OpaqueResponseBlocking, please check browser console for details. DocHackFavicon.png

so all these MIME type errors are unique to hetzner3
google suggests it's actually because the file does not exist
quick check confirms that they're all 404

user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“//g' | tr "\n" " "
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” user@disp7253:/tmp$ 
user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“//g' | tr "\n" " " | xargs firefox

but I think the issue is actually all these weird html-encoded characters at the end
yep; all of these load fine

user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“g' | sed 's/%E2%80%9Dg' | tr "\n" " "
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0 https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1 user@disp7253:/tmp$ 
user@disp7253:/tmp$ 
user@disp7253:/tmp$ 
user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“g' | sed 's/%E2%80%9Dg' | tr "\n" " " | xargs firefox
user@disp7253:/tmp$

that's another red herring; if I click on it in the js console, it opens a new tab with the file. All that buffer crap was just added by firefox when I copied it out of the js console
ah, I think I know the problem: this has some insight https://stackoverflow.com/a/53062522
so the header 'X-Content-Type-Options: nosniff' makes the client not attempt to guess the MIME type
our server should be telling the correct mime type because it's a javascript file
but I think ^ that is determined by the file extension
...and the hetzer3 files are *not* using a '.js' extension – because wget is hard-coding these files with their get variable at the time they were scraped
proof: only '.js' files on hetzner2

[root@opensourceecology wget]# find /var/www/html/oswh.opensourceecology.org/ -type f | grep -i jcarousel
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor.old/script/horizontal/jquery.jcarousel.min.js
[root@opensourceecology wget]#

wheras on hetzer3, they don't end in '.js'

root@hetzner3 /var/www/html/oswh.opensourceecology.org # find /var/www/html/oswh.opensourceecology.org/ -type f | grep -i jcarousel
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
root@hetzner3 /var/www/html/oswh.opensourceecology.org #

in theory if I just manually rename the files by stripping everything after the '?', the webserver will still return the same file. let's try that
alright, this renamed all the files

js_files=$(find /var/www/html/oswh.opensourceecology.org/ -type f -iname '*\.js\?*')
for f in $js_files; do mv $f $(echo $f | sed 's/\.js\?.*/.js/'); done

here's the result after running that on hetzner3

root@hetzner3 /var/www/html/oswh.opensourceecology.org # find /var/www/html/oswh.opensourceecology.org/ -type f -iname '*\.js'
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/contact/contact.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/jscolor/jscolor.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/common.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/spiner/spin.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/menu/superfish.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/quickpager/quickpager.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/countdown/jquery.countdown.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/horizontal/jquery.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/js/imagesloaded.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/js/comment-reply.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/js/wp-embed.min.js
root@hetzner3 /var/www/html/oswh.opensourceecology.org #

well that didn't work; the carousal is still broken and I'm still getting a ton of errors about MIME types
here's a snippet from the actual root index.html file

<link rel='stylesheet' id='tiny_css-css'  href='wp-content/themes/Eventor/inc/shortcodes/css/tinymce.css%3Fver=4.9.1.css' type='text/css' media='all' />
<script type='text/javascript' src='wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/common.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1'></script>

I wonder if the issue is the '%3F' instead of '?'
I edited the file to replaced the '%3F' with '?

root@hetzner3 /var/www/html/oswh.opensourceecology.org # sed 's/\.js%3F/.js?/' htdocs/index.html | grep jquery
<script type='text/javascript' src='wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/countdown/jquery.countdown.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1'></script>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # sed 's/\.js%3F/.js?/' htdocs/index.html | grep jquery^C
You have mail in /var/mail/root
root@hetzner3 /var/www/html/oswh.opensourceecology.org # 

root@hetzner3 /var/www/html/oswh.opensourceecology.org # sed -i 's/\.js%3F/.js?/' htdocs/index.html
root@hetzner3 /var/www/html/oswh.opensourceecology.org #

I reloaded the page and, yeah, it's fixed. So wget is breaking things :(
ah ha! someone else is complaining about this on SO too https://stackoverflow.com/questions/22486047/wget-is-converting-question-marks-to-3f-causing-404s
1. they said that adding '--restrict-file-names="ascii" helped
I tried ^ that, but it still had the same problem; I think we'll just have to clean it up after
let's try this (for both js and css)

find -type f -iname '*\.html' -exec sed -i 's/\.js%3F/.js?/' htdocs/index.html '{}' \;
find -type f -iname '*\.html' -exec sed -i 's/\.css%3F/.css?/' htdocs/index.html '{}' \;
js_files=$(find /var/www/html/oswh.opensourceecology.org/htdocs/ -type f -iname '*\.js\?*')
for f in $js_files; do mv $f $(echo $f | sed 's/\.js\?.*/.js/'); done
css_files=$(find /var/www/html/oswh.opensourceecology.org/htdocs/ -type f -iname '*\.css\?*')
for f in $css_files; do mv $f $(echo $f | sed 's/\.css\?.*/.css/'); done

I updated the process with this on the wiki, re-did the migration on hetzner3, and now it works. great!
I guess that, while I'm editing these files produced by wget, one more thing I could do is to replaced all instances of 'opensourcewarehouse.org' with 'oswh.opensourceecology.org', which would fix some more things like the broken header image

find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse.org/oswh.opensourceecology.org/' '{}' \;

this last one is specific to oswh, so I created a new CHG just for it here https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_oswh
for some reason ^ that command isn't working
1. well there's one issue that 'www.opensourcewarehouse.org' becomes 'www.oswh.opensourceecology.org' – which isn't going to work
2. and another issue where there's a bunch of orphaned 'opensourcewarehouse.org' strings still :(

find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/www\.opensourcewarehouse\.org/oswh.opensourceecology.org/' '{}' \;
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse\.org/oswh.opensourceecology.org/' '{}' \;

oh, I had permissions issues. now the header logo is fixed. nice!
oh wait, but the carousal is broken again :(
here's the current console messages on refresh

This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. index.html
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT

Loading mixed (insecure) display content “http://oswh.opensourceecology.org/wp-content/uploads/2013/02/DocHackFavicon.png%E2%80%9D on a secure page FaviconLoader.sys.mjs:175:20
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT

Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. node.js:416:1
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT

The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1%E2%80%9D. index.html:193:113
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT

The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1%E2%80%9D. index.html:202:116
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://api.flattr.com/js/0.6/load.js?mode=auto index.html:652:26
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0%E2%80%9D index.html
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Uncaught TypeError: jQuery(...).jcarousel is not a function
	<anonymous> https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js?ver=4.9.1:117
	jQuery 11
common.js:117:27
Blocked loading mixed active content “http://player.vimeo.com/video/58165438%E2%80%9D index.html
Loading failed for the <script> with source “https://api.flattr.com/js/0.6/load.js?mode=auto%E2%80%9D. index.html:1:1

ok, that's actually a MIME red-herring. It's returning 'text/html' because it's a 404 error message.

Tue Jan 28, 2025

here's TOFU 3/3 (ISP, exit in Ecuador)

Ecuador
2025-01-28
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . 


https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
################################################################################## 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
################################################################################## 100.0%
2025-01-28
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
0a8dfec431583aac98eaa4c454a8be0641a54dc93704413c56bc861a8816a563  plugin.json
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip

well, a diff of this round of 3TOFUs shows everything matching, except 'woocommerce-gateway-stripe', which changed versions in-between our TOFUs
1. it's not a big deal; we still have an older version TOFUd on the server

root@hetzner3 /var/tmp/wordpress/plugins # ls | grep -i gate
coingate-for-woocommerce
coingate-for-woocommerce.2.1.1.zip
woocommerce-gateway-stripe
woocommerce-gateway-stripe.9.0.0.zip
root@hetzner3 /var/tmp/wordpress/plugins #

I uploaded all these to hetzner3
I updated the CHG article to indicate the newer versions
...
ok, now that 3TOFU is out of the way, we're unblocked on microfactory.opensourceecology.org, www.openbuildinginstitute.org, and www.opensourceecology.org, and phplist.opensourceecology.org
let's start with microfactory
I went through the install script for microfactory again (but this time it should have copied the oshine themes and dependent plugins)
this time after login, the website appeared pretty broken still
but after I logged-in and clicked the db upgrade, the website looked great!
1. oh, except only when I'm logged-in. If I open another browser while not logged-in, it's still broken
2. also, I get an error after logging-in, when I get redirected to here https://microfactory.opensourceecology.org/wp-admin/profile.php

There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.

Learn more about troubleshooting WordPress.

here's the critical error from the apache logs

==> microfactory.opensourceecology.org/error.log <==
[Wed Jan 29 00:04:53.421065 2025] [proxy_fcgi:error] [pid 3214134:tid 3214134] [client 146.70.74.100:0] AH01071: Got error 'PHP message: MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg; PHP message: MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_3_6389662388.jpg; PHP message: MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_5_2197896279.jpg; PHP message: PHP Fatal error:  Uncaught TypeError: ftp_fput(): Argument #1 ($ftp) must be of type FTP\\Connection, null given in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php:212\nStack trace:\n#0 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php(212): ftp_fput()\n#1 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider/admin/includes/msp-admin-functions.php(141): WP_Filesystem_FTPext->put_contents()\n#2 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(141): msp_save_custom_styles()\n#3 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(116): Master_Slider_Admin->after_plugin_update()\n#4 [internal function]: Master_Slider_Admin->admin_init()\n#5 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-includes/c...'

indeed, the file it wants does not exist

root@hetzner3 ~ # ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg
ls: cannot access '/var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg': No such file or directory
root@hetzner3 ~ #

root@hetzner3 ~ # ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/
total 8,0K
drwxrwx--- 2 not-apache www-data 4,0K Sep 26  2018 .
drwxrwx--- 3 not-apache www-data 4,0K Sep 26  2018 ..
root@hetzner3 ~ #

curiously, it doesn't exist on the old server either. hmm. maybe this is one of those issues where it was a warning on old php and its a crit error on new php

[root@opensourceecology ~]# ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/
total 8.0K
drwxrwx--- 2 apache apache 4.0K Sep 26  2018 .
drwxrwx--- 3 apache apache 4.0K Sep 26  2018 ..
[root@opensourceecology ~]#

I tailed the logs on the old sever as I logged-in there, and I got nothing
I also grepped the old logs, and I saw nothing similar

[root@opensourceecology httpd]# zgrep 'MS_Aq_Resize.process' microfactory.opensourceecology.org/error_log-*.gz
[root@opensourceecology httpd]# zgrep 'imageedit_5_2197896279.jpg' microfactory.opensourceecology.org/error_log-*.gz
[root@opensourceecology httpd]#

I'm wondering if we can just disable this master slider plugin, or if our site actually uses it
I checked the master slider section of the wp dashboard on the old site, and it showed one entry named 'v1-home' = "v1 Home copy"
clicking preview just displayed a spinner that stuck forever
I checked the logs and, yep, now it's spitting out errors about missing files

[Wed Jan 29 00:14:05.177783 2025] [:error] [pid 1296] [client 127.0.0.1:40418] MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg, referer: https://microfactory.opensourceecology.org/wp-admin/admin.php?page=masterslider
[Wed Jan 29 00:14:05.178387 2025] [:error] [pid 1296] [client 127.0.0.1:40418] MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_3_6389662388.jpg, referer: https://microfactory.opensourceecology.org/wp-admin/admin.php?page=masterslider
[Wed Jan 29 00:14:05.178944 2025] [:error] [pid 1296] [client 127.0.0.1:40418] MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_5_2197896279.jpg, referer: https://microfactory.opensourceecology.org/wp-admin/admin.php?page=masterslider
[Wed Jan 29 00:14:09.005403 2025] [:error] [pid 2552] [client 127.0.0.1:40742] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 1). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "312"] [id "960012"] [rev "1"] [msg "POST request missing Content-Length Header."] [data "0"] [severity "WARNING"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"] [tag "CAPEC-272"] [hostname "microfactory.opensourceecology.org"] [uri "/wp-cron.php"] [unique_id "Z5ly0cY3Bv263PeWaEUCWwAAAAg"]

so I just disabled the plugin on hetzner3

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider
masterslider/     masterslider.zip  
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # rm -rf /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider*
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #

I refreshed the wp dashboard, and now it loads
at the top, it yells at me saying that it wants Master Slider, as well as other plugins tho

This theme recommends the following plugins: Master Slider, Meta Box Framework and Safe SVG.
The following recommended plugin is currently inactive: WPForms Lite.

if I'm not logged-in, the site still looks pretty broken; showing this at the top

[tatsu_section bg_color= “rgba(229,238,243,1)” bg_image= “https://microfactory.opensourceecology.org/wp-content/uploads/2018/10/ose-workshop.jpg” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= “top left” bg_size= “cover” bg_animation= “none” padding= ‘{“d”:”80px 0px 120px 0px “,”l”:”0px 0px 60px 0px “,”t”:”125px 0px 125px 0px “,”m”:”120px 0px 120px 0px “}’ margin= ‘{“d”:”0-px 0px 0px 0px “}’ border= “0px 0px 0px 0px” border_color= “” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “1” overlay_color= “rgba(255,255,255,0.63)” full_screen= “1” top_divider= “none” bottom_divider= “none” top_divider_height= ‘{“d”:100}’ top_divider_position= “above” bottom_divider_height= ‘{“d”:”165″}’ bottom_divider_position= “below” top_divider_color= “rgba(255,255,255,1)” bottom_divider_color= “#ffffff” invert_top_divider= “0” invert_bottom_divider= “0” flip_top_divider= “0” flip_bottom_divider= “1” section_id= “home” section_class= “” section_title= “” offset_section= “0” offset_value= “0px” full_screen_header_scheme= “background–light” hide_in= “0” key= “ryhdp1iKX”][tatsu_row full_width= “0” bg_color= “rgba(255,255,255,0)” no_margin_bottom= “0” equal_height_columns= “0” gutter= “medium” column_spacing= “px” fullscreen_cols= “0” swap_cols= “0” padding= ‘{“d”:”100px 0px 0px 0px “}’ row_id= “” row_class= “” hide_in= “0” box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” border_radius= “0” layout= “2/3+1/3” key= “BJx2daJsK7”][tatsu_column bg_color= “” bg_image= “” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= ‘{“d”:”top left”}’ bg_size= ‘{“d”:”cover”}’ padding= ‘{“d”:”0px 50px 0% 0% “,”m”:”0% 0% 0% 0% “}’ custom_margin= “0” margin= ‘{“d”:”0px 0px 0px 0px”}’ border= ‘{“d”:”0px 0px 0px 0px”}’ border_color= “” border_radius= “0” enable_box_shadow= “0” box_shadow_custom= “0 0 15px 0 rgba(198,202,202,0.4)” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “0” overlay_color= “” animate_overlay= “none” link_overlay= “” vertical_align= “middle” column_offset= “0” offset= “0px 0px” z_index= “0” column_parallax= “0” column_width= ‘{“d”:66.67,”l”:66.67,”t”:66.67,”m”:100}’ column_mobile_spacing= “0” animate= “0” animation_type= “fadeIn” animation_delay= “0” image_hover_effect= “none” column_hover_effect= “none” hover_box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” col_id= “” column_class= “” hide_in= “0” layout= “2/3” key= “r1-3OTyjF7”][tatsu_inline_text max_width= ‘{“d”:”100″}’ wrap_alignment= “center” animate= “1” animation_type= “fadeIn” animation_delay= “100” margin= ‘{“d”:”0px 0px 20px 0px “}’ bg_color= “” box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” padding= ‘{“d”:”0px 0px 0px 0px”}’ border_radius= “0” key= “Hyzh_pksKm”][/tatsu_inline_text][tatsu_inline_text max_width= ‘{“d”:”100″}’ wrap_alignment= “center” animate= “1” animation_type= “fadeIn” animation_delay= “100” margin= ‘{“d”:”0px 0px 20px 0px “}’ bg_color= “” box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” padding= ‘{“d”:”0px 0px 0px 0px”}’ border_radius= “0” key= “r1m3u6ksF7”]

I confirmed that the tatsu plugin is installed and activated
on the plugin page, we have a bunch of other error messages at the top

The plugin colorhub/colorhub.php has been deactivated due to an error: Plugin file does not exist.

The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist.

The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist.

The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist.

The plugin typehub/typehub.php has been deactivated due to an error: Plugin file does not exist.

Some required plugins are missing or inactive.

oh, I realized the issue with the site being broken while not logged-in is because it was cached; let's clear that

root@hetzner3 ~ # varnishadm 'ban req.url ~ "."'

You have new mail in /var/mail/root
root@hetzner3 ~ #

I did a quick visual check of all the pages, and they look identical on hetzner2 and hetzner3. I'd say this is done!
so the only thing we need to add is to delete the master slider plugin after install
I created a new wiki article for the steps for this migration, which will diverge from store https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_microfactory_to_hetzner3
the versions to change are different now
here's what we have on hetzner2

[root@opensourceecology ~]# sudo -u wp -i wp --path="/var/www/html/microfactory.opensourceecology.org/htdocs/" plugin list
...
+------------------------------------------------+----------+--------+---------+
| name                                           | status   | update | version |
+------------------------------------------------+----------+--------+---------+
| akismet                                        | active   | none   | 4.0.8   |
| be-gdpr                                        | active   | none   | 1.0.0   |
| be-portfolio-post                              | active   | none   | 1.1     |
| colorhub                                       | active   | none   | 1.0.1   |
| duplicate-page                                 | active   | none   | 2.7     |
| force-strong-passwords                         | active   | none   | 1.8.0   |
| google-authenticator                           | active   | none   | 0.48    |
| google-authenticator-encourage-user-activation | active   | none   | 0.2     |
| hello                                          | inactive | none   | 1.7     |
| masterslider                                   | active   | none   | 3.2.2   |
| meta-box-conditional-logic                     | active   | none   | 1.3     |
| meta-box-show-hide                             | active   | none   | 1.0.2   |
| meta-box-tabs                                  | active   | none   | 1.0.0   |
| oshine-core                                    | active   | none   | 1.3.4   |
| oshine-modules                                 | active   | none   | 2.2.5   |
| rename-wp-login                                | active   | none   | 2.5.5   |
| revslider                                      | active   | none   | 5.4.7.4 |
| ssl-insecure-content-fixer                     | active   | none   | 2.7.0   |
| tatsu                                          | active   | none   | 2.6.8   |
| typehub                                        | active   | none   | 1.2     |
| vcaching                                       | active   | none   | 1.6.9   |
+------------------------------------------------+----------+--------+---------+
[root@opensourceecology ~]# 

[root@opensourceecology ~]# sudo -u wp -i wp --path="/var/www/html/microfactory.opensourceecology.org/htdocs/" theme list 
...
+-----------------+----------+--------+---------+
| name            | status   | update | version |
+-----------------+----------+--------+---------+
| oshin           | active   | none   | 6.5     |
| twentyeleven    | inactive | none   | 2.8     |
| twentyfifteen   | inactive | none   | 2.0     |
| twentyfourteen  | inactive | none   | 2.2     |
| twentyseventeen | inactive | none   | 1.7     |
| twentysixteen   | inactive | none   | 1.5     |
| twentyten       | inactive | none   | 2.5     |
| twentythirteen  | inactive | none   | 2.4     |
| twentytwelve    | inactive | none   | 2.5     |
+-----------------+----------+--------+---------+
[root@opensourceecology ~]#

and here's hetzner3

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # sudo -u wp -i wp --path="${docrootDir}" plugin list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+------------------------------------------------+----------+--------+----------+----------------+-------------+
| name                                           | status   | update | version  | update_version | auto_update |
+------------------------------------------------+----------+--------+----------+----------------+-------------+
| activitypub                                    | active   | none   | 4.6.0    |                | off         |
| akismet                                        | active   | none   | 5.3.5    |                | off         |
| aurora-heatmap                                 | active   | none   | 1.7.0    |                | off         |
| be-gdpr                                        | active   | none   | 1.1.6    |                | off         |
| be-portfolio-post                              | active   | none   | 1.1.1    |                | off         |
| bulk-media-register                            | inactive | none   | 1.40     |                | off         |
| advanced-nocaptcha-recaptcha                   | inactive | none   | 7.5.0    |                | off         |
| duplicate-page                                 | active   | none   | 4.5      |                | off         |
| enable-media-replace                           | inactive | none   | 4.1.5    |                | off         |
| google-authenticator-encourage-user-activation | active   | none   | 0.2      |                | off         |
| extensions-leaflet-map                         | inactive | none   | 4.4.3    |                | off         |
| google-authenticator                           | active   | none   | 0.54     |                | off         |
| hcaptcha-for-forms-and-more                    | inactive | none   | 4.9.0    |                | off         |
| hello                                          | inactive | none   | 1.7.2    |                | off         |
| include-mastodon-feed                          | inactive | none   | 1.9.9    |                | off         |
| leaflet-map                                    | inactive | none   | 3.4.1    |                | off         |
| melapress-login-security                       | active   | none   | 2.0.1    |                | off         |
| meta-box-conditional-logic                     | active   | none   | 1.6.23   |                | off         |
| meta-box-show-hide                             | active   | none   | 1.3.1    |                | off         |
| meta-box-tabs                                  | active   | none   | 1.1.18   |                | off         |
| oshine-core                                    | active   | none   | 1.6.1    |                | off         |
| oshine-modules                                 | active   | none   | 3.3.8    |                | off         |
| raw-html                                       | inactive | none   | 1.6.4    |                | off         |
| regenerate-thumbnails                          | inactive | none   | 3.1.6    |                | off         |
| related-posts-by-taxonomy                      | inactive | none   | 2.7.6    |                | off         |
| revslider                                      | active   | none   | 6.7.25   |                | off         |
| smart-slider-3                                 | inactive | none   | 3.5.1.26 |                | off         |
| spam-destroyer                                 | inactive | none   | 2.1.4    |                | off         |
| ssl-insecure-content-fixer                     | active   | none   | 2.7.2    |                | off         |
| tatsu                                          | active   | none   | 3.5.3    |                | off         |
| vcaching                                       | active   | none   | 1.8.3    |                | off         |
| woocommerce-multilingual                       | inactive | none   | 5.3.9    |                | off         |
| woocommerce-gateway-stripe                     | inactive | none   | 9.0.0    |                | off         |
| wp-2fa                                         | inactive | none   | 2.8.0    |                | off         |
| wpforms-lite                                   | inactive | none   | 1.9.2.3  |                | off         |
| wpfront-notification-bar                       | inactive | none   | 3.4.2    |                | off         |
| wp-pgp-encrypted-emails                        | inactive | none   | 0.8.0    |                | off         |
| wp-qrcode                                      | inactive | none   | 1.1.1    |                | off         |
| wps-hide-login                                 | inactive | none   | 1.9.17.1 |                | off         |
| wordpress-seo                                  | inactive | none   | 24.1     |                | off         |
+------------------------------------------------+----------+--------+----------+----------------+-------------+
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # 

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # sudo -u wp -i wp --path="${docrootDir}" theme list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+-----------------+----------+--------+---------+----------------+-------------+
| name            | status   | update | version | update_version | auto_update |
+-----------------+----------+--------+---------+----------------+-------------+
| oshin           | active   | none   | 7.2.1   |                | off         |
| twentyeleven    | inactive | none   | 4.8     |                | off         |
| twentyfifteen   | inactive | none   | 3.9     |                | off         |
| twentyfourteen  | inactive | none   | 4.1     |                | off         |
| twentyseventeen | inactive | none   | 3.8     |                | off         |
| twentysixteen   | inactive | none   | 3.4     |                | off         |
| twentyten       | inactive | none   | 4.3     |                | off         |
| twentythirteen  | inactive | none   | 4.3     |                | off         |
| twentytwelve    | inactive | none   | 4.4     |                | off         |
+-----------------+----------+--------+---------+----------------+-------------+
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #

Fri Jan 17, 2025

here's TOFU 2/3 (VPN, exit in Switzerland)

Switzerland
2025-01-17
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . 


https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
2025-01-17
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
16fb41570f8e66b11358c9dfcd754384018af22b0d8c6a19dff50b2ce005a0f5  plugin.json
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip

Sun Jan 05, 2025

here's TOFU 3/3 (ISP, exit in Ecuador)

Ecuador
2025-01-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
INFO: Determining Latest Version of Wordpress Themes 
	. . . . . . . . . . . . 


WARNING: Failed to download plugin duplicate-page
	null
	null

WARNING: Failed to download plugin hello
	Plugin not found.
	null

WARNING: Failed to download plugin f
	closed
	This plugin has been closed as of March 25, 2012 and is not available for download. Reason: Unknown.

WARNING: Failed to download theme oshin
	Theme not found
	null

https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
-=O=-                                        #     #    #    #                 
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
  -=O=-          #    #     #     #                                            
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
		  -=O#-   #    #     #                                                 
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
##O=#  #                                                                       
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
 -=#=-#  #    #                                                                
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
##O#- #                                                                        
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
									  -=O=-                            #  # ## 
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
		-#O=- #    #    #                                                      
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
		   #    #    #    #                      -=O=-                         
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
##                                                                -=O=-        
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
##O#- #                                                                        
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
##O#- #                                                                        
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/akismet.5.3.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/classic-editor.1.6.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/contact-form-7.6.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.9.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/jetpack.14.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/shareaholic.9.7.13.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.26.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce.9.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyeleven.4.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfifteen.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfourteen.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentynineteen.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyseventeen.3.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentysixteen.3.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyten.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentythirteen.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwelve.4.4.zip
######################################################################### 100.0%
2025-01-06
33656ccc5752ff7c519d328ac4b9819116aa4e2195b436edc132a9362d1a23f6  activitypub.4.6.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
9e72d6c60c8e3adc9b202bd6ad8659605289cd0921d5f9726dabf24e06a50a16  akismet.5.3.5.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0  black-studio-tinymce-widget.2.7.3.zip
371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213  bouquet.1.2.5.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86  chartbeat.2.0.7.zip
4b2b45b19c61f627ff8730222692a691023dea3435b35b8db95a2418b45ece65  classic-editor.1.6.7.zip
8b1475dfaa0f8a73f8a374286a52f59b6f3d3573e80ea1134d3a527498bdd7ea  coingate-for-woocommerce.2.3.0.zip
445152a678b9e307b52e4e96b4acff9de4744efc25c55b16ec45aec731356268  CologneBlue-REL1_42-007b63c.tar.gz
eb0eb22bdfc8b1cb553274064732c406bb173d7f43e73212c8220986e5f6fb1d  CologneBlue-REL1_43-9134d2e.tar.gz
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
32bd9b76da748f0bda96588ffa50d51604365097e21e930325a5bd9817393240  ConfirmAccount-REL1_43-bb470fd.tar.gz
d3880e7405c6842e164b0f974b9ed55e1a48006f86a2b28ed511128865323d89  contact-form-7.6.0.2.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058  google-authenticator.0.54.zip
9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e  google-authenticator-encourage-user-activation.0.2.zip
c61bab4e64eeff3d519e5a24990f5ba8223651f13aed2e0fb69b904419e62c50  hcaptcha-for-forms-and-more.4.9.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
4d824673ad59e74c5bdfc9c6b6c87341b2d0f0879641b72381612aed30335758  jetpack.14.1.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
08f15bb64ddbdf0813ec5e581515cedae5e8aee9d9a3b0c5e4b2726a5a2ac982  meta-box.5.10.5.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
7ccbc9703a0f194ad3219383de1d12f28e373598ed2448e07145a096d6598a0a  Modern-REL1_42-c8a7124.tar.gz
07e855134780a5f1f9925d391f7eaed6f968f553988237ccc261d7e41ffb5829  Modern-REL1_43-5597681.tar.gz
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
7e1fc73f395bc93f958c8fd5418e0b2acc63dcbaf5acba4c044e67c489e546a0  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
2ee4950e6c2bb47ac2614a02529697a59aee8fb967f094a329d78e9626395f92  safe-svg.2.3.1.zip
97960efc1647253080c82739a47ea8b5239257e05065a3a7f24f5e3b5935e7b7  shareaholic.9.7.13.zip
514d1156304ffff54883b4dff57a789b36983d163a12276127ad9c2399f6614d  smart-slider-3.3.5.1.26.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
fb26419580bbc8f0d35842fd98a1a9bf44851453d20c373fadb8b488272547a4  ssl-insecure-content-fixer.2.7.2.zip
576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311  storefront.4.6.0.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
e33538c5a99ac6c6d083f5e70fdf390c5471247e47a44751b6a7a531b903287f  twentyeleven.4.8.zip
dc3c9337a95ff0619f8f4f2bbe494d0caf445af7e050b93f0de818e799e2a850  twentyfifteen.3.9.zip
91b68629634e181b3b5ba2852f97ae9abdbc4cf3455026264a0d5e323db88907  twentyfourteen.4.1.zip
3b7adce871a143bca30dbfb421a69b005cc53e252c9c9a0e4d768571f6c5be3e  twentynineteen.3.0.zip
166035cbf15c3d618b5d2c11d1aa365b93d597f32735aa6f35887f4828673377  twentyseventeen.3.8.zip
04f14d0210cf72797d29c60132d2de4dc2ea00fffffcb824aa9e95fba385c7d4  twentysixteen.3.4.zip
21cd8f5688fc21493becee40bfb99a57837750f5ef6accca7abce5bcb1c37b46  twentyten.4.3.zip
6016a433cd38023181b00d6e44a14b976bc4963ce5fe97033b6af6046360411f  twentythirteen.4.3.zip
f6a5b6c3c8691f9a75f95d9e6675889a217cb57a8339e37b8aef182a7a4a4eaf  twentytwelve.4.4.zip
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
7b64058a82e2a33d452ae8172ae394dce603c788fd3d8b1cad9c8894a5b4bd30  UserMerge-REL1_43-f9d2664.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c  vcaching.1.8.3.zip
543e34e5dbb2ef74af9ada84767e9118e200bb8a1046a5d49a03c7e462b9a13e  Widgets-REL1_42-17dbd92.tar.gz
7946e4c3c970792a9de655dce79fa68bad2023084bb24e8d01694b4d17329778  Widgets-REL1_43-50da5c6.tar.gz
c38427b9bebd799ff97e68593247dbbb1eef3a3ecd0503c0026c32fdc779de74  woocommerce.9.5.1.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
ce79a4dace0e87caa9314d9b773a9663506556cb78cd19d388522fc5401185bd  wordpress-seo.24.1.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d  wp-super-cache.1.12.4.zip
user@disp8573:/tmp/tmp.cHDv7hw2dy$

ok, we do have some files that didn't match on all 3 TOFUs
1. coingate-for-woocommerce wordpress plugin
2. ConfirmAccount MediaWiki extension (for v4.2)
3. extensions-leaflet-map wordpress plugin
4. UserMerge MediaWiki extension (for v4.2)
the coingate plugin says that it changed from v2.2.0 to v2.3.0, so that makes sense that it would change
the leaflet-map extension says it changed from v4.4.2 to v4.4.3, so that also makes sense
the two mediawiki extensions were supposed to be tied to a tag, so I wouldn't expect them to change. But I guess it might make sense if they moved the tags. Not a very good release process, if so.
1. we're probably are going to want the v4.3 versions of both the MediaWiki plugins, so this probably isn't an issue
anyway, I'll keep these ones that were verified and we can re-verify the ones that failed. Here's a follow-up 3TOFU script

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES="https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz"
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='extensions-leaflet-map woocommerce-gateway-stripe'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
sha256sum *

here's TOFU 1/3 (Tor, exit in the Netherlands)

Congratulations. This browser is configured to use Tor.
2025-01-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . 


https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
2025-01-06
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
902ae1c09846dc522c9688d375b944375566b8e1980c38355a131ff48108b555  plugin.json
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@host:/tmp/user/1000/tmp.ddVAgjDjeV$

Fri Jan 03, 2025

here's TOFU 2/3 (VPN, exit in Belgium)

Belgium
2025-01-03
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
INFO: Determining Latest Version of Wordpress Themes 
	. . . . . . . . . . . . 


WARNING: Failed to download plugin duplicate-page
	null
	null

WARNING: Failed to download plugin hello
	Plugin not found.
	null

WARNING: Failed to download plugin f
	closed
	This plugin has been closed as of March 25, 2012 and is not available for download. Reason: Unknown.

WARNING: Failed to download theme oshin
	Theme not found
	null

https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
-#O#- #   #                                                                    
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
   -#O=#   #    #                                                              
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
	   -=#=-  #    #    #                                                      
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
-#O#- #   #                                                                    
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
-#O#- #   #                                                                    
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
-#O#- #   #                                                                    
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
								   -=O=-                             #   # # # 
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
	  -=O#-   #    #    #                                                      
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
							 -=O=-                      #    #    #   #        
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
										-=O=-                               ## 
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
-#O#- #   #                                                                    
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
-=#=#   #   #                                                                  
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/akismet.5.3.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/classic-editor.1.6.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/contact-form-7.6.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.9.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/jetpack.14.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/shareaholic.9.7.13.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.26.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce.9.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyeleven.4.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfifteen.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfourteen.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentynineteen.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyseventeen.3.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentysixteen.3.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyten.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentythirteen.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwelve.4.4.zip
######################################################################### 100.0%
2025-01-03
33656ccc5752ff7c519d328ac4b9819116aa4e2195b436edc132a9362d1a23f6  activitypub.4.6.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
9e72d6c60c8e3adc9b202bd6ad8659605289cd0921d5f9726dabf24e06a50a16  akismet.5.3.5.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0  black-studio-tinymce-widget.2.7.3.zip
371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213  bouquet.1.2.5.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86  chartbeat.2.0.7.zip
4b2b45b19c61f627ff8730222692a691023dea3435b35b8db95a2418b45ece65  classic-editor.1.6.7.zip
8b1475dfaa0f8a73f8a374286a52f59b6f3d3573e80ea1134d3a527498bdd7ea  coingate-for-woocommerce.2.3.0.zip
445152a678b9e307b52e4e96b4acff9de4744efc25c55b16ec45aec731356268  CologneBlue-REL1_42-007b63c.tar.gz
eb0eb22bdfc8b1cb553274064732c406bb173d7f43e73212c8220986e5f6fb1d  CologneBlue-REL1_43-9134d2e.tar.gz
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
32bd9b76da748f0bda96588ffa50d51604365097e21e930325a5bd9817393240  ConfirmAccount-REL1_43-bb470fd.tar.gz
d3880e7405c6842e164b0f974b9ed55e1a48006f86a2b28ed511128865323d89  contact-form-7.6.0.2.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
c58461cb4ab04746356a9bd6809da1cf699ffaad7f3be3987c8e6e2cd1830023  extensions-leaflet-map.4.4.2.zip
b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058  google-authenticator.0.54.zip
9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e  google-authenticator-encourage-user-activation.0.2.zip
c61bab4e64eeff3d519e5a24990f5ba8223651f13aed2e0fb69b904419e62c50  hcaptcha-for-forms-and-more.4.9.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
4d824673ad59e74c5bdfc9c6b6c87341b2d0f0879641b72381612aed30335758  jetpack.14.1.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
08f15bb64ddbdf0813ec5e581515cedae5e8aee9d9a3b0c5e4b2726a5a2ac982  meta-box.5.10.5.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
7ccbc9703a0f194ad3219383de1d12f28e373598ed2448e07145a096d6598a0a  Modern-REL1_42-c8a7124.tar.gz
07e855134780a5f1f9925d391f7eaed6f968f553988237ccc261d7e41ffb5829  Modern-REL1_43-5597681.tar.gz
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
1ed7cade320a6be1aec419d5b30d04e71c9e9692165de5ecc6c1df8b75e21d11  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
2ee4950e6c2bb47ac2614a02529697a59aee8fb967f094a329d78e9626395f92  safe-svg.2.3.1.zip
97960efc1647253080c82739a47ea8b5239257e05065a3a7f24f5e3b5935e7b7  shareaholic.9.7.13.zip
514d1156304ffff54883b4dff57a789b36983d163a12276127ad9c2399f6614d  smart-slider-3.3.5.1.26.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
fb26419580bbc8f0d35842fd98a1a9bf44851453d20c373fadb8b488272547a4  ssl-insecure-content-fixer.2.7.2.zip
576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311  storefront.4.6.0.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
e33538c5a99ac6c6d083f5e70fdf390c5471247e47a44751b6a7a531b903287f  twentyeleven.4.8.zip
dc3c9337a95ff0619f8f4f2bbe494d0caf445af7e050b93f0de818e799e2a850  twentyfifteen.3.9.zip
91b68629634e181b3b5ba2852f97ae9abdbc4cf3455026264a0d5e323db88907  twentyfourteen.4.1.zip
3b7adce871a143bca30dbfb421a69b005cc53e252c9c9a0e4d768571f6c5be3e  twentynineteen.3.0.zip
166035cbf15c3d618b5d2c11d1aa365b93d597f32735aa6f35887f4828673377  twentyseventeen.3.8.zip
04f14d0210cf72797d29c60132d2de4dc2ea00fffffcb824aa9e95fba385c7d4  twentysixteen.3.4.zip
21cd8f5688fc21493becee40bfb99a57837750f5ef6accca7abce5bcb1c37b46  twentyten.4.3.zip
6016a433cd38023181b00d6e44a14b976bc4963ce5fe97033b6af6046360411f  twentythirteen.4.3.zip
f6a5b6c3c8691f9a75f95d9e6675889a217cb57a8339e37b8aef182a7a4a4eaf  twentytwelve.4.4.zip
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
7b64058a82e2a33d452ae8172ae394dce603c788fd3d8b1cad9c8894a5b4bd30  UserMerge-REL1_43-f9d2664.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c  vcaching.1.8.3.zip
543e34e5dbb2ef74af9ada84767e9118e200bb8a1046a5d49a03c7e462b9a13e  Widgets-REL1_42-17dbd92.tar.gz
7946e4c3c970792a9de655dce79fa68bad2023084bb24e8d01694b4d17329778  Widgets-REL1_43-50da5c6.tar.gz
c38427b9bebd799ff97e68593247dbbb1eef3a3ecd0503c0026c32fdc779de74  woocommerce.9.5.1.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
ce79a4dace0e87caa9314d9b773a9663506556cb78cd19d388522fc5401185bd  wordpress-seo.24.1.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d  wp-super-cache.1.12.4.zip
user@disp8342:/tmp/tmp.i0oBc9groR$

Wed Jan 01, 2025

Happy new year!
This year my major goal is to finish the hetzner3 project
The hetzner3 project is a migration of all of OSE's websites from the old server (hetzner2 running CentOS) to the new dedicated server (hetzner3 running Debian)
I finished last year with the following
1. forum.opensourceecology.org - I finished copying a snapshot of the forums to hetnzer3; it passed my initial spot checks and looks OK
2. store.opensourceecology.org - I finished copying a snapshot of this website to hetnzer3; it's horribly broken, but it's an empty website so that's fine. Marcin said we can drop it entirely, to simplify things
3. microfactory.opensourceecology.org - I finished copying a snapshot of this website to hetnzer3; it was pretty broken, and it's blocked until we finish obtaining the oshine theme's required (paid) plugins
4. fef.opensourceecology.org - I finished migrating a snapshot of this website to hetnzer3; it had a critical error and theme was >12 years old. We decided to convert it to a static site, which I generated and finished copying to hetzner3; it passed my initial spot checks and looks OK
5. oswh.opensourceecology.org - I finished migrating a snapshot of this website to hetnzer3; it uses a paid theme called "eventor". I found the website that sells the theme, but couldn't see any version history. I contacted the author 6 days ago, and they didn't (yet) respond. Marcin said we can convert it to a static site. I'm waiting to hear back from the theme author. If they don't reply in a couple weeks, I'll convert it to a static site.
6. seedhome.openbuildinginstitute.org - I finished copying a snapshot of this website to hetnzer3; it's an empty site and looks fine. Marcin said we can drop it entirely, to simplify things.
7. www.openbuildinginstitute.org - I haven't begun on this. It's blocked until we finish obtaining the oshine theme's required (paid) plugins
8. www.opensourceecology.org - I haven't begun on this. Catarina wants to transition it to the oshine theme.
9. phplist.opensourceecology.org - I haven't tried to copy this to hetzner3 yet. I did start a 3TOFU verification of the latest release, but the hash changed after 3TOFU. I created a question on the forums about this, and I'm waiting for a response to clarify what happened.
10. wiki.opensourceecology.org - I have copied this to hetzner2, but it's a long upgrade process that I'm still working on.
...
ok, where we left off before was trying figure out the URLs to download these oshine required paid plugins (which are not on wordpress.org)
I was successfully able to to mitm a wordpress install (using mitmproxy) on a Disposable VM as oshine tried to download its dependent plugins
I found that I could press 'e' to export a given capture in mitmproxy as a cURL command, but that only worked for one request at a time
I found that I could limit all the captures shown to *just* those that downloaded a .zip by pressing the 'f' button (for "filter") and then typing:

: set view_filter '~t application/zip'

there is a complete list of filters and their definitions (that's how I found that ~t is 'content-type') here https://docs.mitmproxy.org/stable/concepts-filters/
ok, the oshine wordpress theme downloaded 10 zip files. Here's an export of all those cURL commands

root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# for file in $(ls -1 | sort -V); do echo "==${file}=="; cat $file; echo; echo; done
==c1==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip

==c2==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip

==c3==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip

==c4==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/oshin-plugins/be-portfolio-post.zip' -H 'Connection: close' https://brandexponents.com/oshin-plugins/be-portfolio-post.zip

==c5==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip

==c6==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip

==c7==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip

==c8==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip

==c9==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip

==c10==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip

root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#

here's just the URLs

root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# for file in $(ls -1 | sort -V); do cat $file | awk '{print $NF}'; echo; echo; done | sort | uniq

https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#

cool, I'm able to download all of them with just a simple wget; this will work for 3TOFU

root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# wget https://brandexponents.com/oshin-plugins/be-portfolio-post.zip https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
...
--2025-01-01 15:57:44--  https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
Reusing existing connection to brandexponents.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘tatsu.zip’

tatsu.zip               [     <=>            ]  15.81M  1.96MB/s    in 7.8s    

2025-01-01 15:57:52 (2.03 MB/s) - ‘tatsu.zip’ saved [16579153]

FINISHED --2025-01-01 15:57:52--
Total wall clock time: 22s
Downloaded: 10 files, 47M in 19s (2.49 MB/s)
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# 

root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# sha256sum *.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# 

root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# du -sh *.zip
308K	be-gdpr.zip
56K	be-portfolio-post.zip
1.6M	masterslider.zip
8.0K	meta-box-conditional-logic.zip
8.0K	meta-box-show-hide.zip
8.0K	meta-box-tabs.zip
18M	oshine-core.zip
1.7M	oshine-modules.zip
11M	revslider.zip
16M	tatsu.zip
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#

so the three plugins that are missing from the above list are the three recommended ones that can be downloaded (most easily & safely) from wordpress.org:
1. meta-box
2. safe-svg
3. wpforms-lite
I used hetzner3:/usr/local/bin/wordpress_3tofu.sh to generate a 3TOFU script

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:10:08+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='  activitypub akismet aurora-heatmap bulk-media-register advanced-nocaptcha-recaptcha duplicate-page enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed leaflet-map melapress-login-security raw-html regenerate-thumbnails related-posts-by-taxonomy smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce-multilingual woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wordpress-seo  activitypub akismet aurora-heatmap bulk-media-register advanced-nocaptcha-recaptcha classic-editor contact-form-7 enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed leaflet-map melapress-login-security meta-box raw-html regenerate-thumbnails related-posts-by-taxonomy smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce woocommerce-multilingual coingate-for-woocommerce woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wordpress-seo  activitypub akismet aurora-heatmap bulk-media-register advanced-nocaptcha-recaptcha enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed leaflet-map melapress-login-security raw-html regenerate-thumbnails related-posts-by-taxonomy smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce-multilingual woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wordpress-seo  activitypub akismet aurora-heatmap black-studio-tinymce-widget bulk-media-register advanced-nocaptcha-recaptcha chartbeat enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed jetpack leaflet-map melapress-login-security raw-html regenerate-thumbnails related-posts-by-taxonomy shareaholic smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce-multilingual woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wp-super-cache wordpress-seo '
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

themes='  oshin twentyeleven twentyfifteen twentyfourteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve  oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve  twentyeleven twentyfifteen twentyfourteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve  bouquet twentyeleven twentyfifteen twentyfourteen twentyseventeen twentysixteen twentyten twentythirteen '
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
sha256sum *

oh shit, not only is 'wpforms-lite' already in the list -- it's in the list 4 times! I guess I need to update the script to do a de-dupe
I updated the script

root@hetzner3 /usr/local/bin # diff wordpress_3tofu.sh.20250101 wordpress_3tofu.sh
5c5
< # Version: 0.2
---
> # Version: 0.3
11c11
< # Updated: 2024-12-12
---
> # Updated: 2025-01-01
53a54,56
> # remove duplicates and trim whitespace
> plugins="$(echo "${plugins}" | tr " " "\n" | sort | uniq | tr "\n" " " | xargs)"
> 
64a68,70
> #
> # remove duplicates and trim whitespace
> themes="$(echo "${themes}" | tr " " "\n" | sort | uniq | tr "\n" " " | xargs)"
root@hetzner3 /usr/local/bin #

ok, this run looks much better

root@hetzner3 /usr/local/bin # wordpress_3tofu.sh

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES=""
WARNINGS=""

# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='activitypub advanced-nocaptcha-recaptcha akismet aurora-heatmap black-studio-tinymce-widget bulk-media-register chartbeat classic-editor coingate-for-woocommerce contact-form-7 duplicate-page enable-media-replace extensions-leaflet-map google-authenticator google-authenticator-encourage-user-activation hcaptcha-for-forms-and-more hello include-mastodon-feed jetpack leaflet-map melapress-login-security meta-box raw-html regenerate-thumbnails related-posts-by-taxonomy shareaholic smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce woocommerce-gateway-stripe woocommerce-multilingual wordpress-seo wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wp-super-cache'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

themes='bouquet oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
sha256sum *

You have mail in /var/mail/root
root@hetzner3 /usr/local/bin #

I manually edited the script above to include 'safe-svg' and 'meta-box' ('wpforms-lite' were already present)
I also added some remote files at the top manually, including the paid plugins and phpList
I also added the extensions for mediawiki
I realized that previously I had chosen the extensions for the release of mediawiki v1.42 because it said that was the latest stable version
in fact, that drop down is wrong. the latest stable LTS version of mediawiki is 1.43. To be safe, I added both.
as for mediawiki skins, it looks like 1.43 ships with 4 already

root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # ls -lah /var/tmp/mediawiki/mediawiki-1.43.0/skins/
total 28K
drwxr-xr-x  6 root root  4,0K Dec 29 18:14 .
drwxr-xr-x 14 root root  4,0K Dec 29 18:14 ..
drwxr-xr-x 11 root root  4,0K Dec 29 18:14 MinervaNeue
drwxr-xr-x  7 root root  4,0K Dec 29 18:14 MonoBook
-rw-r--r--  1  501 staff 1,3K Dec  5 15:41 README
drwxr-xr-x  6 root root  4,0K Dec 29 18:14 Timeless
drwxr-xr-x  9 root root  4,0K Dec 29 18:14 Vector
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #

we're using these

root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep skins LocalSettings.php 
## The relative URL path to the skins directory
# $wgStylePath        = "$wgScriptPath/skins";
#require_once "$IP/skins/CologneBlue/CologneBlue.php";
#require_once "$IP/skins/Modern/Modern.php";
#require_once "$IP/skins/MonoBook/MonoBook.php";
#require_once "$IP/skins/Vector/Vector.php";
#$wgRightsIcon = "${wgScriptPath}/skins/common/images/gnu-fdl.png";
root@hetzner3 /var/www/html/wiki.opensourceecology.org #

so it appears that we just need to download CologneBlue & Modern.
1. https://www.mediawiki.org/wiki/Skin:Cologne_Blue
2. https://www.mediawiki.org/wiki/Skin:Modern
alright, here's the 3TOFU script

################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES="https://brandexponents.com/oshin-plugins/be-portfolio-post.zip https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz"
WARNINGS=""

# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

plugins='activitypub advanced-nocaptcha-recaptcha akismet aurora-heatmap black-studio-tinymce-widget bulk-media-register chartbeat classic-editor coingate-for-woocommerce contact-form-7 duplicate-page enable-media-replace extensions-leaflet-map google-authenticator google-authenticator-encourage-user-activation hcaptcha-for-forms-and-more hello include-mastodon-feed jetpack leaflet-map melapress-login-security f raw-html regenerate-thumbnails related-posts-by-taxonomy shareaholic smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce woocommerce-gateway-stripe woocommerce-multilingual wordpress-seo wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wp-super-cache safe-svg meta-box'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
	echo -n '. '

	json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
	latest_version=$(cat plugin.json | jq -r .version)
	url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
	
	if [ "${url}" = "null" ]; then
		error=$(cat plugin.json | jq -r .error);
		description=$(cat plugin.json | jq -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} ${url}"
	fi
	
done
echo

themes='bouquet oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
sha256sum *

and here's TOFU 1/3 (Tor, exit in Austria)

Congratulations. This browser is configured to use Tor.
2025-01-01
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins 
	. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
INFO: Determining Latest Version of Wordpress Themes 
	. . . . . . . . . . . . 


WARNING: Failed to download plugin duplicate-page
	null
	null

WARNING: Failed to download plugin hello
	Plugin not found.
	null

WARNING: Failed to download plugin f
	closed
	This plugin has been closed as of March 25, 2012 and is not available for download. Reason: Unknown.

WARNING: Failed to download theme oshin
	Theme not found
	null

https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
 -=O=-                   #     #     #     #                                   
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
              -=O=-                            #     #    #    #               
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
               -=O=-                      #     #     #    #                   
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
 -=O=-       #    #    #     #                                                 
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
 -=O=-             #    #     #     #                                          
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
 -=O=-           #    #     #     #                                            
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
                                  -=O=- #     #     #    #                     
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
                                  -=O=-                                     ## 
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
    #  #   #    #                                      -=O=-                   
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
                        -=O=-                                            #  ###
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
-=O=-        #    #    #     #                                                 
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
-=O=-    #    #    #    #                                                      
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/akismet.5.3.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/classic-editor.1.6.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.2.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/contact-form-7.6.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.9.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/jetpack.14.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/shareaholic.9.7.13.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.26.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce.9.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyeleven.4.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfifteen.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfourteen.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentynineteen.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyseventeen.3.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentysixteen.3.4.zip
/######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyten.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentythirteen.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwelve.4.4.zip
######################################################################### 100.0%
2025-01-01
33656ccc5752ff7c519d328ac4b9819116aa4e2195b436edc132a9362d1a23f6  activitypub.4.6.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
9e72d6c60c8e3adc9b202bd6ad8659605289cd0921d5f9726dabf24e06a50a16  akismet.5.3.5.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0  black-studio-tinymce-widget.2.7.3.zip
371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213  bouquet.1.2.5.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86  chartbeat.2.0.7.zip
4b2b45b19c61f627ff8730222692a691023dea3435b35b8db95a2418b45ece65  classic-editor.1.6.7.zip
0c3c2b80926f43c5c7ca77fda54e00240af616a541705ae7118af14f38ab48c6  coingate-for-woocommerce.2.2.0.zip
445152a678b9e307b52e4e96b4acff9de4744efc25c55b16ec45aec731356268  CologneBlue-REL1_42-007b63c.tar.gz
eb0eb22bdfc8b1cb553274064732c406bb173d7f43e73212c8220986e5f6fb1d  CologneBlue-REL1_43-9134d2e.tar.gz
f5c873c0e7aecae24a5e209d589db344ee199b0e7fa78906606ec5705b3b8752  ConfirmAccount-REL1_42-7405319.tar.gz
32bd9b76da748f0bda96588ffa50d51604365097e21e930325a5bd9817393240  ConfirmAccount-REL1_43-bb470fd.tar.gz
d3880e7405c6842e164b0f974b9ed55e1a48006f86a2b28ed511128865323d89  contact-form-7.6.0.2.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
c58461cb4ab04746356a9bd6809da1cf699ffaad7f3be3987c8e6e2cd1830023  extensions-leaflet-map.4.4.2.zip
b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058  google-authenticator.0.54.zip
9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e  google-authenticator-encourage-user-activation.0.2.zip
c61bab4e64eeff3d519e5a24990f5ba8223651f13aed2e0fb69b904419e62c50  hcaptcha-for-forms-and-more.4.9.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
4d824673ad59e74c5bdfc9c6b6c87341b2d0f0879641b72381612aed30335758  jetpack.14.1.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
08f15bb64ddbdf0813ec5e581515cedae5e8aee9d9a3b0c5e4b2726a5a2ac982  meta-box.5.10.5.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
7ccbc9703a0f194ad3219383de1d12f28e373598ed2448e07145a096d6598a0a  Modern-REL1_42-c8a7124.tar.gz
07e855134780a5f1f9925d391f7eaed6f968f553988237ccc261d7e41ffb5829  Modern-REL1_43-5597681.tar.gz
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
1c045983b84a270a305a84ede3a07941037cc3448e9e2e7a51a2cebcd1044715  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
2ee4950e6c2bb47ac2614a02529697a59aee8fb967f094a329d78e9626395f92  safe-svg.2.3.1.zip
97960efc1647253080c82739a47ea8b5239257e05065a3a7f24f5e3b5935e7b7  shareaholic.9.7.13.zip
514d1156304ffff54883b4dff57a789b36983d163a12276127ad9c2399f6614d  smart-slider-3.3.5.1.26.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
fb26419580bbc8f0d35842fd98a1a9bf44851453d20c373fadb8b488272547a4  ssl-insecure-content-fixer.2.7.2.zip
576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311  storefront.4.6.0.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
e33538c5a99ac6c6d083f5e70fdf390c5471247e47a44751b6a7a531b903287f  twentyeleven.4.8.zip
dc3c9337a95ff0619f8f4f2bbe494d0caf445af7e050b93f0de818e799e2a850  twentyfifteen.3.9.zip
91b68629634e181b3b5ba2852f97ae9abdbc4cf3455026264a0d5e323db88907  twentyfourteen.4.1.zip
3b7adce871a143bca30dbfb421a69b005cc53e252c9c9a0e4d768571f6c5be3e  twentynineteen.3.0.zip
166035cbf15c3d618b5d2c11d1aa365b93d597f32735aa6f35887f4828673377  twentyseventeen.3.8.zip
04f14d0210cf72797d29c60132d2de4dc2ea00fffffcb824aa9e95fba385c7d4  twentysixteen.3.4.zip
21cd8f5688fc21493becee40bfb99a57837750f5ef6accca7abce5bcb1c37b46  twentyten.4.3.zip
6016a433cd38023181b00d6e44a14b976bc4963ce5fe97033b6af6046360411f  twentythirteen.4.3.zip
f6a5b6c3c8691f9a75f95d9e6675889a217cb57a8339e37b8aef182a7a4a4eaf  twentytwelve.4.4.zip
d17b77d02d736493413760ab2651745f4fb705ab569ef8c56aea48dee11ad074  UserMerge-REL1_42-41759d0.tar.gz
7b64058a82e2a33d452ae8172ae394dce603c788fd3d8b1cad9c8894a5b4bd30  UserMerge-REL1_43-f9d2664.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c  vcaching.1.8.3.zip
543e34e5dbb2ef74af9ada84767e9118e200bb8a1046a5d49a03c7e462b9a13e  Widgets-REL1_42-17dbd92.tar.gz
7946e4c3c970792a9de655dce79fa68bad2023084bb24e8d01694b4d17329778  Widgets-REL1_43-50da5c6.tar.gz
c38427b9bebd799ff97e68593247dbbb1eef3a3ecd0503c0026c32fdc779de74  woocommerce.9.5.1.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
ce79a4dace0e87caa9314d9b773a9663506556cb78cd19d388522fc5401185bd  wordpress-seo.24.1.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d  wp-super-cache.1.12.4.zip
user@host:/tmp/user/1000/tmp.aQZdXtbcsc$

ok, after 3 days (assuming everything matches), we'll be unblocked on microfactory.opensourceecology.org, www.openbuildinginstitute.org, and www.opensourceecology.org, and phplist.opensourceecology.org
in the meantime, we can continue with the wiki upgrade process
...
I also finished my logs and hours and sent a bill to OSE for 49 hours in the month of Dec 2024

Maltfield Log/2025 Q1: Difference between revisions

Revision as of 22:33, 12 February 2025

Contents

See Also

Thr Feb 06, 2025

Sun Feb 02, 2025

Sat Feb 01, 2025

Fri Jan 31, 2025

Thr Jan 30, 2025

Tue Jan 29, 2025

Tue Jan 28, 2025

Fri Jan 17, 2025

Sun Jan 05, 2025

Fri Jan 03, 2025

Wed Jan 01, 2025

Navigation menu

Maltfield Log/2025 Q1: Difference between revisions

Revision as of 22:33, 12 February 2025

See Also

Thr Feb 06, 2025

Sun Feb 02, 2025

Sat Feb 01, 2025

Fri Jan 31, 2025

Thr Jan 30, 2025

Tue Jan 29, 2025

Tue Jan 28, 2025

Fri Jan 17, 2025

Sun Jan 05, 2025

Fri Jan 03, 2025

Wed Jan 01, 2025

Navigation menu

Search