Maltfield Log/2025 Q1: Difference between revisions

From Open Source Ecology
Jump to navigation Jump to search
No edit summary
(mar 16)
Line 8: Line 8:
# [[Special:Contributions/Maltfield]]
# [[Special:Contributions/Maltfield]]


=Sat Mar 15, 2025=
=Sun Mar 16, 2025=


# Marcin generated a new ssh key for me to auth his access
# I sent a follow-up email to Tom to confirm that his new ssh key was
# I sent him an email asking him to confirm that the key was
## generated following our best-practices as documented on the wiki and https://wiki.opensourceecology.org/wiki/OSE_Server#SSH
## generated following our best-practices as documented on the wiki and https://wiki.opensourceecology.org/wiki/OSE_Server#SSH
## probably backed-up on his veracrypt-encrypted usb drive
## probably backed-up on his veracrypt-encrypted usb drive
# once he confirms, I'll add it to his authorized_keys file, and then we can confirm access is working
# ...
# Marcin sent me an email that seemed to suggest that we should delay the content fixes of osemain until after we migrate to hetzner3. I sent an email asking to confirm if that's his intention
# ...
# ...
# Catarina sent me output of `ls -lah ~/.ssh` which showed she has no ssh keypair
# Here's TOFU 2/3 (VPN, exit in Germany)
# I sent her the commands to generate a new keypair and asked if she has a backup solution in-place
# ...
# Catarina said she found the links to login to her new wiki admin account, but the password didn't work (maybe it expired)
# I logged-into the wiki today and tried to reset the password, but I got an error when trying to query for the username https://wiki.opensourceecology.org/index.php?title=Special%3AUserCredentials&target=CatarinaAdmin
<pre>
No credentials found for this user. Check that the name is spelled correctly.
</pre>
# I checked the list of users in the admin group, and it includes the ones I thought I had deleted!?
# According to my notes, I did this already on 2025-02-16 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2025_Q1#Sun_Feb_16.2C_2025
# my best-guess is that I accidentally made the change on hetzner3, not hetzner2
# again, I edited these accounts, removing the 'administrator' role
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Audrey+Rampone
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Elifarley
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Hart
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Will
# I then tried to create the "CatarinaAdmin" user, but it said it already exists!
<pre>
Username entered already in use. Please choose a different name.
</pre>
# It looks like CatarinaAdmin isn't in the admin group yet, though
# first I confirmed her email address is correct. Ugh, she has 3x accounts already with the same email address
<pre>
MariaDB [osewiki_db]> select user_name,user_email from wiki_user where user_name like '%Catarina%';
+---------------+--------------------------+
| user_name    | user_email              |
+---------------+--------------------------+
| Catarina Mota | REDACTED |
| Catarina      | REDACTED |
| Catarinamota  | REDACTED |
| CatarinaAdmin | REDACTED |
+---------------+--------------------------+
4 rows in set (0.00 sec)
 
MariaDB [osewiki_db]>
 
</pre>
# I used this special page to send her a new password https://wiki.opensourceecology.org/wiki/Special:PasswordReset
# once she's able to confirm access, I'll grant it admin and ask her to setup 2FA.
# ...
# Catarina responded to my email about wordpress themes, asking me to install all of them on the new osemain site on hetzner3 – including twentytwentyfive
## https://wordpress.org/themes/twentytwentyfive/
## https://wordpress.org/themes/twentytwentyfour/
## https://wordpress.org/themes/generatepress/
## https://wordpress.org/themes/astra/
## https://wordpress.org/themes/oceanwp/
## https://wordpress.org/themes/hestia/
## https://wordpress.org/themes/neve/
# time for a new 3TOFU; I needed to do one for a few of the mediawiki plugins I suggested for Marcin earlier, anyway
<pre>
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
#
#          For more info on 3TOFU (and why this is important), see:
#          * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################
 
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --location --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
 
REMOTE_FILES="https://extdist.wmflabs.org/dist/extensions/DeleteBatch-REL1_43-59ffc3f.tar.gz https://extdist.wmflabs.org/dist/extensions/SmiteSpam-REL1_43-d45b08a.tar.gz https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz"
WARNINGS=""
 
# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
 
tmpDir=`mktemp -d`
pushd "${tmpDir}"
 
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
 
# and today's date
date -u +"%Y-%m-%d"
 
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
 
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
 
themes='twentytwentyfive'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
 
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
 
echo -e "${WARNINGS}"
echo
 
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
 
# checksum
date -u +"%Y-%m-%d"
du -sh *
sha256sum *
</pre>
# Here's TOFU 1/3 (Exit in Germany)
<pre>
<pre>
Congratulations. This browser is configured to use Tor.
Germany
2025-03-16
2025-03-16
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Core
Line 157: Line 28:
######################################################################### 100.0%
######################################################################### 100.0%
https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz
https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz
  -=O=-               #    #    #     #                                    
-=O=-                         #    #    #   #                  
https://downloads.wordpress.org/release/wordpress-6.7.2.zip
https://downloads.wordpress.org/release/wordpress-6.7.2.zip
####################                                                      27.6%Crash Annotation GraphicsCriticalError: |[0][GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt (t=51.5764) [GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
######################################################################### 100.0%
######################################################################### 100.0%
[1]+  Done                    torbrowser  (wd: ~)
(wd now: /tmp/user/1000/tmp.sHf2RQALl0)
https://downloads.wordpress.org/theme/twentytwentyfive.1.1.zip
https://downloads.wordpress.org/theme/twentytwentyfive.1.1.zip
######################################################################### 100.0%
######################################################################### 100.0%
Line 176: Line 44:
dfaed40766bedcbf1879949e1a60abf816104725d5c416c1e97a4f2edcf9b7c2  v1.8.22.tar.gz
dfaed40766bedcbf1879949e1a60abf816104725d5c416c1e97a4f2edcf9b7c2  v1.8.22.tar.gz
b6acbc116aabdc5a7c9bc4e861647148a1b444c1f2beec5440261b88f02c3a6e  wordpress-6.7.2.zip
b6acbc116aabdc5a7c9bc4e861647148a1b444c1f2beec5440261b88f02c3a6e  wordpress-6.7.2.zip
user@disp8402:/tmp/tmp.rUpogE2lis$
</pre>
</pre>
=Tue Mar 04, 2025=
# In our meeting yesterday, Marcin and Catarina asked me if GeneratePress (a wordpress theme I recommended) would continue to get updates if you didn't renew the license ($50-$100 in annual fees)
# I said I'd need to double-check, but I had read this on their website before
<pre>
Q: Will my website break if I don’t renew my license key?
A: No, your website will remain intact. Once you purchase GP Premium, you own it for life and are free to use it for as long as you like. The only difference is support and updates
</pre>
# I went to confirm this today
# First I tried to find a sales contact email so I could write them and CC Marcin & Catarina; that failed. There was only a presales form, but I'm not going to give that my email address (and get spammed forever like I did with brand exponents)
# I found a suppot forum, but it was broken with an infinite loop of misconfigured cloudflare https://generate.support/
## I tried in chromium, but it refused to let me register for an account without a license key
# I ended-up resorting to the support forums, which is what it says the plebians without license keys must use (probably better to have this discouse publicly, anyway)
# I realized that the Q&A I read above was actually removed from the site. You can see from Oct 2023 that it was present https://webcf.waybackmachine.org/web/20231006105111/https://generatepress.com/pricing/
# but the current site it's absent https://generatepress.com/pricing/
# I asked why this was removed, and if it means that the website *will* break now if we don't keep paying https://wordpress.org/support/topic/will-my-website-break-if-i-dont-renew-my-license-key-in-2025/
# I also noticed that they don't list lifetime licenses. my notes say there was a lifetime license option for $250, but it's missing
## I found this article that said they removed the lifetime licenses in Jun 2024 (last year) https://generatepress.com/new-pricing-changes-lifetime-licenses/
## but I see that the docs *still* say you can buy a lifetime license after you buy a 1-year license (like an upgrade) https://docs.generatepress.com/article/upgrading-license-keys/
## I asked if this is true or false advertising https://wordpress.org/support/topic/lifetime-licenses-still-possible/
# I also went ahead and asked if they offer discounts for nonprofits https://wordpress.org/support/topic/discounts-for-nonprofits/
# I also asked them to very explicitly enumerate all of the things that need to be updated, so I can better understand exactly what will break and what won't break if we don't renew https://wordpress.org/support/topic/what-updates/
# I also emailed Tom to follow-up on the ssh access confirmation; I haven't heard from him in over 2 weeks
# ...
# ...
# I'm still kinda blocked on osemain
# I tried to load the wiki (on hetzner2), but I got a varnish error
# in our meeting yesterday, Catarina and Marcin stressed that getting Catarina a "sandbox" asap where she can rebuid the osemain site with a new theme is one of our highest priorities
## but this is blocked because Marcin brought-up a desire to fix some broken links in some old posts before we make the fork
# Marcin wasn't exactly sure what was broken. He suggested that it's an http -> https redirect, but I don't think so
# We tabled it during the call because we found it was difficult just to even find a list of all the posts
# I'm going to see if I can make this easier for him
# after some fidgiting with the db, I was able to extract the names, IDs, and title with this
<pre>
<pre>
select ID, post_date, post_title from wp_posts where post_type = 'page';
</pre>
# figuring out the slugs are non-trivial https://permalinkmanager.pro/blog/where-are-permalinks-stored/
# but I found that I can load any page (or post) with just /index.php?page_id=<id>
# so here's some commands to generate a CSV file to help him out
<pre>
sudo su -
cd /home/maltfield
source /root/backups/backup.settings


# dump from db
Error 503 Backend fetch failed
time nice echo "select post_date,ID, post_title from wp_posts where post_type = 'page' order by post_date;" | mysql -u root -p$mysqlPass osemain_db | tr "\t" "," > osemain_pages.$(date "+%Y%m%d").csv
time nice echo "select post_date,ID, post_title from wp_posts where post_type = 'post' order by post_date;" | mysql -u root -p$mysqlPass osemain_db | tr "\t" "," > osemain_posts.$(date "+%Y%m%d").csv


# add URLs
Backend fetch failed
cat osemain_pages.$(date "+%Y%m%d").csv | awk -F',' '{print $1","$2",https://www.opensourceecology.org/index.php?page_id="$2",\""$3"\""}' > osemain_pages2.$(date "+%Y%m%d").csv
Guru Meditation:
cat osemain_posts.$(date "+%Y%m%d").csv | awk -F',' '{print $1","$2",https://www.opensourceecology.org/index.php?page_id="$2",\""$3"\""}' > osemain_posts2.$(date "+%Y%m%d").csv


chown maltfield /home/maltfield/osemain_p*csv
XID: 58196320
</pre>
# then on my local computer, I fetched it with
<pre>
rsync -av --progress opensourceecology.org:/home/maltfield/osemain_p*csv .
</pre>
# here's what I get
<pre>
[root@opensourceecology maltfield]# head osemain_posts2.20250305.csv
post_date,ID,https://www.opensourceecology.org/index.php?page_id=ID,"post_title"
2007-10-19 19:40:53,7057,https://www.opensourceecology.org/index.php?page_id=7057,"The greatest construction in human history."
2007-10-20 11:04:05,7058,https://www.opensourceecology.org/index.php?page_id=7058,"State of Survival"
2007-10-20 20:31:42,9,https://www.opensourceecology.org/index.php?page_id=9,"From Humanure to Hawthorns"
2007-10-21 19:46:44,7059,https://www.opensourceecology.org/index.php?page_id=7059,"Logs and cookies"
2007-10-23 22:05:56,7060,https://www.opensourceecology.org/index.php?page_id=7060,"Lister Ailments"
2007-10-25 19:57:39,7061,https://www.opensourceecology.org/index.php?page_id=7061,"Signs of the season"
2007-10-27 18:26:10,7062,https://www.opensourceecology.org/index.php?page_id=7062,"The Volunteers Experiment."
2007-10-29 21:21:11,7063,https://www.opensourceecology.org/index.php?page_id=7063,"Online presence."
2007-10-29 21:29:08,7064,https://www.opensourceecology.org/index.php?page_id=7064,"Piglet"
[root@opensourceecology maltfield]#


[root@opensourceecology maltfield]# head osemain_pages2.20250305.csv
Varnish cache server
post_date,ID,https://www.opensourceecology.org/index.php?page_id=ID,"post_title"
2013-02-13 05:11:07,91,https://www.opensourceecology.org/index.php?page_id=91,"Home"
2014-02-06 22:30:00,168,https://www.opensourceecology.org/index.php?page_id=168,"Machines: Global Village Construction Set"
2014-02-07 16:10:00,194,https://www.opensourceecology.org/index.php?page_id=194,"Calendar"
2014-02-07 16:10:18,196,https://www.opensourceecology.org/index.php?page_id=196,"Contact"
2014-02-07 16:10:32,198,https://www.opensourceecology.org/index.php?page_id=198,"Press"
2014-02-07 16:10:48,200,https://www.opensourceecology.org/index.php?page_id=200,"Wiki"
2014-02-07 16:23:30,206,https://www.opensourceecology.org/index.php?page_id=206,"About"
2014-02-07 17:04:04,212,https://www.opensourceecology.org/index.php?page_id=212,"Workshops"
2014-02-07 17:04:33,216,https://www.opensourceecology.org/index.php?page_id=216,"Development"
[root@opensourceecology maltfield]#
</pre>
</pre>
# I sent these to Marcin over email
# I'm still able to ssh-in
# I restarted apache, but--wow--it took maybe 2 minutes for this command to finish
<pre>
<pre>
Hey Marcin,
[root@opensourceecology ~]# systemctl restart httpd
 
[root@opensourceecology ~]#
You have 719 posts + 90 pages on osemain.
 
I was hoping to give you a hand with the verification of the broken links that you wanted to fix on hetzner2's live osemain site -- which is blocking me creating the fork for Catarina's "sandbox" on hetzer3, but there's a lot more content than I realized.
 
Catarina, I hope you can find a theme that is able to showcase different categories -- so that this content isn't effectively lost (currently it's nearly impossible to find!)
 
I prepared a list of all these URLs for you in CSV files. See attached.
 
You should be able to open the files in libreoffice calc.
 
You can then highlight some subset of the third column (with the URLs) to copy a chunk of URLs to load into your browser.
 
You can use a plugin like URLs List, which lets you paste into it a line-delimited list of URLs to open in bulk
 
* https://addons.mozilla.org/en-US/firefox/addon/urls-list/
 
Please let me know if you find a theme of broken links that I can try to fix with automation.
</pre>
</pre>
=Mon Mar 03, 2025=
# billed time for February
# ...
# ...
# Met with Marcin & Catarina to discuss osemain and obi and the rest of the hetzer3 migration
# I updated the OBI CHG wiki article for the migration to hetzter3 to include both [a] a static site *and* [b] a broken wordpress site https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_obi_to_hetzner3
# I proposed my idea to migrate two copies of both osemain and obi:
# in order to test this, I need a new IP address and I'll need to update the ansible configs with a new vhost that binds to this new IP addres
## I migrate a curl-built static site, which becomes the publicly accessible website for both on their respective domain names
# I tried to login to the hetzner account using the password in the OSE shared keepass, but I kept getting "Invalid credentials."
## I also migrate the actual wordpress site, but it is *not* referenced by any domain name. OSE buys an additional IPv4 address for these two sites, which is manually set in the /etc/hosts file by Catarina & Marcin to update the obi site's bugs and replace/refresh the osemain site (we called this the "sandbox" in the call)
# I launched a new VM in chromium (to ensure it wasn't some b& due to my hardened firefox), and I the same error; including the account being locked
# Catarina was very happy with this solution, and said we should do it
# Marcin gave me the go-ahead to purchase the second IPv4 address (~5 EUR one-time + ~2 EUR/mo)
## he asked if the cost would be forever, and I said that he could cancel the second IPv4 address after they finish fixing the obi & osemain sites, and make *those* live (to replace the static sites)
# Marcin said that he wanted to be able to make new blog posts on hetzner2 *after* the migration (since the live static site on hetzner3 wouldn't be editable)
## So I guess that means we'll only have a static site of obi, not osemain?
## Nah, I'll go ahead and make a static site of osemain anyway – just in-case.
## I warned about the risks of "split brain fork". Catarina said she'd just manually copy & paste new posts from hetzner2 osemain to hetzner3 osemain. I mentioned there's additional content to sync, not just the posts. For example, comments.
## Marcin said all comments have been disabled on all blogs anyway, which mitigates this issue
## Marcin said there's some broken links on hetzner2's live osemain site, which we agreed he should identify and fix (perhaps I'll help with some mysql sed task, if it's found to be automateable) *before* we do the osemain migtation. He said it was http->https issues on our own site's domains, but I said it's probably something else since we have redirects for http to https on all our nginx vhosts. We'll see.
## Marcin said he'd follow-up with an email by tomorrow identifying the issues. Then we can prioritize the migration of osemain, so Catarina has a sandbox on hetzner3 to being fixing the theme asap.
# We talked about the issue of embedable comment threads, and I mentiond recommended
# I sent Marcin & Catarina a follow-up email about ActivityPub
<pre>
<pre>
Hey Marcin,
Invalid credentials.
 
This is the plugin that turns your wordpress site into an ActivityPub instance:
 
* https://wordpress.org/plugins/activitypub/
 
As stated during our meeting today, I'm already going to be enabling this plugin on all your wordpress sites after they're migrated to hetzner3.
 
This will allow people to subscribe to your wordpress blog from other ActivityPub platforms -- such as Mastodon, PeerTube, Threads, Lemmy, Pixelfed, Friendica, GNU Social, other Wordpress sites with this plugin, etc.
 
* https://en.wikipedia.org/wiki/ActivityPub#Software_using_ActivityPub
 
I don't plan on doing anything other than turning it on, setting the actor name (eg @ose@opensourceecology.org), profile photo, and testing that it's findable on mastodon.
 
I don't have much experience with this plugin myself, but you'll be able to play around with it and see if the federated comments, likes, etc meet your needs. I do think this is the most future-proof way for users to interact between websites (eg for comments).
 
If you want to roll-up your sleeves and dig into the fediverse, I recommend creating an account on these platforms and giving a feel for it. As stated, all these platforms interact with ActivityPub so it doesn't generally matter which platform you use or on which server instance you signup for an account, but here's just some recommendations:
 
Lemmy
 
* https://szmer.info/
* https://slrpnk.net
 
Mastodon


* https://mastodon.com.pl/
There have been too many login attempts! Please wait for 600 seconds before trying to log in again.
* https://climatejustice.social/
 
PeerTube
 
* https://makertube.net
* https://peertube.linuxrocks.online/
</pre>
</pre>
# I brought-up the staging/dev server, and we agreed that I would just make a backup of it, store it on hetzner3, and then kill the dev server (on which the staging server lives)
# unfortunately, the credentials for the email address for this hetzner account is not listed in our ose shared keepass
## it would have to be rebuilt in the future with Debian if Marcin wants to add to the software stack, but it's mostly unused and not worth the risk or cost keeping an old CentOS7 server lying around
# I logged-in as my google admin account, reset the password, and added the password to our keepass
# I asked about Marcin's Dreamhost account, which I need access-to in order to migrate obi
# I then tried to login to the email account, but google demanded an OTP sent to some phone number.
## Catarina mentioned that she has her own Dreamhost account. All I did previously was check that the obi nameserver was on dreamhost; I only *assumed* that it was on OSE's dreamhost account (I can't login to verify). I didn't realize they had different dreamhost accounts
# I sent an email to Marcin asking if he changed the creds, and asking what is the phone number
## Catarina said she uses that dreamhost account to pay for personal hosting for other projects, and she didn't want to migrate OBI into OSE's dreamhost (or OSE's cloudlfare – where the rest of the domains live)
# I found that I could remove the phone number associated with the account from the Google Workspace admin panel
## I said that I would just create an obi-specific dreamhost email address, which Catarina can use for this account temporarily so that I can login and make dns changes for the obi migration. She said she'll probably convert it back to her personal email after the migration is done.
# then it said it sent an email with an otp to – the email I'm trying to login into!?
## Marcin gave approval for me to add Catarina to our ops google group email list
# I found a button next to "login challenge" to disable login challenges for 10 minutes
# Catarina asked me to give her a list of wordpress themes that I recommended. I sent her this email
# that worked; I'm in.
# I discovered that this email account has a *lot* of emails, and many are banking related. So I decided *not* to set it up to foward to the shared operations google group list
# I see an email from Feb 19 indicating that the hetzner password changed
# I tried to reset the password several times. It never sends the email! Part of the issue is that it requires us to enter both [a] the "client number or login" and [b] the "email address"
## the only thing we have entered in our keepass is the email address, which is what we use for login
## there's also several distinct login portals for hetzner. One for hetnzer1. One for the cloud. One for the decdicated servers. We have a client ID for hetzner1, but it's alphanumeric. And we have a distinct client ID that was included in the email from hetzner last month about the password change, but that's also alphanumeric. I tried everything I could think-of for the "client number or login", including stripping the alpha characters (so it's truly just a "number"), but the email never came and eventually I got an error indicating that the account is now disabled
<pre>
<pre>
Hey Catarina,
Account is disabled.


Per our conversation, please see these wordpress themes.
There have been too many login attempts! Please wait for 600 seconds before trying to log in again.
 
I've been looking into many wordpress themes for Eco-Libre for the past couple years. And in the last weeks, I did some more research to narrow-down this list. Currently my #1 is split between Twenty Twenty-Four and GeneratePress. But here's some others
 
https://wordpress.org/themes/twentytwentyfour/
https://wordpress.org/themes/generatepress/
https://wordpress.org/themes/astra/
https://wordpress.org/themes/oceanwp/
https://wordpress.org/themes/hestia/
https://wordpress.org/themes/neve/
 
Also, as mentioned on the call, you might want to pick a theme that supports Wordpress Full Site Editing. This might be important for a future-proof theme. You can read more about Full Site Editing here:
 
* https://wordpress.com/blog/2023/08/16/wordpress-site-editor-updates/
 
Here's a list of Full Site Editor themes
 
* https://fullsiteediting.com/themes/
 
Please check these out and send me a list of which themes you would like me to install for osemain and obi.
</pre>
</pre>
 
# there's no phone number to call on this page
=Sun Feb 23, 2025=
# I found a phone number, but it's a +49. I can only call +1 for free from my Google Voice account :/ https://www.hetzner.com/support-form/
 
# I used the form on the above page to send a cryptographically signed message to hetzner asking them to please send us our "client number" to the email address associated with our account
# updated wiki logs & hours
# ...
# ...
# did a port scan of hetzner3 on ipv6 to make sure my logs weren't exposing something we shoudn't
# while I'm blocked and locked-out of the hetzner wui and I'm logged into the Google Workspace, I'll work on creating those accounts
# crap, I just realized that the shared keepass db doesn't have an entry for the hcaptcha-specific email account that I created on 2025-02-08
## I clearly stated in my log of that day that I added this entry to the keepass db. Is it possible that the file became corrupt somehow? Or maybe I accidentally updated it on hetzner3? https://wiki.opensourceecology.org/wiki/Maltfield_Log/2025_Q1#Sat_Feb_08.2C_2025
# no, the file doesn't even exist on hetzner3 yet
<pre>
<pre>
maltfield@mail:~$ nmap -6 -p- -Pn 2a01:4f8:200:40d7::2
root@hetzner3 ~ # ls -lah /etc/keepass
Starting Nmap 7.93 ( https://nmap.org ) at 2025-02-23 16:43 UTC
ls: cannot access '/etc/keepass': No such file or directory
Nmap scan report for hetzner3.opensourceecology.org (2a01:4f8:200:40d7::2)
Host is up (0.0032s latency).
Not shown: 65531 filtered tcp ports (no-response)
PORT    STATE  SERVICE
80/tcp  open  http
443/tcp  open  https
4443/tcp closed pharos
4444/tcp closed krb524
 
Nmap done: 1 IP address (1 host up) scanned in 105.58 seconds
maltfield@mail:~$
</pre>
# that looks fine.
# ..
# I checked-in on the meidawiki cron
# it still only shows one entry
<pre>
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log
================================================================================
INFO: Beginning mediawiki_generate_captchas.sh Run on 20250223_051847
================================================================================
INFO: generating 500 new captcha images
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
 
real    0m1,450s
user    0m1,389s
sys    0m0,055s
INFO: deleting all but the most-recent 500 captcha images
 
real    0m0,027s
user    0m0,021s
sys    0m0,009s
INFO: exiting at 20250223_051849
root@hetzner3 /usr/local/bin # date -u
2025-02-23T17:15:29 UTC
</pre>
# but here's why: the old ones are getting rotated. So that's fine.
<pre>
root@hetzner3 /usr/local/bin # ls -lah /var/log/cron
total 44K
drwxr-x---  2 www-data www-data 4,0K Feb 23 00:00 .
drwxr-xr-x 16 root    root    4,0K Feb 23 00:00 ..
-rw-r--r--  1 www-data www-data  533 Feb 23 05:18 mediawiki_generate_captchas.log
-rw-r--r--  1 www-data www-data  98 Feb 15 05:04 mediawiki_generate_captchas.log-20250216.gz
-rw-r--r--  1 www-data www-data  98 Feb 16 05:01 mediawiki_generate_captchas.log-20250217.gz
-rw-r--r--  1 www-data www-data  257 Feb 17 05:05 mediawiki_generate_captchas.log-20250218.gz
-rw-r--r--  1 www-data www-data  257 Feb 18 04:39 mediawiki_generate_captchas.log-20250219.gz
-rw-r--r--  1 www-data www-data  258 Feb 19 05:13 mediawiki_generate_captchas.log-20250220.gz
-rw-r--r--  1 www-data www-data  258 Feb 20 05:06 mediawiki_generate_captchas.log-20250221.gz
-rw-r--r--  1 www-data www-data  258 Feb 21 04:37 mediawiki_generate_captchas.log-20250222.gz
-rw-r--r--  1 www-data www-data  533 Feb 22 04:39 mediawiki_generate_captchas.log-20250223
root@hetzner3 /usr/local/bin #
</pre>
 
=Thr Feb 20, 2025=
 
# Marcin sent me an email saying that a wiki user just maliciously deleted a bunch of articles on the wiki, and asked me if there's a way to easily UNdelete in batches
# I asked about this on SE
## https://webmasters.stackexchange.com/questions/146516/mass-undelete-on-mediawiki-unnuke-restore
## https://webapps.stackexchange.com/questions/180833/mass-undelete-on-mediawiki-unnuke-restore
# But previously I had looked at these two extensions, and  found the third. Worth looking into.
## https://www.mediawiki.org/wiki/Extension:DeleteBatch
## https://www.mediawiki.org/wiki/Extension:SmiteSpam
## https://www.mediawiki.org/wiki/Extension:Moderation
 
=Sun Feb 16, 2025=
 
# hmm, I still see only one entry for the mediawiki cron log, despite the fact that it's definitely set to append now
<pre>
root@hetzner3 /etc/nginx # cat /etc/cron.d/mediawiki_cron
# Ansible managed
SHELL=/bin/bash
 
# regenerate new CAPTCHA images
20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &>> /var/log/cron/mediawiki_generate_captchas.log
root@hetzner3 /etc/nginx # cat /var/log/cron/mediawiki_generate_captchas.log
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
root@hetzner3 /etc/nginx #
</pre>
# I updated the script to print the datestamp; hopefully that'll help us debug this now and in the future
<pre>
rroot@hetzner3 /usr/local/bin # diff mediawiki_generate_captchas.sh.20250216 mediawiki_generate_captchas.sh
5c5
< # Version: 0.2
---
> # Version: 0.3
11c11
< # Updated: 2025-02-14
---
> # Updated: 2025-02-16
43a44,48
> stamp=`date -u +%Y%m%d_%H%M%S`
> echo "================================================================================"
> echo "INFO: Beginning mediawiki_generate_captchas.sh Run on ${stamp}"
> echo "================================================================================"
>
64c69,70
< sudo -u www-data bash -c "python3 '${CAPTCHA_SCRIPT_PATH}' --wordlist=${WORDLIST_FILE_PATH} --key='${CAPTCHA_SECRET}' --output='${CAPTCHA_DIR_PATH}' --font='${FONT_FILE_PATH}' --count='${NUM_FILES}'"
---
> echo "INFO: generating ${NUM_FILES} new captcha images"
> sudo -u www-data bash -c "time nice python3 '${CAPTCHA_SCRIPT_PATH}' --wordlist=${WORDLIST_FILE_PATH} --key='${CAPTCHA_SECRET}' --output='${CAPTCHA_DIR_PATH}' --font='${FONT_FILE_PATH}' --count='${NUM_FILES}'"
68c74,75
< sudo -u www-data bash -c "ls -t ${CAPTCHA_DIR_PATH}/* | sed -e '1,${NUM_FILES}d' | xargs rm"
---
> echo "INFO: deleting all but the most-recent ${NUM_FILES} captcha images"
> sudo -u www-data bash -c "time nice ls -t ${CAPTCHA_DIR_PATH}/* | sed -e '1,${NUM_FILES}d' | xargs rm"
69a77
> echo "INFO: exiting at $(date -u +%Y%m%d_%H%M%S)"
root@hetzner3 /usr/local/bin #
root@hetzner3 /usr/local/bin # ./mediawiki_generate_captchas.sh
================================================================================
INFO: Beginning mediawiki_generate_captchas.sh Run on 20250216_213845
================================================================================
INFO: generating 500 new captcha images
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
 
real    0m1,430s
user    0m1,385s
sys    0m0,049s
INFO: deleting all but the most-recent 500 captcha images
 
real    0m0,028s
user    0m0,020s
sys    0m0,011s
INFO: exiting at 20250216_213846
root@hetzner3 /usr/local/bin #
</pre>
# note I also added 'time' and 'nice' and a timestamp at the end.
# ...
# yesterday we fixed a lot of performance configurations of php & apache
# then we began looking at verifying the nginx security configs, which we already found an issue: wp-login.php isn't being blocked
<pre>
user@disp3993:~$ curl -IL https://store.opensourceecology.org/wp-login.php
HTTP/1.1 302 Found
Server: nginx
Date: Sun, 16 Feb 2025 21:02:14 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: /
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 560106
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 16 Feb 2025 21:02:14 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 85328
Connection: keep-alive
X-Pingback: https://store.opensourceecology.org/xmlrpc.php
Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 331395 460383
Age: 79262
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
user@disp3993:~$
</pre>
# ah, looks like I commented it out 4 months ago when I first realized that the 'rename-wp-login' plugin was removed by wordpress due to security issues
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ tail roles/maltfield.apache/templates/security.virtualhost.include.j2
 
  # block access to 'wp-login.php' from brute-forcers;
# see wp plugin 'rename-wp-login'
# TODO: 2024-10: we need to re-enable this after we find a replacement for the
#                (now-deprecated) 'rename-wp-login' wordpress plugin
#              * https://wordpress.org/plugins/rename-wp-login/
#  <LocationMatch ".*wp-login.php">
# Require all denied
#  </LocationMatch>
 
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# I'm actually surprised it's a config in apache, not nginx
# but, anyway, I uncommented it in ansible and pushed it out https://github.com/OpenSourceEcology/ansible/commit/12257eee3bd5134f29af86b11e104a1245144ef3
# after restarting apache and clearing the varnish cache, it's now working as-desired
<pre>
user@disp3993:~$ curl -IL https://store.opensourceecology.org/wp-login.php
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:12:35 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 199
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Varnish: 560114
Age: 0
Via: 1.1 varnish (Varnish/7.1)
 
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sun, 16 Feb 2025 21:12:43 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://store.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=1;includeSubDomains
 
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:12:43 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 199
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Varnish: 35975
Age: 0
Via: 1.1 varnish (Varnish/7.1)
 
user@disp3993:~$
</pre>
# actually, I do have a config that should have blocked it in nginx
<pre>
root@hetzner3 /etc/nginx # cat conf.d/secure.include
# Ansible managed
 
################################################################################
# File:    secure.include
# Version: 0.2
# Purpose: Basic security settings that couldn't be put in the main nginx.conf.
#          This should be included in the server{} blocks nginx vhosts.
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2017-11-23
# Updated: 2024-08-07
################################################################################
 
# whitelist requests to disable TRACE & DELETE
if ($request_method !~ ^(GET|HEAD|POST)$ ) {
# note: 444 is a meta code; it doesn't return anything, actually
#      it just logs, drops, & closes the connection (useful
#      against malware)
return 444;
}
 
## block some bot's useragents (may need to remove some, if impacts SEO)
# 2020-05-03: I'm removing this from my config for my migration to hetzner
#            Logic: If a bot is playing bad, they'll be blocked by DOS rate
#            limits, anyway. Otherwise, let 'em scrape.
#if ($blockedagent) {
#      return 403;
#}
 
# prevent access to any dirs/files starting with a dot (.)
location ~ /\.(?!well\-known) { access_log off; log_not_found off; deny all; }
 
# prevent access to any config files
location ~ config\. { access_log off; log_not_found off; deny all; }
root@hetzner3 /etc/nginx #
</pre>
# looks like it's included. what happened?
<pre>
root@hetzner3 ~ # cat /etc/nginx/sites-enabled/store.opensourceecology.org.conf
# Ansible managed
 
################################################################################
# File:    store.opensourceecology.org.conf
# Version: 0.3
# Purpose: Internet-listening web server for truncating https, basic DOS
#          protection, and passing to varnish cache (varnish then passes to
#          apache)
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
# Updated: 2024-10-04
################################################################################
 
server {
 
access_log /var/log/nginx/store.opensourceecology.org/access.log main;
error_log /var/log/nginx/store.opensourceecology.org/error.log;
 
  include conf.d/secure.include;
  include conf.d/https.opensourceecology.org.include;
 
  listen 127.0.0.1:443;
  listen [::1]:443;
  listen 144.76.164.201:443;
  listen [2a01:4f8:200:40d7::3]:443;
 
  server_name store.opensourceecology.org;
 
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
 
# root /var/www/html/SITE_DOWN/htdocs/;
#  index index.html index.htm;
#
# # force all requests to load exactly this page
# location / {
# try_files $uri /index.html;
# }
 
###################
# SEND TO VARNISH #
###################
 
  location / {
  proxy_pass http://127.0.0.1:6081;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto https;
  proxy_set_header X-Forwarded-Port 443;
  proxy_set_header Host $host;
  }
 
}
 
root@hetzner3 ~ #  
root@hetzner3 ~ #  
</pre>
</pre>
# oh, duh, wp-login.php != wp-config.php
# the file is on hetzner2, and says it's last updated today – but for some reason only I have write permissions?
# let's test that, though
# allright, that's working perfect
<pre>
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-config.php
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:27:53 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
 
user@disp3993:~$ curl -IL store.opensourceecology.org/config.php
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:27:57 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
 
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-config.php.bak
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:28:06 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
 
user@disp3993:~$ curl -IL store.opensourceecology.org/config.php.bak
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:28:10 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
 
user@disp3993:~$ curl -IL store.opensourceecology.org/config.bak.php
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:28:13 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
 
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-config.bak.php
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sun, 16 Feb 2025 21:28:16 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
 
user@disp3993:~$
</pre>
# here's the list of things to test
## wp-login.php shouldn't be accessible
## phplist config.php shouldn't be accessible (I want to confirm because old had ".*" prefix and new doesn't)
## .php files in wp-content uploads dirs shouldn't be executable
##  anything other than GET POST HEAD should be denied
## anything that starts with a dot (other than .well-known) should be denied (test .git & .htaccess)
## indexes on a dir without 'index.php' shouldn't show a directory listing
## .htaccess shouldn't allow changing things
# so the first two are done
# let's test for php executability in wp, phplist, and mediawiki
# first we create some php script and put it in all the dirs that are writeable by the web server
<pre>
dirs="/var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/tmp/ /var/www/html/phplist.opensourceecology.org/public_html/uploadimages/ /var/www/html/wiki.opensourceecology.org/htdocs/images/"
 
for d in $dirs; do
cat > ${d}/test.php <<'EOF'
<pre>
<?php
echo "if this php was processed, that's very bad!";
?>
</pre>
EOF
 
chown not-apache:www-data ${d}/test.php
chmod 0040 ${d}/test.php
 
done
</pre>
# I actually expected it to dump back the raw php, but it looks like it returns 403 for any files that end in .php; that's better
<pre>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/test.php<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/doesnotexist.php
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$
</pre>
# this looks good
<pre>
user@disp3993:~$ curl -L phplist.opensourceecology.org/uploadimages/test.php
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$
 
user@disp3993:~$ curl -L wiki.opensourceecology.org/images/test.php
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$
</pre>
# oh crap, I did find one that worked!
<pre>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/test.php
<pre>
if this php was processed, that's very bad!</pre>
user@disp3993:~$
</pre>
# this is actually a good opportunity. Before, when we used mod_php, we wouldn't just 403 refuse to serve it on nginx, but we'd also configure apache not to pass it to mod_php. So let's take this opportunity to add that layered security at the apache layer too
# we're just using the default php-fpm config file; let's create a backup and see if we can harden it to *not* pass files in these dirs back to the php-fpm daemon/proxy
<pre>
root@hetzner3 /etc/apache2/conf-available # cp php8.2-fpm.conf php8.2-fpm.conf.20250216.conf
root@hetzner3 /etc/apache2/conf-available #
</pre>
# oh, looks like I already configured this here, but for some reason the 'tmp' dir is omitted
</pre>
root@hetzner3 /etc/apache2/conf-available # head -n30 security.virtualhost.include
# Ansible managed
 
################################################################################
# File:    security.virtualhost.include
# Version: 0.4
# Purpose: File includes some common security-hardening that's intended to be
#          Include()d into other vhost files' <VirtualHost> blocks
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-14
# Updated: 2025-02-16
################################################################################
 
# don't execute any php files inside uploads directories
<LocationMatch "/wp-content/uploads/">
SetHandler !
</LocationMatch>
<LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
</LocationMatch>
 
#      <LocationMatch "/wp-content/tmp/">
#              SetHandler !
#      </LocationMatch>
#      <LocationMatch "/wp-content/tmp/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
#              Require all denied
#      </LocationMatch>
 
<LocationMatch "/uploadimages/">
SetHandler !
</LocationMatch>
root@hetzner3 /etc/apache2/conf-available #
</pre>
# ok, so what we needed to test here was that 'SetHandler' would successfully prevent it from being executed by php
# I uncommented just this one block, restarted apache, and cleared the cache
<pre>
root@hetzner3 /etc/apache2/conf-available # head -n30 security.virtualhost.include
# Ansible managed
 
################################################################################
# File:    security.virtualhost.include
# Version: 0.4
# Purpose: File includes some common security-hardening that's intended to be
#          Include()d into other vhost files' <VirtualHost> blocks
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-14
# Updated: 2025-02-16
################################################################################
 
# don't execute any php files inside uploads directories
<LocationMatch "/wp-content/uploads/">
SetHandler !
</LocationMatch>
<LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
</LocationMatch>
 
<LocationMatch "/wp-content/tmp/">
SetHandler !
</LocationMatch>
#      <LocationMatch "/wp-content/tmp/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
#              Require all denied
#      </LocationMatch>
 
<LocationMatch "/uploadimages/">
SetHandler !
</LocationMatch>
root@hetzner3 /etc/apache2/conf-available #
</pre>
# this confirms that that config block is working as-expected
<pre>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/test.php
<pre>
<?php
echo "if this php was processed, that's very bad!";
?>
</pre>
user@disp3993:~$
</pre>
# now we uncomment the second block, and it's totally blocked; perfect
<pre>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/test.php
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$
</pre>
# I updated this in ansible, pushed it out, and re-confirmed it's working as-expected https://github.com/OpenSourceEcology/ansible/commit/d523d0ef4b218fd7189d336f1c8f8fea6dc3dbe8
# the next verification is to make sure only POST, HEAD, and GET requests are allowed
# it looks like we do have this set in our /etc/nginx/conf.d/secure.include
<pre>
root@hetzner3 /etc/nginx # grep -ir POST *
conf.d/secure.include:  if ($request_method !~ ^(GET|HEAD|POST)$ ) {
mime.types:    application/postscript                ps eps ai;
nginx.conf:    # allow large posts for image uploads
nginx.conf.1282157.2024-09-28@23:10:52~:        # allow large posts for image uploads
nginx.conf.3932816.2024-10-05@03:45:22~:        # allow large posts for image uploads
root@hetzner3 /etc/nginx #
You have mail in /var/mail/root
root@hetzner3 /etc/nginx # cat conf.d/secure.include
# Ansible managed
 
################################################################################
# File:    secure.include
# Version: 0.2
# Purpose: Basic security settings that couldn't be put in the main nginx.conf.
#          This should be included in the server{} blocks nginx vhosts.
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2017-11-23
# Updated: 2024-08-07
################################################################################
 
# whitelist requests to disable TRACE & DELETE
if ($request_method !~ ^(GET|HEAD|POST)$ ) {
# note: 444 is a meta code; it doesn't return anything, actually
#      it just logs, drops, & closes the connection (useful
#      against malware)
return 444;
}
 
## block some bot's useragents (may need to remove some, if impacts SEO)
# 2020-05-03: I'm removing this from my config for my migration to hetzner
#            Logic: If a bot is playing bad, they'll be blocked by DOS rate
#            limits, anyway. Otherwise, let 'em scrape.
#if ($blockedagent) {
#      return 403;
#}
 
# prevent access to any dirs/files starting with a dot (.)
location ~ /\.(?!well\-known) { access_log off; log_not_found off; deny all; }
 
# prevent access to any config files
location ~ config\. { access_log off; log_not_found off; deny all; }
root@hetzner3 /etc/nginx #
</pre>
# quick test shows the 444 functioning as expected
<pre>
<pre>
user@disp3993:~$ curl --request OPTIONS www.opensourceecology.org
[maltfield@opensourceecology ~]$ ls -lah /etc/keepass/
curl: (52) Empty reply from server
total 180K
user@disp3993:~$ curl --request OPTIONS www.openbuildinginstitute.org
drwxrwx---   2 root     keepass  4.0K Mar 16 22:55 .
curl: (52) Empty reply from server
drwxr-xr-x. 104 root      root       12K Dec 30 22:33 ..
user@disp3993:~$ curl --request OPTIONS store.opensourceecology.org
-rw-r-----    1 maltfield maltfield  45K Mar 16 22:55 passwords.kdbx
curl: (52) Empty reply from server
-rw-rw----   1 root     keepass  4.5K Jul 29  2017 passwords.kdbx.20170728.bak
user@disp3993:~$ curl --request OPTIONS phplist.opensourceecology.org
-rw-rw----   1 root     keepass  4.5K Jul 13 2017 passwords.kdbx.20170804.bak
curl: (52) Empty reply from server
-rw-rw----   1 root     keepass    33K Aug 20 2019 passwords.kdbx.20190820.bak
user@disp3993:~$ curl --request OPTIONS wiki.opensourceecology.org
-rw-rw----    1 maltfield maltfield 34K Sep 9 2019 passwords.kdbx.20190909.bak
curl: (52) Empty reply from server
-rw-rw----   1 root     keepass    27K May 26 2018 passwords.kdbxs.20180525.bak
user@disp3993:~$
</pre>
# curiously I don't get the same from CONNECT, but I guess this is something built-into nginx?
<pre>
user@disp3993:~$ curl --request CONNECT wiki.opensourceecology.org
<html>
<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl --request CONNECT store.opensourceecology.org
<html>
<head><title>405 Not Allowed</title></head>
<body>
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$
</pre>
# a full list of HTTP methods can be found here https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods
# curiously our varnish configs seem to allow many more types – for some reason. But that's fine.
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ grep -ir HEAD * | grep -i POST | grep -i GET
roles/maltfield.varnish/templates/www.openbuildinginstitute.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/oswh.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/fef.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/forum.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/www.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/store.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/phplist.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/microfactory.opensourceecology.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.varnish/templates/seedhome.openbuildinginstitute.org.vcl.j2: if (req.method !~ "^GET|HEAD|PUT|POST|TRACE|OPTIONS|DELETE$") {
roles/maltfield.nginx/templates/secure.include.j2: if ($request_method !~ ^(GET|HEAD|POST)$ ) {
roles/maltfield.apache/templates/security.directory.include.j2: <LimitExcept GET POST HEAD>
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# as that shows, apache also limits it
<pre>
root@hetzner3 /etc/apache2 # cat conf-available/security.directory.include
# Ansible managed
 
################################################################################
# File:   security.directory.include
# Version: 0.1
# Purpose: File includes some common security-hardening that's intended to be
#          Include()d into other vhost files' <Directory> blocks
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-14
# Updated: 2024-09-14
################################################################################
 
#Options -Indexes -Includes
Options None
AllowOverride None
 
# we can't do this when using mod_remoteip
#Require ip 127.0.0.1
 
# Harden vhost docroot by blocking all request types except the 3 essentials
# Note: The debian wiki actually says not to use this since it can result in
#       disabling auth modules, so be careful
#        * https://wiki.debian.org/Apache/Hardening
<LimitExcept GET POST HEAD>
Require all denied
</LimitExcept>
root@hetzner3 /etc/apache2 #
</pre>
# our next test is that anything starting with a dot is refused (eg .git, .htaccess, .htpasswd, .svn, etc)
# this looks good
<pre>
user@disp3993:~$ curl store.opensourceecology.org/.git
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl store.opensourceecology.org/.htaccess
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl store.opensourceecology.org/.htpasswd
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl store.opensourceecology.org/.svn
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl store.opensourceecology.org/somedir/.svn
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl store.opensourceecology.org/somedir/.git
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl wiki.opensourceecology.org/somedir/.git
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$ curl wiki.opensourceecology.org/.htpasswd
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp3993:~$
</pre>
# and .well-known is the only one that worked
<pre>
user@disp3993:~$ curl -L wiki.opensourceecology.org/.well-known
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
user@disp3993:~$
</pre>
# ok, next test is to make sure Indexes aren't working
# I guess for this, let's just access the uploads dirs; all of these work as expected
<pre>
user@disp3993:~$ curl -L wiki.opensourceecology.org/images/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
user@disp3993:~$
</pre>
# lastly, we need to test that .htaccess files don't work
# I'll create one in all the upload dirs again
<pre>
dirs="/var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/tmp/ /var/www/html/phplist.opensourceecology.org/public_html/uploadimages/ /var/www/html/wiki.opensourceecology.org/htdocs/images/"
 
for d in $dirs; do
cat > ${d}/.htaccess <<'EOF'
Options +Indexes
EOF
 
chown not-apache:www-data ${d}/test.php
chmod 0040 ${d}/test.php
 
done
</pre>
# spot check after running
<pre>
root@hetzner3 /etc/apache2 # ls -lah /var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/.htaccess
-rw-r--r-- 1 root root 17 Feb 16 22:47 /var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/.htaccess
root@hetzner3 /etc/apache2 # cat /var/www/html/store.opensourceecology.org/htdocs/wp-content/uploads/.htaccess
Options +Indexes
root@hetzner3 /etc/apache2 #
</pre>
# looks like it's ignoring the .htaccess files; perfect
<pre>
user@disp3993:~$ curl -L wiki.opensourceecology.org/images/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/uploads/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$ curl -L store.opensourceecology.org/wp-content/tmp/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
user@disp3993:~$
</pre>
# well, that's all the hardening verification I had on my TODO list. Glad we did that; we fixed the wordpress tmp/ dir issue.
# ...
# I'm still blocked on most of the hetzner3 work, but I have two more items on my TODO list:
## setup the staging server with hetnzer3
### that's a huge & high-hanging-fruit task. last I checked the keys expired.
## migrate a script that I use on my personal site for taking down all of my sites = /usr/local/bin/website_maintance.sh
### this basically just moves the main apache2.conf file to apache2_prod.conf, and then we create a symlink named apache2.conf, which points to it.
### And we have a new apache2_maint.conf file, which points all of the websites to some simple index.html file saying that the server is undergoing maintenance.
### then the script just swaps symlinks as-needed
# ah, this is elucidating; it just swaps the includes for all the vhost config files for a single config file
<pre>
root@mail:/etc/apache2# diff apache2_prod.conf apache2_maint.conf
230c230
< IncludeOptional sites-enabled/*.conf
---
> IncludeOptional sites-available/000-maintenance.conf
root@mail:/etc/apache2#
</pre>
# simple
root@mail:/etc/apache2# cat sites-available/000-maintenance.conf
<VirtualHost *:8000>
 
DocumentRoot /var/www/html/maintenance/htdocs/
 
CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorLog ${APACHE_LOG_DIR}/error.log
 
Include 'conf-available/security.virtualhost.include'
 
<Directory "/var/www/html/maintenance/htdocs/">
Include 'conf-available/security.directory.include'
Options +FollowSymLinks
</Directory>
 
# redirects everything to "/index.html"
RewriteEngine On
RewriteRule ^(.*) /index.html
 
</VirtualHost>
root@mail:/etc/apache2#
root@mail:/etc/apache2# ls -lah /var/www/html/maintenance/htdocs/
total 12K
d---r-x--- 2 root www-data 4,0K May 18 2022 .
d---r-x--- 3 root www-data 4,0K May 18  2022 ..
----r----- 1 root www-data  407 May 18 2022 index.html
root@mail:/etc/apache2# cat /var/www/html/maintenance/htdocs/index.html
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Site Under Maintenance</title>
</head>
<body>
Sorry, this website is temporarily down for maintenance. Please check back later.
</body>
</html>
root@mail:/etc/apache2#
# apparently I had already mostly set this up with ansible, I just needed to add the binary
<pre>
root@hetzner3 /etc/apache2 # cat sites-available/000-maintenance.conf
# Ansible managed
 
################################################################################
# File:   000-maintenance.conf
# Version: 0.1
# Purpose: vhost of a simple "site currently under maintenance" landing page
# Author:  Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-10
# Updated: 2024-09-10
################################################################################
<VirtualHost *:8000>
 
DocumentRoot /var/www/html/SITE_DOWN/htdocs/
 
CustomLog ${APACHE_LOG_DIR}/access.log combined
ErrorLog ${APACHE_LOG_DIR}/error.log
 
Include 'conf-available/security.virtualhost.include'
 
<Directory "/var/www/html/SITE_DOWN/htdocs/">
Include 'conf-available/security.directory.include'
Options +FollowSymLinks
</Directory>
 
# redirects everything to "/index.html"
RewriteEngine On
RewriteRule ^(.*) /index.html
 
</VirtualHost>
root@hetzner3 /etc/apache2 #
</pre>
# I added the binary, but it looks like I'm missing the vhost
<pre>
root@hetzner3 /etc/apache2 # website_maintenance.sh onYou have mail in /var/mail/root
 
INFO: Enabling Maintenance Mode
AH00112: Warning: DocumentRoot [/var/www/html/SITE_DOWN/htdocs/] does not exist
Syntax OK
 
INFO: Maintenance Mode is now on
root@hetzner3 /etc/apache2 #
</pre>
# it's on hetzner2
<pre>
[maltfield@opensourceecology ~]$ ls -lah /var/www/html/SITE_DOWN/
total 12K
drwxr-xr-x 3 root root 4.0K Nov 23 2017 .
drwxr-xr-x 25 root root 4.0K May 30 2023 ..
drwxr-xr-x  2 root root 4.0K Nov 23  2017 htdocs
[maltfield@opensourceecology ~]$ ls -lah /var/www/html/SITE_DOWN/htdocs/
total 12K
drwxr-xr-x 2 root root 4.0K Nov 23  2017 .
drwxr-xr-x 3 root root 4.0K Nov 23  2017 ..
-rw-r--r-- 1 root root  303 Nov 23 2017 index.html
[maltfield@opensourceecology ~]$
 
[maltfield@opensourceecology ~]$ cat /var/www/html/SITE_DOWN/htdocs/index.html
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<html xmlns="http://www.w3.org/1999/xhtml">
 
<head>
<meta charset="UTF-8">
</head>
 
<body>
<pre>Sorry, we're down for maintenance. Please check back later.</pre>
</body>
 
</html>
[maltfield@opensourceecology ~]$  
[maltfield@opensourceecology ~]$  
</pre>
</pre>
# but not hetzner3
# according to the wiki https://wiki.opensourceecology.org/wiki/OSE_Server#Keepass
<pre>
root@hetzner3 /etc/apache2 # ls -lah /var/www/html
total 56K
d---r-x--- 13 not-apache www-data 4,0K Feb 14 23:31 .
drwxr-xr-x  3 root      root    4,0K Sep 25 02:37 ..
d---r-x---  3 not-apache www-data 4,0K Dec 31 01:38 fef.opensourceecology.org
d---r-x---  5 not-apache www-data 4,0K Jul 11  2018 forum.opensourceecology.org
----r-----  1 not-apache www-data  138 Mar  3  2018 .htpasswd
d---r-x---  3 not-apache www-data 4,0K Jan 28 23:55 microfactory.opensourceecology.org
drwxr-xr-x  3 root      root    4,0K Feb 14 23:31 new-obi
d---r-x---  3 not-apache www-data 4,0K Jan 30 04:18 oswh.opensourceecology.org
d---r-x---  6 not-apache www-data 4,0K Feb 12 23:30 phplist.opensourceecology.org
d---r-x---  4 not-apache www-data 4,0K Dec 26 20:44 seedhome.openbuildinginstitute.org
d---r-x---  4 not-apache www-data 4,0K Feb 16 00:56 store.opensourceecology.org
d---r-x---  4 not-apache www-data 4,0K Feb 13 23:05 wiki.opensourceecology.org
d---r-x---  3 not-apache www-data 4,0K Jan 30 22:18 www.openbuildinginstitute.org
d---r-x---  3 not-apache www-data 4,0K Feb  1 00:23 www.opensourceecology.org
root@hetzner3 /etc/apache2 #
</pre>
# probably we could create some nice-looking SITE_DOWN page with OSE logos, but I'm just going to keep it simple now. it would be trivial to update later.
<pre>
root@hetzner3 /etc/apache2 # mkdir -p /var/www/html/SITE_DOWN/htdocs/
root@hetzner3 /etc/apache2 #
root@hetzner3 /etc/apache2 # cat > /var/www/html/SITE_DOWN/htdocs/index.html <<'EOF'
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<html xmlns="http://www.w3.org/1999/xhtml">
 
<head>
<meta charset="UTF-8">
</head>
 
<body>
<pre>Sorry, we're down for maintenance. Please check back later.</pre>
</body>
 
</html>
 
EOF
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # time fix_web_permissions.sh
</pre>
# cool, it's working
<pre>
<pre>
user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/
The passwords.kdbx file should be owned by the user 'root' and the group 'keepass'. It should have the file permissions of 660 (such that it can be read & written by 'root' and users in the 'keepass' group, but not accessible in any way from anyone else).  
<!DOCTYPE html PUBLIC "-W3CDTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
<html xmlns="http://www.w3.org/1999/xhtml">
 
<head>
<meta charset="UTF-8">
</head>
 
<body>
<pre>Sorry, we're down for maintenance. Please check back later.</pre>
</body>
 
</html>
 
user@disp3993:~$
</pre>
</pre>
# and I turn it off on the server
# so I fixed this
<pre>
<pre>
root@hetzner3 /usr/local/bin # website_maintenance.sh off
[root@opensourceecology keepass]# ls -lah
INFO: Disabling Maintenance Mode
total 180K
Syntax OK
drwxrwx---    2 root      keepass  4.0K Mar 16 22:55 .
 
drwxr-xr-x. 104 root      root      12K Dec 30 22:33 ..
INFO: Maintenance Mode is now off
-rw-r-----    1 maltfield maltfield  45K Mar 16 22:55 passwords.kdbx
You have mail in /var/mail/root
-rw-rw----    1 root      keepass  4.5K Jul 29  2017 passwords.kdbx.20170728.bak
root@hetzner3 /usr/local/bin #  
-rw-rw----    1 root      keepass  4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]#
[root@opensourceecology keepass]# chown root:keepass passwords.kdbx
[root@opensourceecology keepass]# chmod 0660 passwords.kdbx
[root@opensourceecology keepass]#
[root@opensourceecology keepass]# ls -lah
total 180K
drwxrwx---    2 root      keepass  4.0K Mar 16 22:55 .
drwxr-xr-x. 104 root      root      12K Dec 30 22:33 ..
-rw-rw----    1 root      keepass    45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root     keepass  4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass  4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]#  
</pre>
</pre>
# and the curl works as-expected again
# I also went ahead and made a backup now
<pre>
<pre>
user@disp3993:~$ curl -L phplist.opensourceecology.org/wp-content/uploadimages/
[root@opensourceecology keepass]# cp passwords.kdbx passwords.kdbx.20250316.bak
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
[root@opensourceecology keepass]#
<html><head>
[root@opensourceecology keepass]# ls -lah
<title>404 Not Found</title>
total 228K
</head><body>
drwxrwx---    2 root      keepass  4.0K Mar 16 23:42 .
<h1>Not Found</h1>
drwxr-xr-x. 104 root      root      12K Dec 30 22:33 ..
<p>The requested URL was not found on this server.</p>
-rw-rw----    1 root      keepass    45K Mar 16 22:55 passwords.kdbx
</body></html>
-rw-rw----    1 root      keepass  4.5K Jul 29  2017 passwords.kdbx.20170728.bak
user@disp3993:~$
-rw-rw----    1 root      keepass  4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-r-----    1 root      root      45K Mar 16 23:42 passwords.kdbx.20250316.bak
-rw-rw----   1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]# chown root:keepass passwords.kdbx.20250316.bak
[root@opensourceecology keepass]# ls -lah
total 228K
drwxrwx---   2 root      keepass  4.0K Mar 16 23:42 .
drwxr-xr-x. 104 root      root      12K Dec 30 22:33 ..
-rw-rw----    1 root      keepass    45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root      keepass  4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass  4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-r-----    1 root      keepass    45K Mar 16 23:42 passwords.kdbx.20250316.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]#
</pre>
</pre>
# I updated the wiki, describing how to use this script https://wiki.opensourceecology.org/wiki/OSE_Server#Maintenance_Mode
# my best-guess/hope is that the entry for hcapcha just somehow failed to get added due to a network issue.
# ...
# I went ahead and reset the password on the hcapctha-specific google account and added it to keepass (again)
# ok, the only task on my TODO list that I'm *not* blocked-on is to fix the sync with the staging server
# ok, omg, wtf. I just realized the ops entry is missing. I *definitely* just added that today! What's happening!?!
# the staging server lives on the prod server, and it's only accessible by connecting to the VPN running on the dev server https://wiki.opensourceecology.org/wiki/OSE_Staging_Server
# I created 3 new entries in keepass
# some time ago the keys expired and everything broke, even between hetzner2 and the staging server
## operations-specific email
## obi-dreamhost-specific email
## hetzner-specific email
# I generated random passwords for each, but I didn't set them to be used in the actual accounts yet.
# then I saved keepass, closed keepass, and shutdown my ose vm
# I restated my ose vm, mounted & unlocked the keepass
# the new entries are absent! What's happening!?
# I do see an error from the keepass terminal
<pre>
<pre>
[root@opensourceecology ~]# ip a show tun0
user@ose:~$ keepassxc
Device "tun0" does not exist.
Maximum depth of replacement has been reached. Entry uuid: {REDACTED}
[root@opensourceecology ~]#
</pre>
</pre>
# I tried to connect to the VPN, following the documentation on the wiki; it also failed saying my cert expired
# quick google shows this error; not sure if relevant https://github.com/keepassxreboot/keepassxc/issues/1741
# omg, actually, the ops and hcaptcha entries are back. So it's like the last save is now opening, but the one before it is not. I added the other two accounts, saved, restarted, and reunlocked
# I wonder if somehow there's a local copy that's being put in the mounted dir, which gets written-to during network issues
# yeah, on reboot I do see a file before I mount it!
<pre>
<pre>
user@ose:~/openvpn$ sudo openvpn client.conf
user@ose:~$ ls -lah mnt/ose/keepass/passwords.kdbx
2025-02-16 18:28:17 Note: Kernel support for ovpn-dco missing, disabling data channel offload.
-rw------- 1 user user 44K Mar 16 18:48 mnt/ose/keepass/passwords.kdbx
2025-02-16 18:28:17 WARNING: file '/home/user/openvpn/username.txt' is group or others accessible
user@ose:~$
2025-02-16 18:28:17 OpenVPN 2.6.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]
2025-02-16 18:28:17 library versions: OpenSSL 3.0.15 3 Sep 2024, LZO 2.10
2025-02-16 18:28:17 DCO version: N/A
Enter Auth Password: (no echo)             
2025-02-16 18:28:51 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Enter Private Key Password: ***************       
2025-02-16 18:29:16 WARNING: Your certificate has expired!
2025-02-16 18:29:16 TCP/UDP: Preserving recently used remote address: [AF_INET]195.201.233.113:1194
2025-02-16 18:29:16 Socket Buffers: R=[212992->212992] S=[212992->212992]
2025-02-16 18:29:16 UDPv4 link local: (not bound)
2025-02-16 18:29:16 UDPv4 link remote: [AF_INET]195.201.233.113:1194
2025-02-16 18:29:17 TLS: Initial packet from [AF_INET]195.201.233.113:1194, sid=ef36b092 c9f96236
2025-02-16 18:29:17 VERIFY OK: depth=1, CN=osedev1
2025-02-16 18:29:17 VERIFY ERROR: depth=0, error=certificate has expired: CN=server, serial=17809834222100140758844598132739229483
2025-02-16 18:29:17 OpenSSL: error:0A000086:SSL routines::certificate verify failed
2025-02-16 18:29:17 TLS_ERROR: BIO read tls_read_plaintext error
2025-02-16 18:29:17 TLS Error: TLS object -> incoming plaintext read error
2025-02-16 18:29:17 TLS Error: TLS handshake failed
</pre>
</pre>
# looks like the dev server is still online; I was able to ssh into it
# and, yeah, that local copy has the entries that I had added before. I'm just going to move this out of the way, mount the *real* one on hetnzer2, and confirm my new entries are finally saved
<pre>
<pre>
user@ose:~/openvpn$ ssh osedev1
user@ose:~$ mv mnt/ose/keepass/passwords.kdbx passwords.fail.20250316.kdbx
Enter passphrase for key '/home/user/.ssh/id_rsa':  
user@ose:~$
Last login: Fri Oct 28 20:42:19 2022 from tor-exit-35.for-privacy.net
user@ose:~$ sshfs -p 32415 maltfield@138.201.84.223:/etc/keepass /home/user/mnt/ose/keepass -o IdentityFile=/home/user/.ssh/id_rsa.ose -o uid=1000
[maltfield@osedev1 ~]$  
Enter passphrase for key '/home/user/.ssh/id_rsa.ose':  
user@ose:~$ ls -lah mnt/ose/keepass/passwords.kdbx
-rw-rw---- 1 user 1005 45K Mar 16 18:57 mnt/ose/keepass/passwords.kdbx
user@ose:~$  
</pre>
</pre>
# it's been online for 1,707 days. not bad!
# ok, all the entries are present again. Now that I'm sure the passwords are durably saved in the shared keepass file on the server, I created the two new accounts (for hetzner and obi dreamhost)
<pre>
# I also added catarina's personal gmail to the ops google group list
[maltfield@osedev1 ~]$ uptime
# I sent a test email to the new obi-specific-dreamhost account email to confirm that Catarina is new getting forwarded those emails
00:31:57 up 1707 days, 23:41,  1 user,  load average: 1.31, 0.88, 0.50
# I confirmed that *I* got the email forwarded to me; I'll wait for Catarina's reply
[maltfield@osedev1 ~]$
</pre>
# poor little server's load is high; looks like it's swapping. mostly it's running just ossec
# ah, gross, it's a centos server; yum cron is using the most RAM
# damn, it's CentOS7
<pre>
[maltfield@osedev1 ~]$ cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)  
[maltfield@osedev1 ~]$
</pre>
# I don't know why I didn't think about this before, but this server is now EOL and a risk
# moreover, I'm not sure it's even going to work as-desired because it's going to be so different from prod
# I'm going to have to talk to Marcin about this. it's going to be a lot of work to rebuild it, and I doubt he's going to want to pay for that. so probably we should just retire the ose dev server (and therefore the staging server that runs on it) :'(
# I made a list of items to discuss with Marcin. Until we have this conversation, I can't continue with hetzner3


=Sat Feb 15, 2025=
=Sat Mar 15, 2025=


# Tom responded saying he got "Permission Denied" when trying to ssh into hetzner3
# Marcin generated a new ssh key for me to auth his access
# I asked him to try again with '-vvv' and also send me his public key
# I sent him an email asking him to confirm that the key was
# on the server's side, I saw this
## generated following our best-practices as documented on the wiki and https://wiki.opensourceecology.org/wiki/OSE_Server#SSH
<pre>
## probably backed-up on his veracrypt-encrypted usb drive
root@hetzner3 ~ # journalctl -u ssh
# once he confirms, I'll add it to his authorized_keys file, and then we can confirm access is working
...
Feb 15 02:08:19 hetzner3 sshd[1552191]: Connection from REDACTED port 40998 on 144.76.164.201 port 32415 rdomain ""
Feb 15 02:08:29 hetzner3 sshd[1552191]: Failed publickey for tgriffing from REDACTED port 40998 ssh2: RSA SHA256:OEad8nPjAY1Rwt73cPO7GxvX+I6scrraDZMYwQUldY4
Feb 15 02:08:29 hetzner3 sshd[1552191]: Connection closed by authenticating user tgriffing REDACTED port 40998 [preauth]
Feb 15 02:08:44 hetzner3 sshd[1552199]: Connection from REDACTED port 56426 on 144.76.164.201 port 32415 rdomain ""
Feb 15 02:08:47 hetzner3 sshd[1552199]: Failed publickey for tgriffing from REDACTED port 56426 ssh2: RSA SHA256:OEad8nPjAY1Rwt73cPO7GxvX+I6scrraDZMYwQUldY4
Feb 15 02:08:48 hetzner3 sshd[1552199]: Connection closed by authenticating user tgriffing REDACTED port 56426 [preauth]
</pre>
# ...
# ...
# yesterday I setup a second vhost for obi running on a unique ipv6 address, but I couldn't get it fully working
# Marcin sent me an email that seemed to suggest that we should delay the content fixes of osemain until after we migrate to hetzner3. I sent an email asking to confirm if that's his intention
# one issue was that my laptop might not have an ipv6 address; let's try from my personal server
# yesterday we setup hetzner3 to have two ipv6 addresses
## 2a01:4f8:200:40d7::2, which we had before
## 2a01:4f8:200:40d7::3, which is new
<pre>
root@hetzner3 ~ # ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host noprefixroute
  valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2a01:4f8:200:40d7::3/64 scope global
  valid_lft forever preferred_lft forever
inet6 2a01:4f8:200:40d7::2/64 scope global
  valid_lft forever preferred_lft forever
inet6 fe80::921b:eff:fec4:28b4/64 scope link
  valid_lft forever preferred_lft forever
root@hetzner3 ~ #
</pre>
# I confirmed that I can ping the server on both addresss; and it doesn't work (as expected) on '::4', which was never setup
<pre>
maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::2
PING 2a01:4f8:200:40d7::2(2a01:4f8:200:40d7::2) 56 data bytes
64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=1 ttl=57 time=2.85 ms
64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=2 ttl=57 time=2.91 ms
 
--- 2a01:4f8:200:40d7::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.852/2.879/2.906/0.027 ms
maltfield@mail:~$
 
maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::3
PING 2a01:4f8:200:40d7::3(2a01:4f8:200:40d7::3) 56 data bytes
64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=1 ttl=57 time=3.19 ms
64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=2 ttl=57 time=2.88 ms
 
--- 2a01:4f8:200:40d7::3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 2.877/3.034/3.191/0.157 ms
maltfield@mail:~$
 
maltfield@mail:~$ ping -c2 2a01:4f8:200:40d7::4
PING 2a01:4f8:200:40d7::4(2a01:4f8:200:40d7::4) 56 data bytes
 
--- 2a01:4f8:200:40d7::4 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1011ms
 
maltfield@mail:~$
</pre>
# yeah, I can't do this from my Qubes VM
<pre>
user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::3
ping: connect: Network is unreachable
user@disp3993:~$ ping -c2 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=57 time=394 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=57 time=357 ms
 
--- 1.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 356.708/375.276/393.845/18.568 ms
user@disp3993:~$
 
user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::2
ping: connect: Network is unreachable
user@disp3993:~$
 
user@disp3993:~$ ping -c2 2a01:4f8:200:40d7::3
ping: connect: Network is unreachable
user@disp3993:~$
</pre>
# well, that's a good test. it's not worth continuing to pursue this if Marcin and Catarina can't use IPv6.
# I sent an email to them asking them to run a ping test
<pre>
Hey Marcin,
Hey Catarina,
 
When you're at FeF, can you please run this command and send me the output?
 
  ping -c2 2a01:4f8:200:40d7::3
 
I'm trying to figure out if we can run two copies of osemain and obi (a static site + the broken/upgraded wordpress site). But we need an additional IP address to do this.
 
One way to do this (which would keep the domain name identical for both sites and therefore be best), would be to have each site bound to different IP addresses. With Hetnzer, we get just 1 IPv4 address and 616 IPv6 addresses.
 
But not every ISP (or router, or computer) provides IPv6 to their clients.  So this ping test will let me know if you're able to access a website that's only accessible over IPv6.
 
Please run the above ping command and send me the full output.
 
 
Thank you,
</pre>
# if their test returns "network is unreachable", then I'll suggest we buy a second IPv4 address for ~2 EUR/mo (just to use until they finish the new/replacement websites)
# ...
# ...
# I checked-in on the mediawiki cron. It looks like it ran successfully this time
# Catarina sent me output of `ls -lah ~/.ssh` which showed she has no ssh keypair
<pre>
# I sent her the commands to generate a new keypair and asked if she has a backup solution in-place
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
root@hetzner3 /usr/local/bin #
 
root@hetzner3 /usr/local/bin # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | head
total 2,8M
drwxrwx---  2 www-data www-data  72K Feb 15 05:04 .
drwxrwx--- 30 www-data www-data 4,0K Feb 13 19:48 ..
-rw-r--r--  1 www-data www-data 3,9K Feb 15 05:04 image_006de54f_46a605f583c811fa.png
-rw-r--r--  1 www-data www-data 4,6K Feb 15 05:04 image_00bb6228_3473277933896087.png
-rw-r--r--  1 www-data www-data 3,8K Feb 15 05:04 image_0289ccc4_158c15199e2053dd.png
-rw-r--r--  1 www-data www-data 4,3K Feb 15 05:04 image_02909476_69fb1a6ef8109be7.png
-rw-r--r--  1 www-data www-data 4,9K Feb 15 05:04 image_03bdc132_5d43e1a1df3307de.png
-rw-r--r--  1 www-data www-data 3,6K Feb 15 05:04 image_04f4de33_9b0075a6be38d1ad.png
-rw-r--r--  1 www-data www-data 3,9K Feb 15 05:04 image_0519550c_71efe71f45d70123.png
root@hetzner3 /usr/local/bin #
 
root@hetzner3 /usr/local/bin # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | tail
-rw-r--r--  1 www-data www-data 3,2K Feb 15 05:04 image_fc79e722_7c8a08bcc570879b.png
-rw-r--r--  1 www-data www-data 3,5K Feb 15 05:04 image_fdef3933_121c56952a6c57d3.png
-rw-r--r--  1 www-data www-data 4,2K Feb 15 05:04 image_fe463544_629383d940de704d.png
-rw-r--r--  1 www-data www-data 4,0K Feb 15 05:04 image_fe56e8a6_d360bd714e425344.png
-rw-r--r--  1 www-data www-data 4,6K Feb 15 05:04 image_fe8301f9_4d8d17795ab35d70.png
-rw-r--r--  1 www-data www-data 4,2K Feb 15 05:04 image_feb29944_fda14c66d7316301.png
-rw-r--r--  1 www-data www-data 3,6K Feb 15 05:04 image_fef37631_1c47b152a8639807.png
-rw-r--r--  1 www-data www-data 3,9K Feb 15 05:04 image_ff128933_f9004121b7d624c0.png
-rw-r--r--  1 www-data www-data 3,7K Feb 15 05:04 image_ffb24016_bfc86a5d83776491.png
-rw-r--r--  1 www-data www-data 3,5K Feb 15 05:04 image_fff89283_ee1207b5b2d10c3e.png
root@hetzner3 /usr/local/bin #
 
root@hetzner3 /usr/local/bin # ls /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | wc -l
500
root@hetzner3 /usr/local/bin # date -u
2025-02-15T20:54:29 UTC
root@hetzner3 /usr/local/bin #
</pre>
# so there's exactly 500 images, and they were all generated today. perfect!
# the only issue I do see is that the log was cleared of messages from before, and it wasn't rotated
<pre>
root@hetzner3 /usr/local/bin # ls -lah /var/log/cron
total 12K
drwxr-x---  2 www-data www-data 4,0K Feb 14 05:18 .
drwxr-xr-x 16 root    root    4,0K Feb 13 22:54 ..
-rw-r--r--  1 www-data www-data  86 Feb 15 05:04 mediawiki_generate_captchas.log
root@hetzner3 /usr/local/bin #
</pre>
# looks like the cron is missing a second > sign (for append)
<pre>
root@hetzner3 /usr/local/bin # cat /etc/cron.d/mediawiki_cron
# Ansible managed
SHELL=/bin/bash
 
# regenerate new CAPTCHA images
20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log
root@hetzner3 /usr/local/bin #
</pre>
# I fixed this in ansible and pushed it out https://github.com/OpenSourceEcology/ansible/commit/3aa97d5d27e538ce234aa0354a8a5ac1ae4bc082
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ git diff HEAD^
diff --git a/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2 b/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2
index 28e5bf0..873a137 100644
--- a/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2
+++ b/hetzner3/roles/maltfield.cron/templates/mediawiki_cron.j2
@@ -2,4 +2,4 @@
SHELL=/bin/bash
# regenerate new CAPTCHA images
-20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log
+20 4 * * * root sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &>> /var/log/cron/mediawiki_generate_captchas.log
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# ...
# ...
# another item I have in my TODO list was to confirm that APC is setup properly
# Catarina said she found the links to login to her new wiki admin account, but the password didn't work (maybe it expired)
# I never tested this. If APC isn't running, it'll greatly harm the performance of our PHP websites
# I logged-into the wiki today and tried to reset the password, but I got an error when trying to query for the username https://wiki.opensourceecology.org/index.php?title=Special%3AUserCredentials&target=CatarinaAdmin
# It looks like APC is no longer supported in PHP >5.4
# hetzner2 ran php v5.6 and hetzner3 runs v8.2.26, so I guess there's no change there
# the replacement for APC was APCu https://www.php.net/manual/en/book.apcu.php
# it's not clear to me when APCu was first supported, but they have releases going back to 2013https://pecl.php.net/package/apcu
# here's the mediawiki guide that describes apcu https://www.mediawiki.org/wiki/Manual:Performance_tuning
# it says to install php-acpu, which we've done (but I still want to test that it's actually enabled & working)
<pre>
<pre>
root@hetzner3 /usr/local/bin # apt-cache search apcu
No credentials found for this user. Check that the name is spelled correctly.
apcupsd - APC UPS Power Management (daemon)
apcupsd-cgi - APC UPS Power Management (web interface)
apcupsd-doc - APC UPS Power Management (documentation/examples)
collectd-core - statistics collection and monitoring daemon (core system)
conky-all - highly configurable system monitor (all features enabled)
conky-cli - highly configurable system monitor (basic version)
conky-std - highly configurable system monitor (default version)
php-apcu - APC User Cache for PHP
php-apcu-all-dev - APC User Cache for PHP
php8.2-apcu - APC User Cache for PHP
php-symfony-polyfill-apcu - Symfony polyfill backporting apcu_* functions to lower PHP versions
root@hetzner3 /usr/local/bin #
 
root@hetzner3 /usr/local/bin # dpkg -l | grep -i apcu
ii  php-apcu                      5.1.22+4.0.11-2                        amd64        APC User Cache for PHP
ii  php8.2-apcu                    5.1.22+4.0.11-2                        amd64        APC User Cache for PHP
root@hetzner3 /usr/local/bin #
</pre>
</pre>
# according to the mediawiki docs, that should have installed some script 'apcu.php' that we can use to inspect the apc cache and confirm that mediawiki is using it
# I checked the list of users in the admin group, and it includes the ones I thought I had deleted!?
<pre>
# According to my notes, I did this already on 2025-02-16 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2025_Q1#Sun_Feb_16.2C_2025
A script, apc.php is bundled with the APCu package which can be used to inspect the status of the cache, and also examine the contents of the user cache to verify that MediaWiki is correctly using it.
# my best-guess is that I accidentally made the change on hetzner3, not hetzner2
</pre>
# again, I edited these accounts, removing the 'administrator' role
# uhh, that's just docs
<pre>
root@hetzner3 /usr/local/bin # dpkg-query -L php-apcu
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/php-apcu
/usr/share/doc/php-apcu/README.Debian
/usr/share/doc/php-apcu/changelog.Debian.gz
/usr/share/doc/php-apcu/copyright
root@hetzner3 /usr/local/bin #
</pre>
# ok, it's this one
<pre>
root@hetzner3 /usr/local/bin # dpkg-query -L php8.2-apcu
/.
/etc
/etc/php
/etc/php/8.2
/etc/php/8.2/mods-available
/etc/php/8.2/mods-available/apcu.ini
/usr
/usr/include
/usr/include/php
/usr/include/php/20220829
/usr/include/php/20220829/ext
/usr/include/php/20220829/ext/apcu
/usr/include/php/20220829/ext/apcu/apc.h
/usr/include/php/20220829/ext/apcu/apc_api.h
/usr/include/php/20220829/ext/apcu/apc_arginfo.h
/usr/include/php/20220829/ext/apcu/apc_cache.h
/usr/include/php/20220829/ext/apcu/apc_globals.h
/usr/include/php/20220829/ext/apcu/apc_iterator.h
/usr/include/php/20220829/ext/apcu/apc_lock.h
/usr/include/php/20220829/ext/apcu/apc_mutex.h
/usr/include/php/20220829/ext/apcu/apc_serializer.h
/usr/include/php/20220829/ext/apcu/apc_sma.h
/usr/include/php/20220829/ext/apcu/apc_stack.h
/usr/include/php/20220829/ext/apcu/php_apc.h
/usr/include/php/20220829/ext/apcu/php_apc_legacy_arginfo.h
/usr/lib
/usr/lib/php
/usr/lib/php/20220829
/usr/lib/php/20220829/apcu.so
/usr/share
/usr/share/doc
/usr/share/doc/php8.2-apcu
/usr/share/doc/php8.2-apcu/changelog.Debian.gz
/usr/share/doc/php8.2-apcu/copyright
root@hetzner3 /usr/local/bin #
</pre>
# ...but there's still no apcu.php file :(
# actually, we are missing a package; the MediaWiki docs say we also need this php-igbinary
<pre>
root@hetzner3 /usr/local/bin # apt-cache search igbinary
php-igbinary - igbinary PHP serializer
php-igbinary-all-dev - igbinary PHP serializer
php8.2-igbinary - igbinary PHP serializer
You have mail in /var/mail/root
root@hetzner3 /usr/local/bin # dpkg -l | grep -i igbinary
root@hetzner3 /usr/local/bin #
</pre>
# I added it to the depends listed in the ansible php role
# according to the docs, we're "strongly recommended" to set this
<pre>
$wgMainCacheType = CACHE_ACCEL;
</pre>
# I confirmed that we are using it
<pre>
root@hetzner3 /usr/local/bin # grep -i cache /var/www/html/wiki.opensourceecology.org/LocalSettings.php
$wgCategoryTreeDisableCache = false;
$wgCategoryTreeHTTPCache = true;
$egCache = true;
# sure that cached pages are cleared.
$wgCacheEpoch = max( $wgCacheEpoch, gmdate( 'YmdHis', @filemtime( FILE ) ) );
# VARNISH CACHE #
# INTERNAL MEDIAWIKI CACHE OPTIONS (DISTINCT FROM VARNISH)
# MainCache and MessageCache should use APCU per Aaron Schulz
$wgMainCacheType = CACHE_ACCEL;
# note that if message cache uses the db (per defaults), then it may make every
# not to cache the page (rightfully so), and the result is that varnish (which
# is our most important cache) is rendered useless. For more info, see:
$wgMessageCacheType = CACHE_ACCEL;
$wgUseLocalMessageCache = true;
# Parser Cache should still use the DB per Aaron Schulz
$wgParserCacheType = CACHE_DB;
$wgEnableSidebarCache = true;
# cache interface messages to files in this directory per Aaron Schulz
$wgCacheDirectory = "$IP/../cache";
root@hetzner3 /usr/local/bin #
</pre>
# good to note that the mediawiki performance docs suggest using php-fpm instead of mod_php, something we've done for this migration from hetzner2 to hetzner3
# it also suggests using event MPM instead of prefork MPM
# crap, it looks like we're using prefork
<pre>
root@hetzner3 /etc/apache2 # grep -irl 'mpm' *
magic
mods-available/mpm_event.conf
mods-available/pagespeed.conf
mods-available/mpm_worker.load
mods-available/mpm_prefork.conf
mods-available/mpm_prefork.load
mods-available/mpm_worker.conf
mods-available/php8.2.load
mods-available/mpm_prefork.conf.104647.2024-09-25@01:24:19~
mods-available/mpm_event.load
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # ls -lah mods-enabled/
total 8,0K
drwxr-xr-x 2 root root 4,0K Dec 29 03:27 .
drwxr-xr-x 8 root root 4,0K Feb 13 20:02 ..
lrwxrwxrwx 1 root root  36 Sep 25 01:24 access_compat.load -> ../mods-available/access_compat.load
lrwxrwxrwx 1 root root  28 Sep 25 01:24 alias.conf -> ../mods-available/alias.conf
lrwxrwxrwx 1 root root  28 Sep 25 01:24 alias.load -> ../mods-available/alias.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 auth_basic.load -> ../mods-available/auth_basic.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 authn_core.load -> ../mods-available/authn_core.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 authn_file.load -> ../mods-available/authn_file.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 authz_core.load -> ../mods-available/authz_core.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 authz_host.load -> ../mods-available/authz_host.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 authz_user.load -> ../mods-available/authz_user.load
lrwxrwxrwx 1 root root  32 Sep 25 01:24 autoindex.conf -> ../mods-available/autoindex.conf
lrwxrwxrwx 1 root root  32 Sep 25 01:24 autoindex.load -> ../mods-available/autoindex.load
lrwxrwxrwx 1 root root  30 Sep 25 01:24 deflate.conf -> ../mods-available/deflate.conf
lrwxrwxrwx 1 root root  30 Sep 25 01:24 deflate.load -> ../mods-available/deflate.load
lrwxrwxrwx 1 root root  26 Sep 25 01:24 dir.conf -> ../mods-available/dir.conf
lrwxrwxrwx 1 root root  26 Sep 25 01:24 dir.load -> ../mods-available/dir.load
lrwxrwxrwx 1 root root  26 Sep 25 01:24 env.load -> ../mods-available/env.load
lrwxrwxrwx 1 root root  29 Sep 25 01:24 filter.load -> ../mods-available/filter.load
lrwxrwxrwx 1 root root  40 Sep 25 01:24 headers.load -> /etc/apache2/mods-available/headers.load
lrwxrwxrwx 1 root root  27 Sep 25 01:24 mime.conf -> ../mods-available/mime.conf
lrwxrwxrwx 1 root root  27 Sep 25 01:24 mime.load -> ../mods-available/mime.load
lrwxrwxrwx 1 root root  34 Dec 29 03:27 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf
lrwxrwxrwx 1 root root  34 Dec 29 03:27 mpm_prefork.load -> ../mods-available/mpm_prefork.load
lrwxrwxrwx 1 root root  34 Sep 25 01:24 negotiation.conf -> ../mods-available/negotiation.conf
lrwxrwxrwx 1 root root  34 Sep 25 01:24 negotiation.load -> ../mods-available/negotiation.load
lrwxrwxrwx 1 root root  29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf
lrwxrwxrwx 1 root root  29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load
lrwxrwxrwx 1 root root  43 Sep 26 00:06 proxy_fcgi.load -> /etc/apache2/mods-available/proxy_fcgi.load
lrwxrwxrwx 1 root root  38 Sep 26 00:11 proxy.load -> /etc/apache2/mods-available/proxy.load
lrwxrwxrwx 1 root root  41 Sep 25 01:24 remoteip.load -> /etc/apache2/mods-available/remoteip.load
lrwxrwxrwx 1 root root  33 Sep 25 01:24 reqtimeout.conf -> ../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root  33 Sep 25 01:24 reqtimeout.load -> ../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root  40 Sep 25 01:24 rewrite.load -> /etc/apache2/mods-available/rewrite.load
lrwxrwxrwx 1 root root  32 Sep 25 01:24 security2.conf -> ../mods-available/security2.conf
lrwxrwxrwx 1 root root  32 Sep 25 01:24 security2.load -> ../mods-available/security2.load
lrwxrwxrwx 1 root root  31 Sep 25 01:24 setenvif.conf -> ../mods-available/setenvif.conf
lrwxrwxrwx 1 root root  31 Sep 25 01:24 setenvif.load -> ../mods-available/setenvif.load
lrwxrwxrwx 1 root root  39 Sep 25 01:24 status.conf -> /etc/apache2/mods-available/status.conf
lrwxrwxrwx 1 root root  29 Sep 25 01:24 status.load -> ../mods-available/status.load
lrwxrwxrwx 1 root root  32 Sep 25 01:24 unique_id.load -> ../mods-available/unique_id.load
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # cat mods-enabled/mpm_prefork.load
# Conflicts: mpm_event mpm_worker
LoadModule mpm_prefork_module /usr/lib/apache2/modules/mod_mpm_prefork.so
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # cat mods-enabled/mpm_prefork.conf
# Ansible managed
 
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxRequestWorkers: maximum number of server processes allowed to start
# MaxConnectionsPerChild: maximum number of requests a server process serves
 
<IfModule mpm_prefork_module>
StartServers                5
MinSpareServers              5
MaxSpareServers            10
MaxRequestWorkers          150
MaxConnectionsPerChild    1000
</IfModule>
 
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
 
root@hetzner3 /etc/apache2 #
</pre>
# it looks like the last time I looked at prefork vs event, I realized we were still using mod_php, and that's when I switched us to php-fpm. But apparently I never revisited changing from MPM prefork to event https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Tue_Sep_24.2C_2024
# oh, no, I didn't find it in my logs, but I commented-out the prefork config in ansible on 2024-09-24 https://github.com/OpenSourceEcology/ansible/commit/763d9f996130edd579ad12388ec2861bc5c3cdc3
<pre>
fix apache
 
This commit removes the mpm_prefork module load and config, since Debian 12 actually already ships with mpm_event with Apache 2.4. Note that mpm_event is generally considered more performant, but the downside is that it's not OK for modules that break when threads are used, such as mod_php
 
The solution to that is to switch to php-fpm, which is also more performant -- and thread safe.
 
When we had issues with Apache spawning runaway children and killing the whole server earlier this year (2024), my research suggested that we should use php-fpm, but I didn't make that change for simplicity. What we *did* change was to configure MaxServers in mpm_prefork, but that now conflicts with the defaults on Debian (before it was CentOS), because only one mpm can be selected.
 
Probably our apache configs in ansible will still need to be adjusted to direct php traffic to the cgi process (proxy?)
</pre>
# unfortunately, I guess commenting it out wasn't sufficient to disable it.
# it looks like I need to remove the symlinks in 'mod-enabled/' from prefork and add these instead
<pre>
root@hetzner3 /etc/apache2 # find . | grep -i event
./mods-available/mpm_event.conf
./mods-available/mpm_event.load
root@hetzner3 /etc/apache2 #
</pre>
# here's the default config for mpm_event
<pre>
root@hetzner3 /etc/apache2 # cat ./mods-available/mpm_event.conf
# event MPM
# StartServers: initial number of server processes to start
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestWorkers: maximum number of worker threads
# MaxConnectionsPerChild: maximum number of requests a server process serves
StartServers            2
MinSpareThreads        25
MaxSpareThreads        75
ThreadLimit            64
ThreadsPerChild        25
MaxRequestWorkers      150
MaxConnectionsPerChild  0
You have mail in /var/mail/root
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # cat ./mods-available/mpm_event.load
# Conflicts: mpm_worker mpm_prefork
LoadModule mpm_event_module /usr/lib/apache2/modules/mod_mpm_event.so
root@hetzner3 /etc/apache2 #
</pre>
# I tried to fix this (remove the symlinks for prefork and add new symlinks for event)
user@ose:~/sandbox_local/ansible/hetzner3$ git diff
...
--- a/hetzner3/roles/maltfield.apache/tasks/main.yml
+++ b/hetzner3/roles/maltfield.apache/tasks/main.yml
@@ -140,29 +140,32 @@
  notify: restart apache
# we're using mpm_event, the default in Debian 12 with Apache 2.4
-#- name: enable module = mpm_prefork
-#  file:
-#    src: /etc/apache2/mods-available/mpm_prefork.load
-#    dest: /etc/apache2/mods-enabled/mpm_prefork.load
-#    state: link
-#  notify: restart apache
-#
-#- name: mpm_prefork.conf
...skipping...
+- name: disable module = mpm_prefork
+  file:
+    path: /etc/apache2/mods-enabled/mpm_prefork.load
+    state: absent
+  notify: restart apache
+
+- name: mpm_prefork config symlink
+  file:
+    path: /etc/apache2/mods-enabled/mpm_prefork.conf
+    state: absent
+  notify: restart apache
+
+- name: enable module = mpm_event
+  file:
+    src: /etc/apache2/mods-available/mpm_event.load
+    dest: /etc/apache2/mods-enabled/mpm_event.load
+    state: link
+  notify: restart apache
+
+- name: mpm_event config symlink
+  file:
+    src: /etc/apache2/mods-available/mpm_event.conf
+    dest: /etc/apache2/mods-enabled/mpm_event.conf
+    state: link
+  notify: restart apache
# prod mode config
- name: main apache4_prod.conf
diff --git a/hetzner3/roles/maltfield.php/tasks/main.yml b/hetzner3/roles/maltfield.php/tasks/main.yml
index fecdcd6..75a806a 100644
--- a/hetzner3/roles/maltfield.php/tasks/main.yml
+++ b/hetzner3/roles/maltfield.php/tasks/main.yml
@@ -5,6 +5,7 @@
pkg:
  - php-fpm
  - php-apcu
+      - php-igbinary
  - php-curl
  notify:
- restart php-fpm
user@ose:~/sandbox_local/ansible/hetzner3$
# but apache failed on restart
<pre>
root@hetzner3 /etc/apache2 # journalctl -u apache2 | less
...
root@hetzner3 /etc/apache2 # systemctl restart apache2
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # journalctl -u apache2 | less
...
Feb 15 21:50:31 hetzner3 apachectl[1909056]: [Sat Feb 15 21:50:31.965322 2025] [php:crit] [pid 1909056:tid 1909056] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
Feb 15 21:50:31 hetzner3 apachectl[1909056]: AH00013: Pre-configuration failed
Feb 15 21:50:31 hetzner3 apachectl[1909054]: Action 'graceful-stop' failed.
Feb 15 21:50:31 hetzner3 apachectl[1909054]: The Apache error log may have more information.
Feb 15 21:50:31 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'.
Feb 15 21:50:32 hetzner3 systemd[1]: Stopped apache2.service - The Apache HTTP Server.
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Consumed 8.836s CPU time.
Feb 15 21:50:32 hetzner3 systemd[1]: Starting apache2.service - The Apache HTTP Server...
Feb 15 21:50:32 hetzner3 apachectl[1909062]: [Sat Feb 15 21:50:32.073820 2025] [php:crit] [pid 1909062:tid 1909062] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
Feb 15 21:50:32 hetzner3 apachectl[1909062]: AH00013: Pre-configuration failed
Feb 15 21:50:32 hetzner3 apachectl[1909059]: Action 'start' failed.
Feb 15 21:50:32 hetzner3 apachectl[1909059]: The Apache error log may have more information.
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
Feb 15 21:50:32 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'.
Feb 15 21:50:32 hetzner3 systemd[1]: Failed to start apache2.service - The Apache HTTP Server.
You have mail in /var/mail/root
root@hetzner3 /etc/apache2 #
</pre>
# looks like we're using event. please tell me we're not using mod_php still?
<pre>
root@hetzner3 /etc/apache2 # ls -lah mods-enabled/ |grep -iE 'php|prefork|event'
lrwxrwxrwx 1 root root  42 Feb 15 21:40 mpm_event.conf -> /etc/apache2/mods-available/mpm_event.conf
lrwxrwxrwx 1 root root  42 Feb 15 21:40 mpm_event.load -> /etc/apache2/mods-available/mpm_event.load
lrwxrwxrwx 1 root root  29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf
lrwxrwxrwx 1 root root  29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load
root@hetzner3 /etc/apache2 #
</pre>
# yeah, I think that this is mod_php; let's disable it
<pre>
root@hetzner3 /etc/apache2 # cat mods-enabled/php8.2.conf
# Using (?:pattern) instead of (pattern) is a small optimization that
# avoid capturing the matching pattern (as $1) which isn't used here
<FilesMatch ".+\.ph(?:ar|p|tml)$">
SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch ".+\.phps$">
SetHandler application/x-httpd-php-source
# Deny access to raw php sources by default
# To re-enable it's recommended to enable access to the files
# only in specific virtual host or directory
Require all denied
</FilesMatch>
# Deny access to files without filename (e.g. '.php')
<FilesMatch "^\.ph(?:ar|p|ps|tml)$">
Require all denied
</FilesMatch>
 
# Running PHP scripts in user directories is disabled by default
#
# To re-enable PHP in user directories comment the following lines
# (from <IfModule ...> to </IfModule>.) Do NOT set it to On as it
# prevents .htaccess files from disabling it.
<IfModule mod_userdir.c>
<Directory /home/*/public_html>
php_admin_flag engine Off
</Directory>
</IfModule>
root@hetzner3 /etc/apache2 #
</pre>
# shit, I'm not sure we're even using php-fpm
# oh, it looks like both are enabled? This is very annoying because mod_php is in "mods-enabled" and php-fpm is in "conf-enabled"
<pre>
root@hetzner3 /etc/apache2 # ls -lah mods-available | grep -i php
-rw-r--r-- 1 root root  998 Jun 17  2024 php8.2.conf
-rw-r--r-- 1 root root  101 Jun 17  2024 php8.2.load
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # ls -lah conf-available | grep -i php
-rw-r--r-- 1 root root 1,4K Jun 17  2024 php8.2-fpm.conf
-rw-r--r-- 1 root root 1,2K Sep 25 01:46 phplist.virtualhost.include
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # ls -lah conf-enabled | grep -i php
lrwxrwxrwx 1 root root  43 Sep 25 23:53 php8.2-fpm.conf -> /etc/apache2/conf-available/php8.2-fpm.conf
root@hetzner3 /etc/apache2 #
 
root@hetzner3 /etc/apache2 # ls -lah mods-enabled | grep -i php
lrwxrwxrwx 1 root root  29 Dec 29 03:27 php8.2.conf -> ../mods-available/php8.2.conf
lrwxrwxrwx 1 root root  29 Dec 29 03:27 php8.2.load -> ../mods-available/php8.2.load
root@hetzner3 /etc/apache2 #
</pre>
# anyway, I'll update ansible to remove
## before
<pre>
root@hetzner3 /etc/apache2 # ls *-enabled/* | grep -i php
conf-enabled/php8.2-fpm.conf
mods-enabled/php8.2.conf
mods-enabled/php8.2.load
sites-enabled/phplist.opensourceecology.org.conf
root@hetzner3 /etc/apache2 #
</pre>
## after
<pre>
root@hetzner3 /etc/apache2 # ls *-enabled/* | grep -i php
conf-enabled/php8.2-fpm.conf
sites-enabled/phplist.opensourceecology.org.conf
root@hetzner3 /etc/apache2 #
</pre>
# now it's working
<pre>
root@hetzner3 /etc/apache2 # systemctl restart apache2
root@hetzner3 /etc/apache2 #
</pre>
# I did a quick check of store and wiki, they both load fine
# wow, I'm glad I caught that. So now we're using the much more performant mpm event, and the (prerequisite) php-fpm
# but we still haven't confirmed apc is working!
# this answer on SE says to use 'apc.php' too, but it also provides an alternative https://stackoverflow.com/a/17631545/1174102
<pre>
<pre><?php
print_r(apc_cache_info());
</pre>
# I created a script for this on the wiki
<pre>
cat > /var/www/html/wiki.opensourceecology.org/htdocs/apc.php <<'EOF'
<pre>
<?php
print_r(apc_cache_info());
?>
</pre>
EOF
 
chown not-apache:www-data /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
chmod 0040 /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
</pre>
# crap, that page is empty. And this is in the apache logs
<pre>
[Sat Feb 15 22:19:30.627296 2025] [proxy_fcgi:error] [pid 1917176:tid 1917223] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function apc_cache_info() in /var/www/html/wiki.opensourceecology.org/htdocs/apc.php:3\nStack trace:\n#0 {main}\n  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/apc.php on line 3'
</pre>
# well, it looks like I'm not disabling that function; maybe it just doesn't exist anymore
<pre>
root@hetzner3 /var/log/apache2 # grep -ir apc /etc/php/8.2/
/etc/php/8.2/mods-available/apcu.ini:extension=apcu.so
/etc/php/8.2/mods-available/igbinary.ini:; Use igbinary as serializer in APC cache (3.1.7 or later)
/etc/php/8.2/mods-available/igbinary.ini:;apc.serializer=igbinary
root@hetzner3 /var/log/apache2 #
</pre>
# the php reference docs don't say the function has been retired. So maybe, in fact, apc isn't enabled? https://www.php.net/manual/en/function.apcu-cache-info.php
# oh, duh, ^ that one is 'acpu_' and the one I wrote to the file is 'apc_'
# let's try this
<pre>
cat > /var/www/html/wiki.opensourceecology.org/htdocs/apc.php <<'EOF'
<pre>
<?php
print_r(apcu_cache_info());
?>
</pre>
EOF
 
chown not-apache:www-data /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
chmod 0040 /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
</pre>
# that worked! It spat-out an array that spans 23,041 lines
# here's the top of it
<pre>
Array
(
[num_slots] => 4099
[ttl] => 0
[num_hits] => 1662
[num_misses] => 1867
[num_inserts] => 1701
[num_entries] => 1665
[expunges] => 0
[start_time] => 1739654285
[mem_size] => 2845632
[memory_type] => mmap
[cache_list] => Array
(
</pre>
# the SE post above says that if "num_hits" is increasing, then it's working
# I logged-in on the wiki and edited the main page
# then I refreshed the apc.php page, and now it says
<pre>
Array
(
[num_slots] => 4099
[ttl] => 0
[num_hits] => 1662
[num_misses] => 1867
[num_inserts] => 1701
[num_entries] => 1665
[expunges] => 0
[start_time] => 1739654285
[mem_size] => 2845632
[memory_type] => mmap
[cache_list] => Array
(
</pre>
# fuck; num_hits didn't increase
# wait, maybe it's cached. I tried "/apc.php?nocache=1"
<pre>
Array
(
[num_slots] => 4099
[ttl] => 0
[num_hits] => 6929
[num_misses] => 2080
[num_inserts] => 1870
[num_entries] => 1777
[expunges] => 0
[start_time] => 1739654285
[mem_size] => 3586816
[memory_type] => mmap
[cache_list] => Array
(
</pre>
# oh, that's a huge increase. I guess we can conclude that apc is working. great!
# let's cleanup that script
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # rm /var/www/html/wiki.opensourceecology.org/htdocs/apc.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# ...
# another item on my TODO list is to confirm that my various vhost hardenings are still working (since the upgrades, changes to php-fpm, etc...it's likely some of these broke)
## wp-login.php shouldn't be accessible
## phplist config.php shouldn't be accessible (I want to confirm because old had ".*" prefix and new doesn't)
## .php files in wp-content uploads dirs shouldn't be executable
##  anything other than GET POST HEAD should be denied
## anything that starts with a dot (other than .well-known) should be denied (test .git & .htaccess)
## indexes on a dir without 'index.php' shouldn't show a directory listing
## .htaccess shouldn't allow changing things
# this claims to block access to wp-login.php, but I'm pretty sure it's just the wordpress plugin; you can see the varnish header suggest that it penetrated past nginx
<pre>
user@disp3993:~$ curl -iL store.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 15 Feb 2025 22:34:14 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://store.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
 
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sat, 15 Feb 2025 22:34:17 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 214
Connection: keep-alive
X-Varnish: 29914517
Age: 0
Via: 1.1 varnish-v4
 
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /wp-login.php
on this server.</p>
</body></html>
user@disp3993:~$
</pre>
# also these configs should span apps, and we can see the wiki just gives a 404
<pre>
user@disp3993:~$ curl -iL wiki.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 15 Feb 2025 22:36:54 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://wiki.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=1;includeSubDomains
 
HTTP/1.1 404 Not Found
Server: nginx
Date: Sat, 15 Feb 2025 22:36:56 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 16
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: strict-origin
X-Varnish: 427397 427395
Age: 107
Via: 1.1 varnish (Varnish/7.1)
 
File not found.
user@disp3993:~$
</pre>
# oh, actually, I had an error in /etc/hosts. This is totally broken on hetnzer3
<pre>
user@disp3993:~$ curl -IL store.opensourceecology.org/wp-login.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 15 Feb 2025 23:01:17 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://store.opensourceecology.org/wp-login.php
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Strict-Transport-Security: max-age=1;includeSubDomains
 
HTTP/1.1 302 Found
Server: nginx
Date: Sat, 15 Feb 2025 23:01:20 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: /
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 460388
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 15 Feb 2025 23:01:21 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 85328
Connection: keep-alive
X-Pingback: https://store.opensourceecology.org/xmlrpc.php
Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 200920 460383
Age: 9
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
user@disp3993:~$
</pre>
# I'm not sure if this broke or I intentionally disabled it
# one issue is that the plugin we had been using in wordpress to redirect away from wp-login.php is no longer available due to "Security Issue" https://wordpress.org/plugins/rename-wp-login/
# as a result, there's a narrow window during the migration of sites to hetzner3 where we need to login at wp-login.php and configure the replacement plugin to use a distinct login URL
# I also took a moment to update the plugins list on the ose wiki article for wordpress https://wiki.opensourceecology.org/wiki/Wordpress
# I think the best solution to this is to see if we can configure 'melapress-login-security' from the command-line after migrating, so that we never have to login at wp-login.php
# after some trial-and-error, I found the settings for this plugin are stored in the wordpress options table; there's 4 of them
<pre>
MariaDB [store_db]> select option_name from wp_options where option_name like '%mls_%';
+--------------------+
| option_name        |
+--------------------+
| mls_activation    |
| mls_active_version |
| mls_options        |
| mls_setting        |
+--------------------+
4 rows in set (0,001 sec)
 
MariaDB [store_db]>
</pre>
# the first two are a simple string
# the second two are long json strings
# well it looks like the min password length is set in 'wp_options' and the wp-login.php rename is set in 'mls_setting'
# unfortunately my email address got saved in wp_options, so I think I'm just going to set mls_setting (so that doesn't have to get stored in the wiki CHG migration commands/instructions)
# alright, here's what we have
<pre>
root@hetzner3 ~ # echo "select option_value from wp_options where option_name = 'mls_setting';" | mysql -uroot -p$mysqlPass store_db
option_value
a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}
root@hetzner3 ~ #
</pre>
# it appears that we can either update the db directly (dangerous) or use wp-cli https://developer.wordpress.org/cli/commands/option/
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get siteurl
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
https://store.opensourceecology.org
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
</pre>
# first I tested if 'update' needs to have a key for it to work; it doesn't (that way we don't have to delete first)
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get thisdoesnotexist
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Error: Could not get 'thisdoesnotexist' option. Does it exist?
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update thisdoesnotexist 'somevalue'
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'thisdoesnotexist' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get thisdoesnotexist
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
somevalue
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
</pre>
# so here's my plan to change the login from wp-login.php to ose-hidden-login
<pre>
sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}'
</pre>
# to test this, I deleted the settings on store and deactivated the plugin
# I reactivated it, and confirmed that the "login page hardening" was back at the default (with nothing specified for 'Login page URL'
# I deactivated the plugin again in the wui, and tried the rest with the cli
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" plugin activate melapress-login-security
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'melapress-login-security' activated.
Success: Activated 1 of 1 plugins.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
 
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}'
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'mls_setting' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
</pre>
# I refreshed '/wp-admin/' and re-checked the settings page. crap, it's still empty? https://store.opensourceecology.org/wp-admin/admin.php?page=mls-hide-login
# I confirmed the setting is there
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option get mls_setting
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_password";s:2:"no";s:16:"stop_pw_generate";s:2:"no";s:25:"users_have_multiple_roles";s:2:"no";s:19:"multiple_role_order";a:0:{}s:13:"clear_history";s:2:"no";s:22:"excluded_special_chars";s:0:"";s:25:"password_reset_key_expiry";a:2:{s:5:"value";i:24;s:4:"unit";s:5:"hours";}s:20:"enable_wp_reset_form";s:3:"yes";s:22:"enable_wp_profile_form";s:3:"yes";s:18:"enable_wc_pw_reset";s:2:"no";s:22:"enable_wc_checkout_reg";s:2:"no";s:18:"enable_bp_register";s:2:"no";s:19:"enable_bp_pw_update";s:2:"no";s:18:"enable_ld_register";s:2:"no";s:18:"enable_um_register";s:2:"no";s:19:"enable_um_pw_update";s:2:"no";s:24:"enable_bbpress_pw_update";s:2:"no";s:20:"enable_mepr_register";s:2:"no";s:21:"enable_mepr_pw_update";s:2:"no";s:19:"enable_edd_register";s:2:"no";s:20:"enable_edd_pw_update";s:2:"no";s:19:"enable_pmp_register";s:2:"no";s:20:"enable_pmp_pw_update";s:2:"no";s:19:"enable_pmp_pw_reset";s:2:"no";s:28:"enable_profilepress_register";s:2:"no";s:29:"enable_profilepress_pw_update";s:2:"no";s:28:"enable_profilepress_pw_reset";s:2:"no";s:16:"custom_login_url";s:16:"ose-hidden-login";s:21:"custom_login_redirect";s:0:"";s:24:"enable_login_allowed_ips";b:0;s:26:"restrict_login_allowed_ips";s:0:"";s:27:"restrict_login_redirect_url";s:0:"";s:26:"restrict_login_bypass_slug";s:0:"";s:24:"send_user_unlocked_email";s:3:"yes";s:25:"send_user_unblocked_email";s:3:"yes";s:24:"send_user_pw_reset_email";s:3:"yes";s:26:"send_user_pw_expired_email";s:3:"yes";s:16:"login_geo_method";s:7:"default";s:16:"login_geo_action";s:11:"deny_to_url";s:19:"login_geo_countries";s:0:"";s:22:"login_geo_redirect_url";s:0:"";s:25:"login_geo_blocked_message";s:0:"";s:16:"iplocate_api_key";s:0:"";s:19:"gdpr_banner_message";s:0:"";s:18:"enable_gdpr_banner";b:0;s:33:"disable_user_password_reset_email";s:2:"no";s:41:"disable_user_delayed_password_reset_email";s:2:"no";s:29:"disable_user_pw_expired_email";s:2:"no";s:40:"disable_user_unlocked_reset_needed_email";s:2:"no";s:36:"disable_device_policies_prompt_email";s:2:"no";s:42:"disable_device_policies_prompt_admin_email";s:2:"no";s:27:"disable_user_imported_email";s:2:"no";s:40:"disable_user_imported_forced_reset_email";s:2:"no";s:27:"disable_user_unlocked_email";s:2:"no";s:24:"user_unlocked_email_body";s:0:"";s:25:"user_unblocked_email_body";s:0:"";s:32:"user_reset_next_login_email_body";s:0:"";}
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
</pre>
# idk, maybe somehow I have to set both
# I obfuscated my email address and tried this
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_options 'a:62:{s:13:"master_switch";s:3:"yes";s:26:"activate_password_policies";s:3:"yes";s:37:"activate_password_expiration_policies";s:2:"no";s:34:"activate_password_recycle_policies";s:2:"no";s:16:"enforce_password";s:2:"no";s:10:"min_length";s:2:"20";s:16:"password_history";s:1:"1";s:16:"inherit_policies";s:2:"no";s:15:"password_expiry";a:2:{s:5:"value";s:1:"0";s:4:"unit";s:6:"months";}s:8:"ui_rules";a:7:{s:7:"history";s:2:"no";s:8:"username";s:2:"no";s:6:"length";b:1;s:7:"numeric";s:2:"no";s:8:"mix_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:5:"rules";a:6:{s:6:"length";b:1;s:7:"numeric";s:2:"no";s:10:"upper_case";s:2:"no";s:10:"lower_case";s:2:"no";s:13:"special_chars";s:2:"no";s:21:"exclude_special_chars";s:2:"no";}s:23:"change_initial_password";s:2:"no";s:12:"timed_logins";s:2:"no";s:21:"timed_logins_schedule";a:7:{s:6:"monday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:7:"tuesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:9:"wednesday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"thursday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"friday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:8:"saturday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}s:6:"sunday";a:7:{s:6:"enable";s:2:"no";s:7:"from_hr";i:0;s:8:"from_min";i:0;s:5:"to_hr";i:11;s:6:"to_min";i:59;s:13:"from_am_or_pm";s:2:"am";s:11:"to_am_or_pm";s:2:"pm";}}s:22:"inactive_users_enabled";s:2:"no";s:21:"inactive_users_expiry";a:2:{s:5:"value";i:30;s:4:"unit";s:4:"days";}s:30:"inactive_users_reset_on_unlock";s:2:"no";s:29:"failed_login_policies_enabled";s:3:"yes";s:21:"failed_login_attempts";s:1:"5";s:27:"failed_login_reset_attempts";s:4:"1440";s:27:"failed_login_unlock_setting";s:5:"timed";s:24:"failed_login_reset_hours";s:2:"60";s:29:"failed_login_reset_on_unblock";s:2:"no";s:18:"disable_self_reset";s:3:"yes";s:26:"disable_self_reset_message";b:0;s:27:"deactivated_account_message";s:161:"Your WordPress user has been deactivated. Please contact the <a href="mailto:michael@michaelaltfield.net">website administrator</a> to activate back your user.";s:19:"timed_login_message";b:0;s:30:"locked_user_disable_self_reset";s:2:"no";s:38:"locked_user_disable_self_reset_message";b:0;s:17:"restrict_login_ip";s:2:"no";s:23:"restrict_login_ip_count";i:3;s:22:"restrict_login_message";s:0:"";s:22:"notify_password_expiry";s:2:"no";s:30:"notify_password_reset_on_login";b:0;s:27:"notify_password_expiry_days";i:0;s:27:"notify_password_expiry_unit";s:4:"days";s:26:"restrict_login_credentials";s:7:"default";s:34:"restrict_login_credentials_message";s:0:"";s:24:"enable_sessions_policies";b:0;s:23:"remember_session_expiry";a:2:{s:5:"value";i:14;s:4:"unit";s:4:"days";}s:22:"default_session_expiry";a:2:{s:5:"value";i:2;s:4:"unit";s:4:"days";}s:22:"enable_device_policies";b:0;s:35:"enable_device_policies_admin_alerts";s:2:"no";s:25:"enable_security_questions";b:0;s:17:"enabled_questions";a:0:{}s:36:"device_policies_prompt_email_content";s:0:"";s:41:"device_policies_admin_alert_email_content";s:0:"";s:36:"device_policies_prompt_email_subject";s:0:"";s:41:"device_policies_admin_alert_email_subject";s:0:"";s:25:"min_answered_needed_count";i:3;s:39:"password_reset_request_disabled_message";s:0:"";s:24:"password_expired_message";s:0:"";s:36:"inactive_user_account_locked_message";s:0:"";s:51:"inactive_user_account_locked_reset_disabled_message";s:0:"";s:38:"restrict_logins_prompt_failure_message";s:0:"";s:34:"timed_logins_login_blocked_message";s:0:"";s:39:"restrict_login_ip_login_blocked_message";s:0:"";s:35:"failed_logins_login_blocked_message";s:0:"";s:40:"security_prompt_response_failure_message";s:0:"";s:24:"timed_logins_auto_logout";b:0;s:13:"ppm-user-role";s:0:"";s:22:"excluded_special_chars";s:0:"";}'
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Success: Updated 'mls_options' option.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
</pre>
# refreshing the wui then lead to a critical error
<pre>
==> apache2/store.opensourceecology.org/error.log <==
[Sun Feb 16 00:10:53.811257 2025] [proxy_fcgi:error] [pid 1950272:tid 1950284] [client 146.70.129.180:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught TypeError: Cannot access offset of type string on string in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php:446\nStack trace:\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/class-melapress-login-security.php(362): MLS\\MLS_Options->init()\n#1 [internal function]: MLS_Core->init()\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()\n#4 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()\n#5 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(700): do_action()\n#6 /var/www/html/store.opensourceecology.org/wp-config.php(119): require_once('...')\n#7 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(...'
</pre>
# I think the problem is that this json is encoded with the length of the string for some reason. Yeah, the string before I edited it was 161 chars; I updated this to 159 chars (to reflect the shorter email address)
# crap, this is a catch-22; I can't fix the fatal error because of the fatal error
<pre>
Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php:446
Stack trace:
#0 /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/class-melapress-login-security.php(362): MLS\MLS_Options->init()
#1 [internal function]: MLS_Core->init()
#2 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()
#3 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()
#4 /var/www/html/store.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()
#5 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(700): do_action()
#6 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1375): require('...')
#7 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1294): WP_CLI\Runner->load_wordpress()
#8 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Bootstrap/LaunchRunner.php(28): WP_CLI\Runner->start()
#9 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/bootstrap.php(83): WP_CLI\Bootstrap\LaunchRunner->process()
#10 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/wp-cli.php(32): WP_CLI\bootstrap()
#11 phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/php/boot-phar.php(20): include('...')
#12 /home/wp/.wp-cli/wp-cli-2.11.0.phar(4): include('...')
#13 {main}
  thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/melapress-login-security/app/policies/class-mls-options.php on line 446
Error: There has been a critical error on this website.Learn more about troubleshooting WordPress. There has been a critical error on this website.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 #
 
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2 # sudo -u wp -i wp --path="${docrootDir}" option update mls_setting 'a:62:{s:18:"send_summary_email";s:3:"yes";s:8:"exempted";a:1:{s:5:"users";a:0:{}}s:21:"use_custom_from_email";s:13:"default_email";s:10:"from_email";s:0:"";s:17:"from_display_name";s:0:"";s:26:"terminate_session_passwor
 
=Fri Feb 14, 2025=
 
# I checked on the mediawiki cron that I created yesterday and should have run for the first time last night; it failed :(
<pre>
root@hetzner3 /usr/local/bin # cat /var/log/cron/mediawiki_generate_captchas.log
www-data is not in the sudoers file.
This incident has been reported to the administrator.
www-data is not in the sudoers file.
This incident has been reported to the administrator.
root@hetzner3 /usr/local/bin #
</pre>
# so it's important that we don't execute some python script located in our webroot as root, but I tried to de-escalate in two places (for layered security)
## the cron executes the script as www-data
## the script executes the captcha.py script as www-data with sudo
# that doesn't work because the www-data itself can't execute the 'sudo' command
# I could add that user to sudo, but that sounds risky
# it's probably best just to execute the man bash script as root
# I updated the script with a long comment explaining this
<pre>
root@hetzner3 /usr/local/bin # diff mediawiki_generate_captchas.sh.20250214 mediawiki_generate_captchas.sh
5c5
< # Version: 0.1
---
> # Version: 0.2
8c8
< #          *
---
> #          * https://www.mediawiki.org/wiki/Extension:ConfirmEdit#FancyCaptcha
11c11
< # Updated: 2025-02-13
---
> # Updated: 2025-02-14
42a43,61
>
> ###########################
> # (!!) SECURITY NOTE (!!) #
> ###########################
> # It's critical that any scripts executed inside our web server's document root
> # are executed as a non-privileged user! This is because, if there's a
> # vulnerability in MediaWiki (or wordpress, phpList, etc), then an attacker
> # can modify the contents of such scripts (eg captcha.py). And if we execute
> # that script as root, then it would give the attacker the ability a way to
> # execute arbitrary code as root, escalating their privileges.
> #
> # Also note that this script itself is executed as root by cron. It's better to
> # do the de-escalation in this script, because a human might execute it as
> # root. Also note this script wouldn't work if run as 'www-data', because
> # 'www-data' isn't in sudoers.
> #
> # Therefore, make sure any executions of scripts in /var/www/html/*/htdocs are
> # prefixed with `sudo -u www-data -bash -c "<path to script>"`
> ###########################
root@hetzner3 /usr/local/bin #
</pre>
# and I updated the cron to run the script as root.
# we'll have to check-in on this tomorrow to see if it's working https://github.com/OpenSourceEcology/ansible/commit/720dc468f8bc6ddc08f77fdcdaa31b2a61eb4b35
# ...
# so yesterday I finished my quick verification of the wiki
# we're still blocked on osemain and obi
# otherwise, all the vhost snapshots are done
# I still haven't finished setting up the base server, so let's do that
# first, I'm the only human user who can ssh into this server
<pre>
maltfield@hetzner3:/home$ ls
b2user  maltfield  not-apache  wp
maltfield@hetzner3:/home$
</pre>
# hetzner2 had a few additional users
## cataria
## marcin
## tgriffing
<pre>
[maltfield@opensourceecology home]$ ls
b2user  cmota  crupp  Flipo  hart  jthomas  lberezhny  maltfield  marcin  not-apache  stagingsync  tgriffing  wp
[maltfield@opensourceecology home]$
</pre>
# there's others, but I don't think they're active
# to protect the server from getting broken, it's important that Marcin and Catarina do *not* have root access
# to mitigate the bus factor, it's important that we have at least two users with root access: that's me and tgriffin
<pre>
[root@opensourceecology ~]# grep wheel /etc/group
wheel:x:10:maltfield,crupp,tgriffing,root,jthomas
[root@opensourceecology ~]#
</pre>
# it's not a huge risk, though. If marcin ever needed, he can just login on the hetzner WUI and reboot the server to a rescue distro, reset the root password, and get in that way
## and if he lost hetzner WUI access, he can KYC into it
# we *do* want Marcin and Catarina to have accounts, though. For ssh, access to webserver files (eg to scp a very large file that exceeds our https upload size limits), and keepass
<pre>
[root@opensourceecology ~]# grep marcin /etc/group
marcin:x:1001:
apache:x:48:cmota,crupp,maltfield,wp,apache,marcin
sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas
keepass:x:993:maltfield,marcin,cmota,crupp
apache-admins:x:1012:cmota,maltfield,marcin,crupp,tgriffing,wp,apache
[root@opensourceecology ~]#
 
[root@opensourceecology ~]# grep cmota /etc/group
apache:x:48:cmota,crupp,maltfield,wp,apache,marcin
cmota:x:1002:
sshaccess:x:1006:cmota,marcin,tgriffing,maltfield,lberezhny,crupp,jthomas
keepass:x:993:maltfield,marcin,cmota,crupp
apache-admins:x:1012:cmota,maltfield,marcin,crupp,tgriffing,wp,apache
[root@opensourceecology ~]#
</pre>
# I decided against provisioning users with ansible; I'll just do it manually
# The only thing I do need to keep in ansible is to poke a hole in the firewall for these new users
# I confirmed that ssh has been hardened with the 'sshaccess' groups already
<pre>
root@hetzner3 /usr/local/bin # grep -i group /etc/ssh/sshd_config
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
# In case you don't use PAM (`UsePAM no`), you can alternatively restrict users and groups here. For key-based authentication this is not necessary, since all keys must be explicitely enabled.
AllowGroups sshaccess
root@hetzner3 /usr/local/bin #
</pre>
# currently it's just me
<pre>
root@hetzner3 /usr/local/bin # grep sshaccess /etc/group
sshaccess:x:1001:maltfield
root@hetzner3 /usr/local/bin #
</pre>
# the keepass group doesn't yet exist, but the apache-admins group does
<pre>
root@hetzner3 /usr/local/bin # grep keepass /etc/group
root@hetzner3 /usr/local/bin #
root@hetzner3 /usr/local/bin # grep apache /etc/group
users:x:100:maltfield,wp,not-apache
apache-admins:x:1004:www-data,wp
not-apache:x:1006:
root@hetzner3 /usr/local/bin #
</pre>
# I updated https://wiki.opensourceecology.org/wiki/Hetzner3
# here's the pubkeys we have on hetzner2 for these new users
<pre>
[root@opensourceecology ~]# cat /home/marcin/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500
[root@opensourceecology ~]#
 
[root@opensourceecology ~]# cat /home/cmota/.ssh/authorized_keys
#ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDOhf/CGM3t3sklCPAiwaAl7NQPiyDwy5WwV2oxO6rCsDC3jmwn2JCJ6LZ4jXI9NvQQXwg+1kOqsSv2ZFS6RUq1Rs/EW8jR5lFxy9+FgRFpAIlzglAWIBbjpamX0duCMAwpOH2bK1oEDokNM37TtBynFyNytJqLgIGI48IR45KIU4Ib/IN8KzcQRdfPMPFW3EcO1jHJCMUl5SIKt36rgFvhc6JmHF9sl2fCPZ6Jmbi21DOz8fVQDdxQNAc2ubkAS2Sn27LZS7ww6I1VPK43IKBZlZXm3/FQ5HyHrXsGelIDCwdrwM/GTffClDc9EX1ujEHXZIILBDDVLVbwzI+3QHsHvaU9pY+pXBC1dw4LN8InmVazGeV9MoH6SeSeEx4g4KS+2qMiF6fjViPzpLeDW7hIRiTc2+8pjoivNKcgO0xCIBJ/dvTpTExCOPDBVJisnU+3Psq6Wn0Nb7ts7A0tyCtGOG2rxQEYCRmIy8AvQ6WlKztGFvqrfdHaRO9v7WY0hMtyD7kh0tHhHgxSFNcWSIVAIf8JSfhm3f/Orcsg2JDliT4wJzMz7OAYGOQmpqDI+qYa/ULgp9qlf/XBQWeMrQwAU0YeepRJjx3rz9xMxOLj2ddV3JSS8wHg+BGLpPwvqaoFrVIXIgHkW0AbBoXHmtSIZSZ5t5+odWlT1aLb5GmlXw== catarinamota@catarinas-air.pk5001z
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer
[root@opensourceecology ~]#
 
[root@opensourceecology ~]# cat //home/tgriffing/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0
[root@opensourceecology ~]#
</pre>
# here's the vars
<pre>
USERNAME='marcin'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500'
 
USERNAME='cmota'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer'
 
USERNAME='tgriffing'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0'
</pre>
# and here's the commands I'll use to create these users
<pre>
# create user if it doesn't yet exist
adduser ${USERNAME} --disabled-password --gecos ''
 
# add ssh pubkey
mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
</pre>
# here's Marcin
<pre>
root@hetzner3 ~ # USERNAME='marcin'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDDMiIUel3xYyxuiXAj82PzoJDwRczrEpDgUoRI4W9ceL5FqVcY38Go9q3SF2Nx0FEj+IdCUXc08lyy6ZPUbPcKvscFxWeue4aMM62ikzNxmhGBdjqgT3q3wpJgyjTXmt9AJcglcAm9mcQffSUi3RD9KDlCyc/T923eZdaLAkW/BMhjuOZqY90tjGqs/r/kxN0gf4vI24NMFL/41ct7OMKVnNNsjIpQtceX9fCOCumAx53OdtJEcp46TvzevZk2987Zn0VsONznvVCJ0kmm8B0RJxwIfmiLM73f+reo0pv+sSc2rU7SrpzLfPWLFcM7pkJQc3HtLnktl5form3flp+EkI7fr7348r8A7W+QIifjXk66ohJReDni9H/S4JSX2L1lf8LfJKSHtAqrFRWSPp22MKre5hiH0IybED6XZfz59HT0cgMK2iNcPRj/J+hEbBM0f4zZu62PUad7rr1JI4Vv078/ROaD47fykicxYhauI4R71J1YucSj/vekXf17x3xlO+u8ucSeUhdpMuIAa3Yk16bXsrwo4nIdcApC6rwfNiQDK8Ecx6+M6pV6z+dII4OMHvEYWw92wWJZfIyk7emvAoataqp3DfI0DQagPNBo2ieEZYLvNYny+X9hf6faZ6trsGnR4GfN83PEt3ZfmoEoyTVB2POiBdM8a1GNTlEasQ== marcin@Precision-M6500'
root@hetzner3 ~ #
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `marcin' ...
Adding new group `marcin' (1007) ...
Adding new user `marcin' (1007) with group `marcin (1007)' ...
Creating home directory `/home/marcin' ...
Copying files from `/etc/skel' ...
Adding new user `marcin' to supplemental / extra groups `users' ...
Adding user `marcin' to group `users' ...
root@hetzner3 ~ #
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
</pre>
# And Catarina
<pre>
root@hetzner3 ~ # USERNAME='cmota'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDjEu4tbVMJxAX6VuCIrhLYDh/PBlyFHfgGU5ovuPPZLOWYAYI2xYgl5SCweKgB8g2hNTOoyLKgxj0UF7MH22xYQV/EwEwVxX/isSwNvGGikOaKOfb9rBt6nlW2K6ehJlPpHA2nPiqDcuU/1wT3T1FTpYL+uTtQxr4gF8Ijt4aNLwpRdvzgza5vZ1I1R0yLuC2VI1m0NNqT45yGRyWZpM7thT7YGS5Jr0DQa0kLxEZvhGcgdAL6kYfJLW1IhglOZTcoff5TGY6Q8X/gjNrVv6ZUeNF2QiXz4Gm6I6I1YtUDdEEfndu0bHATkMX9aeNG6qAfcYcUcm8pnK+c/RehE0LAcNSDCg9VozsDGg65ywgYw+k0mTl2sW8V95Igfi8oxf/ulGuzxgyriQlhFA4JckDA6Vz2BCjcYabcRhc0ugG34SBRPOUCxVzdb40FSGftVcxb1FeDxsnHxQkl23W9dCcwMMU1m2ssY6F09TTiqhbIp816MkepfWNkB5QDPbmu6EWgT4jp3zWqjMUNcYz9NmRsb6VZ9G357LPOZgMM36XOQXIePcWo5bCQYSusPDSXXjqeSeEVnrfrJJEpBr2AxFCt1R3Dw/fs/rG+YFGNdFadsgiSHxHs2zJglV+Pj8buI6z/EOuHXylZN/2jfOAT17oRU5QXz0HlT0ToeehwFb1+Gw== catarina@Computer'
root@hetzner3 ~ #
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `cmota' ...
Adding new group `cmota' (1008) ...
Adding new user `cmota' (1008) with group `cmota (1008)' ...
Creating home directory `/home/cmota' ...
Copying files from `/etc/skel' ...
Adding new user `cmota' to supplemental / extra groups `users' ...
Adding user `cmota' to group `users' ...
root@hetzner3 ~ #
 
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys
root@hetzner3 ~ #
</pre>
# And Tom
<pre>
root@hetzner3 ~ # USERNAME='tgriffing'
PUBKEY='ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHS8EYmP85HqJwsP4kJ3D2dIBFBgVY8A8YUFubm+bjOFKHr9mV4nnJoY2TweQsKjsT8Kvg8uRPeThls5/7QK/3gDz/objdWp/2W5kvwhDxlZwEWyf+5a6F23OYLc5oeixwR/TyU13OokXSeZeTxPX3m/It1VBKEz0QUCjwTHEkPrjjhbeVlQ7vFeCAwGlrA8puDF1l8SUIO23hpiU9E+IM/+wTasEP8YblSk9445mLow4BexlvmfrRsXXdg/vrdObchzeo9rhZxMTWPE2nbyVUp86iaNp/PVbeTNKWx0hZF0zr7TjIbsmYmGXlPMZcKaStpcfMlVJ+hJ9NxwTHrqhC0lsfNz9pvPdLkZM3O5Ychevu4xlFb3XddMiO1QHodqf56vZhicMA+9cLfZpFTcwtVGseD+JpURPuG2DBtEDkozGk1szx2SoX5B6ccprZYvfj4HiTW6+qv7XN2uMbRMHw0VMyAPjwSKYC/YzTZ885VAFj8Oo5t5Q6F9VW1oRUF3gWrcLBcvL2XUDQCCUpF3bDlHxQQqJZ3EifW6rDZVHlyLkzq6/FKTUPHuHdX4K5DPdJxEcfdm5zyjiGEtGQ2uzHx3WAJMaykjFsJElsE7avhHagKzneS/b4shReEEseNErhW5d0AyAoPkEoVkCyauS2vOvNAZ29OXc2Yf6DEIdU9Q== tom@thomasgfingsmbp
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDGwcx6l+W/6CTU+bm9gOZM53uEPvSsTk31PqQE/svb9qNzrM+Ny8xBfofbhlbTXYFAldlGC3S3DgBO6yOHCQgnHtf7zqBD+sNsVqMGKSpDhkAn09CmJy9P90p4ovZqHfGpvSPXfyPyBF/ebLgJeS8roxcU9OyTO+iRMXv8rOgK7zLLbdMy+/tXr6muGyaIzHJljYpaebd4kjM4INaycGYY7gEVBmBzC6wHj+PDLcPSeYXTVG6R7RrfGQuvtM61hNY90+pw2di0GR57wqF/0tLvfJ5+QyWJoh4ns4gBhRf8/2QVfcy+DD9ofQ8ILRVVf77IxZRTY8j+zgUBD4YjvBmtx/UB2nJJRwyDjPEB55grC+LjQ8ehwgc2LpE2nVvEWCUZjdw5kFZjD4fHVWRhbcVmusSIAyw47xPpywRtry0+rdbL90i2JTitFMRzqTZLETAOgEfRp50WiPulxh2Gj1bVCHFvx1p/hdxbEWZx2k2s62SOYvZj+yBazK9gBFLwPZWBx5bzeu091Yxvingt+EZ4qGF807trP5e46oJCLmAU1DXD4enWmTfGQxvsallREYj6xbdWjMq+Az35nWmlg7omlvZPVMDZ7S+++dTO9ypxJeeVEfBav/gkghqcY5lGIU51eCiBEric476NQRG7aJp9rakgF2wKj8qWIoOzRysWYw== tom@imac0'
root@hetzner3 ~ #
 
root@hetzner3 ~ # adduser ${USERNAME} --disabled-password --gecos ''
Adding user `tgriffing' ...
Adding new group `tgriffing' (1009) ...
Adding new user `tgriffing' (1009) with group `tgriffing (1009)' ...
Creating home directory `/home/tgriffing' ...
Copying files from `/etc/skel' ...
Adding new user `tgriffing' to supplemental / extra groups `users' ...
Adding user `tgriffing' to group `users' ...
root@hetzner3 ~ #
 
root@hetzner3 ~ # mkdir /home/${USERNAME}/.ssh
echo $PUBKEY > /home/${USERNAME}/.ssh/authorized_keys
chown -R ${USERNAME}:${USERNAME} /home/${USERNAME}/.ssh
chmod 0700 /home/${USERNAME}/.ssh
chmod 0600 /home/${USERNAME}/.ssh/authorized_keys     
root@hetzner3 ~ #
</pre>
# then I added the users to the groups
<pre>
gpasswd -a marcin sshaccess
gpasswd -a cmota sshaccess
gpasswd -a tgriffing sshaccess
 
gpasswd -a marcin www-data
gpasswd -a cmota www-data
gpasswd -a tgriffing www-data
 
gpasswd -a tgriffing apache-admins
</pre>
# then I created the keepass group and added everyone to it
<pre>
groupadd keepass
 
gpasswd -a maltfield keepass
gpasswd -a marcin keepass
gpasswd -a cmota keepass
gpasswd -a tgriffing keepass
</pre>
# finally, I added Tom to the sudoers group
<pre>
gpasswd -a tgriffing sudo
</pre>
# I updated the ansible playbook to permit these human user accounts internet access https://github.com/OpenSourceEcology/ansible/commit/f294f3384f5d4bc47ae55a1631c264950e0e1355
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ git diff
diff --git a/hetzner3/provision.yml b/hetzner3/provision.yml
index 0f8e428..9ae54a1 100644
--- a/hetzner3/provision.yml
+++ b/hetzner3/provision.yml
@@ -85,8 +85,11 @@
- -A OUTPUT -m owner --uid-owner root -j ACCEPT
  820 allow apt:
- -A OUTPUT -m owner --uid-owner _apt -j ACCEPT
-      830 allow maltfield:
+      830 allow human users internet access:
- -A OUTPUT -m owner --uid-owner maltfield -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner marcin -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner cmota -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner tgriffing -j ACCEPT
  840 allow postfix:
- -A OUTPUT -m owner --uid-owner postfix -j ACCEPT
  850 allow stubby:
@@ -128,8 +131,11 @@
- -A OUTPUT -m owner --uid-owner root -j ACCEPT
  820 allow apt:
- -A OUTPUT -m owner --uid-owner _apt -j ACCEPT
-      830 allow maltfield:
+      830 allow human users internet access:
- -A OUTPUT -m owner --uid-owner maltfield -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner marcin -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner cmota -j ACCEPT
+        - -A OUTPUT -m owner --uid-owner tgriffing -j ACCEPT
  840 allow postfix:
- -A OUTPUT -m owner --uid-owner postfix -j ACCEPT
  850 allow stubby:
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# I sent an email to Marcin
<pre>
Hi Marcin,
 
Can you please confirm that ssh access to our new OSE server (hetzner3) is working?
 
We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details:
 
User: marcin
Server: 144.76.164.201
Port: 32415
 
For example, you should be able to login with the following command:
 
  ssh -p 32415 marcin@144.76.164.201
 
The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2.
 
After you login, please set a password for your account with the following command:
 
  passwd
 
Please confirm that you were able to login to hetzner3 with ssh and update your password.
</pre>
# I sent an email to Catarina
<pre>
Hi Catarina,
 
Can you please confirm that ssh access to our new OSE server (hetzner3) is working?
 
We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details:
 
User: cmota
Server: 144.76.164.201
Port: 32415
 
For example, you should be able to login with the following command:
 
  ssh -p 32415 cmota@144.76.164.201
 
The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2.
 
After you login, please set a password for your account with the following command:
 
  passwd
 
Please confirm that you were able to login to hetzner3 with ssh and update your password.
</pre>
# I sent an email to Tom
<pre>
Hi Tom,
 
Can you please confirm that ssh access to our new OSE server (hetzner3) is working?
 
We're in the process of migrating away from OSE's current server (hetzner2) to its replacement server (hetzner3). I just created a user account for you on hetzner3 and granted it ssh access. To make sure you don't loose access to our server, I need you confirm that you're able to login. Here's the login details:
 
User: tgriffing
Server: 144.76.164.201
Port: 32415
 
For example, you should be able to login with the following command:
 
  ssh -p 32415 tgriffing@144.76.164.201
 
The account does not have a password. You should be able to authenticate with the same ssh key that you were using to login on hetzner2.
 
After you login, please set a password for your account with the following command:
 
  passwd
 
After updating your password, you should be able to become root with sudo.
 
  sudo su -
 
Please confirm that you were able to login to hetzner3 with ssh,
update your password, and become root.
</pre>
# that's all the human accounts; I'll just have to wait for their confirmations
# ...
# another thing I wanted to do was to diff the postfix configs on hetzner2 & hetzner3. I did some basic postfix setup just to get wazuh functional. And it appears to work for phpList too, but I never checked the postfix config to make sure it's sane
# hetzner2
<pre>
[root@opensourceecology ~]# postconf > /home/maltfield/postconf_hetzner2_20250214
[root@opensourceecology ~]# chown maltfield /home/maltfield/postconf_hetzner2_20250214
[root@opensourceecology ~]#
</pre>
# hetzner3
<pre>
root@hetzner3 ~ # postconf > /home/maltfield/postconf_hetzner3_20250214
root@hetzner3 ~ # chown maltfield /home/maltfield/postconf_hetzner3_20250214
root@hetzner3 ~ #
</pre>
# and I copied them locally for a visual diff in meld
<pre>
user@ose:~$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/user/.ssh/id_rsa:
Identity added: /home/user/.ssh/id_rsa (guttersnipe@guttersnipe)
user@ose:~$ cd tmp/hetzner3/
user@ose:~/tmp/hetzner3$ mkdir postfix
user@ose:~/tmp/hetzner3$ cd postfix/
user@ose:~/tmp/hetzner3/postfix$
user@ose:~/tmp/hetzner3/postfix$ rsync -av --progress opensourceecology.org:postconf_hetzner2_20250214 .
receiving incremental file list
postconf_hetzner2_20250214
32,676 100%  31.16MB/s    0:00:00 (xfr#1, to-chk=0/1)
 
sent 43 bytes  received 32,799 bytes  4,378.93 bytes/sec
total size is 32,676  speedup is 0.99
user@ose:~/tmp/hetzner3/postfix$
user@ose:~/tmp/hetzner3/postfix$ rsync -av --progress hetzner3:postconf_hetzner3_20250214 .
receiving incremental file list
postconf_hetzner3_20250214
40,776 100%  38.89MB/s    0:00:00 (xfr#1, to-chk=0/1)
 
sent 43 bytes  received 40,908 bytes  7,445.64 bytes/sec
total size is 40,776  speedup is 1.00
user@ose:~/tmp/hetzner3/postfix$
</pre>
# here's the diff
<pre>
user@ose:~/tmp/hetzner3/postfix$ diff postconf_hetzner2_20250214 postconf_hetzner3_20250214
11c11,12
< address_verify_poll_count = ${stress?1}${stress:3}
---
> address_verify_pending_request_limit = 5000
> address_verify_poll_count = ${stress?{1}:{3}}
37c38
< append_dot_mydomain = yes
---
> append_dot_mydomain = ${{$compatibility_level} <level {1} ? {yes} : {no}}
56a58
> cleanup_replace_stray_cr_lf = yes
61a64
> compatibility_level = 0
62a66
> confirm_delay_cleared = no
69c73
< daemon_directory = /usr/libexec/postfix
---
> daemon_directory = /usr/lib/postfix/sbin/
79a84
> default_delivery_status_filter =
95a101
> default_transport_rate_delay = 0s
111c117,118
< disable_vrfy_command = no
---
> disable_vrfy_command = yes
> dns_ncache_ttl_fix_enable = no
113a121
> dnssec_probe = ns:.
117a126
> empty_address_local_login_sender_maps_lookup_key = <>
119a129
> enable_idna2003_compatibility = no
121a132
> enable_threaded_bounces = no
138a150
> error_transport_rate_delay = $default_transport_rate_delay
157a170
> header_from_format = standard
164c177
< import_environment = MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ XAUTHORITY DISPLAY LANG=C
---
> import_environment = MAIL_CONFIG MAIL_DEBUG MAIL_LOGTAG TZ XAUTHORITY DISPLAY LANG=C POSTLOG_SERVICE POSTLOG_HOSTNAME
166c179
< inet_interfaces = localhost
---
> inet_interfaces = 127.0.0.1, [::1], localhost
167a181
> info_log_address_format = external
173a188
> known_tcp_ports = lmtp=24, smtp=25, smtps=submissions=465, submission=587
174a190
> lmdb_map_size = 16777216
175a192
> lmtp_address_verify_target = rcpt
176a194
> lmtp_balance_inet_protocols = yes
178a197
> lmtp_bind_address_enforce = no
184a204
> lmtp_connection_reuse_count_limit = 0
192a213
> lmtp_delivery_status_filter = $default_delivery_status_filter
200a222
> lmtp_dns_reply_filter =
201a224
> lmtp_dns_support_level =
203a227
> lmtp_fallback_relay =
212a237
> lmtp_min_data_rate = 500
217c242
< lmtp_per_record_deadline = no
---
> lmtp_per_request_deadline = ${lmtp_per_record_deadline?{$lmtp_per_record_deadline}:{no}}
253c278,280
< lmtp_tls_ciphers = export
---
> lmtp_tls_chain_files =
> lmtp_tls_ciphers = medium
> lmtp_tls_connection_reuse = no
261c288,289
< lmtp_tls_fingerprint_digest = md5
---
> lmtp_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> lmtp_tls_force_insecure_host_tlsa_lookup = no
266c294
< lmtp_tls_mandatory_protocols = !SSLv2
---
> lmtp_tls_mandatory_protocols = !SSLv2, !SSLv3
270c298
< lmtp_tls_protocols = !SSLv2
---
> lmtp_tls_protocols = !SSLv2, !SSLv3
273a302
> lmtp_tls_servername =
275a305
> lmtp_tls_trust_anchor_file =
276a307,308
> lmtp_tls_wrappermode = no
> lmtp_transport_rate_delay = $default_transport_rate_delay
282a315
> local_delivery_status_filter = $default_delivery_status_filter
291a325
> local_login_sender_maps = static:*
297a332
> local_transport_rate_delay = $default_transport_rate_delay
301c336
< mail_release_date = 20130622
---
> mail_release_date = 20240304
303,304c338,339
< mail_version = 2.10.1
< mailbox_command = /usr/bin/procmail
---
> mail_version = 3.7.11
> mailbox_command =
309a345,348
> maillog_file =
> maillog_file_compressor = gzip
> maillog_file_prefixes = /var, /dev/stdout
> maillog_file_rotate_suffix = %Y%m%d-%H%M%S
321a361
> message_drop_headers = bcc, content-length, resent-bcc, return-path
324a365
> meta_directory = /etc/postfix
326c367
< milter_connect_macros = j {daemon_name} v
---
> milter_connect_macros = j {daemon_name} {daemon_addr} v _
335a377
> milter_macro_defaults =
353,354c395,396
< myhostname = mailer.opensourceecology.org
< mynetworks = 127.0.0.1/32
---
> myhostname = hetzner3.opensourceecology.org
> mynetworks = 127.0.0.1/32 [::1]/128
361a404
> openssl_path = openssl
365a409
> pipe_delivery_status_filter = $default_delivery_status_filter
366a411,412
> postlog_service_name = postlog
> postlogd_watchdog_timeout = 10s
370a417
> postscreen_allowlist_interfaces = ${postscreen_whitelist_interfaces?{$postscreen_whitelist_interfaces}:{static:all}}
374d420
< postscreen_blacklist_action = ignore
381c427,428
< postscreen_command_time_limit = ${stress?10}${stress:300}s
---
> postscreen_command_time_limit = ${stress?{10}:{300}}s
> postscreen_denylist_action = ${postscreen_blacklist_action?{$postscreen_blacklist_action}:{ignore}}
385a433,435
> postscreen_dnsbl_allowlist_threshold = ${postscreen_dnsbl_whitelist_threshold?{$postscreen_dnsbl_whitelist_threshold}:{0}}
> postscreen_dnsbl_max_ttl = ${postscreen_dnsbl_ttl?{$postscreen_dnsbl_ttl}:{1}}h
> postscreen_dnsbl_min_ttl = 60s
389c439
< postscreen_dnsbl_ttl = 1h
---
> postscreen_dnsbl_timeout = 10s
396c446
< postscreen_greet_wait = ${stress?2}${stress:6}s
---
> postscreen_greet_wait = ${stress?{2}:{6}}s
406a457
> postscreen_reject_footer_maps = $smtpd_reject_footer_maps
412,414c463,464
< postscreen_whitelist_interfaces = static:all
< prepend_delivered_header = command, file, forward
< process_id = 5305
---
> prepend_delivered_header =
> process_id = 1460591
419c469
< proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps $alias_maps
---
> proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps $sender_bcc_maps $recipient_bcc_maps $smtp_generic_maps $lmtp_generic_maps $alias_maps $smtpd_client_restrictions $smtpd_helo_restrictions $smtpd_sender_restrictions $smtpd_relay_restrictions $smtpd_recipient_restrictions $address_verify_sender_dependent_default_transport_maps $address_verify_sender_dependent_relayhost_maps $address_verify_transport_maps $fallback_transport_maps $lmtp_discard_lhlo_keyword_address_maps $lmtp_pix_workaround_maps $lmtp_sasl_password_maps $lmtp_tls_policy_maps $mailbox_command_maps $mailbox_transport_maps $postscreen_discard_ehlo_keyword_address_maps $rbl_reply_maps $sender_dependent_default_transport_maps $sender_dependent_relayhost_maps $smtp_discard_ehlo_keyword_address_maps $smtp_pix_workaround_maps $smtp_sasl_password_maps $smtp_tls_policy_maps $smtpd_discard_ehlo_keyword_address_maps $smtpd_milter_maps $virtual_gid_maps $virtual_uid_maps $local_login_sender_maps $postscreen_reject_footer_maps $smtpd_reject_footer_maps $tls_server_sni_maps $tlsproxy_client_policy_maps $default_delivery_status_filter $lmtp_delivery_status_filter $lmtp_dns_reply_filter $lmtp_reply_filter $local_delivery_status_filter $pipe_delivery_status_filter $postscreen_command_filter $smtp_delivery_status_filter $smtp_dns_reply_filter $smtp_reply_filter $smtpd_command_filter $smtpd_dns_reply_filter $virtual_delivery_status_filter $body_checks $header_checks $lmtp_body_checks $lmtp_header_checks $lmtp_mime_header_checks $lmtp_nested_header_checks $milter_header_checks $mime_header_checks $nested_header_checks $smtp_body_checks $smtp_header_checks $smtp_mime_header_checks $smtp_nested_header_checks
458c508
< relay_domains = $mydestination
---
> relay_domains = ${{$compatibility_level} <level {2} ? {$mydestination} : {}}
467a518
> relay_transport_rate_delay = $default_transport_rate_delay
475a527
> respectful_logging = ${{$compatibility_level} <level {3.6} ? {no} : {yes}}
490a543
> retry_transport_rate_delay = $default_transport_rate_delay
500a554
> service_name =
502a557
> shlib_directory = /usr/lib/postfix
505a561
> smtp_address_verify_target = rcpt
506a563
> smtp_balance_inet_protocols = yes
508a566
> smtp_bind_address_enforce = no
514a573
> smtp_connection_reuse_count_limit = 0
522a582
> smtp_delivery_status_filter = $default_delivery_status_filter
530a591
> smtp_dns_reply_filter =
531a593
> smtp_dns_support_level =
543a606
> smtp_min_data_rate = 500
549c612
< smtp_per_record_deadline = no
---
> smtp_per_request_deadline = ${smtp_per_record_deadline?{$smtp_per_record_deadline}:{no}}
579a643
> smtp_tcp_port = smtp
584c648,651
< smtp_tls_ciphers = export
---
> smtp_tls_chain_files =
> smtp_tls_ciphers = medium
> smtp_tls_connection_reuse = no
> smtp_tls_dane_insecure_mx_policy = ${{$smtp_tls_security_level} == {dane} ? {dane} : {may}}
592c659,660
< smtp_tls_fingerprint_digest = md5
---
> smtp_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> smtp_tls_force_insecure_host_tlsa_lookup = no
594c662
< smtp_tls_loglevel = 0
---
> smtp_tls_loglevel = 1
597,598c665,666
< smtp_tls_mandatory_protocols = !SSLv2
< smtp_tls_note_starttls_offer = no
---
> smtp_tls_mandatory_protocols = !SSLv2, !SSLv3
> smtp_tls_note_starttls_offer = yes
601c669
< smtp_tls_protocols = !SSLv2
---
> smtp_tls_protocols = !SSLv2, !SSLv3
604c672,673
< smtp_tls_security_level =
---
> smtp_tls_security_level = may
> smtp_tls_servername =
606a676
> smtp_tls_trust_anchor_file =
607a678,679
> smtp_tls_wrappermode = no
> smtp_transport_rate_delay = $default_transport_rate_delay
614,617c686,690
< smtpd_client_connection_count_limit = 50
< smtpd_client_connection_rate_limit = 0
< smtpd_client_event_limit_exceptions = ${smtpd_client_connection_limit_exceptions:$mynetworks}
< smtpd_client_message_rate_limit = 0
---
> smtpd_client_auth_rate_limit = 0
> smtpd_client_connection_count_limit = 4
> smtpd_client_connection_rate_limit = 120
> smtpd_client_event_limit_exceptions = $mynetworks
> smtpd_client_message_rate_limit = 120
620,621c693,694
< smtpd_client_recipient_rate_limit = 0
< smtpd_client_restrictions =
---
> smtpd_client_recipient_rate_limit = 120
> smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated, check_helo_access hash:/etc/postfix/whitelisted_domains, reject_unknown_client_hostname
627a701
> smtpd_dns_reply_filter =
630c704
< smtpd_error_sleep_time = 1s
---
> smtpd_error_sleep_time = 4s
633,636c707,714
< smtpd_forbidden_commands = CONNECT GET POST
< smtpd_hard_error_limit = ${stress?1}${stress:20}
< smtpd_helo_required = no
< smtpd_helo_restrictions =
---
> smtpd_forbid_bare_newline = no
> smtpd_forbid_bare_newline_exclusions = $mynetworks
> smtpd_forbid_bare_newline_reject_code = 550
> smtpd_forbid_unauth_pipelining = no
> smtpd_forbidden_commands = CONNECT GET POST regexp:{{/^[^A-Z]/ Bogus}}
> smtpd_hard_error_limit = 10
> smtpd_helo_required = yes
> smtpd_helo_restrictions = permit_mynetworks, check_helo_access hash:/etc/postfix/whitelisted_domains, permit_sasl_authenticated
638c716
< smtpd_junk_command_limit = ${stress?1}${stress:100}
---
> smtpd_junk_command_limit = ${stress?{1}:{100}}
639a718
> smtpd_milter_maps =
640a720
> smtpd_min_data_rate = 500
644c724,725
< smtpd_per_record_deadline = ${stress?yes}${stress:no}
---
> smtpd_per_request_deadline = ${smtpd_per_record_deadline?{$smtpd_per_record_deadline}:{${stress?{yes}:{no}}}}
> smtpd_policy_service_default_action = 451 4.3.5 Server configuration problem
646a728,730
> smtpd_policy_service_policy_context =
> smtpd_policy_service_request_limit = 0
> smtpd_policy_service_retry_delay = 1s
647a732
> smtpd_policy_service_try_limit = 2
654c739
< smtpd_recipient_restrictions =
---
> smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, check_recipient_access hash:/etc/postfix/whitelisted_domains, reject_unauth_destination
655a741
> smtpd_reject_footer_maps =
658c744,745
< smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination
---
> smtpd_relay_before_recipient_restrictions = ${{$compatibility_level} <level {3.6} ? {no} : {yes}}
> smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
663a751
> smtpd_sasl_mechanism_filter = !external, static:rest
664a753
> smtpd_sasl_response_limit = 12288
665a755
> smtpd_sasl_service = smtp
669c759
< smtpd_sender_restrictions =
---
> smtpd_sender_restrictions = check_sender_access hash:/etc/postfix/whitelisted_domains, reject_unknown_sender_domain
671,673c761,763
< smtpd_soft_error_limit = 10
< smtpd_starttls_timeout = ${stress?10}${stress:300}s
< smtpd_timeout = ${stress?10}${stress:300}s
---
> smtpd_soft_error_limit = 4
> smtpd_starttls_timeout = ${stress?{10}:{300}}s
> smtpd_timeout = ${stress?{10}:{300}}s
678c768
< smtpd_tls_auth_only = no
---
> smtpd_tls_auth_only = yes
680,681c770,772
< smtpd_tls_cert_file =
< smtpd_tls_ciphers = export
---
> smtpd_tls_cert_file = /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
> smtpd_tls_chain_files =
> smtpd_tls_ciphers = medium
683,684c774,775
< smtpd_tls_dh1024_param_file =
< smtpd_tls_dh512_param_file =
---
> smtpd_tls_dh1024_param_file = /etc/ssl/certs/dhparam.pem
> smtpd_tls_dh512_param_file = /etc/ssl/certs/dhparam.pem
688c779
< smtpd_tls_eecdh_grade = strong
---
> smtpd_tls_eecdh_grade = ultra
690,693c781,784
< smtpd_tls_fingerprint_digest = md5
< smtpd_tls_key_file = $smtpd_tls_cert_file
< smtpd_tls_loglevel = 0
< smtpd_tls_mandatory_ciphers = medium
---
> smtpd_tls_fingerprint_digest = ${{$compatibility_level} <level {3.6} ? {md5} : {sha256}}
> smtpd_tls_key_file = /etc/letsencrypt/live/opensourceecology.org/privkey.pem
> smtpd_tls_loglevel = 1
> smtpd_tls_mandatory_ciphers = high
695,696c786,787
< smtpd_tls_mandatory_protocols = !SSLv2
< smtpd_tls_protocols =
---
> smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
> smtpd_tls_protocols = !SSLv2, !SSLv3
699c790
< smtpd_tls_security_level =
---
> smtpd_tls_security_level = may
705a797,798
> smtputf8_autodetect_classes = sendmail, verify
> smtputf8_enable = ${{$compatibility_level} <level {1} ? {no} : {yes}}
714a808
> strict_smtputf8 = no
718c812
< syslog_name = ${multi_instance_name:postfix}${multi_instance_name?$multi_instance_name}
---
> syslog_name = ${multi_instance_name?{$multi_instance_name}:{postfix}}
720a815,816
> tls_config_file = default
> tls_config_name =
721a818
> tls_dane_digests = sha512 sha256
722a820
> tls_eecdh_auto_curves = X25519 X448 prime256v1 secp521r1 secp384r1
725,726c823,825
< tls_export_cipherlist = aNULL:-aNULL:ALL:+RC4:@STRENGTH
< tls_high_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH
---
> tls_export_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:LOW:EXPORT:+RC4:@STRENGTH
> tls_fast_shutdown_enable = yes
> tls_high_cipherlist = EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA
728,729c827,828
< tls_low_cipherlist = aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH
< tls_medium_cipherlist = aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH
---
> tls_low_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:LOW:+RC4:@STRENGTH
> tls_medium_cipherlist = aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH
736a836,858
> tls_server_sni_maps =
> tls_session_ticket_cipher = aes-256-cbc
> tls_ssl_options = NO_COMPRESSION
> tls_wildcard_matches_multiple_labels = yes
> tlsmgr_service_name = tlsmgr
> tlsproxy_client_CAfile = $smtp_tls_CAfile
> tlsproxy_client_CApath = $smtp_tls_CApath
> tlsproxy_client_cert_file = $smtp_tls_cert_file
> tlsproxy_client_chain_files = $smtp_tls_chain_files
> tlsproxy_client_dcert_file = $smtp_tls_dcert_file
> tlsproxy_client_dkey_file = $smtp_tls_dkey_file
> tlsproxy_client_eccert_file = $smtp_tls_eccert_file
> tlsproxy_client_eckey_file = $smtp_tls_eckey_file
> tlsproxy_client_enforce_tls = $smtp_enforce_tls
> tlsproxy_client_fingerprint_digest = $smtp_tls_fingerprint_digest
> tlsproxy_client_key_file = $smtp_tls_key_file
> tlsproxy_client_loglevel = $smtp_tls_loglevel
> tlsproxy_client_loglevel_parameter = smtp_tls_loglevel
> tlsproxy_client_per_site = $smtp_tls_per_site
> tlsproxy_client_policy_maps = ${tlsproxy_client_policy:$smtp_tls_policy_maps}
> tlsproxy_client_scert_verifydepth = $smtp_tls_scert_verifydepth
> tlsproxy_client_security_level = ${tlsproxy_client_level:$smtp_tls_security_level}
> tlsproxy_client_use_tls = $smtp_use_tls
744a867
> tlsproxy_tls_chain_files = $smtpd_tls_chain_files
763d885
< tlsproxy_tls_session_cache_timeout = $smtpd_tls_session_cache_timeout
789c911,912
< virtual_alias_domains = $virtual_alias_maps
---
> virtual_alias_address_length_limit = 1000
> virtual_alias_domains = openbuildinginstitute.org osedev.org
791c914
< virtual_alias_maps = $virtual_maps
---
> virtual_alias_maps = hash:/etc/postfix/virtual
795a919
> virtual_delivery_status_filter = $default_delivery_status_filter
815a940
> virtual_transport_rate_delay = $default_transport_rate_delay
user@ose:~/tmp/hetzner3/postfix$
</pre>
# here's what I noticed
## disable_vrfy_command changed from 'no' to 'yes'
### docs say enabling this "stops some techniques used to harvest email addresses.", so that's probably good https://www.postfix.org/postconf.5.html
## inet_interfaces now includes 127.0.0.1 and [::1]
### Not sure why that was missing before, but I don't see any issues with it
### note that it's not bound to any external IPs. This seems problematic, but it's as-desired. OSE doesn't use postfix for receiving mail. Our MX records point to our free Google Workplace account. We just use postfix for *sending* mail, which greatly reduces a lot of the risk and complexity.
## lmtp_tls_ciphers changed from "export" to "medium"
### according to the docs " In Postfix ≥ 3.8 this [export] cipher grade is always identical to "medium"" https://www.postfix.org/postconf.5.html#smtp_tls_policy_maps
### I'm guessing this is a shadow of when the US had laws limiting the export of some crypto. Jesus, CentOS 7 is old. That shit changed in, what, the 90s?
### anyway, there's no change here
### the dirty part of TLS encryption in emails is that it's only opportunistic and always vulnerable to downgrade attacks (eg sslstrip). To require encryption between our server and other email servers would be going against the RFC standard, and it would impair our (eg phpList) email delivery :( so we'll leave this at the default
## lmtp_tls_fingerprint_digest changed from md5 to sha256, if possible
### that's good
## lmtp_tls_mandatory_protocols changed from disallowing SSLv2 to disallowing both SSLv2 and SSLv3
### that's good. we should only be using TLS
## same as ^ with lmtp_tls_protocols
## mailbox_command changed from '/usr/bin/procmail' to ''
### idk what that means, but I don't think it matters for us
## myhostname changed from 'mailer.opensourceecology.org' to 'hetzner3.opensourceecology.org'
### yeah, I named the old server 'mail' hoping it would decrease risk of email bounces, but that had some weird consequences. 'hetzner3' seems to work, afaict
## 'mynetworks' changed from '127.0.0.1/32' to '127.0.0.1/32 [::1]/128'
### I see no harm in adding local ipv6
## postscreen_blacklist_action disappeared
### docs say this was renamed to postscreen_denylist_action https://www.postfix.org/postconf.5.html#postscreen_blacklist_action
### looks like it's still 'ignore' in both cases
## postscreen_dnsbl_ttl changed from '1h' to '60s'
### that's a big change
### docs say default is 'https://www.postfix.org/postconf.5.html#postscreen_dnsbl_ttl
### docs say this is basically a cache: "The amount of time that postscreen(8) remembers that a client IP address passed a DNS-based reputation test, before it is required to pass that test again. "
### I see zero impact on OSE's postfix server, which isn't even exposed to the public internet
### docs say
## same for postscreen_dnsbl_ttl
## smtp_tls_fingerprint_digest similar to above, possibly use sha256
## smtp_tls_loglevel changed from 0 to 1
### the default is 0, but I guess debian changed this? Or maybe I did. Anyway, docs say it's ok to use anything <2
## smtp_tls_note_starttls_offer changed from 'no' to 'yes'
### also just affects logging. default is 'no'. So now we just log the hostname. Should be fine.
## smtp_tls_protocols changed, same as above, to disable SSLv3 too
## smtpd_client_message_rate_limit changed from 0 to 120
### docs say the default  is 0. We probably would be fine to have this at 0 for OSE's purposes. But I don't see an issue at 120 either.
## smtpd_client_restrictions changed from '' to 'permit_mynetworks, permit_sasl_authenticated, check_helo_access hash:/etc/postfix/whitelisted_domains, reject_unknown_client_hostname'
### that should be fine
## smtpd_error_sleep_time changed from 1s to 4s
### I think that's fine
## smtpd_helo_required changed from 'no' to 'yes'
### shouldn't matter, and probably a good change
## smtpd_relay_restrictions changed from 'permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination' to 'permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination'
### so 'defer' changed to 'reject'. I guess that's fine.
## smtpd_soft_error_limit changed from 10 to 4
### docs suggest this is a tarpit config. After this number of errors, it slows down processing of requests from that client
### again, shouldn't affect us
## smtpd_tls_auth_only changed from 'no' to 'yes'
### this apparently only affects sasl auth, which is forced to use TLS now
### should be fine; we're not doing any SASL auth
## smtpd_tls_auth_only changed from '' to '/etc/letsencrypt/live/opensourceecology.org/fullchain.pem'
### wow I can't believe this was empty before. So I guess we were one of those servers that was sending all smtp messages without encryption? That or snake oil.
### this is a good change
## smtpd_tls_dh1024_param_file and smtpd_tls_dh512_param_file got set to our ansible-made dhparam file; that's good
## smtpd_tls_eecdh_grade changed from 'strong' to 'ultra'
### docs say that's an increase from 128 bits to 192 bits. It says it's twice as slow. should be fine.
## smtpd_tls_mandatory_ciphers changed from 'medium' to 'high'
### that's probably for the best
## smtpd_tls_security_level changed from '' to 'may'
### yeah, apparently empty means "don't use TLS". So this is a very good change.
### again, 'encrypt' enforces TLS, but that's a violation of RFC 2487 https://tools.ietf.org/html/rfc2487
## tls_export_cipherlist changed from 'aNULL:-aNULL:ALL:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:LOW:EXPORT:+RC4:@STRENGTH'
### looks like "ALL" was removed ,so that's probably good
## tls_high_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:!LOW:!MEDIUM:+RC4:@STRENGTH' to 'EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA'
### ALL was removed. I don't like that it lists SSLv3
### usually my go-to is https://bettercrypto.org/ but it looks like they took the site down saying it's not longer updated :(
### the docs say that you are "strongly encouraged to not change" this setting
### well, it appears that we are changing it
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ grep -ir cipherlist *
roles/maltfield.postfix/templates/main.cf.j2:tls_high_cipherlist=EDH+CAMELLIA:EDH+aRSA:EECDH+aRSA+AESGCM:EECDH+aRSA+SHA256:EECDH:+CAMELLIA128:+AES128:+SSLv3:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!DSS:!RC4:!SEED:!IDEA:!ECDSA:kEDH:CAMELLIA128-SHA:AES128-SHA
roles/maltfield.postfix/templates/master.cf.j2:  -o tls_preempt_cipherlist=yes
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
### here we can view the default settings. that's not very helpful
<pre>
root@hetzner3 ~ # postconf -d | grep -i tls_high_cipherlist
tls_high_cipherlist = aNULL:-aNULL:HIGH:@STRENGTH
root@hetzner3 ~ #
</pre>
### this SE answer cites bettercrypto.org, and it includes SSLv3 too; I guess we'll leave it as-is https://serverfault.com/a/670359
## tls_low_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:+RC4:@STRENGTH' to  'aNULL:-aNULL:HIGH:MEDIUM:LOW:+RC4:@STRENGTH'
## tls_medium_cipherlist changed from 'aNULL:-aNULL:ALL:!EXPORT:!LOW:+RC4:@STRENGTH' to 'aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH'
## tlsproxy_tls_session_cache_timeout disappeared
### docs say it's obsolete and replaced by smtpd_tls_session_cache_timeout
## virtual_alias_domains changed from '$virtual_alias_maps' to 'openbuildinginstitute.org osedev.org'
## virtual_alias_maps changed from '$virtual_maps' to 'hash:/etc/postfix/virtual'
### note that '$virtual_maps' isn't set on hetzner2, so I don't think any alias domains were defined
### it might actually be better to use an external maps file (rather than just hardcoding it), but I don't want to break it; let's leave it as-is
# there were lots of other changes, but those were the ones that stood-out to me
# in general, I'd say the changes were good: err-ing on the side of "more" secure
# ok, postfix looks fine to me
# SPF records and rDNS were already set
# probably after the migration we'd want to remove the old server, but there's nothing to do now until we migrate all the webservers
# oh, actually, one thing I never setup was SPF records for non-ose domains (eg obi, osedev, oswh)
# I opened the shared ose keepass, and I saw 2 domains in ghandi:
## oswarehouse.org
<pre>
user@disp5497:~$ dig +short -t txt oswarehouse.org
"v=spf1 include:_mailcust.gandi.net ?all"
user@disp5497:~$
</pre>
## opensourcewarehouse.org
# I think those are completely unused; the nameserver for the former is ghandi and the nameserver for the later is set to dreamhost
# oh, there *is* an spf record for the ghandi one (oswarehouse.org), but it's not something we ever set = ""v=spf1 include:_mailcust.gandi.net ?all""
# I didn't log into dreamhost, but a query shows the same: we never set it
<pre>
user@disp5497:~$ dig +short -t txt opensourcewarehouse.org
"v=spf1 mx include:netblocks.dreamhost.com include:relay.mailchannels.net -all"
user@disp5497:~$
</pre>
# I'm not going to worry about email records (MX, spf) for these domains
# I think the others are on cloudflare? let's check
# I logged into cloudflare, but I only see 1 domain = opensourceecology.org
# yeah, I think this is why I left a note for myself to research further; I don't know where is OBI
# ah, dig says the nameserver is dreamhost
<pre>
user@disp5497:~$ dig NS openbuildinginstitute.org
 
; <<>> DiG 9.16.50-Debian <<>> NS openbuildinginstitute.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7804
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openbuildinginstitute.org. IN NS
 
;; ANSWER SECTION:
openbuildinginstitute.org. 14400 IN NS ns1.dreamhost.com.
openbuildinginstitute.org. 14400 IN NS ns2.dreamhost.com.
openbuildinginstitute.org. 14400 IN NS ns3.dreamhost.com.
 
;; Query time: 116 msec
;; SERVER: 10.139.1.1#53(10.139.1.1)
;; WHEN: Fri Feb 14 17:19:06 -05 2025
;; MSG SIZE  rcvd: 121
 
user@disp5497:~$
</pre>
# I tried to login to dreamhost, but I couldn't. I entered the creds in the shared ose keepass, then it demanded an OTP sent to some email.
# I decided to create a new gapps account for this, to fix this problem in the future
# I sent an email to Marcin asking him to change the email address with that account from his personal email to this new, shared email.
# I guess the domain/spf work is paused until I can gain access to dreamhost again
# looks like the spf records for obi don't matter either
<pre>
user@disp5497:~$ dig -t txt openbuildinginstitute.org
 
; <<>> DiG 9.16.50-Debian <<>> -t txt openbuildinginstitute.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10607
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;openbuildinginstitute.org. IN TXT
 
;; ANSWER SECTION:
openbuildinginstitute.org. 300 IN TXT "v=spf1 mx include:netblocks.dreamhost.com include:relay.mailchannels.net -all"
 
;; Query time: 133 msec
;; SERVER: 10.139.1.1#53(10.139.1.1)
;; WHEN: Fri Feb 14 17:40:26 -05 2025
;; MSG SIZE  rcvd: 144
 
user@disp5497:~$
</pre>
# ...but I'll still need access to dreamhost, anyway – to update the A and AAAA records for the migration to hetzner3
# ...
# another unknown that I wanted to figure-out was if we could host the same domain (eg for obi and osemain) on two distinct IP addresses
## but we only have one IPv4 address. We could pay for a second one, but we have access to tons of free IPv6 addreses. The question is: does that work with /etc/hosts? Will it work for Marcin and Catarina?
# first I checked the IP addresses we *do* have
<pre>
root@hetzner3 /etc/nginx/sites-enabled # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
  valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 90:1b:0e:c4:28:b4 brd ff:ff:ff:ff:ff:ff
inet 144.76.164.201/27 brd 144.76.164.223 scope global enp0s31f6
  valid_lft forever preferred_lft forever
inet6 2a01:4f8:200:40d7::2/64 scope global
  valid_lft forever preferred_lft forever
inet6 fe80::921b:eff:fec4:28b4/64 scope link
  valid_lft forever preferred_lft forever
root@hetzner3 /etc/nginx/sites-enabled #
</pre>
# woah, that says we have a '/27' IP address. Is that a misconfig? /32 would be 1 IP. A /27 would be 32 IPs https://www.freecodecamp.org/news/subnet-cheat-sheet-24-subnet-mask-30-26-27-29-and-other-ip-address-cidr-network-references/
# I really, really doubt hetzner gave us 32 IP addresses, but we should check
# this subnet calculator says "144.76.164.201/27" would mean that we have "144.76.164.193 - 144.76.164.222" https://www.calculator.net/ip-subnet-calculator.html?cclass=any&csubnet=27&cip=144.76.164.201&ctype=ipv4&x=Calculate
# I logged into the hetzner site for managing hetzner 3 (and hetzner2) https://robot.hetzner.com/server
# It lists IP address = ' 144.76.164.201'
## If I hover over ^ that IP, then a tooltip appears that says
<pre>
Gateway: 144.76.164.193
Netmask: 255.255.255.224
Broadcast: 144.76.164.223
</pre>
# there was a button that said "Other additional IPs / Nets"
## clicking it, it shows prices and options to by more IPs
<pre>
One additional IP with costs (€ 1.70 max. per month / € 0.0027 per hour + € 4.90 once-off setup)
One additional subnet /29 (€ 13.60 max. per month / € 0.0218 per hour + € 34.90 once-off setup)
One additional subnet /28 (€ 27.20 max. per month / € 0.0436 per hour + € 59.90 once-off setup)
One additional subnet /27 (€ 54.40 max. per month / € 0.0872 per hour + € 109.00 once-off setup)
One additional subnet /26 (€ 108.80 max. per month / € 0.1744 per hour + € 199.00 once-off setup)
One additional subnet /25 (€ 217.60 max. per month / € 0.3487 per hour + € 369.00 once-off setup)
One additional subnet /24 (€ 435.20 max. per month / € 0.6974 per hour + € 659.00 once-off setup)
One failover IP (€ 4.20 max. per month / € 0.0067 per hour + € 4.90 once-off setup)
Failover subnet /29 (€ 22.00 max. per month / € 0.0353 per hour + € 34.90 once-off setup)
Failover subnet /28 (€ 35.60 max. per month / € 0.0571 per hour + € 59.90 once-off setup)
Failover subnet /27 (€ 62.80 max. per month / € 0.1006 per hour + € 109.00 once-off setup)
Failover subnet /26 (€ 117.20 max. per month / € 0.1878 per hour + € 199.00 once-off setup)
Failover subnet /25 (€ 226.00 max. per month / € 0.3622 per hour + € 369.00 once-off setup)
Failover subnet /24 (€ 443.60 max. per month / € 0.7109 per hour + € 659.00 once-off setup)
IPv6 Failover subnet /64 (€ 1.00 max. per month / € 0.0016 per hour + € 4.90 once-off setup)
</pre>
# yeah, that says a '/27' would cost € 54.40 per month. That's more than our whole server.
# just one IP would be € 1.70 /mo = € 20.4 per year. That's manageable, but what we're trying to avoid.
# I don't know what a failover subnet is, or why it's more expensive. whatever
# I opened a support request with Hetzner for clarification
<pre>
Hi Hetzner Support,
 
Can you please tell me how many IP addresses our server has?
 
I had assumed we only had 1 IPv4 address, but I was a bit confused when I saw this in the server's config
 
root@hetzner3 ~ # ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
inet 144.76.164.201/27 brd 144.76.164.223 scope global enp0s31f6
  valid_lft forever preferred_lft forever
root@hetzner3 ~ #
 
Is this a misconfiguration?
 
We installed Debian 12 on our server using your installimage script
 
* https://docs.hetzner.com/robot/dedicated-server/operating-systems/installimage/
 
I believe the network was automatically configured during the install process using ^ that script
 
I was shocked to see that it lists a "/27" subnet. I've also seen others with hetzner servers list this on Stack Exchange.
 
Do we really have 32 IPv4 addresses included in our dedicated server?
 
  144.76.164.193 - 144.76.164.222
 
If not, why does our server list "144.76.164.201/27"?
 
 
Thank you,
 
Michael Altfield
Open Source Ecology
</pre>
# anyway, we also have this "/64" on IPv6. That I do not think is a lie
<pre>
root@hetzner3 ~ # ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host noprefixroute
  valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2a01:4f8:200:40d7::2/64 scope global
  valid_lft forever preferred_lft forever
inet6 fe80::921b:eff:fec4:28b4/64 scope link
  valid_lft forever preferred_lft forever
root@hetzner3 ~ #
</pre>
# according to this same site, that means we have 616 IP addresses in this range, from 2a01:04f8:0200:40d7:0000:0000:0000:0000 - 2a01:04f8:0200:40d7:ffff:ffff:ffff:ffff https://www.calculator.net/ip-subnet-calculator.html?c6subnet=64&c6ip=2a01%3A4f8%3A200%3A40d7%3A%3A2&ctype=ipv6&x=Calculate#ipv6
# currently we're binding the OBI site on one of these = 2a01:4f8:200:40d7::2
<pre>
root@hetzner3 /etc/nginx/sites-enabled # grep listen www.openbuildinginstitute.org.conf
# Purpose: Internet-listening web server for truncating https, basic DOS
  listen 144.76.164.201:443;
  listen [2a01:4f8:200:40d7::2]:443;
  listen 144.76.164.201:443;
  listen [2a01:4f8:200:40d7::2]:443;
root@hetzner3 /etc/nginx/sites-enabled #
</pre>
# I wonder why it starts at "::2" and not "::0"
# I tried pinging ourselves at the ipv6 address; that works
<pre>
root@hetzner3 /etc/nginx/sites-enabled # ping -c2 2a01:4f8:200:40d7::2
PING 2a01:4f8:200:40d7::2(2a01:4f8:200:40d7::2) 56 data bytes
64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from 2a01:4f8:200:40d7::2: icmp_seq=2 ttl=64 time=0.045 ms
 
--- 2a01:4f8:200:40d7::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.024/0.034/0.045/0.010 ms
root@hetzner3 /etc/nginx/sites-enabled #
</pre>
# I tried to see if I could just increment by one; that didn't work
<pre>
root@hetzner3 /etc/nginx/sites-enabled # ping -c2 2a01:4f8:200:40d7::3
PING 2a01:4f8:200:40d7::3(2a01:4f8:200:40d7::3) 56 data bytes
From 2a01:4f8:200:40d7::2 icmp_seq=1 Destination unreachable: Address unreachable
From 2a01:4f8:200:40d7::2 icmp_seq=2 Destination unreachable: Address unreachable
 
--- 2a01:4f8:200:40d7::3 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1031ms
 
root@hetzner3 /etc/nginx/sites-enabled #
</pre>
# here's the actual network config for the box; again, it says it was setup by hetzner's installimage
<pre>
root@hetzner3 /etc/nginx/sites-enabled # cat /etc/network/interfaces
### Hetzner Online GmbH installimage
 
source /etc/network/interfaces.d/*
 
auto lo
iface lo inet loopback
iface lo inet6 loopback
 
auto enp0s31f6
iface enp0s31f6 inet static
  address 144.76.164.201
  netmask 255.255.255.224
  gateway 144.76.164.193
  # route 144.76.164.192/27 via 144.76.164.193
  up route add -net 144.76.164.192 netmask 255.255.255.224 gw 144.76.164.193 dev enp0s31f6
 
iface enp0s31f6 inet6 static
  address 2a01:4f8:200:40d7::2
  netmask 64
  gateway fe80::1
root@hetzner3 /etc/nginx/sites-enabled #
</pre>
# if I fuck this up, I'm going to get locked out of the server
<pre>
root@hetzner3 /etc/network # ls
if-down.d  if-post-down.d  if-pre-up.d  if-up.d  interfaces  interfaces.d
root@hetzner3 /etc/network # ls -lah interfaces.d
total 8,0K
drwxr-xr-x 2 root root 4,0K Jan 24  2023 .
drwxr-xr-x 7 root root 4,0K Jun 23  2024 ..
root@hetzner3 /etc/network # cp interfaces interfaces.20250214
root@hetzner3 /etc/network # vim interfaces
...
root@hetzner3 /etc/network # diff interfaces.20250214 interfaces
20a21,25
>
> iface enp0s31f6 inet6 static
>  address 2a01:4f8:200:40d7::3
>  netmask 64
>  gateway fe80::1
root@hetzner3 /etc/network #
 
root@hetzner3 /etc/network # systemctl restart networking
root@hetzner3 /etc/network #
</pre>
# cool, I think that might have worked
<pre>
root@hetzner3 /etc/network # ip -6 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host noprefixroute
  valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2a01:4f8:200:40d7::3/64 scope global
  valid_lft forever preferred_lft forever
inet6 2a01:4f8:200:40d7::2/64 scope global
  valid_lft forever preferred_lft forever
inet6 fe80::921b:eff:fec4:28b4/64 scope link
  valid_lft forever preferred_lft forever
root@hetzner3 /etc/network #
</pre>
# it's pinging!
<pre>
root@hetzner3 /etc/network # ping -c2 2a01:4f8:200:40d7::3
PING 2a01:4f8:200:40d7::3(2a01:4f8:200:40d7::3) 56 data bytes
64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=1 ttl=64 time=0.030 ms
64 bytes from 2a01:4f8:200:40d7::3: icmp_seq=2 ttl=64 time=0.025 ms
 
--- 2a01:4f8:200:40d7::3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.025/0.027/0.030/0.002 ms
root@hetzner3 /etc/network #
</pre>
# I created a new vhost, which I'll bind on that IP
<pre>
root@hetzner3 /etc/nginx/sites-enabled # cp www.openbuildinginstitute.org.conf new-www.openbuildinginstitute.org.conf
root@hetzner3 /etc/nginx/sites-enabled #
 
root@hetzner3 /etc/nginx/sites-enabled # vim new-www.openbuildinginstitute.org.conf
root@hetzner3 /etc/nginx/sites-enabled #
 
root@hetzner3 /etc/nginx/sites-enabled # diff www.openbuildinginstitute.org.conf new-www.openbuildinginstitute.org.conf
23c23
<    listen 144.76.164.201:443;
---
>    #  listen 144.76.164.201:443;
38,39c38,39
<    listen 144.76.164.201:443;
<    listen [2a01:4f8:200:40d7::2]:443; 
---
>    #  listen 144.76.164.201:443;
>    listen [2a01:4f8:200:40d7::3]:443;
49,50c49,50
<      #root /var/www/html/SITE_DOWN/htdocs/;
<    #index index.html index.htm;
---
>      root /var/www/html/new-obi/htdocs/;
>      index index.html index.htm;
53,55c53,55
<      #location / {
<      #      try_files $uri /index.html;
<      #}
---
>      location / {
>              try_files $uri /i
 
=Thr Feb 13, 2025=
 
# Marcin responded to my inquery about Admins on the wiki
# he said to give Catarina a new Admin-specific account, which I suggested for him too. But he didn't want that for his own setup. Sounds good to me.
# I should also remove all users except her, me, Marcin, and Tom Griffin from the sysop (Admin) group.
<pre>
Is the method for 2FA the TOTP app?
 
For Catarina, could we set up one wiki admin account, and another one as
regular user where TOTP is not required, is, its simpler to log in?
 
We can get rid of all the admins except for me, you,  and Tom Griffing.
 
MJ
</pre>
# ...
# yesterday we finished phpList
# we're blocked on osemain and obi until Marcin, Catarina, and I meet to discuss how to proceed. They've tabled the meeting until Marcin returns to FeF
# only site I can continue on is the wiki
# last time I hit issues with verifying the RequestAccount page on the wiki
# for some reason, loading the page produces an error https://wiki.opensourceecology.org/wiki/Special:RequestAccount
<pre>
[Z64oPIiPN52qjU5tBa1PAQAAAAo] 2025-02-13 17:13:32: Fatal exception of type "UnderflowException"
</pre>
# ..yet there's no errors written to the apache log nor the wiki-error.log file
# I noticed the disk was near-full, but not full enough to prevent writing errors
# anyway, I went ahead and cleaned-up some files from /home/maltfield/ and /var/tmp
# I got the disk usage down from 95% full to ?% full
<pre>
root@hetzner3 ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            32G    0  32G  0% /dev
tmpfs          6,3G  1,1M  6,3G  1% /run
/dev/md2        436G  290G  125G  70% /
tmpfs            32G  80K  32G  1% /dev/shm
tmpfs          5,0M    0  5,0M  0% /run/lock
/dev/md1        989M  66M  872M  8% /boot
tmpfs          6,3G    0  6,3G  0% /run/user/1000
root@hetzner3 ~ #
 
root@hetzner3 ~ # du -sh /var/tmp/*
51G    /var/tmp/backups_for_migration_from_hetzner2
59G    /var/tmp/CHG_20241228_wiki_1.35-to-1.43
20G    /var/tmp/CHG_20241229_182546_wiki_1.35-to-1.43
59G    /var/tmp/CHG_20241229_wiki_1.30-to-1.35
245M    /var/tmp/CHG_20250211_phplist
159M    /var/tmp/current
8,0K    /var/tmp/letsencrypt.20240925
835M    /var/tmp/mediawiki
72K    /var/tmp/munin
28K    /var/tmp/munin.20241213.tar.gz
4,0K    /var/tmp/old
181M    /var/tmp/phplist
...
1,2G    /var/tmp/wordpress
root@hetzner3 ~ #
</pre>
## the wiki bakups are the largest, and I'm not deleting those yet (I'm still working on it!)
# anyway, I'm still just getting a 500 server error back from this page load. And no useful errors written to server-side logs
<pre>
user@disp9456:~$ curl -I https://wiki.opensourceecology.org/wiki/Special:RequestAccount
HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Thu, 13 Feb 2025 17:23:17 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 13422
Connection: keep-alive
X-Content-Type-Options: nosniff
Content-language: en
X-Frame-Options: DENY, SAMEORIGIN
Vary: Accept-Encoding,Cookie
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
X-Request-Id: Z64qhIiPN52qjU5tBa1PAgAAAAo
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: strict-origin
X-Varnish: 68976
Age: 0
Via: 1.1 varnish (Varnish/7.1)
 
user@disp9456:~$
</pre>
# I found this guide to debugging info on mediawiki https://www.mediawiki.org/wiki/Manual:How_to_debug
# well, I still don't know why it's not writing to the wiki log, but I was able to get it to spit out logs directly in the WUI
<pre>
$wgShowExceptionDetails = true;
</pre>
# now I see this on the screen
<pre>
[Z64vfE4vD3xHShde_vEOeQAAAAk] /wiki/Special:RequestAccount UnderflowException: Ran out of captcha images
 
Backtrace:
 
from /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/FancyCaptcha/includes/FancyCaptcha.php(473)
#0 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/FancyCaptcha/includes/FancyCaptcha.php(133): MediaWiki\Extension\ConfirmEdit\FancyCaptcha\FancyCaptcha->getCaptcha()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/frontend/specialpages/actions/RequestAccount_body.php(260): MediaWiki\Extension\ConfirmEdit\FancyCaptcha\FancyCaptcha->getFormInformation()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/frontend/specialpages/actions/RequestAccount_body.php(96): RequestAccountPage->showForm()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPage.php(728): RequestAccountPage->execute()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPageFactory.php(1724): MediaWiki\SpecialPage\SpecialPage->run()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ActionEntryPoint.php(504): MediaWiki\SpecialPage\SpecialPageFactory->executePath()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ActionEntryPoint.php(146): MediaWiki\Actions\ActionEntryPoint->performRequest()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#9 {main}
</pre>
# there's a bug report about this here, but not very helpful https://phabricator.wikimedia.org/T91760
# might be easier to just check the ConfirmEdit docs and confirm we have all the depends in-order https://www.mediawiki.org/wiki/Extension:ConfirmEdit#FancyCaptcha
# from ^ that, looks like this is the python script that php uses
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py"
This script requires the Python Imaging Library - http://www.pythonware.com/products/pil/
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I guess pillow got renamed "pil"? The docs aren't updated, but the script is
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # apt-cache search pillow
libpillowfight-dev - Various image processing algorithms library - development files
libpillowfight0 - Various image processing algorithms library
python3-pypillowfight - Python 3 bindings for libpillowfight
luma.core-doc - doc for component library providing a Pillow-compatible drawing canvas
python3-luma.core - component library providing a Pillow-compatible drawing canvas
luma.lcd-doc - doc for library interfacing small LCD displays
python3-luma.lcd - library interfacing small LCD displays
luma.led-matrix-doc - doc for library interfacing LED matrix displays
python3-luma.led-matrix - library interfacing LED matrix displays
luma.oled-doc - doc for library interfacing OLED matrix displays
python3-luma.oled - library interfacing OLED matrix displays
python3-pil - Python Imaging Library (Python3)
python3-av - pythonic bindings for FFmpeg's libraries
python3-pyglfw - Python bindings for GLFW
python3-pylibdmtx - Read Data Matrix barcodes (Python3 version)
python3-pyocr - Python wrapper for OCR engines (Python 3)
python3-tesserocr - Python wrapper for the tesseract-ocr API (Python3 version)
python3-pyzbar - Python bindings for libzbar
python3-typeshed - collection of library stubs for Python, with static types
python-willow-doc - Python image library (documentation)
python3-willow - Python image library combining Pillow, Wand and OpenCV (Python 3)
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# oh, the pypi page says pillow was a fork of PIL. Maybe the package name just stayed the same in the apt repos, to prevent breaking deps https://pypi.org/project/pillow/
# I installed it
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # dpkg -l | grep -i pil
ii  gcc                            4:12.2.0-3                              amd64        GNU C compiler
ii  gcc-12                        12.2.0-14                              amd64        GNU C compiler
ii  gcc-12-base:amd64              12.2.0-14                              amd64        GCC, the GNU Compiler Collection (base package)
ii  liblist-moreutils-xs-perl      0.430-3+b1                              amd64        Perl module providing compiled List::MoreUtils functions
ii  libmagic-mgc                  1:5.44-3                                amd64        File type determination library using "magic" numbers (compiled magic file)
ii  rpcsvc-proto                  1.4.3-1                                amd64        RPC protocol compiler and definitions
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # apt-get install python3-pil
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libimagequant0 liblcms2-2 libopenjp2-7 libraqm0 libwebpdemux2 libwebpmux3
Suggested packages:
  liblcms2-utils python-pil-doc
Recommended packages:
  python3-olefile
The following NEW packages will be installed:
  libimagequant0 liblcms2-2 libopenjp2-7 libraqm0 libwebpdemux2 libwebpmux3 python3-pil
0 upgraded, 7 newly installed, 0 to remove and 77 not upgraded.
Need to get 1.067 kB of archives.
After this operation, 3.077 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
...
Setting up python3-pil:amd64 (9.4.0-1.1+deb12u1) ...
Processing triggers for libc-bin (2.36-9+deb12u8) ...
You have mail in /var/mail/root
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # dpkg -l | grep -i pil
ii  gcc                            4:12.2.0-3                              amd64        GNU C compiler
ii  gcc-12                        12.2.0-14                              amd64        GNU C compiler
ii  gcc-12-base:amd64              12.2.0-14                              amd64        GCC, the GNU Compiler Collection (base package)
ii  liblist-moreutils-xs-perl      0.430-3+b1                              amd64        Perl module providing compiled List::MoreUtils functions
ii  libmagic-mgc                  1:5.44-3                                amd64        File type determination library using "magic" numbers (compiled magic file)
ii  python3-pil:amd64              9.4.0-1.1+deb12u1                      amd64        Python Imaging Library (Python3)
ii  rpcsvc-proto                  1.4.3-1                                amd64        RPC protocol compiler and definitions
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# well this is better
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py"
Need to specify a wordlist
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I tried refreshing the page, but I have the same issue
# here's some files that might be useful for our args
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/extensions/ConfirmEdit/
AUTHORS.txt    captcha.py          ConfirmEdit.alias.php  FancyCaptcha  includes      README.md          SimpleCaptcha  Turnstile
badwordlist    CODE_OF_CONDUCT.md  COPYING                hCaptcha      maintenance    ReCaptchaNoCaptcha  tests
captcha-old.py  composer.json      extension.json        i18n          QuestyCaptcha  resources          tox.ini
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# crap, that only includes a badwordlist. What about the wordlist?
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # find htdocs/extensions/ConfirmEdit/ | grep -Ei 'list|word'
htdocs/extensions/ConfirmEdit/resources/ext.confirmEdit.editPreview.ipwhitelist.styles.css
htdocs/extensions/ConfirmEdit/badwordlist
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# docs say we need to install 'wbritish' or 'wamercian' https://www.mediawiki.org/wiki/Extension:ConfirmEdit#FancyCaptcha
<pre>
wordlist is a path to some word list, for instance /usr/share/dict/words. (Note: on Debian/Ubuntu, the 'wbritish' and 'wamerican' packages provide such lists. On Fedora, use the 'words' package)
</pre>
# looks like we have a choice between US, CA, and GB
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # apt-cache search 'dictionary words' | grep -i english
wamerican - American English dictionary words for /usr/share/dict
wamerican-huge - American English dictionary words for /usr/share/dict
wamerican-insane - American English dictionary words for /usr/share/dict
wamerican-large - American English dictionary words for /usr/share/dict
wamerican-small - American English dictionary words for /usr/share/dict
wbritish - British English dictionary words for /usr/share/dict
wbritish-huge - British English dictionary words for /usr/share/dict
wbritish-insane - British English dictionary words for /usr/share/dict
wbritish-large - British English dictionary words for /usr/share/dict
wbritish-small - British English dictionary words for /usr/share/dict
wcanadian - Canadian English dictionary words for /usr/share/dict
wcanadian-huge - Canadian English dictionary words for /usr/share/dict
wcanadian-insane - Canadian English dictionary words for /usr/share/dict
wcanadian-large - Canadian English dictionary words for /usr/share/dict
wcanadian-small - Canadian English dictionary words for /usr/share/dict
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I installed the normal US word list
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # apt-get install wamerican
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  wamerican
0 upgraded, 1 newly installed, 0 to remove and 77 not upgraded.
Need to get 221 kB of archives.
After this operation, 1.028 kB of additional disk space will be used.
Get:1 http://mirror.hetzner.com/debian/packages bookworm/main amd64 wamerican all 2020.12.07-2 [221 kB]
Fetched 221 kB in 1s (152 kB/s)     
Preconfiguring packages ...
Selecting previously unselected package wamerican.
(Reading database ... 66700 files and directories currently installed.)
Preparing to unpack .../wamerican_2020.12.07-2_all.deb ...
Unpacking wamerican (2020.12.07-2) ...
Setting up wamerican (2020.12.07-2) ...
Processing triggers for man-db (2.11.2-2) ...
You have mail in /var/mail/root
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # dpkg-query -L wamerican
/.
/usr
/usr/share
/usr/share/dict
/usr/share/dict/american-english
/usr/share/doc
/usr/share/doc/wamerican
/usr/share/doc/wamerican/NEWS.Debian.gz
/usr/share/doc/wamerican/README.Debian
/usr/share/doc/wamerican/changelog.Debian.gz
/usr/share/doc/wamerican/copyright
/usr/share/doc/wamerican/wamerican.scowl-word-lists-used
/usr/share/man
/usr/share/man/man5
/usr/share/man/man5/american-english.5.gz
/var
/var/lib
/var/lib/dictionaries-common
/var/lib/dictionaries-common/wordlist
/var/lib/dictionaries-common/wordlist/wamerican
/usr/share/dict/words
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# that gave us over 100,000 words
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # head /usr/share/dict/words
A
AA
AAA
AA's
AB
ABC
ABC's
ABCs
ABM
ABM's
You have mail in /var/mail/root
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail /usr/share/dict/words
zoos
zorch
zucchini
zucchini's
zucchinis
zwieback
zwieback's
zygote
zygote's
zygotes
root@hetzner3 /var/www/html/wiki.opensourceecology.org # wc -l /usr/share/dict/words
104334 /usr/share/dict/words
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# progress
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py" --wordlist /usr/share/dict/words
Need to specify a key
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# progress
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py" --wordlist=/usr/share/dict/words --key="CHANGEME" --output=/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
Need to specify the location of a font
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# the docs example font is FreeSans, but that's not installed /usr/share/fonts/truetype/freefont/FreeSans.ttf
# the only font we have in that dir is dejavu; let's use that
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /usr/share/fonts/truetype/
dejavu
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /usr/share/fonts/truetype/dejavu/
DejaVuSans-Bold.ttf  DejaVuSansMono-Bold.ttf  DejaVuSansMono.ttf  DejaVuSans.ttf  DejaVuSerif-Bold.ttf  DejaVuSerif.ttf
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I don't know what's best here, but I'm guessing bold serif will make more legible words
# this said it worked, but I don't see the actual image files
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py" --wordlist=/usr/share/dict/words --key="CHANGEME" --output=/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha --font=/usr/share/fonts/truetype/dejavu/DejaVuSerif-Bold.ttf
Generating 20 CAPTCHA images separated in 20 image(s) per chunk run by 1 threads...
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
ls: cannot access '/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha': No such file or directory
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I manually created the dir
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # mkdir /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
You have mail in /var/mail/root
root@hetzner3 /var/www/html/wiki.opensourceecology.org # chown www-data:www-data /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
root@hetzner3 /var/www/html/wiki.opensourceecology.org # chmod 0770 /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
total 8,0K
drwxrwx---  2 www-data www-data 4,0K Feb 13 19:48 .
drwxrwx--- 30 www-data www-data 4,0K Feb 13 19:48 ..
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# this time it actually worked!
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py" --wordlist=/usr/share/dict/words --key="CHANGEME" --output=/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha --font=/usr/share/fonts/truetype/dejavu/DejaVuSerif-Bold.ttf
Generating 20 CAPTCHA images separated in 20 image(s) per chunk run by 1 threads...
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha
total 112K
drwxrwx---  2 www-data www-data 4,0K Feb 13 19:49 .
drwxrwx--- 30 www-data www-data 4,0K Feb 13 19:48 ..
-rw-r--r--  1 www-data www-data 4,1K Feb 13 19:49 image_0a55b5dc_2eae4a0549c41fdc.png
-rw-r--r--  1 www-data www-data 3,9K Feb 13 19:49 image_0b962a8b_0a9df8447a57e267.png
-rw-r--r--  1 www-data www-data 3,5K Feb 13 19:49 image_0be2a5fb_408f75224f9dc1a9.png
-rw-r--r--  1 www-data www-data 4,2K Feb 13 19:49 image_123d1a00_3c24c7497e83d719.png
-rw-r--r--  1 www-data www-data 3,6K Feb 13 19:49 image_1de8dd68_ed354602683629e0.png
-rw-r--r--  1 www-data www-data 3,3K Feb 13 19:49 image_23975c7a_0a345deac97e8b7a.png
-rw-r--r--  1 www-data www-data 3,7K Feb 13 19:49 image_31a94cd4_67520db98e6a480b.png
-rw-r--r--  1 www-data www-data 4,3K Feb 13 19:49 image_38cc2218_581c3e95dab9f08a.png
-rw-r--r--  1 www-data www-data 4,0K Feb 13 19:49 image_41c50c4e_f4f2f3b2bac753b8.png
-rw-r--r--  1 www-data www-data 3,6K Feb 13 19:49 image_44859ba7_15f597fc05c196e7.png
-rw-r--r--  1 www-data www-data 4,0K Feb 13 19:49 image_59ad9f3e_299d3257574c116f.png
-rw-r--r--  1 www-data www-data 3,9K Feb 13 19:49 image_87bcfdb0_85f09f4fe8c332c4.png
-rw-r--r--  1 www-data www-data 4,3K Feb 13 19:49 image_96de82cc_363f5bf8be3fcb32.png
-rw-r--r--  1 www-data www-data 4,0K Feb 13 19:49 image_b5958053_4c488bb99fef4751.png
-rw-r--r--  1 www-data www-data 4,0K Feb 13 19:49 image_bf56bb70_6f03b4b393c4cfcd.png
-rw-r--r--  1 www-data www-data 3,8K Feb 13 19:49 image_c4f1a927_163ad5c456e8d7fb.png
-rw-r--r--  1 www-data www-data 3,7K Feb 13 19:49 image_cabdbc25_4968c1264a6c374b.png
-rw-r--r--  1 www-data www-data 4,4K Feb 13 19:49 image_dd6574f7_1b963e299969b0cf.png
-rw-r--r--  1 www-data www-data 4,1K Feb 13 19:49 image_f1f43d57_44d08284f540a8ee.png
-rw-r--r--  1 www-data www-data 3,9K Feb 13 19:49 image_f5da9aa7_9dbf929f4207c479.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I refreshed the page in the browser, and now it loads! https://wiki.opensourceecology.org/wiki/Special:RequestAccount
# yep, I can see the CAPTCHA on the bottom. I'm still not sure if it only works because I manually ran this captcha command – or if it will be able to generate more images after these are deleted. Probably I need to add these config options to LocalSettings.php, delete the images, and test it again
# anyway, let's fix ansible with these required changes
# it's not evident which ansible role should install these depends; we don't have a role specific for MediaWiki. We have apache, nginx, varnish, and php.
# I added a section for MediaWiki apt deps in the php role, just under some deps for generating CAPTCHAs with pear (not sure why MediaWiki chose to use python instead of a php solution for this) https://github.com/OpenSourceEcology/ansible/commit/9c970f3f5a04617b7d6df8ca1c586d26b7d18d53
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ git diff
...
diff --git a/hetzner3/roles/maltfield.php/tasks/main.yml b/hetzner3/roles/maltfield.php/tasks/main.yml
index c468742..fecdcd6 100644
--- a/hetzner3/roles/maltfield.php/tasks/main.yml
+++ b/hetzner3/roles/maltfield.php/tasks/main.yml
@@ -36,6 +36,14 @@
  notify:
- restart php-fpm
+- name: install MediaWiki deps (for FancyCaptcha)
+  apt:
+    update_cache: yes
+    pkg:
+      - fonts-dejavu-core
+      - wamerican
+      - python3-pil
+
- name: php tmp dir
  file:
dest: /var/lib/php/tmp
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# I also updated the permissions script to create and setup this dir properly
<pre>
root@hetzner3 /usr/local/bin # diff fix_web_permissions.sh.20250213 fix_web_permissions.sh
5c5
< # Version: 0.1
---
> # Version: 0.2
11c11
< # Updated: 2025-02-06
---
> # Updated: 2025-02-13
97a98,102
>
> [ -d "${mw_docroot}/images/captcha" ] || mkdir "${mw_docroot}/images/captcha"
> chown -R www-data:www-data "${mw_docroot}/images/captcha"
> find "${mw_docroot}/images/captcha" -type f -exec chmod 0660 {} \;
> find "${mw_docroot}/images/captcha" -type d -exec chmod 0770 {} \;
root@hetzner3 /usr/local/bin #
</pre>
# according to the docs, the secret key is used as a salt for the hash of the captcha, which is the captcha's filename. It mentions the Directory Listing should be off for that dir, which I confirmed that it is
<pre>
user@disp9456:~$ curl https://wiki.opensourceecology.org/images/captcha/
<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>
user@disp9456:~$
</pre>
# I'm just going to use /dev/random, strings, perl, and grep to generate the random salt at install-time
# something like this will work grand
<pre>
root@disp9456:~# pwd=$(cat /dev/random | strings | grep -oE '[A-Za-z0-9]' | perl -p -e 's/\n//g' | head -c 64)
root@disp9456:~# echo $pwd
cz5wx3lxVZUIKLU5eGu3t8ogV6NxXeb9wlAWtGh1VMe9y9GY3CC0fCrLz4C98NgD
root@disp9456:~#
</pre>
# I did some more googling, and I couldn't find anything that said that MediaWiki will delete and re-generate these images
# I also didn't find any info about setting-up a cron
# so I'm pretty sure I was just assuming that's how it worked. I think how it *actually* works is that you generate a bunch of CAPTCHAs on-time, and then MediaWiki will just use those forever
# our 20 images are 108k
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah htdocs/images/captcha/ | wc -l
23
root@hetzner3 /var/www/html/wiki.opensourceecology.org # du -sh htdocs/images/captcha/
108K    htdocs/images/captcha/
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# let's do 500 images. That ought to be enough, right? I'm hoping it'll be <5 MB.
# alright, that made 2.8M
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # rm htdocs/images/captcha/*
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah htdocs/images/captcha/
total 8,0K
drwxrwx---  2 www-data www-data 4,0K Feb 13 21:48 .
drwxrwx--- 30 www-data www-data 4,0K Feb 13 19:48 ..
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py" --wordlist=/usr/share/dict/words --key="CHANGEME" --output=/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha --font=/usr/share/fonts/truetype/dejavu/DejaVuSerif-Bold.ttf --count=500
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
root@hetzner3 /var/www/html/wiki.opensourceecology.org # du -sh htdocs/images/captcha/
2,8M    htdocs/images/captcha/
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I guess there's some risk that, over days or weeks or months or years, bots would learn our CAPTCHAs. So probably it would be a good idea to replace them over-time with some cron
# I'm thinking the most durable solution is to first generate an additional 500 images, then remove the oldest images, leaving 500 images. That way there's always 500 images, at least.
# I confirmed that re-running the command just appends additional 500 images into the dir
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/images/captcha/ | wc -l
500
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data python3 "htdocs/extensions/ConfirmEdit/captcha.py" --wordlist=/usr/share/dict/words --key="CHANGEME" --output=/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha --font=/usr/share/fonts/truetype/dejavu/DejaVuSerif-Bold.ttf --count=500
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/images/captcha/ | wc -l
1000
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I want to make a command that will delete the oldest 500 files, but I want to make sure that it cannot delete the last 500 files – even if the command to generate the new captchas above failed
## so "find files older than 1 day" would be a bad idea
## and "delete the oldest 500 files" would be a bad idea
# instead, let's list the files, ordered by modification date, exclude the first 500 lines, and delete what's left. That way it should always leave at least 500 files in-place
# test: I have 1,000 images now. the first 500 were generated at 21:49. the second 500 were generated at 21:52
# if I run this command once, it should delete 500 images generated at 21:49
# if I run it a second time, it should delete nothing.
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -tl htdocs/images/captcha/ | head
total 5496
-rw-r--r-- 1 www-data www-data 4794 Feb 13 21:52 image_67516d65_06b54c3c8a7f3aee.png
-rw-r--r-- 1 www-data www-data 4083 Feb 13 21:52 image_bc93c0fd_a9683f4fa4e05eee.png
-rw-r--r-- 1 www-data www-data 4555 Feb 13 21:52 image_47856825_667d798d8cff5896.png
-rw-r--r-- 1 www-data www-data 3643 Feb 13 21:52 image_2c658768_e5edd25b2e7d3c55.png
-rw-r--r-- 1 www-data www-data 3680 Feb 13 21:52 image_6780f653_8c36018aca8971e7.png
-rw-r--r-- 1 www-data www-data 4216 Feb 13 21:52 image_5c8ada47_6e354d421d33dd9f.png
-rw-r--r-- 1 www-data www-data 3869 Feb 13 21:52 image_4e3efb1c_c0ec44d9f4d07502.png
-rw-r--r-- 1 www-data www-data 3949 Feb 13 21:52 image_9c9ca3ff_8a3e86623f5c3e39.png
-rw-r--r-- 1 www-data www-data 3245 Feb 13 21:52 image_16f4c787_0fdd0e63f13475d3.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -tl htdocs/images/captcha/ | tail
-rw-r--r-- 1 www-data www-data 4598 Feb 13 21:49 image_167e8aec_fe4025790a1ea33e.png
-rw-r--r-- 1 www-data www-data 3888 Feb 13 21:49 image_0f89ed6a_af1a9526aea0a697.png
-rw-r--r-- 1 www-data www-data 3678 Feb 13 21:49 image_37bf7b7a_f86adc879f5f8d10.png
-rw-r--r-- 1 www-data www-data 3271 Feb 13 21:49 image_d742ced4_8605af0e268ddf18.png
-rw-r--r-- 1 www-data www-data 3538 Feb 13 21:49 image_bc57156a_368b71029bb6435f.png
-rw-r--r-- 1 www-data www-data 4014 Feb 13 21:49 image_c1180428_ea0f0b49eaec39a1.png
-rw-r--r-- 1 www-data www-data 3627 Feb 13 21:49 image_3f87e950_b9ccd0840102075e.png
-rw-r--r-- 1 www-data www-data 3974 Feb 13 21:49 image_f5c2b3df_beb9239dc54859eb.png
-rw-r--r-- 1 www-data www-data 3457 Feb 13 21:49 image_50dfecb9_0604543eff9b5bdf.png
-rw-r--r-- 1 www-data www-data 3982 Feb 13 21:49 image_6444a056_811a69266a7a4d06.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/images/captcha/ | wc -l
1000
You have mail in /var/mail/root
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# ok, this worked as expected on the first run
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -t htdocs/images/captcha/* | sed -e '1,500d' | xargs rm
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -tl htdocs/images/captcha/ | head
total 2752
-rw-r--r-- 1 www-data www-data 4794 Feb 13 21:52 image_67516d65_06b54c3c8a7f3aee.png
-rw-r--r-- 1 www-data www-data 4083 Feb 13 21:52 image_bc93c0fd_a9683f4fa4e05eee.png
-rw-r--r-- 1 www-data www-data 4555 Feb 13 21:52 image_47856825_667d798d8cff5896.png
-rw-r--r-- 1 www-data www-data 3643 Feb 13 21:52 image_2c658768_e5edd25b2e7d3c55.png
-rw-r--r-- 1 www-data www-data 3680 Feb 13 21:52 image_6780f653_8c36018aca8971e7.png
-rw-r--r-- 1 www-data www-data 4216 Feb 13 21:52 image_5c8ada47_6e354d421d33dd9f.png
-rw-r--r-- 1 www-data www-data 3869 Feb 13 21:52 image_4e3efb1c_c0ec44d9f4d07502.png
-rw-r--r-- 1 www-data www-data 3949 Feb 13 21:52 image_9c9ca3ff_8a3e86623f5c3e39.png
-rw-r--r-- 1 www-data www-data 3245 Feb 13 21:52 image_16f4c787_0fdd0e63f13475d3.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -tl htdocs/images/captcha/ | tail
-rw-r--r-- 1 www-data www-data 3138 Feb 13 21:52 image_447a20f7_60f9b106c37d2725.png
-rw-r--r-- 1 www-data www-data 4645 Feb 13 21:52 image_6654e0b9_af21c143bf079a03.png
-rw-r--r-- 1 www-data www-data 3792 Feb 13 21:52 image_f6861b81_cf65f46c3df4dfed.png
-rw-r--r-- 1 www-data www-data 3736 Feb 13 21:52 image_e5dd4d54_d42fdfb49d818ac1.png
-rw-r--r-- 1 www-data www-data 3788 Feb 13 21:52 image_4f001007_c70b837e95e2f171.png
-rw-r--r-- 1 www-data www-data 3709 Feb 13 21:52 image_bb40d233_eb003ac38028a3b7.png
-rw-r--r-- 1 www-data www-data 3922 Feb 13 21:52 image_009f3926_d7cb005e3f0a22e6.png
-rw-r--r-- 1 www-data www-data 4647 Feb 13 21:52 image_4f98ac08_a2c4aafef91652dc.png
-rw-r--r-- 1 www-data www-data 4164 Feb 13 21:52 image_5cb782f7_07114e61180773c6.png
-rw-r--r-- 1 www-data www-data 3273 Feb 13 21:52 image_7102bc7d_bf385847b06089aa.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/images/captcha/ | wc -l
500
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# and it looks like the list was empty on the second run, so it worked perfectly
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -t htdocs/images/captcha/* | sed -e '1,500d' | xargs rm
rm: missing operand
Try 'rm --help' for more information.
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -t htdocs/images/captcha/* | sed -e '1,500d'
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -tl htdocs/images/captcha/ | head
total 2752
-rw-r--r-- 1 www-data www-data 4794 Feb 13 21:52 image_67516d65_06b54c3c8a7f3aee.png
-rw-r--r-- 1 www-data www-data 4083 Feb 13 21:52 image_bc93c0fd_a9683f4fa4e05eee.png
-rw-r--r-- 1 www-data www-data 4555 Feb 13 21:52 image_47856825_667d798d8cff5896.png
-rw-r--r-- 1 www-data www-data 3643 Feb 13 21:52 image_2c658768_e5edd25b2e7d3c55.png
-rw-r--r-- 1 www-data www-data 3680 Feb 13 21:52 image_6780f653_8c36018aca8971e7.png
-rw-r--r-- 1 www-data www-data 4216 Feb 13 21:52 image_5c8ada47_6e354d421d33dd9f.png
-rw-r--r-- 1 www-data www-data 3869 Feb 13 21:52 image_4e3efb1c_c0ec44d9f4d07502.png
-rw-r--r-- 1 www-data www-data 3949 Feb 13 21:52 image_9c9ca3ff_8a3e86623f5c3e39.png
-rw-r--r-- 1 www-data www-data 3245 Feb 13 21:52 image_16f4c787_0fdd0e63f13475d3.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -tl htdocs/images/captcha/ | tail
-rw-r--r-- 1 www-data www-data 3138 Feb 13 21:52 image_447a20f7_60f9b106c37d2725.png
-rw-r--r-- 1 www-data www-data 4645 Feb 13 21:52 image_6654e0b9_af21c143bf079a03.png
-rw-r--r-- 1 www-data www-data 3792 Feb 13 21:52 image_f6861b81_cf65f46c3df4dfed.png
-rw-r--r-- 1 www-data www-data 3736 Feb 13 21:52 image_e5dd4d54_d42fdfb49d818ac1.png
-rw-r--r-- 1 www-data www-data 3788 Feb 13 21:52 image_4f001007_c70b837e95e2f171.png
-rw-r--r-- 1 www-data www-data 3709 Feb 13 21:52 image_bb40d233_eb003ac38028a3b7.png
-rw-r--r-- 1 www-data www-data 3922 Feb 13 21:52 image_009f3926_d7cb005e3f0a22e6.png
-rw-r--r-- 1 www-data www-data 4647 Feb 13 21:52 image_4f98ac08_a2c4aafef91652dc.png
-rw-r--r-- 1 www-data www-data 4164 Feb 13 21:52 image_5cb782f7_07114e61180773c6.png
-rw-r--r-- 1 www-data www-data 3273 Feb 13 21:52 image_7102bc7d_bf385847b06089aa.png
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/images/captcha/ | wc -l
500
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# rather than trying to squeeze this into a cron, I'll make a script and then call that script with cron
# I wrote this
<pre>
root@hetzner3 /usr/local/bin # cat mediawiki_generate_captchas.sh
#!/bin/bash
#set -x
################################################################################
# File:    mediawiki_generate_captchas.sh
# Version: 0.1
# Purpose: Script to replace self-hosted CAPTCHA images. Should probably be run
#          periodically. For consumption by ConfirmEdit & ConfirmAccount
#          *
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-02-13
# Updated: 2025-02-13
################################################################################
 
################################################################################
#                                  SETTINGS                                    #
################################################################################
 
# number of CAPTCHA images to generate
NUM_FILES=500
 
# absolute path to where the CAPTCHA images are stored
CAPTCHA_SCRIPT_PATH='/var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/captcha.py'
 
# absolute path to where the CAPTCHA images are stored
CAPTCHA_DIR_PATH='/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/'
 
# file with one-word-per-line (origin = apt pkg 'wamerican' in debian)
WORDLIST_FILE_PATH='/usr/share/dict/words'
 
# .ttf file path (dejavu origin = apt pkg 'fonts-dejavu-core' in debian)
FONT_FILE_PATH='/usr/share/fonts/truetype/dejavu/DejaVuSerif-Bold.ttf'
 
# secret used to salt image filenames (hashed solution)
CAPTCHA_SECRET='CHANGEME'
 
################################################################################
#                                  FUNCTIONS                                  #
################################################################################
 
################################################################################
#                                  MAIN BODY                                  #
################################################################################
 
# first we generate new CAPTCHA files
sudo -u www-data bash -c "python3 '${CAPTCHA_SCRIPT_PATH}' --wordlist=${WORDLIST_FILE_PATH} --key='${CAPTCHA_SECRET}' --output='${CAPTCHA_DIR_PATH}' --font='${FONT_FILE_PATH}' --count='${NUM_FILES}'"
 
 
# now we delete all the CAPTCHA files, except the number of files we want
sudo -u www-data bash -c "ls -t ${CAPTCHA_DIR_PATH}/* | sed -e '1,${NUM_FILES}d' | xargs rm"
 
exit 0
root@hetzner3 /usr/local/bin #
</pre>
# it worked as expected on the first run
<pre>
root@hetzner3 /usr/local/bin # ls -tl /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | head;  ls -tl /var/www/html/wiki.opensourceecology.o
rg/htdocs/images/captcha/ | tail; ls /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | wc -l
total 2752
-rw-r--r-- 1 www-data www-data 4794 Feb 13 21:52 image_67516d65_06b54c3c8a7f3aee.png
-rw-r--r-- 1 www-data www-data 4083 Feb 13 21:52 image_bc93c0fd_a9683f4fa4e05eee.png
-rw-r--r-- 1 www-data www-data 4555 Feb 13 21:52 image_47856825_667d798d8cff5896.png
-rw-r--r-- 1 www-data www-data 3643 Feb 13 21:52 image_2c658768_e5edd25b2e7d3c55.png
-rw-r--r-- 1 www-data www-data 3680 Feb 13 21:52 image_6780f653_8c36018aca8971e7.png
-rw-r--r-- 1 www-data www-data 4216 Feb 13 21:52 image_5c8ada47_6e354d421d33dd9f.png
-rw-r--r-- 1 www-data www-data 3869 Feb 13 21:52 image_4e3efb1c_c0ec44d9f4d07502.png
-rw-r--r-- 1 www-data www-data 3949 Feb 13 21:52 image_9c9ca3ff_8a3e86623f5c3e39.png
-rw-r--r-- 1 www-data www-data 3245 Feb 13 21:52 image_16f4c787_0fdd0e63f13475d3.png
-rw-r--r-- 1 www-data www-data 3138 Feb 13 21:52 image_447a20f7_60f9b106c37d2725.png
-rw-r--r-- 1 www-data www-data 4645 Feb 13 21:52 image_6654e0b9_af21c143bf079a03.png
-rw-r--r-- 1 www-data www-data 3792 Feb 13 21:52 image_f6861b81_cf65f46c3df4dfed.png
-rw-r--r-- 1 www-data www-data 3736 Feb 13 21:52 image_e5dd4d54_d42fdfb49d818ac1.png
-rw-r--r-- 1 www-data www-data 3788 Feb 13 21:52 image_4f001007_c70b837e95e2f171.png
-rw-r--r-- 1 www-data www-data 3709 Feb 13 21:52 image_bb40d233_eb003ac38028a3b7.png
-rw-r--r-- 1 www-data www-data 3922 Feb 13 21:52 image_009f3926_d7cb005e3f0a22e6.png
-rw-r--r-- 1 www-data www-data 4647 Feb 13 21:52 image_4f98ac08_a2c4aafef91652dc.png
-rw-r--r-- 1 www-data www-data 4164 Feb 13 21:52 image_5cb782f7_07114e61180773c6.png
-rw-r--r-- 1 www-data www-data 3273 Feb 13 21:52 image_7102bc7d_bf385847b06089aa.png
500
root@hetzner3 /usr/local/bin #
 
root@hetzner3 /usr/local/bin # mediawiki_generate_captchas.sh
Generating 500 CAPTCHA images separated in 500 image(s) per chunk run by 1 threads...
root@hetzner3 /usr/local/bin #
 
root@hetzner3 /usr/local/bin # ls -tl /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | head;  ls -tl /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | tail; ls /var/www/html/wiki.opensourceecology.org/htdocs/images/captcha/ | wc -l
total 2804
-rw-r--r-- 1 www-data www-data 4035 Feb 13 22:33 image_435659ee_cb266ea06c4df56a.png
-rw-r--r-- 1 www-data www-data 3648 Feb 13 22:33 image_46d9ae64_5376df5ce73e87d0.png
-rw-r--r-- 1 www-data www-data 3962 Feb 13 22:33 image_03fd0a99_d23817ac2b69e2ca.png
-rw-r--r-- 1 www-data www-data 3918 Feb 13 22:33 image_373700bb_4ee44572db424a54.png
-rw-r--r-- 1 www-data www-data 3908 Feb 13 22:33 image_ecf10b3b_b0557b353d02633c.png
-rw-r--r-- 1 www-data www-data 4398 Feb 13 22:33 image_19a990a3_a25303b98a5e3c14.png
-rw-r--r-- 1 www-data www-data 4304 Feb 13 22:33 image_b67101a0_52f12f202edcfce5.png
-rw-r--r-- 1 www-data www-data 3702 Feb 13 22:33 image_dadb9910_3f3d4688f93e7e7e.png
-rw-r--r-- 1 www-data www-data 4651 Feb 13 22:33 image_25851cb4_0d1ae6e9705e5c04.png
-rw-r--r-- 1 www-data www-data 3407 Feb 13 22:33 image_6b8fdb25_cf432a75a6d7a95c.png
-rw-r--r-- 1 www-data www-data 3923 Feb 13 22:33 image_de909ea8_520d463d9055ce4e.png
-rw-r--r-- 1 www-data www-data 4039 Feb 13 22:33 image_89127234_70db328b38f75014.png
-rw-r--r-- 1 www-data www-data 3736 Feb 13 22:33 image_a5c28da3_b8f94f92cb42a306.png
-rw-r--r-- 1 www-data www-data 3720 Feb 13 22:33 image_10c743b6_2abfa7c229085922.png
-rw-r--r-- 1 www-data www-data 3639 Feb 13 22:33 image_557b71f3_a603416758e2da88.png
-rw-r--r-- 1 www-data www-data 3675 Feb 13 22:33 image_fe67e0c0_721a8705f69dedcf.png
-rw-r--r-- 1 www-data www-data 4219 Feb 13 22:33 image_3642f580_a6908783c9400e93.png
-rw-r--r-- 1 www-data www-data 3274 Feb 13 22:33 image_495a8abb_239f5ea45fd72128.png
-rw-r--r-- 1 www-data www-data 3360 Feb 13 22:33 image_0c79bc8c_3ca5a2580650c88a.png
500
root@hetzner3 /usr/local/bin #
</pre>
# I created a cron to call this daily
<pre>
root@hetzner3 /usr/local/bin # cat /etc/cron.d/mediawiki_cron
# Ansible managed
SHELL=/bin/bash
 
# regenerate new CAPTCHA images
20 4 * * * www-data sleep $(( RANDOM \% 3600 )) && /usr/local/bin/mediawiki_generate_captchas.sh &> /var/log/cron/mediawiki_generate_captchas.log
root@hetzner3 /usr/local/bin #
</pre>
# I'll have to wait and see if this is working as desired
# I'm also worried that there might be cache issues. we'll just have to wait and test and see.
# ok, with all that in-place, I tried to submit the RequestAccount page. I got an error
<pre>
[Z656KOyg1u-trWcf6QUD6AAAAAc] /wiki/Special:RequestAccount Error: Call to undefined function Wikimedia\FileBackend\chmod()
 
Backtrace:
 
from /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FSFileBackend.php(920)
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FSFileBackend.php(339): Wikimedia\FileBackend\FSFileBackend->chmod()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FileBackendStore.php(212): Wikimedia\FileBackend\FSFileBackend->doStoreInternal()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fileop/StoreFileOp.php(84): Wikimedia\FileBackend\FileBackendStore->storeInternal()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fileop/FileOp.php(252): Wikimedia\FileBackend\FileOps\StoreFileOp->doAttempt()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FileOpBatch.php(162): Wikimedia\FileBackend\FileOps\FileOp->attempt()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FileOpBatch.php(118): Wikimedia\FileBackend\FileOpBatch::runParallelBatches()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FileBackendStore.php(1333): Wikimedia\FileBackend\FileOpBatch::attempt()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FileBackend.php(505): Wikimedia\FileBackend\FileBackendStore->doOperationsInternal()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/filerepo/FileRepo.php(999): Wikimedia\FileBackend\FileBackend->doOperations()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/business/AccountRequestSubmission.php(228): FileRepo->storeBatch()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/frontend/specialpages/actions/RequestAccount_body.php(352): AccountRequestSubmission->submit()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/frontend/specialpages/actions/RequestAccount_body.php(92): RequestAccountPage->doSubmit()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPage.php(728): RequestAccountPage->execute()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPageFactory.php(1724): MediaWiki\SpecialPage\SpecialPage->run()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ActionEntryPoint.php(504): MediaWiki\SpecialPage\SpecialPageFactory->executePath()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ActionEntryPoint.php(146): MediaWiki\Actions\ActionEntryPoint->performRequest()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#18 {main}
</pre>
# well, shit, I definitely don't want MediaWiki to be executing chmod() commands!
# that error comes from here, line 920 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/FSFileBackend.php
<pre>
/**
* Chmod a file, suppressing the warnings
*
* @param string $fsPath Absolute file system path
* @return bool Success
*/
protected function chmod( $fsPath ) {
if ( $this->os === 'Windows' ) {
return true;
}
 
AtEase::suppressWarnings();
$ok = chmod( $fsPath, $this->fileMode );
AtEase::restoreWarnings();
 
return $ok;
}
</pre>
# I added this to the LocalSettings.php file and re-submitted the form
<pre>
if( ! function_exists("chmod") ){
function chmod(){
return;
}
}
</pre>
# the page said I needed to re-enter the CV and enter a new captcha. I did it, and it finally submitted!
<pre>
Your account request has been sent and is now pending review. A confirmation email has been sent to your email address.
 
Return to Main Page.
</pre>
# in another VM, lgged-in as my Maltfield admin account, I checked Special:ConfirmAccounts
# I saw the entry. It worked great!
# the only thing that was off was that it said the IP address was 127.0.0.1.
# I went ahead and approved the user, and created their account. it worked!
# I decided that it doesn't really make sense to setup the admin accounts after the migration; let's just put it in-place before the migration
# I edited these accounts, removing the 'administrator' role
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Audrey+Rampone
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Audrey+Rampone
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Elifarley
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Elifarley
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Hart
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Hart
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Will
## https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Will
# I also created an account for Catarina. It's called 'CatarinaAdmin', but it's not yet an admin
# I then tried to create the "CatarinaAdmin" user, but it said it already exists!
# after she confirms that she's secured the account (by logging-in and resetting the temp password that was mailed to her in plaintext), then I'll add it to the sysop group.
# I sent her an email
<pre>
<pre>
Hey Catarina,
Username entered already in use. Please choose a different name.
 
I created a new account for you on the live wiki (on hetzner2) called "CatarinaAdmin"
 
You should have received an email with your new account's temp password. The account isn't yet an admin. Once you've logged-in and changed your password, I'll promote it to be an admin.
 
Please generate a new long, 20+ random password in your password manager for this account, login, change the password, and setup 2FA.
 
* https://wiki.opensourceecology.org/wiki/Special:ChangeCredentials
* https://wiki.opensourceecology.org/wiki/Special:Manage_Two-factor_authentication
 
After you do this, please make a backup of your password manager and your 2FA app and store it them in a safe place.
 
Please let me know once you've logged-in and changed your password on your new 'CatarinaAdmin' account.
 
 
Thank you,
</pre>
</pre>
# I also sent an email to Marcin asking him to setup 2FA before we migrate
# It looks like CatarinaAdmin isn't in the admin group yet, though
# first I confirmed her email address is correct. Ugh, she has 3x accounts already with the same email address
<pre>
<pre>
Hey Marcin,
MariaDB [osewiki_db]> select user_name,user_email from wiki_user where user_name like '%Catarina%';
 
+---------------+--------------------------+
Can you please setup 2FA on your wiki account before we migrate?
| user_name    | user_email              |
 
+---------------+--------------------------+
I realized it would be better if you already setup 2FA *before* the migration -- otherwise you'll end-up with lots of confusing entries in your 2FA app for the wiki as we test-out snapshots on hetzner3.
| Catarina Mota | REDACTED |
 
| Catarina     | REDACTED |
* https://wiki.opensourceecology.org/wiki/Special:Manage_Two-factor_authentication
| Catarinamota | REDACTED |
 
| CatarinaAdmin | REDACTED |
After you do this, please make a backup of your 2FA app and store it them in a safe place (eg your veracrypt-encrypted usb drives)
+---------------+--------------------------+
 
4 rows in set (0.00 sec)
 
Thank you,
</pre>
# ah crap, I realized I created the CatarinaAdmin account in the wrong browser! It's on hetzner3
# I created it again on hetzner2. She might get two emails now. Whoops. Hopefully she figures it out.
# honestly I don't know what else to test. I think the wiki is basically ready to hand-off for validation to Marcin
# but before I do that, I want to go through the entire list of sites and migrate new snapshots over. I want to test all of our CHG tickets from start-to-finish and make sure all is well
# I think before I do that, I might also test my plan to do a double-migration of OBI and osemain – one that's' a static site and publicly accessible on the main IP and one that's the broken & updated wordpress site that's on some non-dns-tied IPv6 address
# oh, I guess I also need to do some other items, like granting Marcin and Tom Griffin ssh access to hetzner3
# one more thing: I noticed during the scrolling wall of text during the upgrade process that we have a bunch of spam on the wiki still
# many of them are named "quickbooks" just a search for this shows a ton of spam pages https://wiki.opensourceecology.org/index.php?search=quickbooks&title=Special%3ASearch&go=Go
# I figured it would be good to test the Nuke/mass delete special page, but it didn't work. I couldn't match any pages with "quickbooks" or "quickbooks%" or "%quickbooks%".
# I even found the (now-deleted) spam user that created them https://wiki.opensourceecology.org/wiki/User:Davis25Miller
# I can see ^ that user has a ton of not-yet-deleted spam pages, but I can't get them to show-up!
# I think the reason is that MassDelete/Nuke only shows "recent" changes. We might want to upgrade our capabilities by installing some better extension that can more easily find & delete this spam. Here's a few that may be of interest
## https://www.mediawiki.org/wiki/Extension:DeleteBatch
## https://www.mediawiki.org/wiki/Extension:SmiteSpam
# I sent Marcin an email asking for his input
<pre>
(sorry, forgot to uncheck the encryption button on the last email)
 
Hey Marcin,
 
I noticed you still have a lot of spam on the wiki. For example:
 
* https://wiki.opensourceecology.org/index.php?search=quickbooks
 
I was testing out the Nuke/MassDelete Special Page on the wiki on hetzner3, and I found that I couldn't query to find all of the pages named "%quickbooks%" for some reason. Even if I search by the user who created the pages, there were no results.
 
Do you know why? I'm guessing it's because the pages are not "recent".
 
Now might be a good time for me to look at installing some other extensions that might help with moderation.
 
Can you tell me about moderating spam on the wiki? What works? What doesn't work? Are there any moderation-helper extensions I should look at installing?
</pre>
# oh, I found a better page called "Deleted User Contributions" https://wiki.opensourceecology.org/wiki/Special:DeletedContributions
# here I can search for the deleted user and it does show all of the pages they created (lots of spam), but there's no way here to easily delete them all
# at the top is a link to "mass delete", but it just links back to Nuke – which again shows zero results https://wiki.opensourceecology.org/index.php?title=Special:Nuke&target=Davis25Miller
# oh, no, I'm wrong. That DeletedContributions page only shows the pages that were already deleted. For some reason there's like 20 pages that are missing
 
=Wed Feb 12, 2025=
 
# I realized that the reason we never enforced 2FA on the wiki was because that wasn't a supported option on the old version!
# they only recently added this; great!
# I was thinking that it would be helpful if Catarina could reset Marcin's 2FA the next time he breaks his phone, and I checked and saw that Catarina isn't and Admin on the wiki
# I sent Marcin an email asking to review the current list of admins, and asking if we could add Catarina and maybe should remove others
<pre>
Hey Marcin,
 
What are your thoughts on promoting Catarina's wiki account to Administrator (sysop)?
 
Per our previous discussion, I'm setting the wiki on hetzner3 to require 2FA for all Administrators.
 
I do hope that you'll setup backups so that if your phone breaks, you'll be able to self-restore all your 2FA accounts on your phone's TOTP-app-of-choice. But, even still, it would be a good idea to have another Administrator at FeF that can temp reset 2FA on your account.
 
I noticed today that Catarina is not currently an Administrator on the OSE wiki. She seems like the obvious person who can help you regain access to your account if you ever get locked out of the wiki (and vice-versa).
 
Also, here's a list of current administrators:
 
Audrey Rampone
Elifarley
Hart
Maltfield
Marcin
Tom Griffing
Will
 
We probably want to keep this list small.
 
What do you think about adding Catarina to the Administrator list, and should anyone be removed from the Administrator list?
</pre>
# ...
# I entered my hours & logs so far for Feb
# ...
# we left-off yesterday with a phpList error trying to hit this page https://phplist.opensourceecology.org/lists/
<pre>
Error: please make sure that index.php is your default document for a directory
 
If you have just installed PHPlist and get this message, make sure that your Apache configuration has somewhere
 
DirectoryIndex index.php index.html
 
or that at least index.php is mentioned before index.html
 
For other webservers please consult your manual to find how to make index.php be the default document for a directory.
 
Alternatively you can delete the file "index.html" in the lists directory of PHPlist
 
You probably want to be here or here.
</pre>
# attempting to hit the index.html returns the same error
# attempting to hit the index.php results in a 500 error with a blank page https://phplist.opensourceecology.org/lists/index.php
<pre>
user@disp9456:~$ curl -i https://phplist.opensourceecology.org/lists/index.php
HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Wed, 12 Feb 2025 23:06:22 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Referrer-Policy: strict-origin
X-VC-Req-Host: phplist.opensourceecology.org
X-VC-Req-URL: /lists/index.php
X-VC-Req-URL-Base: /lists/index.php
X-VC-Cacheable: NO:Not cacheable, ttl: 0.000
X-Varnish: 68745
Age: 0
Via: 1.1 varnish (Varnish/7.1)
 
user@disp9456:~$
</pre>
# there's no error logs being written to apache, which is an extremely annoying default behaviour of phpList iirc
# I believe I was actually the one who submitted the PR describing how to fix this here https://www.phplist.org/manual/books/phplist-manual/page/troubleshooting-techniques
<pre>
# enable error logging https://www.phplist.org/manual/books/phplist-manual/page/troubleshooting-techniques
cp ${docrootDir}/lists/admin/index.php ${docrootDir}/lists/admin/index.$(date "+%Y%m%d_%H%M%S").php
cp ${docrootDir}/lists/admin/init.php ${docrootDir}/lists/admin/init.$(date "+%Y%m%d_%H%M%S").php
sed -i 's/error_reporting(0)/error_reporting(1)/g' ${docrootDir}/lists/admin/init.php
sed -i 's/error_reporting(0)/error_reporting(1)/g' ${docrootDir}/lists/admin/init.php
</pre>
# ok, now I do get an error on page refresh
<pre>
==> phplist.opensourceecology.org/error.log <==
[Wed Feb 12 23:19:05.443701 2025] [proxy_fcgi:error] [pid 237609:tid 237609] [client 146.70.116.205:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function ini_set() in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/init.php:17\nStack trace:\n#0 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php(96): require_once()\n#1 {main}\n  thrown in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/init.php on line 17'
</pre>
# fixed with
<pre>
# fix phpList bugs
# TODO: fix URL to bug report, after creation
grep 'ini_set' ${vhostDir}/config.php || sed -i 's%^<?php%<?php\n# fix mediawiki bugs\n# * https://phabricator.wikimedia.org/T385965\nif( ! function_exists("ini_set") ){\n\tfunction ini_set(){\n\t\treturn;\n\t}\n}\n%' ${vhostDir}/config.php
</pre>
# that was followed-up with another fail
<pre>
==> phplist.opensourceecology.org/error.log <==
[Wed Feb 12 23:25:28.388693 2025] [proxy_fcgi:error] [pid 237624:tid 237624] [client 146.70.116.205:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function parse_ini_file() in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php:19\nStack trace:\n#0 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php(103): include_once()\n#1 {main}\n  thrown in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php on line 19'
</pre>
# and then we had
<pre>
==> phplist.opensourceecology.org/error.log <==
[Wed Feb 12 23:28:46.139645 2025] [proxy_fcgi:error] [pid 237607:tid 237607] [client 146.70.116.205:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function putenv() in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php:244\nStack trace:\n#0 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php(507): phplist_I18N->gettext()\n#1 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php(529): phplist_I18N->getTranslation()\n#2 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php(571): phplist_I18N->get()\n#3 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/defaultconfig.php(106): s()\n#4 /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/index.php(104): require_once('...')\n#5 {main}\n  thrown in /var/www/html/phplist.opensourceecology.org/public_html/lists/admin/languages.php on line 244'
</pre>
# ok, after this it's working
<pre>
# fix phpList bugs
# TODO: fix URL to bug report, after creation
grep 'ini_set' ${vhostDir}/config.php || sed -i 's%^<?php%<?php\n# fix mediawiki bugs\n# * https://phabricator.wikimedia.org/T385965\nif( ! function_exists("ini_set") ){\n\tfunction ini_set(){\n\t\treturn;\n\t}\n}\nif( ! function_exists("parse_ini_file") ){\n\tfunction parse_ini_file(){\n\t\treturn;\n\t}\n}\nif( ! function_exists("putenv") ){\n\tfunction putenv(){\n\t\treturn;\n\t}\n}\n%' ${vhostDir}/config.php
</pre>
# this loads, and I confirmed that it has our styles (eg colors and OSE logo) https://phplist.opensourceecology.org/lists/index.php
# ..but clicking the links returns the same errors; we need to fix the Index to include index.php somehow
# oh, that file's contents *is* index.html
<pre>
root@hetzner3 /var/www/html/phplist.opensourceecology.org # cat public_html/lists/index.html
<html>
<head><title>Nothing here</title></head>
<body>
<h1>Error: please make sure that index.php is your default document for a directory</h1>
<p>If you have just installed PHPlist and get this message, make sure that
your Apache configuration has somewhere
<pre>
<b>DirectoryIndex index.php index.html</b>
</pre>
or that at least index.php is mentioned before index.html
</p>
<p>For other webservers please consult your manual to find how to make
index.php be the default document for a directory.</p>
<p>Alternatively you can delete the file "index.html" in the lists directory of PHPlist</p>
 
<p>You probably want to be <a href="../">here</a> or <a href="admin/">here</a>.</p>
 
</body>
</html>
 
 
root@hetzner3 /var/www/html/phplist.opensourceecology.org #
</pre>
# so one option is to just delete all the 'index.html' files, but that's not what we did on hetzner2
<pre>
[root@opensourceecology phplist.opensourceecology.org]# ls public_html/lists/
admin  config  dl.php  images  index.html  index.php  js  lt.php  styles  texts  ut.php
[root@opensourceecology phplist.opensourceecology.org]#
</pre>
# I don't see this getting explicitly set on hetzner3
<pre>
[root@opensourceecology httpd]# grep -i index conf.d/00-phplist.opensourceecology.org.conf
Options -Indexes -Includes
[root@opensourceecology httpd]#
</pre>
# and it's not including anything
<pre>
[root@opensourceecology httpd]# grep -i include conf.d/00-phplist.opensourceecology.org.conf
#Include /etc/httpd/conf.d/ssl.opensourceecology.org
Options -Indexes -Includes
[root@opensourceecology httpd]#
</pre>
# I only found two other relevant places. I guess it's possible that the php one overrides the main one?
<pre>
[root@opensourceecology httpd]# grep -i index conf/httpd.conf
Options -Indexes -Includes
#    #  Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#    Options Indexes FollowSymLinks
# DirectoryIndex: sets the file that Apache will serve if a directory
DirectoryIndex index.html
[root@opensourceecology httpd]# grep -i index conf.d/php.conf
# Add index.php to the list of files that will be served as directory
# indexes.
DirectoryIndex index.php
[root@opensourceecology httpd]#
</pre>
# this could be a consequence of switching for mod_php to php-fpm
# anyway, I fixed this in ansible's apache role for the phplist vhost https://github.com/OpenSourceEcology/ansible/commit/b5bc261a8bd403a2e14ca77891ab3bad685b0e7f
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ git diff HEAD
...
diff --git a/hetzner3/roles/maltfield.apache/templates/phplist.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.apache/templates/phplist.opensourceecology.org.conf.j2
index e8e5411..cc27e23 100644
--- a/hetzner3/roles/maltfield.apache/templates/phplist.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.apache/templates/phplist.opensourceecology.org.conf.j2
@@ -26,6 +26,9 @@
<Directory "/var/www/html/phplist.opensourceecology.org/public_html">
Include 'conf-available/security.directory.include'
+
+              # phpList requires us to explicitly set this order
+              DirectoryIndex index.php index.html
</Directory>
Include 'conf-available/phplist.virtualhost.include'
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# it's looking good now
# I tried to signup as a new user, and I confirmed that I got the GDPR confirmation email in my inbox, so emails are working on the server
## I'm actually a bit surprised because the RDNS and MX records haven't all been migrated yet
# I logged-in. It worked.
# After login, it prompted me to update the DB. I clicked "Upgrade" https://phplist.opensourceecology.org/lists/admin/?page=upgrade&tk=REDACTED
# It demanded I make a backup; well that's implicitly been done before restoring this site on hetzner3. I clicked the "upgrade" button again
# after a few seconds, it said "success"
# I clicked around and, as far as I can tell, everything is working
# I'm calling this CHG process fine. Cool.
# oh, I need to create the ticket
# oh, looks like I opened a ticket for them about this in Oct 2024 https://github.com/phpList/phplist3/issues/1054
# I updated the ticket with the two additional functions, and I update our CHG wiki article with a link to the ticket
# ...
# ok, back to the wiki
# well, I'm still getting this error on trying to load the request account page https://wiki.opensourceecology.org/wiki/Special:RequestAccount
<pre>
[Z60-rmK6kgwUWA7e4A_UxQAAAAM] 2025-02-13 00:37:02: Fatal exception of type "UnderflowException"
</pre>
# curiously there's no errors getting written to the apache logs or the wiki-error.log file
# hmm..disk isn't full, but we've almost filled the disk
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            32G    0  32G  0% /dev
tmpfs          6,3G  1,1M  6,3G  1% /run
/dev/md2        436G  390G  24G  95% /
tmpfs            32G  80K  32G  1% /dev/shm
tmpfs          5,0M    0  5,0M  0% /run/lock
/dev/md1        989M  66M  872M  8% /boot
tmpfs          6,3G    0  6,3G  0% /run/user/1000
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
 
=Tue Feb 11, 2025=
 
# one person responded to my question about phpList saying I should use the sourceforge version, not the github version https://discuss.phplist.org/t/should-we-download-releases-from-github-or-sourceforge/9983
## they're not from the official dev team, but it took 5 days just to get *any* reply. the forums are pretty dead; I guess we take what we can get.
# the docs also said to download from sourceforge, so let's proceed with that
# here's the doc page describing how to update phpList https://www.phplist.org/manual/books/phplist-manual/page/upgrading-a-manual-installation
## this is actually pretty shitty; it basically says the only file we need to keep from before is config.php
## but I KNOW ^ that isn't true, because it doesn't even mention the image uploads dir
<pre>
[maltfield@opensourceecology ~]$ find /var/www/html/phplist.opensourceecology.org/public_html/uploadimages/
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/800px-OSE_logo_2014-grey-1.png
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/osemail_header_logo_2012.png
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/files
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/.htaccess
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/like-glyph.png
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/forward-glyph.png
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/.thumbs
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/.thumbs/files
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/.thumbs/image
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/.thumbs/image/hello.jpg
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/tweet-glyph.png
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/image
/var/www/html/phplist.opensourceecology.org/public_html/uploadimages/image/hello.jpg
[maltfield@opensourceecology ~]$
</pre>
# I guess we'll just have to trial and error
# I created a CHG article on the wiki for this phplist migration from hetzer2 to hetnzer3 https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_phplist_to_hetzner3
# I migrated a snapshot, fixed a few bugs in the commands in the wiki article above
# I uncommitted the phplist vhost and pushed it with ansible https://github.com/OpenSourceEcology/ansible/commit/9cf0e11f4ddaae813b8f216db52d630683b16f3f
# after pushing this, I now get this error trying to load the main page of phplist https://phplist.opensourceecology.org/lists/
<pre>
Error: please make sure that index.php is your default document for a directory
 
If you have just installed PHPlist and get this message, make sure that your Apache configuration has somewhere
 
DirectoryIndex index.php index.html
 
or that at least index.php is mentioned before index.html
 
For other webservers please consult your manual to find how to make index.php be the default document for a directory.
 
Alternatively you can delete the file "index.html" in the lists directory of PHPlist
 
You probably want to be here or here.
</pre>
 
=Sun Feb 09, 2025=
 
# yesterday I finished updating the wiki skins, extensions, and fixed read errors
# today let's see if I can login
# nope; login fails (before it even prompts me for the TOTP)
<pre>
[Z6jiQdBARKwS5eO0NyyPzwAAAAI] 2025-02-09 17:13:37: Fatal exception of type "DomainException"
</pre>
# strange, the apache error log doesn't have a cooresponding entry
# I confirmed that varnish *is* calling the apache backend, and it's returning with a 500 Internal Server Error
# the error log doesn't exist; I wonder if I create it, will it start writing to it?
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail LocalSettings.php
#
# note: do *not* enable any of the following variables, or it may leak
#      sensitive user or server data to web clients:
#        $wgDebugComments, $wgShowDBErrorBacktrace, $wgShowDebug,
#        $wgShowSQLErrors, $wgShowExceptionDetails
#
#    instead, just tail the log file outside the docroot:
#    `tail -f /var/www/html/wiki.opensourceecology.org/wiki-error.log`
 
$wgDebugLogFile = "/var/www/html/wiki.opensourceecology.org/wiki-error.log";
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls
cache  htdocs  LocalSettings.20250206_220118.php  LocalSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # touch wiki-error.log
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # chown not-apache:www-data wiki-error.log
root@hetzner3 /var/www/html/wiki.opensourceecology.org # chmod 0060 wiki-error.log
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# yep, that worked. I refresh the login page's POST in the browser, and now there's 91 lines of data written to this file
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # wc -l wiki-error.log
91 wiki-error.log
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# alright, here's the exception
<pre>
[exception] [Z6jk6sI43HtXVYKRFwoLOQAAAAY] /index.php?title=Special:UserLogin&returnto=Special%3ASearch&returntoquery=search%3DCHG%26go%3DGo  DomainException: Invalid password policy config. No check defined for 'PasswordCannotBePopular'.
</pre>
# looks like that's from our "HARDENING" section of LocalSettings.php
<pre>
#############
# HARDENING #
#############
 
$wgSecureLogin = true;
$wgSecretKey = 'REDACTED';
 
# PASSWORD POLICIES
 
$wgPasswordPolicy['policies']['default']['MinimalPasswordLength'] = 10;
$wgPasswordPolicy['policies']['default']['PasswordCannotBePopular'] = PHP_INT_MAX;
$wgPasswordPolicy['policies']['default']['PasswordCannotMatchUsername'] = true;
 
$wgPasswordPolicy['policies']['sysop']['MinimalPasswordLength'] = 20;
</pre>
# according to the docs, PasswordCannotBePopular was removed in MediaWiki v1.35 https://www.mediawiki.org/wiki/Manual:$wgPasswordPolicy
## it says to use "PasswordNotInCommonList" instead
# there's also a few new ones that I like
## PasswordCannotBeSubstringInUsername
## PasswordCannotMatchBlacklist
# here's a command to fix this
<pre>
# remove deprecated password policy and add new https://www.mediawiki.org/wiki/Manual:$wgPasswordPolicy
grep 'PasswordNotInCommonList' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%\\\$wgPasswordPolicy\['policies'\]\['default'\]\['PasswordCannotBePopular'\] = PHP_INT_MAX;%#\\\$wgPasswordPolicy['policies']['default']['PasswordCannotBePopular'] = PHP_INT_MAX;\n\\\$wgPasswordPolicy['policies']['default']['PasswordNotInCommonList'] = true;\n\\\$wgPasswordPolicy['policies']['default']['PasswordCannotMatchBlacklist'] = true;\n\\\$wgPasswordPolicy['policies']['default']['PasswordCannotBeSubstringInUsername'] = true;%igs" ${vhostDir}/LocalSettings.php
</pre>
# I refreshed the login POST in the browser, but now I have a new error in wiki-error.log
<pre>
[exception] [Z6jryiT2KExyu6R5tRR0twAAAAc] /index.php?title=Special:UserLogin&returnto=Special%3ASearch&returntoquery=search%3DCHG%26go%3DGo  DomainException: Invalid password policy config. No check defined for 'PasswordCannotMatchBlacklist'.
</pre>
# well, the docs say it's available but I'll just remove it manually
# I get an error on this one too!
<pre>
[exception] [Z6jsfUWy7Z1fzmrwWGWXiwAAAAA] /index.php?title=Special:UserLogin&returnto=Special%3ASearch&returntoquery=search%3DCHG%26go%3DGo  DomainException: Invalid password policy config. No check defined for 'PasswordCannotMatchUsername'.
</pre>
# ok, I'm wondering if it needs to be a sub-value 'value'
# my best-guess is that this was also removed (replaced by PasswordCannotBeSubstringInUsername), but the docs haven't been updated.
# anyway, I would guess that PasswordCannotBeSubstringInUsername would also apply for an exact match of username, so I'm going to use that one instead only
<pre>
# remove deprecated password policy and add new https://www.mediawiki.org/wiki/Manual:$wgPasswordPolicy
grep 'PasswordNotInCommonList' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%\\\$wgPasswordPolicy\['policies'\]\['default'\]\['PasswordCannotBePopular'\] = PHP_INT_MAX;\n\\\$wgPasswordPolicy\['policies'\]\['default'\]\['PasswordCannotMatchUsername'\] = true;%#\\\$wgPasswordPolicy['policies']['default']['PasswordCannotBePopular'] = PHP_INT_MAX;\n#\\\$wgPasswordPolicy['policies']['default']['PasswordCannotMatchUsername'] = true;\n\\\$wgPasswordPolicy['policies']['default']['PasswordNotInCommonList'] = true;\n\\\$wgPasswordPolicy['policies']['default']['PasswordCannotBeSubstringInUsername'] = true;%igs" ${vhostDir}/LocalSettings.php
</pre>
# I tried to edit the docs page's discussion tab, but I got an error again about no open proxies https://www.mediawiki.org/wiki/Manual_talk:$wgPasswordPolicy
## I already have an exception on my account for wikipedia, but I guess my mediawiki one expired. It usually takes them a few months to approve and it expires every year. Very frustrating.
# ok, after that I'm now prompted for the TOTP token
# and I'm able to login!
# I checked some special pages
# the 2fa page was just for my own account. I couldn't see how to require 2fa for privileged users
# I visited the ConfrmAccount list (of pending account requests). There's definitely spam in there. I hope that the transition to hCaptcha will reduce this.
# I also went ahead and created an account on the MediaWiki "phabricator" – to submit feature requests and bug reports
## I found there's already a feature request for mCaptcha support, opened last year https://phabricator.wikimedia.org/T376393
# I also opened a ticket about the ini_set and putenv bug https://phabricator.wikimedia.org/T385965
## To generate the error for the bug report, I had found that the php_uname function override was no longer necessary in the latest version of MediaWiki (1.43), so the bug report only mentions ini_set and putenv
# I went to this Special page, which told me that Marcin does *not* have 2FA enabled https://wiki.opensourceecology.org/wiki/Special:VerifyOATHForUser
# cool, this page translates the password policies I was configuring earlier https://wiki.opensourceecology.org/wiki/Special:PasswordPolicies
# ok, looks like the docs for 2fa in mediawiki are here https://www.mediawiki.org/wiki/Extension:OATHAuth#Configuration
# apparently I just enabled it before and never set it up; let's do that now
## looks like, by default, the range of tokens accepted deviates from the current time by 1+(2*4)* (30 seconds) = 270 seconds. So that's +/- 2.25 minutes.
### That actually doesn't seem too bad, but I'll increase it a bit to prevent login issues (which ultimately cause people to disable 2FA and leave their accounts less secure)
### Looks like the "Relaxed Mode" that we for Google Authenticator is +/- 4 minutes https://wordpress.org/plugins/google-authenticator-encourage-user-activation/
### So let's change this from 4 to 8 to get 1+(2*8)* (30 seconds) = 481 seconds = 241 seconds = +/- 4 minutes
<pre>
# Relaxed mode with +/- 4 minutes time drift tolerance. This is important or users will complain and/or disable 2FA entirely
# The security consequences are small. The usability benefits are huge.
$wgOATHAuthWindowRadius = 8
 
# make admins require 2FA
$wgOATHRequiredForGroups = ['sysop', 'interface-admin', 'bureaucrat', 'suppress', 'widgeteditor' ]
 
# make "powerful" rights require 2FA
$wgOATHExclusiveRights =  [ 'apihighlimits', 'applychangetags', 'autoconfirmed', 'autopatrol', 'bigdelete', 'block', 'blockemail', 'bot', 'changetags', 'createaccount', 'createtalk', 'delete', 'deletechangetags', 'deletedhistory', 'deletedtext', 'deletelogentry', 'deleterevision', 'editcontentmodel', 'editinterface', 'editprotected', 'editsemiprotected', 'editsitecss', 'editsitejs', 'editsitejson', 'editusercss', 'edituserjs', 'edituserjson', 'hideuser', 'import', 'importupload', 'ipblock-exempt', 'managechangetags', 'markbotedits', 'mergehistory', 'move-categorypages', 'move-rootuserpages', 'move-subpages', 'nominornewtalk', 'noratelimit', 'patrol', 'protect', 'renameuser', 'reupload-shared', 'sendemail', 'suppressionlog', 'suppressredirect', 'suppressrevision', 'unblockself', 'unwatchedpages', 'userrights', ]
 
# full list for reference (taken from htdocs/includes/MainConfigSchema.php) https://www.mediawiki.org/wiki/Extension:OATHAuth#Configuration
# $wgOATHExclusiveRights =  [ 'apihighlimits', 'applychangetags', 'autoconfirmed', 'autopatrol', 'bigdelete', 'block', 'blockemail', 'bot', 'browsearchive', 'changetags', 'createaccount', 'createpage', 'createtalk', 'delete', 'deletechangetags', 'deletedhistory', 'deletedtext', 'deletelogentry', 'deleterevision', 'edit', 'editcontentmodel', 'editinterface', 'editmyoptions', 'editmyprivateinfo', 'editmyusercss', 'editmyuserjs', 'editmyuserjson', 'editmyuserjsredirect', 'editmywatchlist', 'editprotected', 'editsemiprotected', 'editsitecss', 'editsitejs', 'editsitejson', 'editusercss', 'edituserjs', 'edituserjson', 'hideuser', 'import', 'importupload', 'ipblock-exempt', 'managechangetags', 'markbotedits', 'mergehistory', 'minoredit', 'move', 'move-categorypages', 'movefile', 'move-rootuserpages', 'move-subpages', 'nominornewtalk', 'noratelimit', 'patrol', 'protect', 'read', 'renameuser', 'reupload', 'reupload-shared', 'rollback', 'sendemail', 'suppressionlog', 'suppressredirect', 'suppressrevision', 'unblockself', 'undelete', 'unwatchedpages', 'upload', 'userrights', 'viewmyprivateinfo', 'viewmywatchlist', 'viewsuppressed' ]
</pre>
# set with
<pre>
# configure OATH (2FA) requirements
grep 'wgOATHAuthWindowRadius' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%wfLoadExtension\( 'OATHAuth' \);%wfLoadExtension( 'OATHAuth' );\n\n# Relaxed mode with +/- 4 minutes time drift tolerance. This is important or users will complain and/or disable 2FA entirely\n# The security consequences are small. The usability benefits are huge.\n\\\$wgOATHAuthWindowRadius = 8;\n\n# make admins require 2FA\n\\\$wgOATHRequiredForGroups = ['sysop', 'interface-admin', 'bureaucrat', 'suppress', 'widgeteditor' ];\n\n# make 'powerful' rights require 2FA\n\\\$wgOATHExclusiveRights =  [ 'apihighlimits', 'applychangetags', 'autoconfirmed', 'autopatrol', 'bigdelete', 'block', 'blockemail', 'bot', 'changetags', 'createaccount', 'createtalk', 'delete', 'deletechangetags', 'deletedhistory', 'deletedtext', 'deletelogentry', 'deleterevision', 'editcontentmodel', 'editinterface', 'editprotected', 'editsemiprotected', 'editsitecss', 'editsitejs', 'editsitejson', 'editusercss', 'edituserjs', 'edituserjson', 'hideuser', 'import', 'importupload', 'ipblock-exempt', 'managechangetags', 'markbotedits', 'mergehistory', 'move-categorypages', 'move-rootuserpages', 'move-subpages', 'nominornewtalk', 'noratelimit', 'patrol', 'protect', 'renameuser', 'reupload-shared', 'sendemail', 'suppressionlog', 'suppressredirect', 'suppressrevision', 'unblockself', 'unwatchedpages', 'userrights', ];\n\n# full list for reference (taken from htdocs/includes/MainConfigSchema.php) https://www.mediawiki.org/wiki/Extension:OATHAuth#Configuration\n#\\\$wgOATHExclusiveRights =  [ 'apihighlimits', 'applychangetags', 'autoconfirmed', 'autopatrol', 'bigdelete', 'block', 'blockemail', 'bot', 'browsearchive', 'changetags', 'createaccount', 'createpage', 'createtalk', 'delete', 'deletechangetags', 'deletedhistory', 'deletedtext', 'deletelogentry', 'deleterevision', 'edit', 'editcontentmodel', 'editinterface', 'editmyoptions', 'editmyprivateinfo', 'editmyusercss', 'editmyuserjs', 'editmyuserjson', 'editmyuserjsredirect', 'editmywatchlist', 'editprotected', 'editsemiprotected', 'editsitecss', 'editsitejs', 'editsitejson', 'editusercss', 'edituserjs', 'edituserjson', 'hideuser', 'import', 'importupload', 'ipblock-exempt', 'managechangetags', 'markbotedits', 'mergehistory', 'minoredit', 'move', 'move-categorypages', 'movefile', 'move-rootuserpages', 'move-subpages', 'nominornewtalk', 'noratelimit', 'patrol', 'protect', 'read', 'renameuser', 'reupload', 'reupload-shared', 'rollback', 'sendemail', 'suppressionlog', 'suppressredirect', 'suppressrevision', 'unblockself', 'undelete', 'unwatchedpages', 'upload', 'userrights', 'viewmyprivateinfo', 'viewmywatchlist', 'viewsuppressed' ];\n%igs" ${vhostDir}/LocalSettings.php
</pre>
# I also wanted to understand why I keep getting de-auth'd (it appears to have happened both on hetzner2 and on hetzner3). There's a million reasons this could be
## I confirmed that session cookies are set to not expire on the server's php.ini config (session.cookie_lifetime = 0)
## I realized there's a "keep me logged in" checkbox on the login page. Miraculously, I never noticed this. I logged-out and logged-in again, this time with that set. It still might de-auth me, but we'll see if I'm still logged-in tomorrow.
## according to this doc, if the user does *not* check that "keep me logged-in" box, then they're de-auth'd after 1 hour of inactivity (by default). This can be changed by updating wgObjectCacheSessionExpiry https://www.mediawiki.org/wiki/Manual:$wgObjectCacheSessionExpiry
## see also https://www.mediawiki.org/wiki/Manual:$wgExtendedLoginCookieExpiration
## see also https://www.mediawiki.org/wiki/Manual:How_to_debug/Login_problems#For_site_administrators/2
# I tried to edit a page, but I got an error :(
<pre>
[Z6k4RyT2KExyu6R5tRR07QAAAAc] 2025-02-09 23:20:39: Fatal exception of type "Error"
</pre>
# there's no errors emitted to the apache error logs, but this was in the wiki-error.log
<pre>
[exception] [Z6k4dcI43HtXVYKRFwoLbAAAAAY] /index.php?title=File:Altfield_michael_2011_ohio1.jpg&action=submit  Error: Call to undefined function Wikimedia\RequestTimeout\Detail\set_time_limit()
</pre>
# damn, that's another bug. I fixed it by updating the bugfix lines at the top of LocalSettings.php with this
<pre>
if( ! function_exists("putenv") ){
function putenv(){
return;
}
}
</pre>
# ok, that fixed the edit issue
# I also confirmed a successful edit of the home page
# I then tried to signup for a new account, and it failed
<pre>
[Z6k8FtBARKwS5eO0NyyQSwAAAAI] 2025-02-09 23:36:54: Fatal exception of type "RuntimeException"
</pre>
# I got a ton of error in the apache error log. oh, actually it's just one long line (with 7,382 chars)
<pre>
root@hetzner3 /var/log/apache2 # tail -n1 wiki.opensourceecology.org/error.log | wc -m
7382
root@hetzner3 /var/log/apache2 #
 
root@hetzner3 /var/log/apache2 # tail -n1 wiki.opensourceecology.org/error.log | head -c4096
[Sun Feb 09 23:37:53.959463 2025] [proxy_fcgi:error] [pid 2315303:tid 2315303] [client 127.0.0.1:0] AH01071: Got error 'firmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find theroot@hetzner3 /var/log/apache2 #
 
root@hetzner3 /var/log/apache2 # tail -n1 wiki.opensourceecology.org/error.log | tail -c4096
mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30; PHP message: PHP Warning:  file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to enable it when you configured PHP? in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php on line 30', referer: https://wiki.opensourceecology.org/
root@hetzner3 /var/log/apache2 #
</pre>
# some discussion here suggets its a filesystem permission issue, but it would be nice if I knew where it was trying to read/write https://www.mediawiki.org/wiki/Extension_talk:ConfirmAccount/archive_3
## ah man, that's 11 years old those comments
# mwstore is apparently a reference to an internal URI "mwstore://", but I still don't know what it resolves-to https://www.mediawiki.org/wiki/Mwstore
# the wiki-error.log file has more info
<pre>
[error] [Z6k8UdkIYao5-602-ohiTQAAAAE] /wiki/Special:RequestAccount  PHP Warning: file_exists(): Unable to find the wrapper &quot;mwstore&quot; - did you forget to en
able it when you configured PHP?
#0 [internal function]: MWExceptionHandler::handleError()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/backend/ConfirmAccount.class.php(30): file_exists()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/business/AccountRequestSubmission.php(75): ConfirmAccount::runAutoMaintenance()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/frontend/specialpages/actions/RequestAccount_body.php(352): AccountRequestSubmis
sion->submit()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmAccount/includes/frontend/specialpages/actions/RequestAccount_body.php(92): RequestAccountPage->d
oSubmit()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPage.php(728): RequestAccountPage->execute()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPageFactory.php(1724): MediaWiki\SpecialPage\SpecialPage->run()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ActionEntryPoint.php(504): MediaWiki\SpecialPage\SpecialPageFactory->executePath()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ActionEntryPoint.php(146): MediaWiki\Actions\ActionEntryPoint->performRequest()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWikiEntryPoint.php(200): MediaWiki\Actions\ActionEntryPoint->execute()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#11 {main}
[rdbms] ConfirmAccount::runAutoMaintenance [0.132ms] localhost: DELETE FROM `wiki_account_requests` WHERE acr_id = '634'
[rdbms] ConfirmAccount::runAutoMaintenance [0.614ms] localhost: UPDATE  `wiki_account_requests` SET acr_rejected = '20250209233753',acr_user = 0,acr_comment = '(this request has automatically been discarded due to inactivity)',acr_deleted = 1 WHERE (acr_rejected IS NULL) AND (acr_registration < '20250110233753') AND (acr_held < '20250110233753' OR acr_held IS NULL)
[BlockManager] Create account block cache hit with key BlockCacheKey{request=#262,user=#372,replica}
[rdbms] MWExceptionHandler::rollbackPrimaryChanges [0.559ms] localhost: ROLLBACK
[exception] [Z6k8UdkIYao5-602-ohiTQAAAAE] /wiki/Special:RequestAccount  RuntimeException: GuzzleHttp requires cURL, the allow_url_fopen ini setting, or a custom HTTP handler.
</pre>
# GuzzleHTTP appears to be this php library, which I guess MediaWiki uses https://docs.guzzlephp.org/en/stable/overview.html
# It says PHP 7.2.5 is a min req, so I guess they didn't use it on the version we have installed on MediaWiki; it's new
# It says either it needs allow_url_fopen (which we disable for security reasons) or "a recent version of cURL >= 7.19.4 compiled with OpenSSL and zlib"
# well, we have curl 7.88 installed
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # dpkg -l | grep -i curl
ii  curl                          7.88.1-10+deb12u6                      amd64        command line tool for transferring data with URL syntax
ii  libcurl3-gnutls:amd64          7.88.1-10+deb12u6                      amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
ii  libcurl4:amd64                7.88.1-10+deb12u6                      amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
ii  python3-pycurl                7.45.2-3                                amd64        Python bindings to libcurl (Python 3)
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# but I guess we don't have the php bindings for it
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # apt-cache search php | grep -i curl
php-curl - CURL module for PHP [default]
php8.2-curl - CURL module for PHP
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I remembered encountering this issue recently, but I realized it was when I was doing the mitm on wordpress in a DispVM. I had to install 'php-curl' for wordpress https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Tue_Dec_31.2C_2024
# I updated the php role in ansible to install 'php-curl' as a depend
# cool, now it's installed
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # dpkg -l | grep -i curl
ii  curl                          7.88.1-10+deb12u6                      amd64        command line tool for transferring data with URL syntax
ii  libcurl3-gnutls:amd64          7.88.1-10+deb12u6                      amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
ii  libcurl4:amd64                7.88.1-10+deb12u6                      amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
ii  php-curl                      2:8.2+93                                all          CURL module for PHP [default]
ii  php8.2-curl                    8.2.26-1~deb12u1                        amd64        CURL module for PHP
ii  python3-pycurl                7.45.2-3                                amd64        Python bindings to libcurl (Python 3)
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I restarted apache and php and refreshed the POST in the browser, but I'm still getting the error
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # systemctl restart php8.2-fpm apache2
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # 
</pre>
# there's some more notes here suggesting that this mwstore error message is a warning, so it might be a red herring https://www.edegan.com/mediawiki/index.php?title=Research_Computing_Configuration&mobileaction=toggle_view_desktop
## yeah, it's a known/open bug since 2019 https://phabricator.wikimedia.org/T219859
# so I do think the issue is GuzzleHttp, but why doesn't it see that php-curl is available now?
# perhaps I have to enable it
# I see the mod files are present
<pre>
root@hetzner3 /etc/php/8.2 # ls mods-available/
apcu.ini      curl.ini  ffi.ini      gd.ini      intl.ini      mysqlnd.ini  pdo_mysql.ini  readline.ini  sockets.ini  sysvshm.ini    xml.ini        xsl.ini
calendar.ini  dom.ini  fileinfo.ini  gettext.ini  mbstring.ini  opcache.ini  phar.ini      shmop.ini      sysvmsg.ini  tideways.ini  xmlreader.ini
ctype.ini    exif.ini  ftp.ini      iconv.ini    mysqli.ini    pdo.ini      posix.ini      simplexml.ini  sysvsem.ini  tokenizer.ini  xmlwriter.ini
root@hetzner3 /etc/php/8.2 # cat mods-available/curl.ini
; configuration for php curl module
; priority=20
extension=curl.so
root@hetzner3 /etc/php/8.2 #
</pre>
# perhaps this "extension=curl" line needs to be uncommented
<pre>
root@hetzner3 /etc/php/8.2 # grep -ir curl apache2/php.ini
disable_functions = ini_set,php_uname,getmyuid,getmypid,passthru,leak,listen,diskfreespace,tmpfile,link,ignore_user_abord,shell_exec,dl,set_time_limit,exec,system,highlight_file,source,show_source,fpaththru,virtual,posix_ctermid,posix_getcwd,posix_getegid,posix_geteuid,posix_getgid,posix_getgrgid,posix_getgrnam,posix_getgroups,posix_getlogin,posix_getpgid,posix_getpgrp,posix_getpid,posix,_getppid,posix_getpwnam,posix_getpwuid,posix_getrlimit,posix_getsid,posix_getuid,posix_isatty,posix_kill,posix_mkfifo,posix_setegid,posix_seteuid,posix_setgid,posix_setpgid,posix_setsid,posix_setuid,posix_times,posix_ttyname,posix_uname,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,phpinfo,popen,curl_exec,curl_multi_exec,parse_ini_file,allow_url_fopen,allow_url_include,pcntl_exec,chgrp,chmod,chown,lchgrp,lchown,putenv
; The ldap extension must be before curl if OpenSSL 1.0.2 and OpenLDAP is used
;extension=curl
[curl]
; A default value for the CURLOPT_CAINFO option. This is required to be an
;curl.cainfo =
You have mail in /var/mail/root
root@hetzner3 /etc/php/8.2 # grep -ir curl fpm/php.ini
disable_functions = ini_set,php_uname,getmyuid,getmypid,passthru,leak,listen,diskfreespace,tmpfile,link,ignore_user_abord,shell_exec,dl,set_time_limit,exec,system,highlight_file,source,show_source,fpaththru,virtual,posix_ctermid,posix_getcwd,posix_getegid,posix_geteuid,posix_getgid,posix_getgrgid,posix_getgrnam,posix_getgroups,posix_getlogin,posix_getpgid,posix_getpgrp,posix_getpid,posix,_getppid,posix_getpwnam,posix_getpwuid,posix_getrlimit,posix_getsid,posix_getuid,posix_isatty,posix_kill,posix_mkfifo,posix_setegid,posix_seteuid,posix_setgid,posix_setpgid,posix_setsid,posix_setuid,posix_times,posix_ttyname,posix_uname,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,phpinfo,popen,curl_exec,curl_multi_exec,parse_ini_file,allow_url_fopen,allow_url_include,pcntl_exec,chgrp,chmod,chown,lchgrp,lchown,putenv
; The ldap extension must be before curl if OpenSSL 1.0.2 and OpenLDAP is used
;extension=curl
[curl]
; A default value for the CURLOPT_CAINFO option. This is required to be an
;curl.cainfo =
root@hetzner3 /etc/php/8.2 #
</pre>
# well I guess we have to choose if we want to expose either curl or allow_url_fopen
# the OWASP php hardening guide says to disable allow_url_fopen, but it does not say to disable curl https://cheatsheetseries.owasp.org/cheatsheets/PHP_Configuration_Cheat_Sheet.html
# import to note that we already have a firewall that prevents the web server from initiating public web requests, so even if we allow curl functions to run, it still *shouldn't* permit malicious code from downloading (and running) malicious assets (or uploading for exfiltration)
## that said, layered security is good. I found a bug at some point where ipset caused my firewall rules to get cleared, so that protection could disappear at some point. layered security is good.
# alright, I'm going to remove the curl functions from the disabled_functions list
# yeah, that worked
# I removed them from the fpm php.ini file, restarted fpm & apache, and refreshed the page. Now it tells me the hCaptcha is expired (yeah it is), but at least the error is gone.
# let's try to fix this in ansbile https://github.com/OpenSourceEcology/ansible/commit/8259a838b21a10e08db1130da60f9b0a05b43d39
# I pushed that out with ansible and tried again
# this time I solved the CAPTCHA and submitted it; I got an error again complaining about the CAPTCHA
<pre>
Incorrect or missing CAPTCHA.
</pre>
# I thought maybe it was stale since I first loaded the page, so I solved it again. I got the same error.
# god damn it, now the hCaptcha wants to make server-side queries using cURL (taken from wiki-errro.log)
<pre>
[http] POST https://hcaptcha.com/siteverify HTTP/1.1 - NULL cURL error 28: Failed to connect to hcaptcha.com port 443 after 5001 ms: Timeout was reached (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://hcaptcha.com/siteverify
[captcha] Unable to validate response: http-timed-out
</pre>
# well, I was thinking I might switch from a math CAPTCHA to a self-hosted image of characters (FancyCaptcha), but they both have the same "Low" efficacy rating on the docs https://www.mediawiki.org/wiki/Extension:ConfirmEdit#FancyCaptcha
# FancyCAPTCHA requires more depends and load on the server. If it's not more effective, then let's just switch back to SimpleCaptcha
# In the future, hopefully MediaWiki will support a self-hosted mCaptcha option. I left my $0.02 on both project's issues
## https://phabricator.wikimedia.org/T376393
## https://phabricator.wikimedia.org/T314352
## https://github.com/mCaptcha/mCaptcha/issues/87
# crap, changing back from "hCaptcha" to "SimpleCaptcha" lead to an error
<pre>
MediaWiki
SimpleCaptcha configuration error
 
MediaWiki is unable to load the extension SimpleCaptcha. Please check that the extension's name is correct and all of its files are properly installed.
 
Details:
Error reading /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/SimpleCaptcha/extension.json. filemtime(): stat failed for /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/SimpleCaptcha/extension.json.
 
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(77): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(283): wfLoadExtensions()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(85): require_once('...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(50): require('...')
#6 {main}
</pre>
# is there supposed to be some sort of extension.json file here?
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/extensions/ConfirmEdit/
AUTHORS.txt    captcha.py          ConfirmEdit.alias.php  FancyCaptcha  includes      README.md          SimpleCaptcha  Turnstile
badwordlist    CODE_OF_CONDUCT.md  COPYING                hCaptcha      maintenance    ReCaptchaNoCaptcha  tests
captcha-old.py  composer.json      extension.json        i18n          QuestyCaptcha  resources          tox.ini
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/extensions/ConfirmEdit/SimpleCaptcha/
resources  SimpleCaptcha.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/extensions/ConfirmEdit/hCaptcha/
COPYING  extension.json  i18n  includes  resources
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# if that's the case, then I guess these are our only options
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # find htdocs/extensions/ConfirmEdit/ | grep -i extension.json
htdocs/extensions/ConfirmEdit/ReCaptchaNoCaptcha/extension.json
htdocs/extensions/ConfirmEdit/extension.json
htdocs/extensions/ConfirmEdit/QuestyCaptcha/extension.json
htdocs/extensions/ConfirmEdit/FancyCaptcha/extension.json
htdocs/extensions/ConfirmEdit/Turnstile/extension.json
htdocs/extensions/ConfirmEdit/hCaptcha/extension.json
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# well, then I guess the only self-hosted options are "QuestyCaptcha" and "FancyCaptcha"
# refresh of Specail:RequestAccount has error
<pre>
[Z6lQ8xfDLrGT7Qv2czN2UwAAAAE] 2025-02-10 01:05:55: Fatal exception of type "UnderflowException"
</pre>
# and this from wiki-error.log
<pre>
[captcha] ConfirmEdit: new captcha session; [Null]
[rdbms] Wikimedia\Rdbms\LoadBalancer::reuseOrOpenConnectionForNewRef: reusing connection for 0/osewiki_db-wiki_
[rdbms] MediaWiki\Page\PageStore::getPageByNameViaLinkCache [0.051ms] localhost: SELECT  page_id,page_namespace,page_title,page_is_redirect,page_is_new,page_touched,p
age_links_updated,page_latest,page_len,page_content_model  FROM `wiki_page`    WHERE page_namespace = 8 AND page_title = 'Requestaccount-page'  LIMIT 1 
[rdbms] Wikimedia\Rdbms\LoadBalancer::reuseOrOpenConnectionForNewRef: reusing connection for 0/osewiki_db-wiki_
[rdbms] MediaWiki\Parser\LinkHolderArray::replaceInternal [0.059ms] localhost: SELECT  page_id,page_namespace,page_title,page_is_redirect,page_is_new,page_latest,page
_touched,page_len,page_content_model  FROM `wiki_page`    WHERE ((page_namespace = 4 AND page_title = 'Terms_of_Service')) 
Cache miss for mwstore://captcha-backend/captcha-render captcha listing.
[silenced-error] [Z6lQ8xfDLrGT7Qv2czN2UwAAAAE] /wiki/Special:RequestAccount  PHP Warning: rmdir(/var/www/html/wiki.opensourceecology.org/htdocs/images/captcha): No s
uch file or directory
</pre>
# yeah, there is no images/captcha/ dir. And, wtf, it's owned by root:root; that's not going to work
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah htdocs/images/
total 356K
drwxr-xr-x 29 root    root    4,0K Dec 29 18:14 .
drwxr-xr-x 14 root    root    4,0K Dec 29 18:14 ..
drwxrwx--- 18 www-data www-data 4,0K Sep 18  2010 0
drwxrwx--- 18 www-data www-data 4,0K Sep 18  2010 1
drwxrwx--- 18 www-data www-data 4,0K Sep 18  2010 2
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 3
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 4
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 5
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 6
drwxrwx--- 18 www-data www-data 4,0K Sep 25  2010 7
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 8
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 9
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 a
drwxrwx--- 19 www-data www-data 4,0K Jun 21  2017 accountcreds
drwxrwx--- 18 www-data www-data 4,0K Jun 20  2017 accountreqs
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 archive
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 b
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 c
drwxrwx---  2 www-data www-data 4,0K Jun 19  2015 ConfirmAccount
-rw-rw----  1 www-data www-data  56K Jan 13  2010 cycle.jpg
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 d
drwxrwx--- 33 www-data www-data 4,0K May 14  2019 deleted
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 e
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 f
-rw-r--r--  1      501 staff    232 Dec  5 15:41 .htaccess
drwxrwx---  2 www-data www-data 4,0K Dec 22 06:54 lockdir
drwxrwx--- 12 www-data www-data 4,0K Jun  3  2011 math
-rw-rw----  1 www-data www-data  23K Oct  4  2010 ose-logo.png
-rw-rw----  1 www-data www-data  84K Jan 13  2010 products.jpg
drwxrwx---  2 www-data www-data 4,0K Sep 19  2010 proposal
-rw-r--r--  1      501 staff      84 Dec  5 15:41 README
drwxrwx--- 18 www-data www-data 4,0K May 24  2018 temp
drwxrwx--- 19 www-data www-data 4,0K Sep 20  2010 thumb
drwxrwx---  2 www-data www-data  68K Dec 23  2012 tmp
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I can't see why that would happen, considering our permissions script
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep -A40 MediaWiki /usr/local/bin/fix_web_permissions.sh
# MediaWiki #
#############
 
vhost_dir="/var/www/html/wiki.opensourceecology.org"
mw_docroot="${vhost_dir}/htdocs"
 
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
 
chown not-apache:apache-admins "${vhost_dir}/LocalSettings.php"
chmod 0040 "${vhost_dir}/LocalSettings.php"
 
[ -d "${mw_docroot}/images" ] || mkdir "${mw_docroot}/images"
chown -R www-data:www-data "${mw_docroot}/images"
find "${mw_docroot}/images" -type f -exec chmod 0660 {} \;
find "${mw_docroot}/images" -type d -exec chmod 0770 {} \;
 
[ -d "${vhost_dir}/cache" ] || mkdir "${vhost_dir}/cache"
chown -R www-data:www-data "${vhost_dir}/cache"
find "${vhost_dir}/cache" -type f -exec chmod 0660 {} \;
find "${vhost_dir}/cache" -type d -exec chmod 0770 {} \;
 
exit 0
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I gave it a manual try
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # vhost_dir="/var/www/html/wiki.opensourceecology.org"
mw_docroot="${vhost_dir}/htdocs"
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # echo $mw_docroot
/var/www/html/wiki.opensourceecology.org/htdocs
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls $mw_docroot
api.php            composer.json              docker-compose.yml  HISTORY      index.php  load.php          opensearch_desc.php  rest.php  thumb_handler.php
autoload.php        composer.local.json-sample  docs                images        INSTALL    LocalSettings.php  README.md            SECURITY  thumb.php
cache              COPYING                    extensions          img_auth.php  jsdoc.json  maintenance        RELEASE-NOTES-1.43  skins    UPGRADE
CODE_OF_CONDUCT.md  CREDITS                    FAQ                includes     languages  mw-config          resources            tests    vendor
root@hetzner3 /var/www/html/wiki.opensourceecology.org # [ -d "${mw_docroot}/images" ] || mkdir "${mw_docroot}/images"
chown -R www-data:www-data "${mw_docroot}/images"
find "${mw_docroot}/images" -type f -exec chmod 0660 {} \;
find "${mw_docroot}/images" -type d -exec chmod 0770 {} \;
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# the results are the same; wtf
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah htdocs/images/
total 356K
drwxr-xr-x 29 root    root    4,0K Dec 29 18:14 .
drwxr-xr-x 14 root    root    4,0K Dec 29 18:14 ..
drwxrwx--- 18 www-data www-data 4,0K Sep 18  2010 0
drwxrwx--- 18 www-data www-data 4,0K Sep 18  2010 1
drwxrwx--- 18 www-data www-data 4,0K Sep 18  2010 2
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 3
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 4
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 5
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 6
drwxrwx--- 18 www-data www-data 4,0K Sep 25  2010 7
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 8
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 9
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 a
drwxrwx--- 19 www-data www-data 4,0K Jun 21  2017 accountcreds
drwxrwx--- 18 www-data www-data 4,0K Jun 20  2017 accountreqs
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 archive
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 b
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 c
drwxrwx---  2 www-data www-data 4,0K Jun 19  2015 ConfirmAccount
-rw-rw----  1 www-data www-data  56K Jan 13  2010 cycle.jpg
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 d
drwxrwx--- 33 www-data www-data 4,0K May 14  2019 deleted
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 e
drwxrwx--- 18 www-data www-data 4,0K Sep 19  2010 f
-rw-r--r--  1      501 staff    232 Dec  5 15:41 .htaccess
drwxrwx---  2 www-data www-data 4,0K Dec 22 06:54 lockdir
drwxrwx--- 12 www-data www-data 4,0K Jun  3  2011 math
-rw-rw----  1 www-data www-data  23K Oct  4  2010 ose-logo.png
-rw-rw----  1 www-data www-data  84K Jan 13  2010 products.jpg
drwxrwx---  2 www-data www-data 4,0K Sep 19  2010 proposal
-rw-r--r-- 1      501 staff      84 Dec  5 15:41 README
drwxrwx--- 18 www-data www-data 4,0K May 24  2018 temp
drwxrwx--- 19 www-data www-data 4,0K Sep 20  2010 thumb
drwxrwx---  2 www-data www-data  68K Dec 23  2012 tmp
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
 
=Sat Feb 08, 2025=
 
# here's tofu 3/3 (ISP, exit in Ecuador)
<pre>
Ecuador
2025-02-08
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. . . .
INFO: Determining Latest Version of Wordpress Themes
. .
 
 
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/polylang.3.6.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/translatepress-multilingual.2.9.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-language-translator.6.0.20.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/gtranslate.3.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.1.zip
######################################################################### 100.0%
2025-02-08
296K google-language-translator.6.0.20.zip
700K gtranslate.3.0.7.zip
8.0M hestia.3.2.8.zip
7.1M neve.4.0.1.zip
64K plugin.json
592K polylang.3.6.6.zip
2.1M translatepress-multilingual.2.9.4.zip
28M wordpress-6.7.1.zip
01246c9c90f1373ee83d8e5884e5abba264c62e7301d99a468fd8eb95144be05  google-language-translator.6.0.20.zip
171eb362801ea28e74b302cd0fee472dbd5025a89f6c1634b5ca1029362a678c  gtranslate.3.0.7.zip
241b8c804ed1af72b1c9aa52f603730a52ebf7850383ac2e4d9dd163f6cfc3ca  hestia.3.2.8.zip
2b51e758d61b9d78ebd57d2afd9a967335bcb6866c5bff5a0d7157eecb6ec8cb  neve.4.0.1.zip
553e0118251211ef13964a88562bc6b859885f886811164d4a2751d9fe66f5e2  plugin.json
3a4a7f8872d16cb3538e948b4f85acee7d823c04fe8c820259ef6e2735093bbb  polylang.3.6.6.zip
16f2e24d14af341c6208fb9cc08d0c4db33bf813a5b1668930e30c6179a1111c  translatepress-multilingual.2.9.4.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@disp8537:/tmp/tmp.bSWwWjZNTu$
</pre>
# cool, all of these files match across all 3 TOFUs this time; I copied them to hetzer3
# ...
# back on the wiki, it seems that we need to update LocalSettings.php such that:
### the following skins are enabled: Cologne Blue, Modern, MonoBook, Vector
### and the following extensions are enabled: CategoryTree, ConfirmEdit, Confirm User Accounts, ReCaptcha, UserMerge, Widgets
# I used this command to add the skins
<pre>
# add skins (with wfLoadSkin() instead of require_once)
grep 'wfSkinLoad' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s/$wgDefaultSkin = 'Vector';/\$wgDefaultSkin = 'Vector';\n\nwfLoadSkin( 'CologneBlue' );\nwfLoadSkin( 'Modern' );\nwfLoadSkin( 'MonoBook' );\nwfLoadSkin( 'Vector' );/igs" ${vhostDir}/LocalSettings.php | grep -C4 wfLoadSkin
</pre>
# and this for the extensions
<pre>
# re-enable CategoryTree extension
grep 'wfLoadExtension.*CategoryTree' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once\("{\$IP}/extensions/CategoryTree/CategoryTree.php"\);%#require_once("{$IP}/extensions/CategoryTree/CategoryTree.php");\n\nwfLoadExtension( "CategoryTree" );\n%igs' ${vhostDir}/LocalSettings.php
 
# re-enable ConfirmEdit and switch to hCaptcha
grep 'hCaptcha' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e "s%\\\$wgCaptchaClass = 'SimpleCaptcha';%\#\\\$wgCaptchaClass = 'SimpleCaptcha';\n\nwfLoadExtensions([ 'ConfirmEdit', 'ConfirmEdit/hCaptcha' ]);\n\\\$wgHCaptchaSiteKey = 'your public/site key here';\n\\\$wgHCaptchaSecretKey = 'your private key here';\n%igs" ${vhostDir}/LocalSettings.php
 
# re-enable ConfirmAccount extension
grep 'wfLoadExtension.*ConfirmAccount' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once "\$IP/extensions/ConfirmAccount/ConfirmAccount.php";%#require_once "$IP/extensions/ConfirmAccount/ConfirmAccount.php";\n\nwfLoadExtension( "ConfirmAccount" );\n%igs' ${vhostDir}/LocalSettings.php
 
# re-enable UserMerge extension
sed -i 's%^\(\s*\)#wfLoadExtension\(.*\)UserMerge\(.*\)%\1wfLoadExtension\2UserMerge\3%' ${vhostDir}/LocalSettings.php
 
# re-enable Widgets extension
grep 'wfLoadExtension.*Widgets' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once\("\$IP/extensions/Widgets/Widgets.php"\);%#require_once("$IP/extensions/Widgets/Widgets.php");\n\nwfLoadExtension( "Widgets" );\n%igs' ${vhostDir}/LocalSettings.php
</pre>
# I also had an issue with the script to rsync over the extensions, but I fixed that in the CHG commands for migrating the wiki
# curiously the definition for ConfirmEdit also specifies a captcha
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep -iC2 confirmedit LocalSettings.php
 
 
# ConfirmEdit
# reCaptcha settings and keys
 
#wfLoadExtensions([ 'ConfirmEdit', 'ConfirmEdit/ReCaptcha' ]);
#$wgCaptchaClass = 'ReCaptcha';
$wgCaptchaClass = 'SimpleCaptcha';
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
<pre>
# looks like we disabled ReCaptcha and swiched to SimpleCaptcha
## the wiki article for this plugin says SimpleCaptcha presents a simple math challenge and isn't very effective https://www.mediawiki.org/wiki/Extension:ConfirmEdit#SimpleCaptcha_(calculation)
<pre>
Note that the display of a trivial maths problem as plaintext yields a captcha which can be trivially solved by automated means; as of 2012, sites using SimpleCaptcha are receiving significant amounts of spam and many automated registrations of spurious new accounts. Wikis currently using this as the default should therefore migrate to one of the other CAPTCHAs.
</pre>
## we should probably switch to hCaptcha, if anything https://www.mediawiki.org/wiki/Extension:ConfirmEdit#hCaptcha
## it does not support mCaptcha
## it does not support friendly captcha
## we do need an API key for hCAPTCHA, but (as far as I can tell) it's free https://www.hcaptcha.com/plans
# I created a new google account for this new service, and added its creds to our ose shared keepass
# I created a new account on the hcaptcha website using ^ that gapps account, and added its creds to our ose shared keepass
# I generated a new set of API keys (niknamed "wiki") and added the public "Site Key" and "Secret Key" to the notes section of the hcaptcha entry of our ose shared keepass
# I granted michael and marcin gapps accounts access to the new hcaptcha-service-specific gapps gmail account, and I set it to forward all its emails to our shared ops google group list per https://wiki.opensourceecology.org/wiki/Google_Workspace#Google_Groups
# ok, after enabling the skins & extensions && adding the hCaptcha creds to LocalSettings && clearing cache, I refreshed the Special:Version page Special:Version
<pre>
Installed software
Product Version
MediaWiki 1.43.0
PHP 8.2.26 (fpm-fcgi)
ICU 72.1
MariaDB 10.11.6-MariaDB-0+deb12u1
Entry point URLs
Entry point URL
Article path /wiki/$1
Script path /
index.php /index.php
api.php /api.php
rest.php /rest.php
Installed skins
Skin Version License Description Authors
Cologne Blue – (9134d2e) 06:04, 24 December 2024 GPL-2.0-or-later A lightweight skin with minimal formatting Lee Daniel Crocker and others
Modern – (5597681) 12:21, 14 December 2024 GPL-2.0-or-later A blue/gray skin with sidebar and top bar. Derived from MonoBook River Tarnell and others
MonoBook – GPL-2.0-or-later The classic MediaWiki skin since 2004, named after the black-and-white photo of a book in the page background Gabriel Wicke, Isarra Yos and others
Vector 1.0.0 GPL-2.0-or-later
 
Provides 2 Vector skins:
 
2011 - The Modern version of MonoBook with fresh look and many usability improvements.
2022 - The Vector built as part of the WMF Desktop Improvements project.
 
Readers Web Team, Trevor Parscal, Roan Kattouw, Alex Hollender, Bernard Wang, Clare Ming, Jan Drewniak, Jon Robson, Nick Ray, Sam Smith, Stephen Niedzielski and Volker E.
Installed extensions
Special pagesExtension Version License Description Authors
Confirm User Accounts – (bb470fd) 05:57, 31 December 2024 GPL-2.0-or-later Gives bureaucrats the ability to confirm account requests Aaron Schulz
Interwiki 3.2 GPL-2.0-or-later Adds a special page to view and edit the interwiki table Stephanie Amanda Stevens, Alexandre Emsenhuber, Robin Pepermans, Siebrand Mazeland, Platonides, Raimond Spekking, Sam Reed, Jack Phoenix, Calimonius the Estrange and others
Nuke – GPL-2.0-or-later Gives administrators the ability to mass delete pages Brion Vibber and Jeroen De Dauw
Replace Text 1.8 GPL-2.0-or-later Provides a special page to allow administrators to do a global string find-and-replace on all the content pages of a wiki Yaron Koren, Niklas Laxström and others
UserMerge 1.10.2 (f9d2664) 06:00, 31 December 2024 GPL-2.0-or-later Merges references from one user to another user in the wiki database - will also delete old users following merge. Requires usermerge privileges Tim Laqua, Thomas Gries and Matthew April
Parser hooksExtension Version License Description Authors
CategoryTree – GPL-2.0-or-later Dynamically navigate the category structure Daniel Kinzler
Cite – GPL-2.0-or-later Adds <ref> and <references> tags for citations Ævar Arnfjörð Bjarmason, Andrew Garrett, Brion Vibber, Ed Sanders, Marius Hoch, Steve Sanbeg, Trevor Parscal and others
Widgets 1.6.0 (50da5c6) 06:09, 26 November 2024 GPL-2.0-or-later Allows wiki administrators to add free-form widgets to the wiki by editing pages within the Widget namespace. Community-contributed widgets can be found on MediaWikiWidgets.org Sergey Chernyshev, Yaron Koren and others
Spam preventionExtension Version License Description Authors
ConfirmEdit 1.6.0 GPL-2.0-or-later Provides CAPTCHA techniques to protect against spam and password-guessing Brion Vibber, Florian Schmidt, Sam Reed and others
hCaptcha – GPL-2.0-or-later hCaptcha module for Confirm Edit Sam Reed and others
OtherExtension Version License Description Authors
Gadgets – GPL-2.0-or-later Lets users select custom CSS and JavaScript gadgets in their preferences Daniel Kinzler, Max Semenik, Timo Tijhof and Siddharth VP
OATHAuth 0.5.0 GPL-2.0-or-later AND GPL-3.0-or-later Provides authentication support using HMAC based one-time passwords Ryan Lane, Robert Vogel <vogel@hallowelt.com>, Dejan Savuljesku <savuljesku@hallowelt.com> and Taavi Väänänen
Installed libraries
Installed server-side libraries
Library Version License Description Authors
bacon/bacon-qr-code 2.0.8 BSD-2-Clause BaconQrCode is a QR code generator for PHP. Ben Scholzen 'DASPRiD'
christian-riesen/base32 1.6.0 MIT Base32 encoder/decoder according to RFC 4648 Christian Riesen
composer/semver 3.4.3 MIT Semver library that offers utilities, version constraint parsing and validation. Nils Adermann, Jordi Boggiano and Rob Bast
cssjanus/cssjanus 2.3.0 Apache-2.0 Convert CSS stylesheets between left-to-right and right-to-left. Roan Kattouw, Trevor Parscal and Timo Tijhof
dasprid/enum 1.0.5 BSD-2-Clause PHP 7.1 enum implementation Ben Scholzen 'DASPRiD'
endroid/qr-code 4.6.1 MIT Endroid QR Code Jeroen van den Enden
guzzlehttp/guzzle 7.9.2 MIT Guzzle is a PHP HTTP client library Graham Campbell, Michael Dowling, Jeremy Lindblom, George Mponos, Tobias Nyholm, Márk Sági-Kazár and Tobias Schultze
guzzlehttp/promises 2.0.4 MIT Guzzle promises library Graham Campbell, Michael Dowling, Tobias Nyholm and Tobias Schultze
guzzlehttp/psr7 2.7.0 MIT PSR-7 message implementation that also provides common utility methods Graham Campbell, Michael Dowling, George Mponos, Tobias Nyholm, Márk Sági-Kazár, Tobias Schultze and Márk Sági-Kazár
jakobo/hotp-php 2.0.0 BSD-3-Clause HOTP simplifies One Time Password systems for PHP Authentication Jakob Heuser
justinrainbow/json-schema 5.3.0 MIT A library to validate a json schema. Bruno Prieto Reis, Justin Rainbow, Igor Wiedler and Robert Schönthal
liuggio/statsd-php-client 1.0.18 MIT Statsd (Object Oriented) client library for PHP Giulio De Donato
mck89/peast 1.16.3 BSD-3-Clause Peast is PHP library that generates AST for JavaScript code Marco Marchiò
monolog/monolog 2.9.3 MIT Sends your logs to files, sockets, inboxes, databases and various web services Jordi Boggiano
oojs/oojs-ui 0.51.2 MIT Provides library of common widgets, layouts, and windows. Bartosz Dziewoński, Ed Sanders, James D. Forrester, Kirsten Menger-Anderson, Kunal Mehta, Moriel Schottlender, Prateek Saxena, Roan Kattouw, Thiemo Kreuz, Timo Tijhof, Trevor Parscal and Volker E.
pear/console_getopt 1.4.3 BSD-2-Clause More info available on: http://pear.php.net/package/Console_Getopt Andrei Zmievski, Stig Bakken and Greg Beaver
pear/mail 2.0.0 BSD-3-Clause Class that provides multiple interfaces for sending emails. Chuck Hagenbuch, Armin Graefe, Richard Heyes and Aleksander Machniak
pear/mail_mime 1.10.12 BSD-3-Clause Mail_Mime provides classes to create MIME messages Cipriano Groenendal and Aleksander Machniak
pear/net_smtp 1.12.1 BSD-2-Clause An implementation of the SMTP protocol Jon Parise, Chuck Hagenbuch and Armin Graefe
pear/net_socket 1.2.2 PHP License More info available on: http://pear.php.net/package/Net_Socket Chuck Hagenbuch, Aleksander Machniak and Stig Bakken
pear/net_url2 2.2.2 BSD-3-Clause Class for parsing and handling URL. Provides parsing of URLs into their constituent parts (scheme, host, path etc.), URL generation, and resolving of relative URLs. David Coallier, Tom Klingenberg and Christian Schmidt
pear/pear-core-minimal 1.10.15 BSD-3-Clause Minimal set of PEAR core files to be used as composer dependency Christian Weiske
pear/pear_exception 1.0.2 BSD-2-Clause The PEAR Exception base class. Helgi Thormar and Greg Beaver
psr/container 1.1.2 MIT Common Container Interface (PHP FIG PSR-11) PHP-FIG
psr/http-client 1.0.3 MIT Common interface for HTTP clients PHP-FIG
psr/http-factory 1.1.0 MIT PSR-17: Common interfaces for PSR-7 HTTP message factories PHP-FIG
psr/http-message 1.1 MIT Common interface for HTTP messages PHP-FIG
psr/log 1.1.4 MIT Common interface for logging libraries PHP-FIG
ralouphie/getallheaders 3.0.3 MIT A polyfill for getallheaders. Ralph Khattar
smarty/smarty 4.5.5 LGPL-3.0 Smarty - the compiling PHP template engine Monte Ohrt, Uwe Tews, Rodney Rehm and Simon Wisselink
symfony/deprecation-contracts 2.5.3 MIT A generic function and convention to trigger deprecation notices Nicolas Grekas and Symfony Community
symfony/polyfill-php80 1.31.0 MIT Symfony polyfill backporting some PHP 8.0+ features to lower PHP versions Ion Bazan, Nicolas Grekas and Symfony Community
symfony/polyfill-php81 1.31.0 MIT Symfony polyfill backporting some PHP 8.1+ features to lower PHP versions Nicolas Grekas and Symfony Community
symfony/polyfill-php82 1.31.0 MIT Symfony polyfill backporting some PHP 8.2+ features to lower PHP versions Nicolas Grekas and Symfony Community
symfony/polyfill-php83 1.31.0 MIT Symfony polyfill backporting some PHP 8.3+ features to lower PHP versions Nicolas Grekas and Symfony Community
symfony/yaml 5.4.45 MIT Loads and dumps YAML files Fabien Potencier and Symfony Community
wikimedia/assert 0.5.1 MIT Provides runtime assertions Daniel Kinzler and Thiemo Kreuz
wikimedia/at-ease 3.0.0 GPL-2.0-or-later Safe replacement to @ for suppressing warnings. Tim Starling and MediaWiki developers
wikimedia/base-convert 2.0.2 GPL-2.0-or-later Convert an arbitrarily-long string from one numeric base to another, optionally zero-padding to a minimum column width. Brion Vibber and Tyler Romeo
wikimedia/bcp-47-code 2.0.0 GPL-2.0-or-later Simple interface representing languages which have a BCP 47 code C. Scott Ananian
wikimedia/cdb 3.0.0 GPL-2.0-or-later Constant Database (CDB) wrapper library for PHP. Provides pure-PHP fallback when dba_* functions are absent. Tim Starling, Chad Horohoe, Ori Livneh and Daniel Kinzler
wikimedia/cldr-plural-rule-parser 2.0.0 GPL-2.0-or-later Evaluates plural rules specified in the CLDR project notation. Tim Starling and Niklas Laxström
wikimedia/common-passwords 0.5.0 MIT List of the 100,000 most commonly used passwords Sam Reed
wikimedia/composer-merge-plugin 2.1.0 MIT Composer plugin to merge multiple composer.json files Bryan Davis
wikimedia/equivset 1.7.0 GPL-2.0-or-later Visually Equivalent Set of UTF-8 Characters Brooke Vibber, David Barratt, Thiemo Kreuz and Umherirrender
wikimedia/html-formatter 4.1.0 GPL-2.0-or-later Performs transformations of HTML by wrapping around libxml2 and working around its countless bugs. MediaWiki contributors
wikimedia/idle-dom 1.0.0 MIT DOM interfaces automatically generated from WebIDL C. Scott Ananian
wikimedia/ip-utils 5.0.0 GPL-2.0-or-later Parse, match, and analyze IP addresses and CIDR ranges MediaWiki developers
wikimedia/json-codec 3.0.3 GPL-2.0-or-later Interfaces to serialize and deserialize PHP objects to/from JSON Petr Pchelko, Daniel Kinzler and C. Scott Ananian
wikimedia/less.php 5.1.2 Apache-2.0 PHP port of the LESS processor Timo Tijhof, Josh Schmidt, Matt Agar and Martin Jantošovič
wikimedia/minify 2.8.0 Apache-2.0 Minification of JavaScript code and CSS stylesheets. Paul Copperman, Trevor Parscal, Timo Tijhof and Roan Kattouw
wikimedia/normalized-exception 2.0.0 MIT A helper for making exceptions play nice with PSR-3 logging Gergő Tisza
wikimedia/object-factory 5.0.1 GPL-2.0-or-later Construct objects from configuration instructions Bryan Davis
wikimedia/parsoid 0.20.1 GPL-2.0-or-later Parsoid, a bidirectional parser between wikitext and HTML5 Wikimedia Content Transform Team and the broader MediaWiki community
wikimedia/php-session-serializer 3.0.0 GPL-2.0-or-later Provides methods like PHP's session_encode and session_decode that don't mess with $_SESSION Brad Jorsch
wikimedia/purtle 2.0.0 GPL-2.0-or-later Fast streaming RDF serializer Daniel Kinzler, Stanislav Malyshev, Thiemo Kreuz and C. Scott Ananian
wikimedia/relpath 4.0.1 MIT Work with file paths to join or find the relative path Ori Livneh
wikimedia/remex-html 4.1.1 MIT Fast HTML 5 parser Tim Starling
wikimedia/request-timeout 2.0.0 MIT Request timeout library for Excimer with plain PHP fallback Tim Starling
wikimedia/running-stat 2.1.0 GPL-2.0-or-later PHP implementations of online statistical algorithms Ori Livneh
wikimedia/scoped-callback 5.0.0 GPL-2.0-or-later Make a callback run when a dummy object leaves the scope. Aaron Schulz
wikimedia/services 4.0.0 GPL-2.0-or-later Generic service to manage named services using lazy instantiation based on instantiator callback functions Daniel Kinzler
wikimedia/shellbox 4.1.1 MIT Library and server for containerized shell execution Tim Starling, Kunal Mehta and Max Semenik
wikimedia/timestamp 4.1.1 GPL-2.0-or-later Creation, parsing, and conversion of timestamps Tyler Romeo
wikimedia/utfnormal 4.0.0 GPL-2.0-or-later Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present Brion Vibber
wikimedia/wait-condition-loop 2.0.2 GPL-2.0-or-later Wait loop that reaches a condition or times out Aaron Schulz
wikimedia/wikipeg 4.0.0 MIT Parser generator for JavaScript and PHP
wikimedia/wrappedstring 4.0.1 MIT Automatically compact sequentially-outputted strings that share a common prefix / suffix pair. Timo Tijhof
wikimedia/xmp-reader 0.9.4 GPL-2.0-or-later Reader for XMP data containing properties relevant to images Brian Wolff
wikimedia/zest-css 3.0.1 MIT Fast, lightweight, extensible CSS selector engine for PHP Christopher Jeffrey and C. Scott Ananian
zordius/lightncandy 1.2.6 MIT An extremely fast PHP implementation of handlebars ( http://handlebarsjs.com/ ) and mustache ( http://mustache.github.io/ ). Zordius Chen
Installed client-side libraries
Library Version License Authors Source
CLDRPluralRuleParser 1.3.1-0dda851 MIT Santhosh Thottingal MediaWiki
codex-design-tokens 1.14.0 GPL-2.0+ Design System team, Wikimedia Foundation MediaWiki
codex-icons 1.14.0 MIT Design System team, Wikimedia Foundation MediaWiki
codex 1.14.0 GPL-2.0+ Design System team, Wikimedia Foundation MediaWiki
fetch-polyfill 3.6.2 MIT GitHub, Inc. MediaWiki
intersection-observer 0.12.0 Apache-2.0 Philip Walton MediaWiki
jquery.chosen 1.8.2 MIT Patrick Filler for Harvest, Matthew Lettini, Patrick Filler, Ken Earley, Christophe Coevoet, Koen Punt, and T.J. Schuck. MediaWiki
jquery.client 3.0.0 MIT Trevor Parscal, Timo Tijhof, and Roan Kattouw MediaWiki
jquery.i18n 1.0.10 MIT OR GPL-2.0-or-later Language Engineering team, Wikimedia Foundation MediaWiki
jquery.ui 1.9.2 MIT OpenJS Foundation and other contributors MediaWiki
jquery 3.7.1 MIT OpenJS Foundation and other contributors MediaWiki
moment 2.25.2 MIT JS Foundation and other contributors MediaWiki
mustache 4.2.0 MIT Michael Jackson, Jan Lehnardt, Phillip Johnsen, and other contributors MediaWiki
oojs 7.0.1 MIT OOjs Team and other contributors MediaWiki
ooui 0.51.2 MIT OOUI Team and other contributors MediaWiki
pako 2.1.0 MIT AND Zlib Andrei Tuputcyn, Vitaly Puzrin, Friedel Ziegelmayer, Kirill Efimov, Jean-loup Gailly, and Mark Adler MediaWiki
pinia 2.0.16 MIT Eduardo San Martin Morote MediaWiki
qunitjs 2.20.0 MIT OpenJS Foundation and other contributors MediaWiki
sinonjs 1.17.7 BSD-3-Clause Christian Johansen and other contributors MediaWiki
swagger-ui 4.15.5 Apache-2.0 Kyle Shockey, Tony Tam, Vladimír Gorej, and others MediaWiki
url 3.111.0-0ece79ce32 MIT Financial Times MediaWiki
vue-demi 0.14.7 MIT Anthony Fu MediaWiki
vue 3.4.27 MIT Yuxi (Evan) You MediaWiki
vuex 4.0.2 MIT Yuxi (Evan) You MediaWiki
Parser extension tags
<categorytree>, <gallery>, <html>, <indicator>, <langconvert>, <nowiki>, <pre>, <ref> and <references>
Parser function hooks
{{anchorencode}}, {{BASEPAGENAME}}, {{BASEPAGENAMEE}}, {{#bcp47}}, {{bidi}}, {{canonicalurl}}, {{canonicalurle}}, {{CASCADINGSOURCES}}, {{#categorytree}}, {{DEFAULTSORT}}, {{#dir}}, {{DISPLAYTITLE}}, {{filepath}}, {{#FORMAL}}, {{#formatdate}}, {{formatnum}}, {{FULLPAGENAME}}, {{FULLPAGENAMEE}}, {{fullurl}}, {{fullurle}}, {{gender}}, {{grammar}}, {{int}}, {{#language}}, {{lc}}, {{lcfirst}}, {{localurl}}, {{localurle}}, {{NAMESPACE}}, {{NAMESPACEE}}, {{NAMESPACENUMBER}}, {{ns}}, {{nse}}, {{NUMBERINGROUP}}, {{NUMBEROFACTIVEUSERS}}, {{NUMBEROFADMINS}}, {{NUMBEROFARTICLES}}, {{NUMBEROFEDITS}}, {{NUMBEROFFILES}}, {{NUMBEROFPAGES}}, {{NUMBEROFUSERS}}, {{padleft}}, {{padright}}, {{pageid}}, {{PAGENAME}}, {{PAGENAMEE}}, {{PAGESINCATEGORY}}, {{PAGESIZE}}, {{plural}}, {{PROTECTIONEXPIRY}}, {{PROTECTIONLEVEL}}, {{REVISIONDAY}}, {{REVISIONDAY2}}, {{REVISIONID}}, {{REVISIONMONTH}}, {{REVISIONMONTH1}}, {{REVISIONTIMESTAMP}}, {{REVISIONUSER}}, {{REVISIONYEAR}}, {{ROOTPAGENAME}}, {{ROOTPAGENAMEE}}, {{#special}}, {{#speciale}}, {{SUBJECTPAGENAME}}, {{SUBJECTPAGENAMEE}}, {{SUBJECTSPACE}}, {{SUBJECTSPACEE}}, {{SUBPAGENAME}}, {{SUBPAGENAMEE}}, {{#tag}}, {{TALKPAGENAME}}, {{TALKPAGENAMEE}}, {{TALKSPACE}}, {{TALKSPACEE}}, {{uc}}, {{ucfirst}}, {{urlencode}} and {{#widget}}
</pre>
# side-note: the wiki actually looks nice now that the skins are fixed; and there's no error at the top of every page
# diffing that with hetzner2, and looks like we're still missing
## extension: Renameuser
## extension: ParserFunctions
# according to this page, ParserFunctions should be included with MediaWiki core. I guess it just doesn't show up as an "extension" anymore like some of the others that were merged with core https://www.mediawiki.org/wiki/Extension:ParserFunctions
<pre>
This extension comes with MediaWiki 1.18 and above. Thus you do not have to download it again. However, you still need to follow the other instructions provided.
</pre>
# oh, I think I need to activate it with
<pre>
wfLoadExtension( 'ParserFunctions' );
</pre>
# yeah, I'm missing this
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep -i parser LocalSettings.php
# ParserFunctions
#require_once( "$IP/extensions/ParserFunctions/ParserFunctions.php"  );
# Parser Cache should still use the DB per Aaron Schulz
$wgParserCacheType = CACHE_DB;
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
#  I added it with
<pre>
# re-enable ParserFunctions extension
grep 'wfLoadExtension.*ParserFunctions' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%#require_once\( "\$IP/extensions/ParserFunctions/ParserFunctions.php"  \);%#require_once( "$IP/extensions/ParserFunctions/ParserFunctions.php"  );\n\nwfLoadExtension( "ParserFunctions" );\n%igs' ${vhostDir}/LocalSettings.php
</pre>
# it looks like RenameUser was also merged to core https://www.mediawiki.org/wiki/Manual:Renameuser
# it's not clear if this one requires activation with wfLoadExtension() or not
# the way to check would be to login and go to Special:Renameuser
# well, it looks like the wiki is more broken than I realized
# clicking "Login" redirects to the osemain site for some reason
## http://opensourceecology.org/index.php?title=Special:UserLogin ->
## https://www.opensourceecology.org/
# and loading the frontpage has an error
<pre>
[Z6gB-0saz-Rit8IwiiNJYQAAAAk] 2025-02-09 01:16:44: Fatal exception of type "RuntimeException"
</pre>
# at the same time, this got spat-out to the apache error log
<pre>
[Sun Feb 09 01:17:04.465571 2025] [proxy_fcgi:error] [pid 2169915:tid 2169915] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Warning:  fopen(): open_basedir restriction in effect. File(/var/lib/php/tmp_upload/mw-GlobalIdGenerator-UUID-128) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp:/var/lib/php/sessions:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/staging.opensourceecology.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/var/www/html/store.opensourceecology.org:/var/www/html/microfactory.opensourceecology.org:/var/www/html/phplist.opensourceecology.org) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/uuid/GlobalIdGenerator.php on line 451; PHP message: PHP Warning:  fopen(/var/lib/php/tmp_upload/mw-GlobalIdGenerator-UUID-128): Failed to open stream: Operation not permitted in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/uuid/GlobalIdGenerator.php on line 451', referer: https://wiki.opensourceecology.org/
</pre>
# yeah, this looks like an error in my php.ini config
# on one hand, I defined 'upload_tmp_dir' to be '/var/lib/php/tmp_upload'
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ grep upload_tmp_dir roles/maltfield.php/templates/php_8.2.ini.j2
;upload_tmp_dir =
upload_tmp_dir = /var/lib/php/tmp_upload
user@ose:~/sandbox_local/ansible/hetzner3$ grep basedir roles/maltfield.php/templates/php_8.2.ini.j2
; open_basedir, if set, limits all file operations to the defined directory
; https://php.net/open-basedir
open_basedir = "/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp:/var/lib/php/sessions:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/staging.opensourceecology.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/var/www/html/store.opensourceecology.org:/var/www/html/microfactory.opensourceecology.org:/var/www/html/phplist.opensourceecology.org"
; Note: if open_basedir is set, the cache is disabled
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# but on another hand, that's not included in the open_basedir var
# I fixed this in ansible https://github.com/OpenSourceEcology/ansible/commit/01d8dd8ea0a49a4306c22d96eb5878907196f3f4
# I pushed it with ansible and restarted both php and apache
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # systemctl restart php8.2-fpm apache2
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# but I'm still getting an error
<pre>
[Sun Feb 09 07:15:57.201538 2025] [proxy_fcgi:error] [pid 2310677:tid 2310677] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Warning:  fopen(): open_basedir restriction in effect. File(/var/lib/php/tmp_upload/mw-GlobalIdGenerator-UUID-128) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp:/var/lib/php/tmp_upload:/var/lib/php/sessions:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/staging.opensourceecology.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/var/www/html/store.opensourceecology.org:/var/www/html/microfactory.opensourceecology.org:/var/www/html/phplist.opensourceecology.org) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/uuid/GlobalIdGenerator.php on line 451; PHP message: PHP Warning:  fopen(/var/lib/php/tmp_upload/mw-GlobalIdGenerator-UUID-128): Failed to open stream: Operation not permitted in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/uuid/GlobalIdGenerator.php on line 451'
</pre>
# it's curious that the diff between those last two error messages now shows that the relevant dir is, in fact, in the open_basedir list. what gives?
# oh, the dir doesn't exist!
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # ls -lah /var/lib/php
total 24K
drwxr-xr-x  6 root    root 4,0K Sep 25 01:23 .
drwxr-xr-x 36 root    root 4,0K Oct  3 04:29 ..
drwxr-xr-x  3 root    root 4,0K Sep 25 01:23 modules
drwx-wx-wt  2 root    root 4,0K Feb  8 23:39 sessions
drwx------  2 www-data root 4,0K Sep 25 01:23 soap_cache
drwx------  2 www-data root 4,0K Jan 30 22:26 tmp
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# I updated the php role in ansible to create this dir https://github.com/OpenSourceEcology/ansible/commit/66cdaec9f5fcd716f6d97295c946ce90dfb85f1c
# I pushed ansible, restarted apache & php, and cleared varnish
# this time the front page loads. horray!
# I clicked the "login" button, but I'm still being redirected to www.opensourceecology.org for some reason
# yeah, for some reason attempting to load the login page just 301 redirects to osemain
<pre>
user@disp1568:~$ curl -IL "https://opensourceecology.org/index.php?title=Special:UserLogin"
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sun, 09 Feb 2025 07:26:09 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://www.opensourceecology.org/index.php
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sun, 09 Feb 2025 07:25:09 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
Location: https://www.opensourceecology.org/
X-XSS-Protection: 1; mode=block
X-Varnish: 16885187 3647332
Age: 7994
Via: 1.1 varnish-v4
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 09 Feb 2025 07:25:09 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 56627
Connection: keep-alive
Link: <https://www.opensourceecology.org/wp-json/>; rel="https://api.w.org/"
Link: <https://www.opensourceecology.org/>; rel=shortlink
X-XSS-Protection: 1; mode=block
X-Varnish: 16885189 15314564
Age: 7994
Via: 1.1 varnish-v4
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
 
user@disp1568:~$
</pre>
# I tried the "Request Account" page instead. This one works.
## It also has the hCaptcha at the bottom of the page, which is working great too ☺
# wtf, I found that if I click the "login" link on the "register" page, then I can get the login page! https://wiki.opensourceecology.org/wiki/Special:UserLogin
# ah, duh, I just realized from the curl above that I was loading the naked domain. For some reason the big "log in" link on the top-right of the site is missing the 'wiki.' subdomain. That's the problem.
# when I search the LocalSettings.php file for 'Login', this looks suspicious
<pre>
###################
# FUNCTION HOOKDS #
###################
 
# See http://www.mediawiki.org/wiki/Manual:Hooks/GetLocalURL
 
function fnMakeAbsoluteURL( &$title, &$url, $query ) {
  global $wgServer;
  $indexRe = "/title=Special:.*Login|\/Special:Contact/";
  if ( preg_match( $indexRe, $url ) )
# $url = preg_replace( $indexRe, "", $url );
$url = $wgServer . $url;
  return true;
} 
$wgHooks['GetLocalURL'][] = 'fnMakeAbsoluteURL';
</pre>
# yeah, if I comment-out that block, then the issue goes away.
# ah, I think the issue here is that $wgServer is actually the naked domain
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep wgServer LocalSettings.php
$wgServer = "http://opensourceecology.org";
#$wgScriptPath      = "$wgServer/w";
#  global $wgServer;
#    $url = $wgServer . $url;
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# oh, looks like I introduced this bug because the wiki was totally broken if wgServer wasn't defined, so I had added to the CHG process to uncomment this line
<pre>
# uncomment wgServer (now defaults to 'false') https://www.mediawiki.org/wiki/Manual:$wgServer#Default_value
sed -i 's%^\(\s*\)#$wgServer\(.*\)%\1$wgServer\2%' ${vhostDir}/LocalSettings.php
</pre>
# but that defined it with http and the old naked domain, so I follow that up with this
<pre>
# fix $wgServer
grep '#$wgServer' ${vhostDir}/LocalSettings.php || perl -0777 -pi -e 's%\$wgServer = "http://opensourceecology.org";
%#\$wgServer = "http://opensourceecology.org";\n\$wgServer = "https://wiki.opensourceecology.org";\n%igs' ${vhostDir}/LocalSettings.php
</pre>
# now I can load the login page. Yay!
 
=Fri Feb 07, 2025=
 
# here's tofu 2/3 (VPN, exit in Hungary)
<pre>
Hungary
2025-02-07
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. . . .
INFO: Determining Latest Version of Wordpress Themes
. .
 
 
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/polylang.3.6.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/translatepress-multilingual.2.9.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-language-translator.6.0.20.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/gtranslate.3.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.1.zip
######################################################################### 100.0%
2025-02-07
296K google-language-translator.6.0.20.zip
700K gtranslate.3.0.7.zip
8.0M hestia.3.2.8.zip
7.1M neve.4.0.1.zip
64K plugin.json
592K polylang.3.6.6.zip
2.1M translatepress-multilingual.2.9.4.zip
28M wordpress-6.7.1.zip
01246c9c90f1373ee83d8e5884e5abba264c62e7301d99a468fd8eb95144be05  google-language-translator.6.0.20.zip
171eb362801ea28e74b302cd0fee472dbd5025a89f6c1634b5ca1029362a678c  gtranslate.3.0.7.zip
241b8c804ed1af72b1c9aa52f603730a52ebf7850383ac2e4d9dd163f6cfc3ca  hestia.3.2.8.zip
2b51e758d61b9d78ebd57d2afd9a967335bcb6866c5bff5a0d7157eecb6ec8cb  neve.4.0.1.zip
002bb1e14326a02cf2a8e06a0f3e0ca2326256fd9a5941a659b5f4bf9341ed78  plugin.json
3a4a7f8872d16cb3538e948b4f85acee7d823c04fe8c820259ef6e2735093bbb  polylang.3.6.6.zip
16f2e24d14af341c6208fb9cc08d0c4db33bf813a5b1668930e30c6179a1111c  translatepress-multilingual.2.9.4.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@disp8772:/tmp/tmp.7XNQH4tjs9$
</pre>
# ...
# back to the wiki
# yesterday I was finally able to confirm the upgrade to 1.35.0 on Special:Version https://wiki.opensourceecology.org/wiki/Special:Version
## the fix was to do a double-tap: first run the update script, then run the populateContentTables script, then run the update script again
# today I'll try to see if I can finally do the second upgrade from 1.35 to 1.43
# well, after installing the files and fixing permissions, second upgrade fails with error
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.43.log
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Cite extension cannot be loaded. Check that all of its files are installed properly.
 
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(282): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Cite/extension.json: filemtime(): stat failed for /Cite/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# yeah, I think somehow I copied-in an old LocalSettings.php file, but I don't understand how. I'll have to investigate this further with a future/fresh snapshot.
# For now I'm just going to manually copy these
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /var/tmp/CHG_20241228_wiki_1.35-to-1.43/pre/wiki.opensourceecology.org.20241228
cache  htdocs  LocalSettings.php  wiki.opensourceecology.org
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /var/tmp/CHG_20241228_wiki_1.35-to-1.43/pre/wiki.opensourceecology.org.20241228/wiki.opensourceecology.org/
cache  htdocs  LocalSettings.20250206_220118.php  LocalSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
 
root@hetzner3 /var/www/html/wiki.opensourceecology.org # rsync -av --progress /var/tmp/CHG_20241228_wiki_1.35-to-1.43/pre/wiki.opensourceecology.org.20241228/wiki.opensourceecology.org/LocalSettings.* /var/www/html/wiki.opensourceecology.org/
sending incremental file list
LocalSettings.20250206_220118.php
17.440 100%    0,00kB/s    0:00:00 (xfr#1, to-chk=1/2)
LocalSettings.php
17.946 100%  17,11MB/s    0:00:00 (xfr#2, to-chk=0/2)
 
sent 35.604 bytes  received 54 bytes  71.316,00 bytes/sec
total size is 35.386  speedup is 0,99
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# ok, that fixed that error
# after this upgrade is finished, I need to enumerate all of the skins and extensions that are currently active, diff that with the same output from hetzner2, and then update the LocalSettings.php file as-needed
# geez, it took 52 minutes for that upgrade to run, and it exited with an error
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.43.log
 
real    52m7,118s
user    0m0,004s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
<pre>
# here's the tail of the update log
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail -n30 /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.43.log
Table pagelinks contains pl_title field. Dropping...done.
Modifying page_links_updated field of table page...done.
Changing table options of 'searchindex'.
...migrating searchindex table...done.
Running MediaWiki\Extension\OATHAuth\Maintenance\UpdateTOTPScratchTokensToArray...
Done.
done.
Running MediaWiki\Extension\OATHAuth\Maintenance\UpdateForMultipleDevicesSupport...
Wikimedia\Rdbms\DBQueryError from line 1198 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php: Error 1146: Table 'osewiki_db.wiki_oathauth_types' doesn't exist
Function: MediaWiki\Extension\OATHAuth\OATHAuthModuleRegistry::getModuleIdsFromDatabase
Query: SELECT  oat_id,oat_name  FROM `wiki_oathauth_types`   
 
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1182): Wikimedia\Rdbms\Database->getQueryException()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1156): Wikimedia\Rdbms\Database->getQueryExceptionAndLog()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(647): Wikimedia\Rdbms\Database->reportQueryError()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1345): Wikimedia\Rdbms\Database->query()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\Database->select()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DBConnRef.php(351): Wikimedia\Rdbms\DBConnRef->__call()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(744): Wikimedia\Rdbms\DBConnRef->select()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/OATHAuth/src/OATHAuthModuleRegistry.php(132): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/OATHAuth/src/OATHAuthModuleRegistry.php(93): MediaWiki\Extension\OATHAuth\OATHAuthModuleRegistry->getModuleIdsFromDatabase()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php(56): MediaWiki\Extension\OATHAuth\OATHAuthModuleRegistry->getModuleIds()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(51): MediaWiki\Extension\OATHAuth\Maintenance\UpdateForMultipleDevicesSupport->doDBUpdates()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1140): MediaWiki\Maintenance\LoggedUpdateMaintenance->execute()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(595): MediaWiki\Installer\DatabaseUpdater->runMaintenance()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(552): MediaWiki\Installer\DatabaseUpdater->runUpdates()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(195): MediaWiki\Installer\DatabaseUpdater->doUpdates()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/MaintenanceRunner.php(703): UpdateMediaWiki->execute()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#17 {main}
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# The internet suggests that we should disable the 'oatuh' extension, but apparently that's required for 2fa – which we want https://www.mediawiki.org/wiki/Topic:Yakailc03f0u43c0
# looks like this OATHAuth is part of core
<pre>
root@hetzner3 /var/tmp/mediawiki # ls mediawiki-1.43.0/extensions/
AbuseFilter  CodeEditor      Gadgets    Linter            Nuke            PdfHandler  Scribunto              TemplateData    VisualEditor
CategoryTree  ConfirmEdit      ImageMap  LoginNotify      OATHAuth        Poem        SecureLinkFixer        TextExtracts    WikiEditor
Cite          DiscussionTools  InputBox  Math              PageImages      README      SpamBlacklist          Thanks
CiteThisPage  Echo            Interwiki  MultimediaViewer  ParserFunctions  ReplaceText  SyntaxHighlight_GeSHi  TitleBlacklist
root@hetzner3 /var/tmp/mediawiki #
 
root@hetzner3 /var/tmp/mediawiki # ls mediawiki-1.43.0/extensions/OATHAuth/
CODE_OF_CONDUCT.md  composer.json  COPYING  extension.json  i18n  maintenance  modules  OATHAuth.alias.php  ServiceWiring.php  sql  src  tests
root@hetzner3 /var/tmp/mediawiki #
</pre>
## one person in ^ that thread suggests they had to update to v1.41 before upgrading to 1.43. So I may have to do three updates :(
## actually, it reads like you can just do the second upgrade with the version of this one extension from 1.41 and then run the update script. So core doesn't need to be updated with 3 different versions, at least
## a couple people said you can get around this by running "(wiki folder)/extensions/OATHAuth/sql/sqlite/tables-generated.sql"
# this isn't a script; it's just an sql file
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat $docrootDir/extensions/OATHAuth/sql/sqlite/tables-generated.sql
-- This file is automatically generated using maintenance/generateSchemaSql.php.
-- Source: sql/tables.json
-- Do not modify this file directly.
-- See https://www.mediawiki.org/wiki/Manual:Schema_changes
CREATE TABLE /*_*/oathauth_types (
  oat_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
  oat_name BLOB NOT NULL
);
 
CREATE UNIQUE INDEX oat_name ON /*_*/oathauth_types (oat_name);
 
 
CREATE TABLE /*_*/oathauth_devices (
  oad_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
  oad_user INTEGER NOT NULL, oad_type INTEGER NOT NULL,
  oad_name BLOB DEFAULT NULL, oad_created BLOB DEFAULT NULL,
  oad_data BLOB DEFAULT NULL
);
 
CREATE INDEX oad_user ON /*_*/oathauth_devices (oad_user);
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# I tried running this using `mysql`, but it failed
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # mysql -u $dbUser -p${dbPass} $dbName < ${docrootDir}/extensions/OATHAuth/sql/sqlite/tables-generated.sql
ERROR 1064 (42000) at line 5: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NOT NULL,
  oat_name BLOB NOT NULL
)' at line 2
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
 
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat ${docrootDir}/extensions/OATHAuth/sql/sqlite/tables-generated.sql | mysql -u $dbUser -p${dbPass} $dbName
ERROR 1064 (42000) at line 5: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NOT NULL,
  oat_name BLOB NOT NULL
)' at line 2
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# I tried pasting it manually into the command-line, and I got the same result
<pre>
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1081219
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [osewiki_db]> CREATE TABLE /*_*/oathauth_types (
->  oat_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
->  oat_name BLOB NOT NULL
-> );
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NOT NULL,
  oat_name BLOB NOT NULL
)' at line 2
MariaDB [osewiki_db]>
</pre>
# after several trial-and-errors, I got this to work
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # mysql -u $dbUser -p${dbPass} $dbName
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1081332
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [osewiki_db]> CREATE TABLE wiki_oathauth_types (
->  oat_id INTEGER PRIMARY KEY AUTO_INCREMENT NOT NULL,
->  oat_name BLOB NOT NULL
-> );
Query OK, 0 rows affected (0,028 sec)


MariaDB [osewiki_db]>  
MariaDB [osewiki_db]>  
</pre>
#I tried many things, but I think what worked was changing AUTOINCREMENT to AUTO_INCREMENT, per https://mariadb.com/kb/en/create-table/#column-definitions
# I tried again, this time doing a sed substitue for AUTOINCREMENT to AUTO_INCREMENT, and this time I had no errors
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat $docrootDir/extensions/OATHAuth/sql/sqlite/tables-generated.sql | sed 's/AUTOINCREMENT/AUTO_INCREMENT/g' | mysql -u $dbUser -p${dbPass} $dbName
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# after this, I tried to run the upgrade scripts again, but it failed in 5 seconds with this error
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail -n30 /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.43.log
...ipblocks doesn't exist.
...pagelinks table does not contain pl_title field.
...page_links_updated in table page already modified by patch patch-page-page_links_updated-noinfinite.sql.
...searchindex table has already been migrated.
Running MediaWiki\Extension\OATHAuth\Maintenance\UpdateTOTPScratchTokensToArray...
...Update 'MediaWiki\Extension\OATHAuth\Maintenance\UpdateTOTPScratchTokensToArray' already logged as completed. Use --force to run it again.
done.
Running MediaWiki\Extension\OATHAuth\Maintenance\UpdateForMultipleDevicesSupport...
Wikimedia\Rdbms\DBQueryError from line 1198 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php: Error 1146: Table 'osewiki_db.wiki_oathauth_types' doesn't exist
Function: MediaWiki\Extension\OATHAuth\OATHAuthModuleRegistry::getModuleIdsFromDatabase
Query: SELECT  oat_id,oat_name  FROM `wiki_oathauth_types`   


#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1182): Wikimedia\Rdbms\Database->getQueryException()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1156): Wikimedia\Rdbms\Database->getQueryExceptionAndLog()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(647): Wikimedia\Rdbms\Database->reportQueryError()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1345): Wikimedia\Rdbms\Database->query()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\Database->select()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DBConnRef.php(351): Wikimedia\Rdbms\DBConnRef->__call()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(744): Wikimedia\Rdbms\DBConnRef->select()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/OATHAuth/src/OATHAuthModuleRegistry.php(132): Wikimedia\Rdbms\SelectQueryBuilder->fetchResultSet()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/OATHAuth/src/OATHAuthModuleRegistry.php(93): MediaWiki\Extension\OATHAuth\OATHAuthModuleRegistry->getModuleIdsFromDatabase()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php(56): MediaWiki\Extension\OATHAuth\OATHAuthModuleRegistry->getModuleIds()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(51): MediaWiki\Extension\OATHAuth\Maintenance\UpdateForMultipleDevicesSupport->doDBUpdates()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1140): MediaWiki\Maintenance\LoggedUpdateMaintenance->execute()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(595): MediaWiki\Installer\DatabaseUpdater->runMaintenance()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(552): MediaWiki\Installer\DatabaseUpdater->runUpdates()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(195): MediaWiki\Installer\DatabaseUpdater->doUpdates()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/MaintenanceRunner.php(703): UpdateMediaWiki->execute()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#17 {main}
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
</pre>
# I checked the db, and I see the tables it created are missing the 'wiki_' prefix. I guess that's what the '/*_*/' was for? not sure how that's *supposed* to work
# I used this special page to send her a new password https://wiki.opensourceecology.org/wiki/Special:PasswordReset
<pre>
# once she's able to confirm access, I'll grant it admin and ask her to setup 2FA.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # mysql -u $dbUser -p${dbPass} $dbName
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1081392
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [osewiki_db]> show tables limit 5;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'limit 5' at line 1
MariaDB [osewiki_db]> show tables;
+-----------------------------+
| Tables_in_osewiki_db        |
+-----------------------------+
| oathauth_devices            |
| oathauth_types              |
| wiki_account_credentials    |
| wiki_account_requests      |
...
</pre>
# a second sed to fix that
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat $docrootDir/extensions/OATHAuth/sql/sqlite/tables-generated.sql | sed 's/AUTOINCREMENT/AUTO_INCREMENT/g' | sed 's%/\*_\*/%wiki_%g'
-- This file is automatically generated using maintenance/generateSchemaSql.php.
-- Source: sql/tables.json
-- Do not modify this file directly.
-- See https://www.mediawiki.org/wiki/Manual:Schema_changes
CREATE TABLE wiki_oathauth_types (
  oat_id INTEGER PRIMARY KEY AUTO_INCREMENT NOT NULL,
  oat_name BLOB NOT NULL
);
 
CREATE UNIQUE INDEX oat_name ON wiki_oathauth_types (oat_name);
 
 
CREATE TABLE wiki_oathauth_devices (
  oad_id INTEGER PRIMARY KEY AUTO_INCREMENT NOT NULL,
  oad_user INTEGER NOT NULL, oad_type INTEGER NOT NULL,
  oad_name BLOB DEFAULT NULL, oad_created BLOB DEFAULT NULL,
  oad_data BLOB DEFAULT NULL
);
 
CREATE INDEX oad_user ON wiki_oathauth_devices (oad_user);
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# and we run it to add the DBs
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat $docrootDir/extensions/OATHAuth/sql/sqlite/tables-generated.sql | sed 's/AUTOINCREMENT/AUTO_INCREMENT/g' | sed 's%/\*_\*/%wiki_%g' | mysql -u $dbUser -p${dbPass} $dbName
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# it ran in 5 seconds again, but this time it exited with "Done" – no errors
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.43.log
 
real    0m5,290s
user    0m0,005s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# here's the run log output
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail -n30 /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.43.log
done.
...ipblocks doesn't exist.
...pagelinks table does not contain pl_title field.
...page_links_updated in table page already modified by patch patch-page-page_links_updated-noinfinite.sql.
...searchindex table has already been migrated.
Running MediaWiki\Extension\OATHAuth\Maintenance\UpdateTOTPScratchTokensToArray...
...Update 'MediaWiki\Extension\OATHAuth\Maintenance\UpdateTOTPScratchTokensToArray' already logged as completed. Use --force to run it again.
done.
Running MediaWiki\Extension\OATHAuth\Maintenance\UpdateForMultipleDevicesSupport...
Now processing rows with id between 0 and 500... (updated 0 users so far)
Now processing rows with id between 500 and 1000... (updated 0 users so far)
Now processing rows with id between 1000 and 1500... (updated 0 users so far)
Now processing rows with id between 1500 and 2000... (updated 0 users so far)
Now processing rows with id between 2000 and 2500... (updated 0 users so far)
Now processing rows with id between 2500 and 3000... (updated 0 users so far)
Now processing rows with id between 3000 and 3500... (updated 0 users so far)
Now processing rows with id between 3500 and 4000... (updated 0 users so far)
Done, updated data for 1 users.
done.
...oathauth_types table already exists.
Dropping table oathauth_users ...done.
...site_stats is populated...done.
...Update 'cleanup empty categories' already logged as completed. Use --force to run it again.
Fixing log entries with log_title starting with 'User:#'
...Processing unblock rows with IDs 16061 to 110080
done.
Set the local repo temp zone container to be private.
Purging caches...done.
 
Done in 0.1 s.
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I tried to update the thread with this https://www.mediawiki.org/wiki/Topic:Yakailc03f0u43c0
<pre>
> I've probably found solution about this. (maybe?) Before run update.php, run <code>(wiki folder)/extensions/OATHAuth/sql/sqlite/tables-generated.sql</code> manually
 
fwiw, we found that the sql file had errors
 
<pre>
ERROR 1064 (42000) at line 5: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NOT NULL,
  oat_name BLOB NOT NULL
)' at line 2
</pre>
 
The solution was to replace "AUTOINCREMENT" with "AUTO_INCREMENT".
 
And we also had to replace "/*_*/" with our actual table prefix "wiki_".
 
Running this command before the upgrade works for us
 
<pre>
cat $docrootDir/extensions/OATHAuth/sql/sqlite/tables-generated.sql | sed 's/AUTOINCREMENT/AUTO_INCREMENT/g' | sed 's%/\*_\*/%wiki_%g' | mysql -u $dbUser -p${dbPass} $dbName
</pre>
</pre>
# ...but I got an error
<pre>
'Your IP address is in a range that has been [[m:Special:MyLanguage/Global blocks|blocked on all Wikimedia Foundation wikis]].' The block was made by ‪JJMC89‬. The reason given is [[m:Special:MyLanguage/NOP|Open proxy/Webhost]]: See the [[m:WM:OP/H|help page]] if you are affected . * Start of block: 01:01, 3 July 2024 * Expiry of block: 01:01, 3 July 2027 Your current IP address is REDACTED. The blocked range is REDACTED/20. Please include all above details in any queries you make. If you believe you were blocked by mistake, you can find additional information and instructions in the No open proxies global policy. Otherwise, to discuss the block please post a request for review on Meta-Wiki. You could also send an email to the stewards VRT queue at "stewards@wikimedia.org" including all above details.
</pre>
# I tried to load the Special:Version, but it still says it's running v1.35.0 https://wiki.opensourceecology.org/wiki/Special:Version
# oh, I cleared the varnish cache, refreshed, and now it's saying it's v1.43.0. Great!
# ok, here's the contents of the version page now
<pre>
Installed software
Product Version
MediaWiki 1.43.0
PHP 8.2.26 (fpm-fcgi)
ICU 72.1
MariaDB 10.11.6-MariaDB-0+deb12u1
Entry point URLs
Entry point URL
Article path /wiki/$1
Script path /
index.php /index.php
api.php /api.php
rest.php /rest.php
Installed skins
 
No skins currently installed.
Installed extensions
Special pagesExtension Version License Description Authors
Interwiki 3.2 GPL-2.0-or-later Adds a special page to view and edit the interwiki table Stephanie Amanda Stevens, Alexandre Emsenhuber, Robin Pepermans, Siebrand Mazeland, Platonides, Raimond Spekking, Sam Reed, Jack Phoenix, Calimonius the Estrange and others
Nuke – GPL-2.0-or-later Gives administrators the ability to mass delete pages Brion Vibber and Jeroen De Dauw
Replace Text 1.8 GPL-2.0-or-later Provides a special page to allow administrators to do a global string find-and-replace on all the content pages of a wiki Yaron Koren, Niklas Laxström and others
Parser hooksExtension Version License Description Authors
Cite – GPL-2.0-or-later Adds <ref> and <references> tags for citations Ævar Arnfjörð Bjarmason, Andrew Garrett, Brion Vibber, Ed Sanders, Marius Hoch, Steve Sanbeg, Trevor Parscal and others
OtherExtension Version License Description Authors
Gadgets – GPL-2.0-or-later Lets users select custom CSS and JavaScript gadgets in their preferences Daniel Kinzler, Max Semenik, Timo Tijhof and Siddharth VP
OATHAuth 0.5.0 GPL-2.0-or-later AND GPL-3.0-or-later Provides authentication support using HMAC based one-time passwords Ryan Lane, Robert Vogel <vogel@hallowelt.com>, Dejan Savuljesku <savuljesku@hallowelt.com> and Taavi Väänänen
Installed libraries
Installed server-side libraries
Library Version License Description Authors
bacon/bacon-qr-code 2.0.8 BSD-2-Clause BaconQrCode is a QR code generator for PHP. Ben Scholzen 'DASPRiD'
christian-riesen/base32 1.6.0 MIT Base32 encoder/decoder according to RFC 4648 Christian Riesen
composer/semver 3.4.3 MIT Semver library that offers utilities, version constraint parsing and validation. Nils Adermann, Jordi Boggiano and Rob Bast
cssjanus/cssjanus 2.3.0 Apache-2.0 Convert CSS stylesheets between left-to-right and right-to-left. Roan Kattouw, Trevor Parscal and Timo Tijhof
dasprid/enum 1.0.5 BSD-2-Clause PHP 7.1 enum implementation Ben Scholzen 'DASPRiD'
endroid/qr-code 4.6.1 MIT Endroid QR Code Jeroen van den Enden
guzzlehttp/guzzle 7.9.2 MIT Guzzle is a PHP HTTP client library Graham Campbell, Michael Dowling, Jeremy Lindblom, George Mponos, Tobias Nyholm, Márk Sági-Kazár and Tobias Schultze
guzzlehttp/promises 2.0.4 MIT Guzzle promises library Graham Campbell, Michael Dowling, Tobias Nyholm and Tobias Schultze
guzzlehttp/psr7 2.7.0 MIT PSR-7 message implementation that also provides common utility methods Graham Campbell, Michael Dowling, George Mponos, Tobias Nyholm, Márk Sági-Kazár, Tobias Schultze and Márk Sági-Kazár
jakobo/hotp-php 2.0.0 BSD-3-Clause HOTP simplifies One Time Password systems for PHP Authentication Jakob Heuser
justinrainbow/json-schema 5.3.0 MIT A library to validate a json schema. Bruno Prieto Reis, Justin Rainbow, Igor Wiedler and Robert Schönthal
liuggio/statsd-php-client 1.0.18 MIT Statsd (Object Oriented) client library for PHP Giulio De Donato
mck89/peast 1.16.3 BSD-3-Clause Peast is PHP library that generates AST for JavaScript code Marco Marchiò
monolog/monolog 2.9.3 MIT Sends your logs to files, sockets, inboxes, databases and various web services Jordi Boggiano
oojs/oojs-ui 0.51.2 MIT Provides library of common widgets, layouts, and windows. Bartosz Dziewoński, Ed Sanders, James D. Forrester, Kirsten Menger-Anderson, Kunal Mehta, Moriel Schottlender, Prateek Saxena, Roan Kattouw, Thiemo Kreuz, Timo Tijhof, Trevor Parscal and Volker E.
pear/console_getopt 1.4.3 BSD-2-Clause More info available on: http://pear.php.net/package/Console_Getopt Andrei Zmievski, Stig Bakken and Greg Beaver
pear/mail 2.0.0 BSD-3-Clause Class that provides multiple interfaces for sending emails. Chuck Hagenbuch, Armin Graefe, Richard Heyes and Aleksander Machniak
pear/mail_mime 1.10.12 BSD-3-Clause Mail_Mime provides classes to create MIME messages Cipriano Groenendal and Aleksander Machniak
pear/net_smtp 1.12.1 BSD-2-Clause An implementation of the SMTP protocol Jon Parise, Chuck Hagenbuch and Armin Graefe
pear/net_socket 1.2.2 PHP License More info available on: http://pear.php.net/package/Net_Socket Chuck Hagenbuch, Aleksander Machniak and Stig Bakken
pear/net_url2 2.2.2 BSD-3-Clause Class for parsing and handling URL. Provides parsing of URLs into their constituent parts (scheme, host, path etc.), URL generation, and resolving of relative URLs. David Coallier, Tom Klingenberg and Christian Schmidt
pear/pear-core-minimal 1.10.15 BSD-3-Clause Minimal set of PEAR core files to be used as composer dependency Christian Weiske
pear/pear_exception 1.0.2 BSD-2-Clause The PEAR Exception base class. Helgi Thormar and Greg Beaver
psr/container 1.1.2 MIT Common Container Interface (PHP FIG PSR-11) PHP-FIG
psr/http-client 1.0.3 MIT Common interface for HTTP clients PHP-FIG
psr/http-factory 1.1.0 MIT PSR-17: Common interfaces for PSR-7 HTTP message factories PHP-FIG
psr/http-message 1.1 MIT Common interface for HTTP messages PHP-FIG
psr/log 1.1.4 MIT Common interface for logging libraries PHP-FIG
ralouphie/getallheaders 3.0.3 MIT A polyfill for getallheaders. Ralph Khattar
symfony/deprecation-contracts 2.5.3 MIT A generic function and convention to trigger deprecation notices Nicolas Grekas and Symfony Community
symfony/polyfill-php80 1.31.0 MIT Symfony polyfill backporting some PHP 8.0+ features to lower PHP versions Ion Bazan, Nicolas Grekas and Symfony Community
symfony/polyfill-php81 1.31.0 MIT Symfony polyfill backporting some PHP 8.1+ features to lower PHP versions Nicolas Grekas and Symfony Community
symfony/polyfill-php82 1.31.0 MIT Symfony polyfill backporting some PHP 8.2+ features to lower PHP versions Nicolas Grekas and Symfony Community
symfony/polyfill-php83 1.31.0 MIT Symfony polyfill backporting some PHP 8.3+ features to lower PHP versions Nicolas Grekas and Symfony Community
symfony/yaml 5.4.45 MIT Loads and dumps YAML files Fabien Potencier and Symfony Community
wikimedia/assert 0.5.1 MIT Provides runtime assertions Daniel Kinzler and Thiemo Kreuz
wikimedia/at-ease 3.0.0 GPL-2.0-or-later Safe replacement to @ for suppressing warnings. Tim Starling and MediaWiki developers
wikimedia/base-convert 2.0.2 GPL-2.0-or-later Convert an arbitrarily-long string from one numeric base to another, optionally zero-padding to a minimum column width. Brion Vibber and Tyler Romeo
wikimedia/bcp-47-code 2.0.0 GPL-2.0-or-later Simple interface representing languages which have a BCP 47 code C. Scott Ananian
wikimedia/cdb 3.0.0 GPL-2.0-or-later Constant Database (CDB) wrapper library for PHP. Provides pure-PHP fallback when dba_* functions are absent. Tim Starling, Chad Horohoe, Ori Livneh and Daniel Kinzler
wikimedia/cldr-plural-rule-parser 2.0.0 GPL-2.0-or-later Evaluates plural rules specified in the CLDR project notation. Tim Starling and Niklas Laxström
wikimedia/common-passwords 0.5.0 MIT List of the 100,000 most commonly used passwords Sam Reed
wikimedia/composer-merge-plugin 2.1.0 MIT Composer plugin to merge multiple composer.json files Bryan Davis
wikimedia/equivset 1.7.0 GPL-2.0-or-later Visually Equivalent Set of UTF-8 Characters Brooke Vibber, David Barratt, Thiemo Kreuz and Umherirrender
wikimedia/html-formatter 4.1.0 GPL-2.0-or-later Performs transformations of HTML by wrapping around libxml2 and working around its countless bugs. MediaWiki contributors
wikimedia/idle-dom 1.0.0 MIT DOM interfaces automatically generated from WebIDL C. Scott Ananian
wikimedia/ip-utils 5.0.0 GPL-2.0-or-later Parse, match, and analyze IP addresses and CIDR ranges MediaWiki developers
wikimedia/json-codec 3.0.3 GPL-2.0-or-later Interfaces to serialize and deserialize PHP objects to/from JSON Petr Pchelko, Daniel Kinzler and C. Scott Ananian
wikimedia/less.php 5.1.2 Apache-2.0 PHP port of the LESS processor Timo Tijhof, Josh Schmidt, Matt Agar and Martin Jantošovič
wikimedia/minify 2.8.0 Apache-2.0 Minification of JavaScript code and CSS stylesheets. Paul Copperman, Trevor Parscal, Timo Tijhof and Roan Kattouw
wikimedia/normalized-exception 2.0.0 MIT A helper for making exceptions play nice with PSR-3 logging Gergő Tisza
wikimedia/object-factory 5.0.1 GPL-2.0-or-later Construct objects from configuration instructions Bryan Davis
wikimedia/parsoid 0.20.1 GPL-2.0-or-later Parsoid, a bidirectional parser between wikitext and HTML5 Wikimedia Content Transform Team and the broader MediaWiki community
wikimedia/php-session-serializer 3.0.0 GPL-2.0-or-later Provides methods like PHP's session_encode and session_decode that don't mess with $_SESSION Brad Jorsch
wikimedia/purtle 2.0.0 GPL-2.0-or-later Fast streaming RDF serializer Daniel Kinzler, Stanislav Malyshev, Thiemo Kreuz and C. Scott Ananian
wikimedia/relpath 4.0.1 MIT Work with file paths to join or find the relative path Ori Livneh
wikimedia/remex-html 4.1.1 MIT Fast HTML 5 parser Tim Starling
wikimedia/request-timeout 2.0.0 MIT Request timeout library for Excimer with plain PHP fallback Tim Starling
wikimedia/running-stat 2.1.0 GPL-2.0-or-later PHP implementations of online statistical algorithms Ori Livneh
wikimedia/scoped-callback 5.0.0 GPL-2.0-or-later Make a callback run when a dummy object leaves the scope. Aaron Schulz
wikimedia/services 4.0.0 GPL-2.0-or-later Generic service to manage named services using lazy instantiation based on instantiator callback functions Daniel Kinzler
wikimedia/shellbox 4.1.1 MIT Library and server for containerized shell execution Tim Starling, Kunal Mehta and Max Semenik
wikimedia/timestamp 4.1.1 GPL-2.0-or-later Creation, parsing, and conversion of timestamps Tyler Romeo
wikimedia/utfnormal 4.0.0 GPL-2.0-or-later Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present Brion Vibber
wikimedia/wait-condition-loop 2.0.2 GPL-2.0-or-later Wait loop that reaches a condition or times out Aaron Schulz
wikimedia/wikipeg 4.0.0 MIT Parser generator for JavaScript and PHP
wikimedia/wrappedstring 4.0.1 MIT Automatically compact sequentially-outputted strings that share a common prefix / suffix pair. Timo Tijhof
wikimedia/xmp-reader 0.9.4 GPL-2.0-or-later Reader for XMP data containing properties relevant to images Brian Wolff
wikimedia/zest-css 3.0.1 MIT Fast, lightweight, extensible CSS selector engine for PHP Christopher Jeffrey and C. Scott Ananian
zordius/lightncandy 1.2.6 MIT An extremely fast PHP implementation of handlebars ( http://handlebarsjs.com/ ) and mustache ( http://mustache.github.io/ ). Zordius Chen
Installed client-side libraries
Library Version License Authors Source
CLDRPluralRuleParser 1.3.1-0dda851 MIT Santhosh Thottingal MediaWiki
codex-design-tokens 1.14.0 GPL-2.0+ Design System team, Wikimedia Foundation MediaWiki
codex-icons 1.14.0 MIT Design System team, Wikimedia Foundation MediaWiki
codex 1.14.0 GPL-2.0+ Design System team, Wikimedia Foundation MediaWiki
fetch-polyfill 3.6.2 MIT GitHub, Inc. MediaWiki
intersection-observer 0.12.0 Apache-2.0 Philip Walton MediaWiki
jquery.chosen 1.8.2 MIT Patrick Filler for Harvest, Matthew Lettini, Patrick Filler, Ken Earley, Christophe Coevoet, Koen Punt, and T.J. Schuck. MediaWiki
jquery.client 3.0.0 MIT Trevor Parscal, Timo Tijhof, and Roan Kattouw MediaWiki
jquery.i18n 1.0.10 MIT OR GPL-2.0-or-later Language Engineering team, Wikimedia Foundation MediaWiki
jquery.ui 1.9.2 MIT OpenJS Foundation and other contributors MediaWiki
jquery 3.7.1 MIT OpenJS Foundation and other contributors MediaWiki
moment 2.25.2 MIT JS Foundation and other contributors MediaWiki
mustache 4.2.0 MIT Michael Jackson, Jan Lehnardt, Phillip Johnsen, and other contributors MediaWiki
oojs 7.0.1 MIT OOjs Team and other contributors MediaWiki
ooui 0.51.2 MIT OOUI Team and other contributors MediaWiki
pako 2.1.0 MIT AND Zlib Andrei Tuputcyn, Vitaly Puzrin, Friedel Ziegelmayer, Kirill Efimov, Jean-loup Gailly, and Mark Adler MediaWiki
pinia 2.0.16 MIT Eduardo San Martin Morote MediaWiki
qunitjs 2.20.0 MIT OpenJS Foundation and other contributors MediaWiki
sinonjs 1.17.7 BSD-3-Clause Christian Johansen and other contributors MediaWiki
swagger-ui 4.15.5 Apache-2.0 Kyle Shockey, Tony Tam, Vladimír Gorej, and others MediaWiki
url 3.111.0-0ece79ce32 MIT Financial Times MediaWiki
vue-demi 0.14.7 MIT Anthony Fu MediaWiki
vue 3.4.27 MIT Yuxi (Evan) You MediaWiki
vuex 4.0.2 MIT Yuxi (Evan) You MediaWiki
Parser extension tags
<gallery>, <html>, <indicator>, <langconvert>, <nowiki>, <pre>, <ref> and <references>
Parser function hooks
{{anchorencode}}, {{BASEPAGENAME}}, {{BASEPAGENAMEE}}, {{#bcp47}}, {{bidi}}, {{canonicalurl}}, {{canonicalurle}}, {{CASCADINGSOURCES}}, {{DEFAULTSORT}}, {{#dir}}, {{DISPLAYTITLE}}, {{filepath}}, {{#FORMAL}}, {{#formatdate}}, {{formatnum}}, {{FULLPAGENAME}}, {{FULLPAGENAMEE}}, {{fullurl}}, {{fullurle}}, {{gender}}, {{grammar}}, {{int}}, {{#language}}, {{lc}}, {{lcfirst}}, {{localurl}}, {{localurle}}, {{NAMESPACE}}, {{NAMESPACEE}}, {{NAMESPACENUMBER}}, {{ns}}, {{nse}}, {{NUMBERINGROUP}}, {{NUMBEROFACTIVEUSERS}}, {{NUMBEROFADMINS}}, {{NUMBEROFARTICLES}}, {{NUMBEROFEDITS}}, {{NUMBEROFFILES}}, {{NUMBEROFPAGES}}, {{NUMBEROFUSERS}}, {{padleft}}, {{padright}}, {{pageid}}, {{PAGENAME}}, {{PAGENAMEE}}, {{PAGESINCATEGORY}}, {{PAGESIZE}}, {{plural}}, {{PROTECTIONEXPIRY}}, {{PROTECTIONLEVEL}}, {{REVISIONDAY}}, {{REVISIONDAY2}}, {{REVISIONID}}, {{REVISIONMONTH}}, {{REVISIONMONTH1}}, {{REVISIONTIMESTAMP}}, {{REVISIONUSER}}, {{REVISIONYEAR}}, {{ROOTPAGENAME}}, {{ROOTPAGENAMEE}}, {{#special}}, {{#speciale}}, {{SUBJECTPAGENAME}}, {{SUBJECTPAGENAMEE}}, {{SUBJECTSPACE}}, {{SUBJECTSPACEE}}, {{SUBPAGENAME}}, {{SUBPAGENAMEE}}, {{#tag}}, {{TALKPAGENAME}}, {{TALKPAGENAMEE}}, {{TALKSPACE}}, {{TALKSPACEE}}, {{uc}}, {{ucfirst}} and {{urlencode}}
</pre>
# important differences from above:
## skins
### hetzner2 lists the following skins: Cologne Blue, Modern, MonoBook, Vector
### hetzner3 lists 0 skins
## extensions
### hetzner2 lists CategoryTree, Cite, ConfirmEdit, Confirm User Accounts, Gadgets, Interwiki, Nuke, OATHAuth, ReCaptcha, Replace Text, UserMerge, Widgets
### hetzner3 lists Cite, Gadgets, Interwiki, Nuke, OATHAuth, and Replace Text
### therefore, we need to add the following missing extensions to hetzner3: CategoryTree, ConfirmEdit, Confirm User Accounts, ReCaptcha, UserMerge, Widgets
### but I also might want to look into seeing if I can find some extension that serves as a replacement for ReCaptcha, such as hCaptcha or mCaptcha or friendlycaptcha
# I still get an error at the top of the page complaining about the skin "Vector" not being installed; let's do that
<pre>
# UPDATE SKINS
 
rsync -av --progress /var/tmp/mediawiki/themes/CologneBlue ${docrootDir}/skins/
rsync -av --progress /var/tmp/mediawiki/themes/Modern ${docrootDir}/skins/
 
# UPDATE EXTENSIONS
 
extensions="ConfirmAccount UserMerge Widgets"
 
for extension in ${extensions}; do
extension_path="${docrootDir}/extensions/${extension}"
source_path="/var/tmp/wordpress/extensions/${extensions}"
       
if [ -d "${source_path}" ]; then
echo "${extension}"
rsync -a ${source_path}/ "${extension_path}/"
fi
done
 
# FIX PERMISSIONS
time /usr/local/bin/fix_web_permissions.sh
 
# clear varnish cache again
varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'
</pre>
# after that, I refreshed the wiki and I still have the error.
# here's the error
<pre>
Whoops! The default skin for your wiki, defined in $wgDefaultSkin as Vector, is not available.
 
Your installation seems to include the following skins. See Manual: Skin configuration for information how to enable them and choose the default.
 
cologneblue / CologneBlue (disabled)
minervaneue / MinervaNeue (disabled)
modern / Modern (disabled)
monobook / MonoBook (disabled)
timeless / Timeless (disabled)
vector / Vector (disabled)
 
If you have just installed MediaWiki
You probably installed from git, or directly from the source code using some other method. This is expected. Try installing some skins from mediawiki.org's skin directory, by:
 
Downloading the tarball installer, which comes with several skins and extensions. You can copy and paste the skins/ directory from it.
Downloading individual skin tarballs from mediawiki.org.
Using Git to download skins.
 
Doing this should not interfere with your git repository if you're a MediaWiki developer.
 
If you have just upgraded MediaWiki
MediaWiki 1.24 and newer no longer automatically enables installed skins (see Manual: Skin autodiscovery). You can paste the following lines into LocalSettings.php to enable all installed skins:
 
wfLoadSkin( 'CologneBlue' );
wfLoadSkin( 'MinervaNeue' );
wfLoadSkin( 'Modern' );
wfLoadSkin( 'MonoBook' );
wfLoadSkin( 'Timeless' );
wfLoadSkin( 'Vector' );
 
If you have just modified LocalSettings.php
Double-check the skin names for typos.
</pre>
# so it looks like installing the theme isn't enough, I have to activate it in LocalSettings.php
 
=Thr Feb 06, 2025=
 
# Here's TOFU 3/3 (ISP, exit in Ecuador)
<pre>
Ecuador
2025-02-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
.
INFO: Determining Latest Version of Wordpress Themes
. . . . .
 
 
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
  -=O=-                                    #  # ##
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
-=O=-                                #  #  #  # 
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/astra.4.8.11.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/generatepress.3.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/oceanwp.4.0.5.zip
######################################################################### 100.0%
2025-02-06
5.9M astra.4.8.11.zip
1.1M generatepress.3.5.1.zip
8.0M hestia.3.2.8.zip
7.1M neve.4.0.1.zip
6.0M oceanwp.4.0.5.zip
21M phplist-3.6.15.tgz
29M phplist-3.6.15.zip
32K plugin.json
4.6M v3.6.15.tar.gz
5.0M v3.6.15.zip
1.4M woocommerce-gateway-stripe.9.1.1.zip
28M wordpress-6.7.1.zip
0e106338e48c9d4d023a46a368f0069b807ce2118a9bf51ddf04c76070867aba  astra.4.8.11.zip
a5e1d7478cce21b7bbef511bbea44156f59acd40cc7e51ad469c403013ba29f5  generatepress.3.5.1.zip
241b8c804ed1af72b1c9aa52f603730a52ebf7850383ac2e4d9dd163f6cfc3ca  hestia.3.2.8.zip
2b51e758d61b9d78ebd57d2afd9a967335bcb6866c5bff5a0d7157eecb6ec8cb  neve.4.0.1.zip
d1392dbe5f729178c968f5551855a412b41866e876dcf2a79ffd4e07298c4d13  oceanwp.4.0.5.zip
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15.tgz
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15.zip
69708af323f3bad6654dd7b52bd385df4f0fa600d79261b608626ce7109136ef  plugin.json
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15.tar.gz
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15.zip
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@disp4288:/tmp/tmp.aqIzX10FIC$
</pre>
# meld diff of these 3 TOFUs had two issues
## hestia changed from v3.2.7 to v3.2.8
## neve changed from v4.0.0 to v4.0.1
# well that sucks; neve was the most important reason we were doing this.
# so let's do this again. And this time I'm adding some translation plugins.
## I recently did a ton of research (for my personal sites) for the best free wordpress i18nl plugins, and narrowed it down to 4, which I need to play-with
## so I'm mostly doing this TOFU for my own benefit, but I'll do the comparison for free and I'll probably install the winner (but not activate it) on all of OSE's wordpress sites, in-case they want to have posts or pages in >1 language in the future (useful when doing collaboration with German, Italian, Spanish, etc-speaking orgs)
# the most important result of this last 3TOFU is that we now finally have our phpList releases
# unfortunately, there's 4 files and they all differ
<pre>
user@ose:~/tmp/hetzner3/3tofu3$ sha256sum */*.zip
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15-zip/phplist-3.6.15.zip
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15-zip/v3.6.15.zip
user@ose:~/tmp/hetzner3/3tofu3$ sha256sum */*.tar.gz
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15-tar-gz/v3.6.15.tar.gz
user@ose:~/tmp/hetzner3/3tofu3$ sha256sum */*.tgz
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15-tgz/phplist-3.6.15.tgz
user@ose:~/tmp/hetzner3/3tofu3$
</pre>
# this is somewhat expected, especially from the compression algorithms of the archives being totally distinct
# I confirmed that he actual contents of phplist-3.6.15.zip and phplist-3.6.15.tgz are identical
<pre>
user@ose:~/tmp/hetzner3/3tofu3$ diff -r phplist-3.6.15-zip/phplist-3.6.15 phplist-3.6.15-tgz/phplist-3.6.15
user@ose:~/tmp/hetzner3/3tofu3$
</pre>
# and I also confirmed that the contents of the other set of files (I think these ones came from github -- as opposed to sourceforge) were also identical
<pre>
user@ose:~/tmp/hetzner3/3tofu3$ diff -r v3.6.15-zip/phplist3-3.6.15/ v3.6.15-tar-gz/phplist3-3.6.15/
user@ose:~/tmp/hetzner3/3tofu3$
</pre>
# but it looks like there's lots of differences between the ones named 'phplist-3.6.15' and the ones named 'v3.6.15'
<pre>
user@ose:~/tmp/hetzner3/3tofu3$ diff -r phplist-3.6.15-zip/phplist-3.6.15 v3.6.15-zip/phplist3-3.6.15/
Only in v3.6.15-zip/phplist3-3.6.15/bin: fake-sendmail.sh
Only in v3.6.15-zip/phplist3-3.6.15/bin: imgur-uploader.sh
Only in v3.6.15-zip/phplist3-3.6.15/bin: start-selenium
Only in v3.6.15-zip/phplist3-3.6.15/: composer.json
Only in v3.6.15-zip/phplist3-3.6.15/: composer.lock
Only in v3.6.15-zip/phplist3-3.6.15/: default.behat.yml
Only in v3.6.15-zip/phplist3-3.6.15/: .dotgitlab-ci.yml
Only in v3.6.15-zip/phplist3-3.6.15/: .github
Only in v3.6.15-zip/phplist3-3.6.15/: .gitignore
Only in v3.6.15-zip/phplist3-3.6.15/: .gitmodules
Only in v3.6.15-zip/phplist3-3.6.15/: .gitsvnextmodules
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin: help
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin: info
diff -r phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/init.php v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin/init.php
11c11,44
< define("VERSION","3.6.15");
---
> //## remove on rollout ###
> if (is_file(dirname(__FILE__).'/../../../VERSION')) {
>    $fd = fopen(dirname(__FILE__).'/../../../VERSION', 'r');
>    while ($line = fscanf($fd, '%[a-zA-Z0-9,. ]=%[a-zA-Z0-9,. ]')) {
>        list($key, $val) = $line;
>        if ($key == 'VERSION') {
>            $version = $val;
>        }
>    }
>    fclose($fd);
> } else {
>    $version = 'dev';
> }
>
> if (!defined('VERSION')) {
>    if (!ini_get('open_basedir') && is_dir(dirname(__FILE__).'/../../../.git')) {
>        define('VERSION', $version.'-dev');
>        define('DEVVERSION', true);
>    } else {
>        define('VERSION', $version);
>        define('DEVVERSION', false);
>    }
> } else {
>    define('DEVVERSION', false);
> }
>
> if (empty($GLOBALS['commandline']) && isset($GLOBALS['developer_email']) && $_SERVER['HTTP_HOST'] != 'dev.phplist.com' && !empty($GLOBALS['show_dev_errors'])) {
>    error_reporting(E_ALL);
>    ini_set('display_errors', 1);
>    foreach ($_REQUEST as $key => $val) {
>        unset($$key);
>    }
> }
> //## end remove on rollout ###
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: campaignslicer.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CaptchaPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CaptchaPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CKEditorPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CKEditorPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: Common
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CommonPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: CommonPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: COPYING.txt
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: dateplaceholder.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: disposablemailblock.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: domainthrottlemap.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: embedremoteimages.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: inviteplugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: SegmentPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: SegmentPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: subjectLinePlaceholdersPlugin.php
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: UpdaterPlugin
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/plugins: UpdaterPlugin.php
diff -r phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/structure.php v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin/structure.php
16c16
<    define('STRUCTUREVERSION',"3.6.15");
---
>    define('STRUCTUREVERSION', 'dev');
Only in v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin: tests
Only in v3.6.15-zip/phplist3-3.6.15/public_html/lists/admin/ui: default
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/admin/ui: phplist-ui-bootlist
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists: base
diff -r phplist-3.6.15-zip/phplist-3.6.15/public_html/lists/config/config.php v3.6.15-zip/phplist3-3.6.15/public_html/lists/config/config.php
35,38c35
< define('PHPMAILERHOST', 'localhost');
< define('PHPMAILERPORT',2500);
< define('PHPMAILER_SECURE',false);
<
---
> define('PHPMAILERHOST', '');
44c41
< define('TEST', 0);
---
> define('TEST', 1);
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists: texts
Only in phplist-3.6.15-zip/phplist-3.6.15/public_html/lists: updater
Only in v3.6.15-zip/phplist3-3.6.15/: scripts
Only in v3.6.15-zip/phplist3-3.6.15/: TESTING.md
Only in v3.6.15-zip/phplist3-3.6.15/: tests
Only in v3.6.15-zip/phplist3-3.6.15/: Vagrantfile
user@ose:~/tmp/hetzner3/3tofu3$
</pre>
# I asked the community which release we should use. I'm thinking sourceforge, but I want to be sure
## at first I'd think "don't use the one that has 'TEST' set to '1'), which is the sourceforge one
## but then there's the code block that says "remove on rollout", which also is in the sourceforge one.
## so basically there's a couple signs that say I should use one instead of the other and other signs that say I should use the other instead of one
# I'll wait to hear back from the phpList team before proceeding
# anyway, we do have all the files we need. 3TOFU is done for phpList
# ...
# ...
# here's our new 3TOFU script
# Catarina responded to my email about wordpress themes, asking me to install all of them on the new osemain site on hetzner3 – including twentytwentyfive
## https://wordpress.org/themes/twentytwentyfive/
## https://wordpress.org/themes/twentytwentyfour/
## https://wordpress.org/themes/generatepress/
## https://wordpress.org/themes/astra/
## https://wordpress.org/themes/oceanwp/
## https://wordpress.org/themes/hestia/
## https://wordpress.org/themes/neve/
# time for a new 3TOFU; I needed to do one for a few of the mediawiki plugins I suggested for Marcin earlier, anyway
<pre>
<pre>
################################################################################
################################################################################
Line 6,669: Line 297:
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)


REMOTE_FILES=""
REMOTE_FILES="https://extdist.wmflabs.org/dist/extensions/DeleteBatch-REL1_43-59ffc3f.tar.gz https://extdist.wmflabs.org/dist/extensions/SmiteSpam-REL1_43-d45b08a.tar.gz https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz"
WARNINGS=""
WARNINGS=""


# in tails, we must torify
# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
PYTHON="/usr/bin/torify ${PYTHON}"
Line 6,693: Line 321:
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"


plugins='polylang translatepress-multilingual google-language-translator gtranslate'
themes='twentytwentyfive'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
 
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
 
themes='hestia neve'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
for theme in $themes; do
Line 6,750: Line 356:
sha256sum *
sha256sum *
</pre>
</pre>
# and here's TOFU 1/3 (Tor, exit in Poland)
# Here's TOFU 1/3 (Exit in Germany)
<pre>
<pre>
Congratulations. This browser is configured to use Tor.
Congratulations. This browser is configured to use Tor.
2025-02-06
2025-03-16
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. . . .
INFO: Determining Latest Version of Wordpress Themes  
INFO: Determining Latest Version of Wordpress Themes  
. .
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/polylang.3.6.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/translatepress-multilingual.2.9.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-language-translator.6.0.20.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/gtranslate.3.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.1.zip
######################################################################### 100.0%
2025-02-06
296K google-language-translator.6.0.20.zip
700K gtranslate.3.0.7.zip
8.0M hestia.3.2.8.zip
7.1M neve.4.0.1.zip
64K plugin.json
592K polylang.3.6.6.zip
2.1M translatepress-multilingual.2.9.4.zip
28M wordpress-6.7.1.zip
01246c9c90f1373ee83d8e5884e5abba264c62e7301d99a468fd8eb95144be05  google-language-translator.6.0.20.zip
171eb362801ea28e74b302cd0fee472dbd5025a89f6c1634b5ca1029362a678c  gtranslate.3.0.7.zip
241b8c804ed1af72b1c9aa52f603730a52ebf7850383ac2e4d9dd163f6cfc3ca  hestia.3.2.8.zip
2b51e758d61b9d78ebd57d2afd9a967335bcb6866c5bff5a0d7157eecb6ec8cb  neve.4.0.1.zip
4a87243dd3afb7c15bc8e1c51df6f63604996110691caa0d80d2119ccc59057a  plugin.json
3a4a7f8872d16cb3538e948b4f85acee7d823c04fe8c820259ef6e2735093bbb  polylang.3.6.6.zip
16f2e24d14af341c6208fb9cc08d0c4db33bf813a5b1668930e30c6179a1111c  translatepress-multilingual.2.9.4.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@host:/tmp/user/1000/tmp.JC5exXrKLS$
</pre>
# ...
# I was supposed to have a meeting with Marcin & Catarina today, but they asked to reschedule
# so I'm blocked on phplist, obi, and osemain
# let's continue with the wiki!
# ok, here's the last error we got (when trying to do the second upgrade)
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Cite extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(282): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Cite/extension.json: filemtime(): stat failed for /Cite/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
real    0m0,077s
user    0m0,002s
sys    0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# I fixed this with
<pre>
# don't yet load Cite (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Cite\(.*\)%\1#wfLoadExtension\2Cite\3%' ${vhostDir}/LocalSettings.php
</pre>
# subsequent upgrade attempts resulted in an error with the interwiki extension
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Interwiki extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(289): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Interwiki/extension.json: filemtime(): stat failed for /Interwiki/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
real    0m0,052s
user    0m0,008s
sys    0m0,000s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# fixed with
<pre>
# don't yet load Interwiki (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Interwiki\(.*\)%\1#wfLoadExtension\2Interwiki\3%' ${vhostDir}/LocalSettings.php
</pre>
# then it complains about Gadgets
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Gadgets extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(308): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Gadgets/extension.json: filemtime(): stat failed for /Gadgets/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
real    0m0,051s
user    0m0,004s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# fixed with
<pre>
# don't yet load Gadgets (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Gadgets\(.*\)%\1#wfLoadExtension\2Gadgets\3%' ${vhostDir}/LocalSettings.php
</pre>
# then ReplaceText
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The ReplaceText extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(311): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /ReplaceText/extension.json: filemtime(): stat failed for /ReplaceText/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
real    0m0,053s
user    0m0,002s
sys    0m0,007s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# fixed with
<pre>
# don't yet load ReplaceText (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)ReplaceText\(.*\)%\1#wfLoadExtension\2ReplaceText\3%' ${vhostDir}/LocalSettings.php
</pre>
# then RenameUser
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Renameuser extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(314): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Renameuser/extension.json: filemtime(): stat failed for /Renameuser/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
real    0m0,052s
user    0m0,004s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# fixed with
<pre>
# don't yet load Renameuser (until after upgrades)
sed -i 's%^\(\s*\)[^#]*wfLoadExtension\(.*\)Renameuser\(.*\)%\1#wfLoadExtension\2Renameuser\3%' ${vhostDir}/LocalSettings.php
</pre>
# tried again, and it complained about Nuke
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Nuke extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(326): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Nuke/extension.json: filemtime(): stat failed for /Nuke/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
real    0m0,053s
user    0m0,003s
sys    0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# this seems a bit excessive; these dirs are in-place
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # ls -lah ${vhostDir}/htdocs/extensions
total 144K
d---r-x--- 35 not-apache www-data 4,0K Dec 29 18:14 .
d---r-x--- 14 not-apache www-data 4,0K Feb  2 00:39 ..
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 AbuseFilter
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 CategoryTree
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 Cite
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 CiteThisPage
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 CodeEditor
d---r-x--- 13 not-apache www-data 4,0K Dec 29 18:14 ConfirmEdit
d---r-x---  9 not-apache www-data 4,0K Dec 29 18:14 DiscussionTools
d---r-x--- 10 not-apache www-data 4,0K Dec 29 18:14 Echo
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 Gadgets
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 ImageMap
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 InputBox
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 Interwiki
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 Linter
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 LoginNotify
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 Math
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 MultimediaViewer
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 Nuke
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 OATHAuth
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 PageImages
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 ParserFunctions
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 PdfHandler
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 Poem
----r-----  1 not-apache www-data 1,1K Dec  5 15:41 README
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 ReplaceText
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 Scribunto
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 SecureLinkFixer
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 SpamBlacklist
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 SyntaxHighlight_GeSHi
d---r-x---  8 not-apache www-data 4,0K Dec 29 18:14 TemplateData
d---r-x---  5 not-apache www-data 4,0K Dec 29 18:14 TextExtracts
d---r-x---  7 not-apache www-data 4,0K Dec 29 18:14 Thanks
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 TitleBlacklist
d---r-x--- 12 not-apache www-data 4,0K Dec 29 18:14 VisualEditor
d---r-x---  6 not-apache www-data 4,0K Dec 29 18:14 WikiEditor
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# I thought maybe there was another issue with the path to the 'extensions' dir itself, but that looks like I put that in-place already
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep wgExtensionsDirectory ${vhostDir}/LocalSettings.php
$wgExtensionsDirectory = "$IP/extensions";
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# ah, crap, I just realized the var name according to the wiki is wgExtensionDirectory, not wgExtensionsDirectory https://www.mediawiki.org/wiki/Manual:$wgExtensionDirectory
# I updated this, and now I get a new error
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
PHP Fatal error:  Uncaught FatalError: $wgBaseDirectory must not be modified in settings files! Use the MW_INSTALL_PATH environment variable to override the installation root directory. in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php:274
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once()
#1 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 274
real    0m0,053s
user    0m0,008s
sys    0m0,000s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# ok, I don't understand this error; we are *not* using that var
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep wgBaseDirectory ${vhostDir}/LocalSettings.php
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# here's where the error came-from
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep -C4 wgBaseDirectory ${vhostDir}/htdocs/includes/Setup.php
*    from executing.
*
* This file does:
* - run-time environment checks,
* - define MW_INSTALL_PATH, $IP, and $wgBaseDirectory,
* - load autoloaders, constants, default settings, and global functions,
* - load the site configuration (e.g. LocalSettings.php),
* - load the enabled extensions (via ExtensionRegistry),
* - trivial expansion of site configuration defaults and shortcuts
--
if ( defined( 'MW_AUTOLOAD_TEST_CLASSES' ) ) {
        require_once __DIR__ . '/../tests/common/TestsAutoLoader.php';
}
if ( $wgBaseDirectory !== MW_INSTALL_PATH ) {
        throw new FatalError(
                '$wgBaseDirectory must not be modified in settings files! ' .
                'Use the MW_INSTALL_PATH environment variable to override the installation root directory.'
        );
}
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# we didn't modify that file; it's identical to the file as included in the release of mediawiki-1.43.0
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # diff ${vhostDir}/htdocs/includes/Setup.php /var/tmp/mediawiki/mediawiki-1.43.0/includes/Setup.php
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# ok, looks like we do try to set the "IP" dir
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # grep -C4 MW_INSTALL_PATH ${vhostDir}/LocalSettings.php
# http://www.mediawiki.org/wiki/Manual:Configuration_settings
# If you customize your file layout, set $IP to the directory that contains
# the other MediaWiki files. It will be used as a base to locate files.
if( defined( 'MW_INSTALL_PATH' ) ) {
        $IP = MW_INSTALL_PATH;
} else {
        $IP = dirname( __FILE__ ) . "/htdocs" ;
}
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# the docs on the IP var suggest that you used to be able to update it, but that won't work since Mediawiki >=v1.18
# let's see what happens if we comment-out all these lines. if it still has issues, I guess we should set 'MW_INSTALL_PATH' to `dirname( __FILE__ ) . "/htdocs"` instead
# well, after commenting them out, I *still* have an error complaining about this setting
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
PHP Fatal error:  Uncaught FatalError: $wgBaseDirectory must not be modified in settings files! Use the MW_INSTALL_PATH environment variable to override the installation root directory. in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php:274
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once()
#1 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 274
real    0m0,055s
user    0m0,004s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# looking at the error above it, I wonder if it's possible that the issue is coming from here
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep 'DefaultSettings.php' LocalSettings.php
# See includes/DefaultSettings.php for all configurable settings
require_once( "$IP/includes/DefaultSettings.php" );
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# nah, looks like this file is present both in our docroot *and* in the release dir. And they're identical
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/includes/DefaultSettings.php
htdocs/includes/DefaultSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls /var/tmp/mediawiki/mediawiki-1.43.0/includes/DefaultSettings.php
/var/tmp/mediawiki/mediawiki-1.43.0/includes/DefaultSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org # diff htdocs/includes/DefaultSettings.php /var/tmp/mediawiki/mediawiki-1.43.0/includes/DefaultSettings.php
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# ok, yeah, the only thing in this file is the error about using this file
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # cat htdocs/includes/DefaultSettings.php
<?php
/**
* THIS IS A DEPRECATED STUB FILE!
*
* Default settings are now defined in the MainConfigSchema class.
*
* To get default values for configuration variables, use MainConfigSchema::listDefaultValues()
* or MainConfigSchema::getDefaultValue().
*
* @file
* @deprecated since 1.39
*/
use MediaWiki\MainConfigSchema;
if ( function_exists( 'wfDeprecatedMsg' ) ) {
        wfDeprecatedMsg(
                'DefaultSettings.php is deprecated and will be removed. '
                . 'Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead.',
                '1.39'
        );
}
// Extract the defaults into the current scope
foreach ( MainConfigSchema::listDefaultValues( 'wg' ) as $defaultSettingsVar => $defaultSettingsValue ) {
        $$defaultSettingsVar = $defaultSettingsValue;
}
unset( $defaultSettingsVar );
unset( $defaultSettingsValue );
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# oh, actually there is some stuff at the end
# yeah, that *was* it. If I comment-out that line, then it continues
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
MediaWiki 1.43.0 Updater
Your composer.lock file is up to date with current dependencies!
Going to run database updates for osewiki_db-wiki_
Depending on the size of your database this may take a while!
Abort with control-c in the next five seconds (skip this countdown with --quick) ...0
Can not upgrade from versions older than 1.35, please upgrade to that version or later first.
real    0m5,144s
user    0m0,004s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# but, crap, this is the same problem we had before. It claims we didn't upgrade yet. Even though we did!
# maybe the update failed somewhere in the middle because of this same error about DefaultSettings.php
# I updated the wiki CHG to store all the output from the upgrade runs to log files in the $chg_dir so we can review it after it runs
# I'm also pretty annoyed about these huge permissions commands being stored in all the CHG files separately; I'm just going to create a script in /usr/local/bin/ for this
# ok, let's restore to our snapshot and try the whole process again
# ugh, for some reason I got a new error this time
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # cat $chg_dir/update-to-v1.35.log
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Fatal error:  Uncaught Exception: Unable to open file /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha/extension.json: filemtime(): stat failed for /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php:177
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(71): ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(257): wfLoadExtensions()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(143): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(91): require_once('...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#6 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php on line 177
PHP Fatal error:  Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63
Stack trace:
#0 [internal function]: MediaWiki\HeaderCallback::callback()
#1 {main}
  thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# curiously, the path to the ReCaptcha plugin is present; just the ReCaptcha dir is missing
## I hate ReCaptcha, and I noticed there's a dir for hCaptcha instead
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha
ls: cannot access '/var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit/ReCaptcha': No such file or directory
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/extensions/ConfirmEdit
total 140K
d---r-x--- 13 not-apache www-data 4,0K Dec 26 23:53 .
d---r-x--- 30 not-apache www-data 4,0K Dec 26 23:53 ..
----r-----  1 not-apache www-data 3,1K Jun  7  2019 AUTHORS.txt
----r-----  1 not-apache www-data  510 Sep 25  2020 blacklist
----r-----  1 not-apache www-data 9,8K Sep 24  2020 captcha-old.py
----r-----  1 not-apache www-data  11K Sep 24  2020 captcha.py
----r-----  1 not-apache www-data  135 Sep 25  2020 CODE_OF_CONDUCT.md
----r-----  1 not-apache www-data  439 Sep 25  2020 composer.json
----r-----  1 not-apache www-data 2,7K Sep 25  2020 ConfirmEdit.alias.php
----r-----  1 not-apache www-data  18K Dec 12  2018 COPYING
----r-----  1 not-apache www-data 4,4K Sep 25  2020 extension.json
d---r-x---  5 not-apache www-data 4,0K Dec 26 23:53 FancyCaptcha
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 hCaptcha
d---r-x---  3 not-apache www-data  12K Dec 26 23:53 i18n
d---r-x---  5 not-apache www-data 4,0K Dec 26 23:53 includes
d---r-x---  2 not-apache www-data 4,0K Dec 26 23:53 maintenance
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 MathCaptcha
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 QuestyCaptcha
----r-----  1 not-apache www-data 5,1K Sep 25  2020 README.md
d---r-x---  5 not-apache www-data 4,0K Dec 26 23:53 ReCaptchaNoCaptcha
d---r-x---  4 not-apache www-data 4,0K Dec 26 23:53 resources
d---r-x---  3 not-apache www-data 4,0K Dec 26 23:53 SimpleCaptcha
d---r-x---  3 not-apache www-data 4,0K Dec 26 23:53 tests
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# whatever, I'm going to disable this pesky extension again during our upgrades
# ok, the upgrade appears to have exited with an error 27 minutes-in
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.35.log
real    27m59,071s
user    0m0,008s
sys    0m0,000s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# here's the error log
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail -n40 /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.35.log
Completed migration, inserted 1 row(s) with 0 new actor(s), 0 error(s)
errors were encountered.
Modifying rev_text_id field of table revision ...done.
Modifying table site_stats ...done.
Populating ar_rev_id.
Populating ar_rev_id...
MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1217): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1528): MessageCache->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1167): WANObjectCache->getWithSetCallback()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1163): BagOStuff->getWithSetCallback()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1106): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1016): MessageCache->getMsgFromNamespace()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(927): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(661): Message->text()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(213): Title::newMainPage()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(118): PopulateArchiveRevId::makeDummyRevisionRow()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(63): PopulateArchiveRevId::checkMysqlAutoIncrementBug()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(45): PopulateArchiveRevId->doDBUpdates()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1377): LoggedUpdateMaintenance->execute()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(512): DatabaseUpdater->populateArchiveRevId()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(475): DatabaseUpdater->runUpdates()
#29 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(181): DatabaseUpdater->doUpdates()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(107): UpdateMediaWiki->execute()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# I believe this is why we had the second upgrade command; let's try that and then maybe try the first command again
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php" &> ${chg_dir}/populateContentTables-to-v1.35.log
real    0m13,673s
user    0m0,004s
sys    0m0,004s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# that only took 13 seconds; let's do the first command again
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php" &> ${chg_dir}/update-to-v1.35b.log
real    1m52,641s
user    0m0,004s
sys    0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# cool, on double-tap it finished in less than 2 minutes, and this time it ended with a "Done" message instead of an error
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # tail /var/tmp/CHG_20241228_wiki_1.35-to-1.43/update-to-v1.35b.log
el_id 30000 - 40000 of 76644
el_id 40000 - 50000 of 76644
el_id 50000 - 60000 of 76644
el_id 60000 - 70000 of 76644
el_id 70000 - 76644 of 76644
Done, 689 rows updated, 2 deleted.
Set the local repo temp zone container to be private.
Purging caches...done.
Done in 1 min 48 s.
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# now I went to the version page, and I confirmed that it lists the MediaWiki version as "1.35.0". Perfect! https://wiki.opensourceecology.org/wiki/Special:Version
<pre>
Installed software
Product Version
MediaWiki 1.35.0
PHP 8.2.26 (fpm-fcgi)
MariaDB 10.11.6-MariaDB-0+deb12u1
ICU 72.1
</pre>
=Sun Feb 02, 2025=
# Here's TOFU 2/3 (VPN, exit in US)
<pre>
United States
2025-02-03
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
.  
.  
INFO: Determining Latest Version of Wordpress Themes
. . . . .




https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
https://extdist.wmflabs.org/dist/extensions/DeleteBatch-REL1_43-59ffc3f.tar.gz
######################################################################### 100.0%######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
  -=O=-                                #  #  # # 
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
  -=O=-                            #    #  #  #   
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
https://extdist.wmflabs.org/dist/extensions/SmiteSpam-REL1_43-d45b08a.tar.gz
######################################################################### 100.0%
######################################################################### 100.0%
https://downloads.wordpress.org/theme/astra.4.8.11.zip
https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz
  -=O=-              #    #    #    #                                   
https://downloads.wordpress.org/release/wordpress-6.7.2.zip
####################                                                      27.6%Crash Annotation GraphicsCriticalError: |[0][GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt (t=51.5764) [GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
######################################################################### 100.0%
######################################################################### 100.0%
https://downloads.wordpress.org/theme/generatepress.3.5.1.zip
[1]+  Done                    torbrowser  (wd: ~)
(wd now: /tmp/user/1000/tmp.sHf2RQALl0)
https://downloads.wordpress.org/theme/twentytwentyfive.1.1.zip
######################################################################### 100.0%
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.7.zip
2025-03-16
######################################################################### 100.0%
120K DeleteBatch-REL1_43-59ffc3f.tar.gz
https://downloads.wordpress.org/theme/neve.4.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/oceanwp.4.0.5.zip
######################################################################### 100.0%
2025-02-03
5.9M astra.4.8.11.zip
1.1M generatepress.3.5.1.zip
8.0M hestia.3.2.7.zip
7.1M neve.4.0.0.zip
6.0M oceanwp.4.0.5.zip
21M phplist-3.6.15.tgz
29M phplist-3.6.15.zip
32K plugin.json
4.6M v3.6.15.tar.gz
5.0M v3.6.15.zip
1.4M woocommerce-gateway-stripe.9.1.1.zip
28M wordpress-6.7.1.zip
0e106338e48c9d4d023a46a368f0069b807ce2118a9bf51ddf04c76070867aba  astra.4.8.11.zip
a5e1d7478cce21b7bbef511bbea44156f59acd40cc7e51ad469c403013ba29f5  generatepress.3.5.1.zip
cf89c4f50301f2d030ae3662d4c8b4359445b8ee31fcdd5f67cb0c2c1c50adf8  hestia.3.2.7.zip
cb4cee1ad2cdd020729884d3bd97ecabd16194f3388e301bc9164c3f980d10a3  neve.4.0.0.zip
d1392dbe5f729178c968f5551855a412b41866e876dcf2a79ffd4e07298c4d13  oceanwp.4.0.5.zip
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15.tgz
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15.zip
c0a5eabed1f1317a58dcaeff254865764c071eb03aa670c0084b3dc6767de489  plugin.json
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15.tar.gz
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15.zip
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@disp2327:/tmp/tmp.0MQDfRxGyc$
</pre>
 
=Sat Feb 01, 2025=
 
# Catarina responded to my email.
# It sounds like she doesn't want to pay someone to convert the existing osemain to a new theme because she wants to do a complete relaunch of the website
<pre>
Sorry I didn't explain myself. What I meant is that regardless of any
issues with the migration, we were planning to re-do OSE's main website
because: 1) it's outdated and needs a refresh, and 2) we're about to launch
a new phase of the project and need a website that reflects that. There is
no point in hiring someone to recreate the old site when we're going to
create a new one anyway. We just need the old one to be online while we
work on the new one (I guess?)
 
OBI's website is outdated too (in terms of content) and a new website is in
the cards, but I just don't know if we want to tackle that right now or
just let the old one in place for a while longer. Regardless of timing, I
agree that the page-builder thing is not the way to go and we would want to
switch to what you suggest (native wordpress galleries and a popular theme)
when we update the website. The question is just whether to do that now or
later.
 
I'll take a look then. Can you send me the instructions? My main computer
runs linux, but I've been having trouble with the browser on that one. My
laptop runs windows 11 (🤢).
 
On Sat, Feb 1, 2025 at 10:08 AM Michael Altfield <REDACTED>
wrote:
 
>  > 1) If we decide to create a new website for OBI, what is the best
>  > way to access the visual assets (images, animations, graphics)
>  > from the old site?
>  > I think we'd want to re-use quite a few.
>
> I'm not an expert at wordpress design, but I think what you've built
> with "portfolios" and the "be-page-builder" is non-trivial to replace. I
> was hoping (since you built it) you would know better after looking at
> it -- maybe it's just some quirk of oshine that you know how to quickly
> fix? I think you should take a look at it.
>
> The OSE site would be pretty cheap to be re-built with a more popular
> theme (I recommend Neve), but I think it would be quite a large
> undertaking for the OBI website.
>
> Migrating the images is fine, but the interactive tabs is the type of
> complexity that I'd avoid. But if you just want to replace all that with
> native wordpress galleries, that's not too hard.
>
> What I'm suggesting for the "osemain" (www.opensourceecology.org)
> website is that we don't create a "new" site -- but we just migrate the
> site over to our new server, switch to a popular, flexible, lightweight,
> and freemium theme (I recommend Neve), and then hire some freelancer to
> hack the theme config and content until it looks more-or-less identical
> to the old site, but using only wordpress-built-in Gutenberg blocks --
> so we don't have this issue in the future.
>
>  > 2) If we decide that it doesn't make sense to create a new website
>  > for OBI at this moment, would it be possible to turn the existing
>  > one into a static site (like you did for FeF)? If so, how much
>  > time would that take you?
>
> This is pretty trivial. It depends a lot on the website, but it took me
> a few hours for fef and half a day for oswh.
>
> I can definitely do it for OBI, but then you can't edit the content
> anymore. Of course, as with the other sites that I'm converting to
> static-site, I'll make a full backup of the db and files, so you'd be
> able to use these backups to create a new DB and wordpress site to
> create the website again in wordpress. This restored site would have the
> same rendering issues I'm describing, but all the content will be
> present and you could then try to fix the rendering issues within oshine
> or switch to another theme and rebuild it, as I'm suggesting for osemain.
>
> But I think it would be best if you just access the site to know what
> I'm talking about. Are you running linux? Do you have firefox installed?
>
>
> Cheers,
>
> Michael Altfield
> https://www.michaelaltfield.net
> PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41
>
> Note: If you cannot reach me via email, please check to see if I have
> changed my email address by visiting my website at
> https://email.michaelaltfield.net
>
> On 2/1/25 10:43, Catarina Mota wrote:
>> Thank you Michael. I was planning on re-doing OSE's site very soon, but
> was
>> hoping OBI's would hang in there a while longer.
>>
>> A couple questions:
>>
>> 1) If we decide to create a new website for OBI, what is the best way to
>> access the visual assets (images, animations, graphics) from the old
> site?
>> I think we'd want to re-use quite a few.
>>
>> 2) If we decide that it doesn't make sense to create a new website for
> OBI
>> at this moment, would it be possible to turn the existing one into a
> static
>> site (like you did for FeF)? If so, how much time would that take you?
>>
>> Thank you,
>> Catarina
>>
>> On Fri, Jan 31, 2025 at 4:44 PM Michael Altfield <REDACTED>
>> wrote:
>>
>>> Hey Catarina,
>>>
>>> I just finished migrating a snapshot of OBI to hetzner3. Can you please
>>> review it?
>>>
>>> I'm sorry to say that OBI didn't handle the upgrades very well. About
>>> half of the content is missing. I was able to fix some of the issues by
>>> changing the web server config, but it appears that most of the issues
>>> are theme- or page-builder-related, and I'll need to defer to you as the
>>> SME.
>>>
>>> Can you please tell me:
>>>
>>> 1. What OS does your computer use?
>>> 2. What web browser do you use?
>>>
>>> And I can send instructions for you to view/edit the OBI site on
> hetzner3.
>>>
>>>
>>> Thank you,
>>>
>>> Michael Altfield
>>> https://www.michaelaltfield.net
>>> PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41
>>>
>>> Note: If you cannot reach me via email, please check to see if I have
>>> changed my email address by visiting my website at
>>> https://email.michaelaltfield.net
</pre>
# before I send her instructions for accessing the websites on hetzner3, I created some files named 'is_hetzner3' at the docroot of each site, so we can confirm it easily
<pre>
echo "true" > /var/www/html/fef.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/www.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/microfactory.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/forum.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/store.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/wiki.opensourceecology.org/htdocs/is_hetzner3
echo "true" > /var/www/html/www.openbuildinginstitute.org/htdocs/is_hetzner3
echo "true" > /var/www/html/seedhome.openbuildinginstitute.org/htdocs/is_hetzner3
echo "true" > /var/www/html/oswh.opensourceecology.org/htdocs/is_hetzner3
</pre>
# I sent an email describing how to access the new sites
<pre>
So the way your computer knows to connect to the old server (hetzner2) vs the new server (hetzner3) is via DNS.
 
You can override your DNS settings locally on your computer by editing the /etc/hosts file.
 
Add these lines to your /etc/hosts file
 
144.76.164.201 forum.opensourceecology.org
144.76.164.201 store.opensourceecology.orgc
144.76.164.201 microfactory.opensourceecology.org
144.76.164.201 seedhome.openbuildinginstitute.org
144.76.164.201 fef.opensourceecology.org
144.76.164.201 oswh.opensourceecology.org
144.76.164.201 www.openbuildinginstitute.org
144.76.164.201 openbuildinginstitute.org
144.76.164.201 www.opensourceecology.org
144.76.164.201 opensourceecology.org
144.76.164.201 phplist.opensourceecology.org
144.76.164.201 wiki.opensourceecology.org
 
You can edit the /etc/hosts file with nano, vim, or your fav editor:
 
  sudo nano /etc/hosts
  sudo vim /etc/hosts
 
Windows also has a hosts file, but it's instead located at
 
  C:\Windows\System32\Drivers\etc\hosts
 
* https://www.wikihow.com/Edit-the-Hosts-File-on-Windows
 
After making that change, open firefox and tell it to clear its cache by going to about:networking -> DNS -> Clear DNS Cache
 
* https://www.groovypost.com/howto/flush-dns-cache-on-windows-11/
 
Now when you visit any of the OSE sites listed in your hosts file, you'll view it on the new server (hetzner3).
 
To confirm that you're viewing the site on the new server, I've created a file called '/is_hetnzer3' with the contents "true" on every host. For example:
 
* https://www.openbuildinginstitute.org/is_hetzner3
 
If ^ that page returns a 404 error, you're on hetzner2 (the old server).
 
If ^ that page returns a page that says "true", then you're on hetzner3 (the new server).
 
When you're finished, just delete those lines from your hosts file and clear your DNS cache again, and you'll be able to access the sites on hetzner2 again.
 
Note that these sites have been migrated as a snapshot. They will be deleted. Feel free to edit them however you want, but know that anything you change will be deleted.
 
Please review all of the webpages of obi and osemain. Login, make test changes, and let me know how you want to proceed.
 
 
Thank you,
 
Michael Altfield
https://www.michaelaltfield.net
PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41
 
Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
</pre>
# ...
# at this point, I'm thinking we're probably going to make static sites of both bi and osemain and make those the "live" sites on hetzner3 when we migrate
## ...but we'll *also* migrate the actual wordpress site. To avoid the name collision, we can just expose it on the Internet over a different IP address, which DNS doesn't point-to
## either we can buy another IPv4 address (expensive, but cheaper than having a second server or keeping hetzner2 around) or we can try to see if Marcin can access it over IPv6
## that will buy them some time, so they can hack away at a new site on hetzner3 as much as they want without the old site going down. it just means a content freeze until they finish with the new site
## and when they do finally finish it, we just have to make a simple nginx config change to swap the old static site for the new site
# Catarina wrote me back saying she'd review the sites and discuss with Marcin how they want to proceed
# I'm hoping that we can have a call and I can explain my suggestion above after they've had time to review it, so they have a better understanding of how it's broken
# anyway, I'm blocked on obi and osemain until they've reviewed it
# ...
# I finished my logs and hours and sent a bill to OSE for 18 hours in the month of Jan 2025
#...
# next-up is phplist
# I 3TOFU'd two distinct files for phpList
<pre>
root@hetzner3 /var/tmp/phplist # sha256sum *
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
root@hetzner3 /var/tmp/phplist #
</pre>
# it's surprising that the tarball and the zip file have the same hash; that tells me the file extension is probably a lie
# but I don't understand why the two versions (I think one came from sourceforge and one came from github) don't match
# ok, apparently neither is a gzip tarball file
<pre>
root@hetzner3 /var/tmp/phplist # tar -xzvf phplist-3.6.15.zip
 
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
root@hetzner3 /var/tmp/phplist # tar -xzvf phplist-3.6.15.tgz
 
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
root@hetzner3 /var/tmp/phplist #
</pre>
# wtf, neither is a zip file either
<pre>
root@hetzner3 /var/tmp/phplist # unzip phplist-3.6.15.tgz
Archive:  phplist-3.6.15.tgz
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of phplist-3.6.15.tgz or
phplist-3.6.15.tgz.zip, and cannot find phplist-3.6.15.tgz.ZIP, period.
root@hetzner3 /var/tmp/phplist # unzip phplist-3.6.15.zip
Archive:  phplist-3.6.15.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of phplist-3.6.15.zip or
phplist-3.6.15.zip.zip, and cannot find phplist-3.6.15.zip.ZIP, period.
root@hetzner3 /var/tmp/phplist #
</pre>
# ah, crap, they look empty
<pre>
root@hetzner3 /var/tmp/phplist # du -sh *
4,0K    phplist-3.6.15.tgz
4,0K    phplist-3.6.15.zip
0      v3.6.15.tar.gz
0      v3.6.15.zip
root@hetzner3 /var/tmp/phplist #
</pre>
# yeah, two of the files are literally empty and the other two are just 301 redirect errors. crap
<pre>
root@hetzner3 /var/tmp/phplist # for file in $(ls -1); do echo $file; cat $file; echo; done
phplist-3.6.15.tgz
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
 
phplist-3.6.15.zip
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
 
v3.6.15.tar.gz
 
v3.6.15.zip
 
root@hetzner3 /var/tmp/phplist #
</pre>
# yeah, so apparently I need to update the curl command in my 3TOFU script
<pre>
user@disp6255:/tmp/tmp.t8DUYsEVAi$ CURL="/usr/bin/curl --retry 5 --retry-all-errors"
user@disp6255:/tmp/tmp.t8DUYsEVAi$ ${CURL} --progress-bar -O https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
user@disp6255:/tmp/tmp.t8DUYsEVAi$ ls
phplist-3.6.15.tgz
user@disp6255:/tmp/tmp.t8DUYsEVAi$ cat phplist-3.6.15.tgz
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
user@disp6255:/tmp/tmp.t8DUYsEVAi$
</pre>
# let's try this instead
<pre>
CURL="/usr/bin/curl --retry 5 --retry-all-errors --location"
${CURL} --progress-bar -O https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
</pre>
# ok, that worked
<pre>
user@disp6255:/tmp/tmp.t8DUYsEVAi$ rm phplist-3.6.15.tgz
user@disp6255:/tmp/tmp.t8DUYsEVAi$ ls
user@disp6255:/tmp/tmp.t8DUYsEVAi$ CURL="/usr/bin/curl --retry 5 --retry-all-errors --location"
${CURL} --progress-bar -O https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
user@disp6255:/tmp/tmp.t8DUYsEVAi$ du -sh *
21M phplist-3.6.15.tgz
user@disp6255:/tmp/tmp.t8DUYsEVAi$
</pre>
# that also appears to fix the github files too
# I updated the hetzner3:/usr/local/bin/wordpress_3tofu.sh file to include a `du -sh` and `--location` arg to CURL to prevent this in the future
# alright, let's do another 3TOFU for phpList and also for some wordpress themes to pre-install for the replacement obi & osemain sites (might not be needed)
# here's the 3TOFU script
<pre>
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
#
#          For more info on 3TOFU (and why this is important), see:
#          * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################
 
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --location --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
 
REMOTE_FILES="https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz"
WARNINGS=""
 
# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
 
tmpDir=`mktemp -d`
pushd "${tmpDir}"
 
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
 
# and today's date
date -u +"%Y-%m-%d"
 
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
 
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
 
plugins='woocommerce-gateway-stripe'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
 
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
 
themes='astra generatepress hestia neve oceanwp'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
 
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
 
echo -e "${WARNINGS}"
echo
 
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
 
# checksum
date -u +"%Y-%m-%d"
du -sh *
sha256sum *
</pre>
# and here's TOFU 1/3 (Tor, exit in Poland)
<pre>
Congratulations. This browser is configured to use Tor.
2025-02-01
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
.
INFO: Determining Latest Version of Wordpress Themes
. . . . .
 
 
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
  #    #    #    #                -=O=-                   
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
# #                                                                      -=O=-
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/astra.4.8.11.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/generatepress.3.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/hestia.3.2.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/neve.4.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/oceanwp.4.0.5.zip
######################################################################### 100.0%
2025-02-01
5.9M astra.4.8.11.zip
1.1M generatepress.3.5.1.zip
8.0M hestia.3.2.7.zip
7.1M neve.4.0.0.zip
6.0M oceanwp.4.0.5.zip
21M phplist-3.6.15.tgz
29M phplist-3.6.15.zip
32K plugin.json
4.6M v3.6.15.tar.gz
5.0M v3.6.15.zip
1.4M woocommerce-gateway-stripe.9.1.1.zip
28M wordpress-6.7.1.zip
0e106338e48c9d4d023a46a368f0069b807ce2118a9bf51ddf04c76070867aba  astra.4.8.11.zip
a5e1d7478cce21b7bbef511bbea44156f59acd40cc7e51ad469c403013ba29f5  generatepress.3.5.1.zip
cf89c4f50301f2d030ae3662d4c8b4359445b8ee31fcdd5f67cb0c2c1c50adf8  hestia.3.2.7.zip
cb4cee1ad2cdd020729884d3bd97ecabd16194f3388e301bc9164c3f980d10a3  neve.4.0.0.zip
d1392dbe5f729178c968f5551855a412b41866e876dcf2a79ffd4e07298c4d13  oceanwp.4.0.5.zip
dfe441583f7f72b116c2f7db24821259df4fdc991ab52a7078ba3293729d71b9  phplist-3.6.15.tgz
fcbe14b2770832d2788f3a8a5c9c6c18b178bf069559ca30c947bac78ca51e19  phplist-3.6.15.zip
88ee2ac6c1c04d4ebbae269d1117b6727a7152a6dd683aa444e4452333cd9c13  plugin.json
31e4a733aa481fe483f5513931d04607b14243b7f2cc2c3c210a6abe508e3265  v3.6.15.tar.gz
16c3bc98c6d4acd52478042b733c90ac8d64d31762b884856e6cf3c620b4b82e  v3.6.15.zip
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@host:/tmp/user/1000/tmp.WJ1lwTXhlC$
</pre>
# ...
# ok, back to the wiki
# I'm not sure where we left-off. I don't think I was ever able to successfully do the double upgrade
# previously I couldn't figure out the MediaWiki version without logging-in and dumping it from some special page
# well, if I try to load the login page, get an error https://wiki.opensourceecology.org/index.php?title=Special:UserLogin
<pre>
[Z56tEJjVN3m_4ExsymEpZwAAAAQ] 2025-02-01 23:24:01: Fatal exception of type "mysqli_sql_exception"
</pre>
# I'm just going to start with a fresh reinstall from the snapshot taken on 2024-12-28
# first half of the first upgrade took 28 minutes
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php"
...
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}
 
real    28m16,791s
user    0m0,168s
sys    0m0,268s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# the next command finished in in 15 seconds
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php"
...
... archive processed up to revision id 302632 of 302632 (3178 rows in 1.3690750598907 seconds)
Done populating archive table. Processed 3178 rows in 1.3690841197968 seconds
Done. Processed 302790 rows in 14.684278964996 seconds
 
real    0m14,776s
user    0m0,009s
sys    0m0,039s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# loading the site still fails with the same error as before; let's continue with the second upgrade
<pre>
[Z56-K4l3m_5hOvSLliKhkgAAAAM] 2025-02-02 00:37:00: Fatal exception of type "mysqli_sql_exception"
</pre>
# well my next documented step is to visit https://wiki.opensourceecology.org/wiki/Special:Version
## but that just spits out the same error..
# I rsynce'd in the next version's files, fixed permissions, and attempted an upgrade, but it failed
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning:  Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated:  DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The Cite extension cannot be loaded. Check that all of its files are installed properly.
 
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(282): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/run.php(49): require_once('...')
#5 {main}
PHP Fatal error:  Error Loading extension. Unable to open file /Cite/extension.json: filemtime(): stat failed for /Cite/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
 
real    0m0,077s
user    0m0,002s
sys    0m0,005s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
 
=Fri Jan 31, 2025=
 
# yesterday we mostly migrated obi, but we got stuck on verification because one of the sliders on the frontpage was missing
# I realized that second slider was buried under a "page builder" on hetzner2, and 'be-page-builder' is missing on hetzner3
# yesterday I emailed brand exponents support asking where I can download the latest version of 'be-page-builder'. I have not received a response.
# I confirmed that the files they gave me (via google drive) before, didn't include be-page-builder https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Tue_Dec_31.2C_2024
# but later that day I had managed to get some error output that gave me the path it was using to 'download be-portfolio-post' https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
# I tried to just sub the plugin slug for 'be-page-builder', and it worked! https://brandexponents.com/oshin-plugins/be-page-builder.zip
<pre>
user@disp6255:~$ wget https://brandexponents.com/oshin-plugins/be-page-builder.zip
--2025-01-31 15:37:46--  https://brandexponents.com/oshin-plugins/be-page-builder.zip
Resolving brandexponents.com (brandexponents.com)... 188.114.96.12, 188.114.97.12, 2a06:98c1:3120::c, ...
Connecting to brandexponents.com (brandexponents.com)|188.114.96.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘be-page-builder.zip’
 
be-page-builder.zip    [      <=>          ]  1.61M  1.36MB/s    in 1.2s   
 
2025-01-31 15:37:48 (1.36 MB/s) - ‘be-page-builder.zip’ saved [1691575]
 
user@disp6255:~$ unzip be-page-builder.zip
Archive:  be-page-builder.zip
  creating: be-page-builder/
...
user@disp6255:~$ grep -i version be-page-builder/be-page-builder.php
Version: 4.6.1
user@disp6255:~$
</pre>
# well shit, that's the same version that we have installed on hetzner2. So no need to update or 3TOFU
# I just copied the plugin dir that I had copied from hetzner2 into the dir of plugins on hetzner3 (so that future migrations won't delete the dir)s
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # rsync -av --progress var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/be-page-builder /var/tmp/wordpress/plugins/
...
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
</pre>
# great. after I copied the plugin back to the wordpress site's 'plugins/' dir, activated it in the wp dashboard, cleared the varnish cache, and refreshed – the second slider re-appeared ☺
# oh, but the pages aren't exact.
## there's a white box around the second slider in the new site that says "OPEN BUILDING INSTITUTE" honestly it looks better on the new site than the old imho, but it is an important difference
## also the content in the footer has less padding in the new site. not a big deal
# there's also differences on /about-who-we-are/
## curiously the profiles of everyone looks good *except* for Catarina and Marcin :/
## oh, it looks like that's because Catarina & Marcin have some sort of "tabs" that the rest don't: bio, talks, press & publications, and education
# if I try to edit this page on hetzner3, I can see all the tabs in the page builder. so that seems ok
# if I open the js console, I don't see any obvious errors
## here's the console output on-load on hetzner2
<pre>
GET
https://f.vimeocdn.com/js/froogaloop2.min.js?ver=4.8.1
NS_BINDING_ABORTED
 
This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. about-who-we-are
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. node.js:416:1
A resource is blocked by OpaqueResponseBlocking, please check browser console for details. froogaloop2.min.js
JQMIGRATE: Migrate is installed, version 1.4.1 jquery-migrate.min.js:2:552
Google Analytics and Tag Manager is being shimmed by Firefox. See https://bugzilla.mozilla.org/show_bug.cgi?id=1713687 for details. sandbox eval code:1:9
GET
https://f.vimeocdn.com/js/froogaloop2.min.js?ver=4.8.1
NS_BINDING_ABORTED
 
A resource is blocked by OpaqueResponseBlocking, please check browser console for details. froogaloop2.min.js
Loading failed for the <script> with source “https://f.vimeocdn.com/js/froogaloop2.min.js?ver=4.8.1%E2%80%9D. about-who-we-are:2032:93
Google Maps JavaScript API has been loaded directly without loading=async. This can result in suboptimal performance. For best-practice loading patterns please see https://goo.gle/js-api-loading js:278:278
unreachable code after return statement be-modules-plugin.js:35:12433
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://maps.googleapis.com/maps/api/mapsjs/gen_204?csp_test=true. (Reason: CORS request did not succeed). Status code: (null).
downloadable font: OS/2: Bad sTypoLineGap, setting it to 0: -32 (font-family: "icomoon" style:normal weight:400 stretch:100 src index:1) source: https://www.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i
downloadable font: Glyph bbox was incorrect (glyph ids 7 9 10 13 14 15 19 20 21 23 24 26 30 33 44 52 57 58 59 60 63 72 77 80 83 84 87 88 90 91 93 94 95 99 106 114 121 129 142 143 146 147 149 150 151 152 156 159 160 161 163 166 167 168 169 170 171 172 176 177 182 183 184 185 191 197 207 208 209 212 214 215 216 218 220 222 226 227 228 236 237 239 240 241 242 243 244 245 246 247 248 250 252 254 256 257 258 259 260 262 263 264 266 269 271 273 276 278 279 281 283 284 285 286 287 288 299 302 305 309 310 316 317 318 321 322 326 327 328 329 330 335 336 339 341 342 343 344 345 347 348 351 355 357 361 363 367 369 370 372 375 376 377 378 382 383 385 386 388 389 390 391 393 394 407 408 411 413 414 416 417 418 419 420 421 422 423 428 430 432 438 443 445 449 451 458 464 465 468 470 474 475 477 478 482 483 484 485 486 488 489 490 491 494 497 498 500 501 503 504 506 507 509 510 513 514 515 517 519 525 526 527 528 529 530 531 532 534 535 536 537 538 541 544 547 550 552 553 554 561 564 573 577 578 579 580 583 588 591 593 598 599 601 611 612 624 627 628 631 632 648 656 661 662 680 704 706 718 720 721 729 731 738 743 744 746 747 748 749 750 755 770 771 775 790 814 820 826 841 844) (font-family: "icomoon" style:normal weight:400 stretch:100 src index:1) source: https://www.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i
downloadable font: Glyph bbox was incorrect (glyph ids 91 223) (font-family: "Economica" style:normal weight:700 stretch:100 src index:0) source: https://fonts.gstatic.com/s/economica/v15/Qw3aZQZaHCLgIWa29ZBTjecUDXx4.woff2
Google Maps JavaScript API warning: NoApiKeys https://developers.google.com/maps/documentation/javascript/error-messages#no-api-keys util.js:81:65
</pre>
## and here's it on hetzner3
<pre>
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://fonts.googleapis.com/css?family=Economica%3A400%2C700%7COpen+Sans%3A400%7CMontserrat%3A400%2C700%7CCrimson+Text%3A400italic%7CRaleway%3A400%2C600&subset=latin&ver=1738275627 css
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.google-analytics.com/analytics.js about-who-we-are:26:67
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://i.creativecommons.org/l/by-sa/4.0/88x31.png 88x31.png
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps/api/js?ver=6.7.1 js
This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. about-who-we-are
JQMIGRATE: Migrate is installed, version 3.4.1 jquery-migrate.min.js:2:981
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. modernizr.js:4:3947
unreachable code after return statement be-modules-plugin.js:35:12433
Google Maps JavaScript API has been loaded directly without loading=async. This can result in suboptimal performance. For best-practice loading patterns please see https://goo.gle/js-api-loading js:278:278
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps/api/mapsjs/gen_204?csp_test=true js:303:207
downloadable font: Glyph bbox was incorrect (glyph ids 91 223) (font-family: "Economica" style:normal weight:700 stretch:100 src index:0) source: https://fonts.gstatic.com/s/economica/v15/Qw3aZQZaHCLgIWa29ZBTjecUDXx4.woff2
downloadable font: OS/2: Bad sTypoLineGap, setting it to 0: -32 (font-family: "icomoon" style:normal weight:400 stretch:100 src index:1) source: https://www.openbuildinginstitute.org/wp-content/themes/oshin/fonts/icomoon/fonts/icomoon.woff?85pf5i
Some cookies are misusing the recommended “SameSite“ attribute 4
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.google-analytics.com/j/collect?v=1&_v=j101&a=847737016&t=pageview&_s=1&dl=https%3A%2F%2Fwww.openbuildinginstitute.org%2Fabout-who-we-are%2F%3Fnocache%3D3&ul=en-us&de=UTF-8&dt=About%3A%20Who%20We%20Are%20%E2%80%93%20Open%20Building%20Institute&sd=24-bit&sr=1920x1200&vp=1289x510&je=0&_u=AACAAEABAAAAACAAI~&jid=383980088&gjid=939186075&cid=1363777290.1738276114&tid=UA-78141409-1&_gid=1042838251.1738276114&_r=1&_slc=1&z=931548647 analytics.js:36:32
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/gtag/js?id=G-2XNSR98CNW&cx=c&_slc=1 analytics.js:24:55
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtm.init_consent&eid=1&tr=1ogtdma&ti=2ogtdma&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtm.init&eid=2&tr=1ogtgasend.1ogtreferralexclusion.1ogtsessiontimeout.1ogt1pdatav2.1ccdgafirst.1setproductsettings.1ccdgaregscope.1ccdconversionmarking.1ccdautoredact.1ccdgalast&ti=2ogtgasend.2ogtreferralexclusion.2ogtsessiontimeout.2ogt1pdatav2.2ccdgafirst.2setproductsettings.2ccdgaregscope.2ccdconversionmarking.2ccdautoredact.2ccdgalast&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtm.js&eid=3&tr=1gct&ti=1gct&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=gtag.config&eid=4&u=AAAAAAAI&epr=1G.3G&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://www.googletagmanager.com/a?id=G-2XNSR98CNW&v=3&t=t&pid=1426927297&cv=3&rv=51u0&tc=12&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&es=1&e=*&eid=5&u=AAAAAAAIAAAAAIA&ut=Ag&h=Ag&epr=1G.2G&z=0 js:172:144
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps-api-v3/api/js/59/7/common.js js:167:267
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://maps.googleapis.com/maps-api-v3/api/js/59/7/util.js js:167:267
Google Maps JavaScript API warning: NoApiKeys https://developers.google.com/maps/documentation/javascript/error-messages#no-api-keys util.js:81:65
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://region1.google-analytics.com/g/collect?v=2&tid=G-2XNSR98CNW&gtm=45je51u0v9124516828za200&_p=1738357726052&gcd=13l3l3l3l2l1&npa=0&dma=0&tag_exp=102067808~102081485~102123608~102482433~102528644~102539968~102546754~102558063&ul=en-us&sr=1920x1200&cid=1363777290.1738276114&ir=1&frm=0&pscdl=noapi&_eu=EBAI&_s=1&dl=https%3A%2F%2Fwww.openbuildinginstitute.org%2Fabout-who-we-are%2F%3Fnocache%3D3&dt=About%3A%20Who%20We%20Are%20%E2%80%93%20Open%20Building%20Institute&sid=1738356502&sct=3&seg=1&en=page_view&_ee=1&tfd=12191 js:169:286
</pre>
# I don't see anything obvious. I'm going to keep going.
# on '/supporters/' I see there's a new header that says SUPPORTERS and has breadcrumbs at the top (just below the navbar)
## this is a thing on all the pages, it seems. maybe a change to oshine theme, idk
# the '/library/' page has a broken shortcode
<pre>
[do_widget id=nav_menu-2]
</pre>
# looks like this is supposed to be from a plugin named 'amr-shortcode-any-widget'
<pre>
[root@opensourceecology www.openbuildinginstitute.org]# grep -irl 'do_widget' *
htdocs/wp-content/plugins/amr-shortcode-any-widget/amr-utilities.php
htdocs/wp-content/plugins/amr-shortcode-any-widget/readme.txt
htdocs/wp-content/plugins/amr-shortcode-any-widget/amr-admin-form-html.php
htdocs/wp-content/plugins/amr-shortcode-any-widget/amr-shortcode-any-widget.php
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget-sr_RS.po
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget-sr_RS.mo
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget.pot
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr-shortcode-any-widget-id_ID.po
htdocs/wp-content/plugins/amr-shortcode-any-widget/languages/amr_shortcode_any_widget_id_ID.mo
[root@opensourceecology www.openbuildinginstitute.org]#
</pre>
# yeah, we don't have a version of this plugin because it was closed by wordpress due to a "security issue" https://wordpress.org/plugins/amr-shortcode-any-widget/
<pre>
This plugin has been closed as of January 19, 2023 and is not available for download. Reason: Security Issue.
</pre>
# this other plugin claims to offer a 'do_wiget' shortcode, but it only has 400 installs and it hasn't been updated for 4 years https://wordpress.org/plugins/shortcodely/
# looks like the security issue was a 6.4/10 XSS vuln = CVE-2022-4458 https://www.wordfence.com/threat-intel/vulnerabilities/wordpress-plugins/amr-shortcode-any-widget/amr-shortcode-any-widget-40-authenticated-contributor-stored-cross-site-scripting-via-shortcode
# curiously, I found a repo that claims to be a mirror on github, but it was last updated 3 months ago https://github.com/common-repository/amr-shortcode-any-widget
## oh, it was created 3 months ago. it was probably just copied from the old (vulnerable?) codebase
# there's more problems on '/structures/'
## I can't change between tabs, and the images aren't showing
## the div class says "portfolio", so I assume this is "be-portfolio-post', which we do have installed.
## oh, each of these tabs actually cooresponds to an entry in the wp dashboard's "portfoilo" section https://www.openbuildinginstitute.org/wp-admin/edit.php?post_type=portfolio
## huh, actually, some of these are missing
## oh, it looks like the "checkboxes" that you choose from on the actual page are found in the Portfolio -> Categories slugs https://www.openbuildinginstitute.org/wp-admin/edit-tags.php?taxonomy=portfolio_categories&post_type=portfolio
## well shit, if I view the category it works fine https://www.openbuildinginstitute.org/portfolio_categories/roof/
## so it appears to just be the tab selection thing that wraps these portfolios?
# this page has issues too https://www.openbuildinginstitute.org/library-utilities-appliances/
## the HYRDOPONICS CONTROL PANEL text is missing on the gold label
# lots of the same issues exist on the other "libraries" pages
# on the '/buildings/' page there's issues
## none of the sliders show up
# on the '/use/' page
## the iframe of the 3d studio doesn't load at all
## js console gives this relevant message
<pre>
Invalid X-Frame-Options header was found when loading “https://www.openbuildinginstitute.org/wp-content/3Dmodels/example_studio/%E2%80%9D: “invalid” is not a valid directive.example_studio
</pre>
## in the networking tab, I see a failed load of 'https://www.openbuildinginstitute.org/wp-content/3Dmodels/example_studio/' due to NS_ERROR_XFO_VIOLATION. I guess XFO = X-Frame-Options
## the response headers for that request include both 'SAMEORIGIN' and 'deny'
<pre>
X-Frame-Options: SAMEORIGIN
X-Frame-Options: deny
</pre>
## here's what mozilla has to say https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
<pre>
If you specify DENY, not only will the browser attempt to load the page in a frame fail when loaded from other sites, attempts to do so will fail when loaded from the same site. On the other hand, if you specify SAMEORIGIN, you can still use the page in a frame as long as the site including it in a frame is the same as the one serving the page.
</pre>
## so it sounds like we want 'sameorigin'
## well, shit, it looks like we've setup nginx to set it to 'sameorigin' and apache to 'deny'
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ grep -ir 'x-frame-options' *
roles/maltfield.nginx/templates/nginx.conf.j2: add_header X-Frame-Options "SAMEORIGIN";
roles/maltfield.apache/templates/security.conf.j2:#Header set X-Frame-Options: "sameorigin"
roles/maltfield.apache/templates/security.conf.j2:Header always append X-Frame-Options "SAMEORIGIN"
roles/maltfield.apache/templates/security.conf.j2:Header set X-Frame-Options "deny"
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
## it looks like our old server never used 'deny'
<pre>
[root@opensourceecology ~]# grep -ir 'x-frame-options' /etc/nginx
/etc/nginx/nginx.conf:  add_header X-Frame-Options "SAMEORIGIN";
/etc/nginx/nginx.conf.20241230:  add_header X-Frame-Options "SAMEORIGIN";
[root@opensourceecology ~]# grep -ir 'x-frame-options' /etc/apache2
grep: /etc/apache2: No such file or directory
[root@opensourceecology ~]# grep -ir 'x-frame-options' /etc/httpd
/etc/httpd/conf/httpd.20170826.bak:Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf.20170720.bak:Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf:#Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf.20170901:#Header always append X-Frame-Options SAMEORIGIN
/etc/httpd/conf/httpd.conf.20170811.bak:Header always append X-Frame-Options SAMEORIGIN
[root@opensourceecology ~]#
</pre>
## since I know Marcin loves to use iframes, I'm going to make this 'sameorigin' the default for all our configs https://github.com/OpenSourceEcology/ansible/commit/5efbec50f239817be54764fcada4dafeade7a034
## I made that change in ansible, pushed, restarted nginx/apache, cleared varnish, and refreshed. now the 3d model of the studio loads in the iframe fine ☺
# loads of stuff is broken on '/workshops-events/'
## I don't see any obvious errors
# well, I think that's about all I can do for OBI
# I want Catarina to look at it and see how she feels; maybe she just needs to do some simple change to fix them? Or maybe this is a disaster
<pre>
Hey Catarina,
 
I just finished migrating a snapshot of OBI to hetzner3. Can you please review it?
 
I'm sorry to say that OBI didn't handle the upgrades very well. About half of the content is missing. I was able to fix some of the issues by changing the web server config, but it appears that most of the issues are theme- or page-builder-related, and I'll need to defer to you as the SME.
 
Can you please tell me:
 
1. What OS does your computer use?
2. What web browser do you use?
 
And I can send instructions for you to view/edit the OBI site on hetzner3.
 
 
Thank you,
</pre>
# ...
# I'm moving onto osemain
# I migrated a snapshot of www.opensourceecology.org to hetnzer3 (surprisingly the first time) following the same guide as with the other sites
# I updated ansible to use this vhost (and make it the default vhost for the server) and pushed it out https://github.com/OpenSourceEcology/ansible/commit/674a16babf2f250f80c7acdd06f1ac4d7a1888c7
# the website actually looks ok on first load, except the social media links in the footer fail to load as expected (plugin no longer available) and instead just says '
[smbtoolbar]'
# but after I login to the wp dashboard wui, I just get a crit error
<pre>
There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.
 
Learn more about troubleshooting WordPress.
</pre>
# apache error log indicates it's the theme; damn
<pre>
[Fri Jan 31 23:39:03.032865 2025] [proxy_fcgi:error] [pid 266776:tid 266776] [client 193.32.127.139:0] AH01071: Got error 'PHP message: PHP Deprecated:  Creation of dynamic property MO_Framework::$slider_manager is deprecated in /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes/enigmatic/framework/framework.php on line 292; PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function create_function() in /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes/enigmatic/framework/option-tree/includes/ot-functions-admin.php:69\nStack trace:\n#0 [internal function]: ot_register_theme_options_page()\n#1 /var/www/html/www.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(324): call_user_func_array()\n#2 /var/www/html/www.opensourceecology.org/htdocs/wp-includes/class-wp-hook.php(348): WP_Hook->apply_filters()\n#3 /var/www/html/www.opensourceecology.org/htdocs/wp-includes/plugin.php(517): WP_Hook->do_action()\n#4 /var/www/html/www.opensourceecology.org/htdocs/wp-settings.php(704): do_action()\n#5 /var/www/html/www.opensourceecology.org/wp-config.php(126): require_once('...')\n#6 /var/www/html/www.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\n#7 /var/www/html/www.opensourceecology.org/htdocs/wp-admin/admin.php(34): require_once('...')\n#8 /var/www/html/www.opensourceecology.org/htdo...'
</pre>
# allriht, it looks like there used to be a built-in function called 'create_function()' in php, but it was removed in PHP 8.0 https://www.php.net/manual/en/function.create-function.php
# hetzner2 had php v5.6 and hetzner3 has v8.2, so that tracks
# looks like we used to run theme v3.5 and now we run theme v3.6
## hetzner2
<pre>
[root@opensourceecology current]# sudo -u wp -i wp --path="/var/www/html/www.opensourceecology.org/htdocs/" theme list
...
+------------------------+----------+-----------+---------+
| name                  | status  | update    | version |
+------------------------+----------+-----------+---------+
| enigmatic.20170714.bak | inactive | none      | 3.1    |
| enigmatic              | active  | none      | 3.5    |
| twentyeleven          | inactive | available | 2.7    |
| twentyfifteen          | inactive | available | 1.9    |
| twentyfourteen        | inactive | available | 2.1    |
| twentyseventeen        | inactive | available | 1.4    |
| twentysixteen          | inactive | available | 1.4    |
| twentyten              | inactive | available | 2.4    |
| twentythirteen        | inactive | available | 2.3    |
| twentytwelve          | inactive | available | 2.4    |
+------------------------+----------+-----------+---------+
[root@opensourceecology current]#
</pre>
## hetzner3
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.opensourceecology.org_20250131/current # sudo -u wp -i wp --path="${docrootDir}" theme list
...
+-----------------+----------+-----------+---------+----------------+-------------+
| name            | status  | update    | version | update_version | auto_update |
+-----------------+----------+-----------+---------+----------------+-------------+
| enigmatic      | active  | none      | 3.6    |                | off        |
| twentyeleven    | inactive | available | 4.8    | 3.3            | off        |
| twentyfifteen  | inactive | available | 3.9    | 2.5            | off        |
| twentyfourteen  | inactive | available | 4.1    | 2.7            | off        |
| twentyseventeen | inactive | available | 3.8    | 2.2            | off        |
| twentysixteen  | inactive | available | 3.4    | 2.0            | off        |
| twentyten      | inactive | available | 4.3    | 2.9            | off        |
| twentythirteen  | inactive | available | 4.3    | 2.9            | off        |
| twentytwelve    | inactive | available | 4.4    | 3.0            | off        |
+-----------------+----------+-----------+---------+----------------+-------------+
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.opensourceecology.org_20250131/current #
</pre>
# PHP v8.0 came out in 2020 and became EOL in 2023 https://en.wikipedia.org/wiki/PHP
# geez..I wonder how long it's been since the latest version of this theme (v3.6) was released if it doesn't work on a version of PHP that was EOL 2 years ago?
# the URL listed for the theme points here, but it doesn't have any info about the theme; it's just a "Lorem ipsum" demo of the theme
<pre>
root@hetzner3 /var/www/html/www.opensourceecology.org # head htdocs/wp-content/themes/enigmatic/style.css
/*
Theme Name: Enigmatic
Theme URI: https://www.livemeshthemes.com/enigmatic
Description: A clean, responsive, seo-optimized Corporate theme from LiveMesh. Follow us on <a href="http://twitter.com/live_mesh">Twitter</a> for updates
Version: 3.6
Author: <a href="http://themeforest.net/user/livemesh">LiveMesh</a>
Author URI: http://themeforest.net/user/livemesh
License: GNU General Public License version 3.0
License URI: http://www.gnu.org/licenses/gpl-3.0.html
Tags: one-column, two-columns, three-columns, left-sidebar, right-sidebar, woocommerce, ecommerce, fixed-width, theme-options, threaded-comments, translation-ready
root@hetzner3 /var/www/html/www.opensourceecology.org #
</pre>
# this is their profile on themeforest, which lists all of the themes they have for sale http://themeforest.net/user/livemesh
# unfortunately, enigmatic is at the bottom of the list, and when you click it, it brings you to this page – which says "this item is no longer available" https://themeforest.net/item/enigmatic-responsive-multipurpose-wp-theme/3919108
# I can't find any changelog in the theme dir that suggets when it was last updated
# Catarina anticpated this, and she wanted to change to the oshine theme. I'm not a huge fan of using premium themes like this with their weird & bespoke page builders. it seems fragile and not future-proof
# in my own research for eco-libre, I've been looking at Neve https://wordpress.org/themes/neve/
## It's freemium. pro is $69-$259/yr
## 300,000+ active installs
## 1,023 5-star reviews
## responsive
## lots of starter sites https://themeisle.com/themes/neve/starter-sites/
## I had good interactions with the devs to my bug on github https://github.com/Codeinwp/neve/issues/4250
## in my trials, I couldn't install their "starter sties", and support wasn't very helpful https://wordpress.org/support/topic/how-to-download-starter-site/
# another great theme is just twentytwentyone https://wordpress.org/themes/twentytwentyone/
# but let's see if we just install oshine and switch to it
# here's how I copied the oshine theme and all its required plugins
<pre>
rsync -av --progress /var/tmp/wordpress/themes/oshin /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes/
rsync -av --progress /var/tmp/wordpress/plugins/be-portfolio-post /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box-conditional-logic /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box-show-hide /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/meta-box-tabs /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/oshine-core /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/oshine-modules /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
rsync -av --progress /var/tmp/wordpress/plugins/tatsu /var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/
</pre>
# oh, crap, well, I can't change the theme because I can't login because the existing theme is broken. So I have to get rid of the old theme first
<pre>
root@hetzner3 /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes # mv enigmatic enigmatic.die
root@hetzner3 /var/www/html/www.opensourceecology.org/htdocs/wp-content/themes #
</pre>
# alright, now I can refresh the wp wui after I logged-in, and the error goes away. now it's prompting me about a db upgrade. I did it.
# after switching to the oshine plugin, the website is pretty horribly broken.
## I mean at least I can login
## and the navbar at the top is there
## and the content is mostly there
## but the footer and lots of naked shortcodes are around the page#
# I sent an email to Marcin about this
<pre>
Hey Marcin,
 
I just finished migrating a snapshot of osemain to hetzner3. It's bad, and I don't know how you want to proceed.
 
I'm sorry to say that osemain didn't handle the upgrades very well.
 
Most of the website's content is surprisingly present, but the worst part is that you can't access the wp dashboard. After you login, you get a critical error.
 
The critical error is caused by your enigmatic theme trying to call a PHP function that was removed from PHP in 2020.
 
It looks like the version of your enigmatic theme that we downloaded from themeforest is v3.6. You have v3.5 installed on hetzner2. I checked the theme's websites. I couldn't find any datestampped changelog, but it appears that the theme is no longer for sale. I guess it hasn't been updated for many, many years.
 
* https://www.livemeshthemes.com/enigmatic
* http://themeforest.net/user/livemesh
* https://themeforest.net/item/enigmatic-responsive-multipurpose-wp-theme/3919108
 
Catarina mentioned wanting to switch from enigmatic to oshine. I went ahead and installed the latest version of oshine (and its many dependent plugins). After that, you're able to login. But the website becomes pretty horribly broken.
 
We should probably talk about what you want to do with this website as there's no easy solution.
 
Best case would be for you to hire someone to build you a new website with a better theme. Catarina suggested oshine. That would work, but I have a similar worry about the future of oshine. You might want to consider a more popular freemium theme, such as Neve, Astra, Hestia, OceanWP, TwentyTwentyOne, etc..
</pre>
# of course I could convert the website over to one of these themes, but I'm not an expert at it. So we could probably find someone cheaper (and faster) than me
# quick search on upwork shows some folks who cost half as much as I'm charging, and specifically mention experience with some of these themes (including neve, which is at the top of my list)
## for example Bhavesh J in India https://www.upwork.com/freelancers/~01e0c758d97b22e145
# if I could figure out some way to make ose on hetzner2 read-only (I tried researching this before, and didn't find a solution I liked), then perhaps I can actually do the fork and just give a new admin user to a wordpress dev like this and tell them "make this website (with neve installed) look like this old website"
## then, after they're done and it's verified and approved by me and marcin, we just change DNS over to the new site
## also, one alt to "making the wordpress site read-only is to just create an actual static site of osemain with wget and just replaced the docroot of the wordpress site with it on hetzner2 while we do the migration.
 
=Thr Jan 30, 2025=
 
# so yesterday I realized that my oswh static site backup had some 404 errors loading js depend assets
# today I checked and confirmed that there's a lot of those missing
## here's a list of the js files we have in the website's theme dir on hetzner2
<pre>
[root@opensourceecology oswh.opensourceecology.org]# find htdocs/wp-content/themes/Eventor -iname '*\.js'
htdocs/wp-content/themes/Eventor/script/countdown/jquery.countdown.js
htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js
htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors-min.js
htdocs/wp-content/themes/Eventor/script/animate-colors/my-animate-colors.js
htdocs/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js
htdocs/wp-content/themes/Eventor/script/horizontal/my-horizontal.js
htdocs/wp-content/themes/Eventor/script/spiner/spin.min.js
htdocs/wp-content/themes/Eventor/script/common.js
htdocs/wp-content/themes/Eventor/script/datepicker/js/jquery-ui-1.9.2.custom.min.js
htdocs/wp-content/themes/Eventor/script/datepicker/js/jquery-1.7.2.min.js
htdocs/wp-content/themes/Eventor/script/datepicker/js/jquery-ui-1.8.20.custom.min.js
htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.min.js
htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.js
htdocs/wp-content/themes/Eventor/script/anythingslider/my-anythingslider.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/swfobject.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.min.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.video.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.video.min.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.fx.min.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.easing.1.2.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.fx.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/newjavascript.js
htdocs/wp-content/themes/Eventor/script/contact/contact.js
htdocs/wp-content/themes/Eventor/script/menu/supersubs.js
htdocs/wp-content/themes/Eventor/script/menu/superfish.js
htdocs/wp-content/themes/Eventor/script/imagesloaded/imagesloaded.js
htdocs/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js
htdocs/wp-content/themes/Eventor/script/mailchimp/js/jquery-1.4.2.min.js
htdocs/wp-content/themes/Eventor/script/mailchimp/js/mailing-list.js
htdocs/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js
htdocs/wp-content/themes/Eventor/script/jscolor/jscolor.js
htdocs/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js
htdocs/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/plugin.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/js/base64.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/js/jquery.livequery.js
htdocs/wp-content/themes/Eventor/inc/shortcodes/js/popup.js
[root@opensourceecology oswh.opensourceecology.org]#
</pre>
## and here's the same results on hetzner3
<pre>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # find htdocs/wp-content/themes/Eventor -iname '*\.js'
htdocs/wp-content/themes/Eventor/script/contact/contact.js
htdocs/wp-content/themes/Eventor/script/jscolor/jscolor.js
htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.js
htdocs/wp-content/themes/Eventor/script/common.js
htdocs/wp-content/themes/Eventor/script/spiner/spin.min.js
htdocs/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js
htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js
htdocs/wp-content/themes/Eventor/script/menu/superfish.js
htdocs/wp-content/themes/Eventor/script/quickpager/quickpager.js
htdocs/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js
htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js
htdocs/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js
htdocs/wp-content/themes/Eventor/script/countdown/jquery.countdown.js
htdocs/wp-content/themes/Eventor/script/horizontal/jquery.js
root@hetzner3 /var/www/html/oswh.opensourceecology.org #
</pre>
# the files are missing too in the wget-produced source files on hetzner2 (before we did the mv to remove the version pinned get var from the filename)
<pre>
[root@opensourceecology oswh.opensourceecology.org]# find wp-content/themes/Eventor -iname '*\.js*'
wp-content/themes/Eventor/script/countdown/jquery.countdown.js?ver=4.9.1
wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js?ver=4.9.1
wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
wp-content/themes/Eventor/script/spiner/spin.min.js?ver=4.9.1
wp-content/themes/Eventor/script/pirobox/js/pirobox.js?ver=4.9.1
wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js?ver=4.9.1
wp-content/themes/Eventor/script/common.js?ver=4.9.1
wp-content/themes/Eventor/script/contact/contact.js?ver=4.9.1
wp-content/themes/Eventor/script/menu/superfish.js?ver=4.9.1
wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js?ver=4.9.1
wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1
wp-content/themes/Eventor/script/jscolor/jscolor.js?ver=4.9.1
wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js?ver=4.9.1
wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js?ver=4.9.1
[root@opensourceecology oswh.opensourceecology.org]#
</pre>
# I'll add this to add those missing files too
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/oswh.opensourceecology.org_20250129/current # rsync -av --progress --include="*.js" --include="*/" --exclude="*" var/www/html/${vhost_name}/htdocs/wp-content/themes/Eventor/ ${vhostDir}/htdocs/wp-content/themes/Eventor/
...
 
sent 392.159 bytes  received 765 bytes  785.848,00 bytes/sec
total size is 618.066  speedup is 1,57
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/oswh.opensourceecology.org_20250129/current #
</pre>
# after running that and fixing the permissions (and reloading the page), the carossel is working again. Now the errors look more sane:
<pre>
This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. oswh.opensourceecology.org
Loading mixed (insecure) display content “http://oswh.opensourceecology.org/wp-content/uploads/2013/02/DocHackFavicon.png” on a secure page FaviconLoader.sys.mjs:175:20
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. 2 oswh.opensourceecology.org
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://api.flattr.com/js/0.6/load.js?mode=auto oswh.opensourceecology.org:491:35
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
The resource at “https://api.flattr.com/js/0.6/load.js?mode=auto” was blocked because content blocking is enabled. oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org:491:35
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” oswh.opensourceecology.org
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” 2 oswh.opensourceecology.org
</pre>
# we can probably fix those last errors with some sed that replaces youtube and vimeo embeds from http to https
<pre>
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://www.youtube.com%https://www.youtube.com%g' '{}' \;
 
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://player.vimeo.com%https://player.vimeo.com%g' '{}' \;
 
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's%http://oswh.opensourceecology.org%https://oswh.opensourceecology.org%g' '{}' \;
<pre>
# ok, now the site looks good. I think we can call oswh done, pending Marcin's review.
# ...
# Let's move-on to OBI
# I followed the migration guide (as was used by microfactory), but the last steps (to activate the new plugins) failed
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
Error: Error establishing a database connection.
Error: Error establishing a database connection.
Error: Error establishing a database connection.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
</pre>
# loading it in the web browser failed with https cert error
# here's the logs for when I copied them over last https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Wed_Sep_25.2C_2024
# that says I had copied the entire dir, so it should include OBI's certs
# yep, hetnzer3 says it has them, and they're still valid until 2025-03-11
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: openbuildinginstitute.org
Serial Number: 37201823a671e8c6da8373cddd4efde6c6a
Key Type: RSA
Domains: www.openbuildinginstitute.org awstats.openbuildinginstitute.org openbuildinginstitute.org seedhome.openbuildinginstitute.org
Expiry Date: 2025-03-11 08:00:42+00:00 (VALID: 39 days)
Certificate Path: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem
  Certificate Name: opensourceecology.org
Serial Number: 3ec0988ac3af1baa0909ce9a9f4a6409c21
Key Type: RSA
Domains: fef.opensourceecology.org awstats.opensourceecology.org forum.opensourceecology.org microfactory.opensourceecology.org munin.opensourceecology.org opensourceecology.org oswh.opensourceecology.org phplist.opensourceecology.org staging.opensourceecology.org store.opensourceecology.org wiki.opensourceecology.org www.opensourceecology.org
Expiry Date: 2025-03-11 08:00:47+00:00 (VALID: 39 days)
Certificate Path: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/opensourceecology.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
</pre>
# ah, here's the error in the browser; looks like it's configured to use the wrong cert
<pre>
Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for www.openbuildinginstitute.org. The certificate is only valid for the following names: awstats.opensourceecology.org, fef.opensourceecology.org, forum.opensourceecology.org, microfactory.opensourceecology.org, munin.opensourceecology.org, opensourceecology.org, oswh.opensourceecology.org, phplist.opensourceecology.org, staging.opensourceecology.org, store.opensourceecology.org, wiki.opensourceecology.org, www.opensourceecology.org
Error code: SSL_ERROR_BAD_CERT_DOMAIN
</pre>
# https truncates at nginx
# here's the obi nginx config
<pre>
root@hetzner3 /etc/nginx # ls sites-enabled/
00-default.conf                    fef.opensourceecology.org.conf.20241230  munin.opensourceecology.org.conf        store.opensourceecology.org.conf
awstats.opensourceecology.org.conf  forum.opensourceecology.org.conf        oswh.opensourceecology.org.conf          wiki.opensourceecology.org.conf
fef.opensourceecology.org.conf      microfactory.opensourceecology.org.conf  seedhome.openbuildinginstitute.org.conf
root@hetzner3 /etc/nginx #
</pre>
# uhh, it's not there. right, so it's probably just the catchall fallback vhost that's triggering. we need to push it out with ansible.
# I uncommented this vhost and pushed it out with ansible https://github.com/OpenSourceEcology/ansible/commit/ec1ea4da19722268fdabd095ab7b7e2deee5246a
# I had to manaully restart nginx & apache on the server (before that the stale config redirected us to the forums, which is currently the default catchall site – until we migrate osemain)
# loading the site just returns an error
<pre>
Error establishing a database connection
</pre>
# so that's the same error that we saw on wp-cli
# I get the same error trying to login on the wui https://www.openbuildinginstitute.org/wp-login.php
# I actually don't have issues logging into the db using the creds copied from wp-config.php
<pre>
root@hetzner3 /var/www/html/www.openbuildinginstitute.org # mysql -u obi_user -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 950748
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> show databases;
ERROR 1227 (42000): Access denied; you need (at least one of) the SHOW DATABASES privilege(s) for this operation
MariaDB [(none)]>
 
MariaDB [(none)]> use obi_db;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
MariaDB [obi_db]> show tables;
+-------------------------------+
| Tables_in_obi_db              |
+-------------------------------+
| wp_commentmeta                |
| wp_comments                  |
...
+-------------------------------+
21 rows in set (0,000 sec)
 
MariaDB [obi_db]>
</pre>
# I checked to see if there's any  differences between users in this wp site vs others (maybe it's a host 127.0.0.1 vs localhost issue vs the old public ipv4 address)
<pre>
root@hetzner3 ~ # source /root/backups/backup.settings
root@hetzner3 ~ # mysql -u root -p${mysqlPass}
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 951030
Server version: 10.11.6-MariaDB-0+deb12u1 Debian 12
 
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]>
 
MariaDB [(none)]> use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
 
Database changed
MariaDB [mysql]>
 
MariaDB [mysql]> select Host, User from user;
+-----------+------------------+
| Host      | User            |
+-----------+------------------+
...
| localhost | obi2_user        |
| localhost | obi3_user        |
| localhost | obi_staging_user |
| localhost | obi_user        |
...
| localhost | oswh_user        |
...
25 rows in set (0,009 sec)
 
MariaDB [mysql]>
</pre>
# curiously, there's four distinct obi users, but all looks good with it
# ah, I found this in the wp-config.php
<pre>
/** MySQL hostname */
#define('DB_HOST', 'localhost');
define('DB_HOST', 'localhost:/var/lib/mysql/mysql.sock');
</pre>
# I'm actually surprised that I'm not already doing this for others, but it looks like mariadb ignores 'skip-networking' (which I have set) and instead listens on 'localhost' if you set that https://github.com/OpenSourceEcology/ansible/blob/0dc853cd08ce504ae5ccf7f51948edf7a3cce7dd/hetzner3/roles/maltfield.mariadb/templates/50-server.cnf.j2#L37-L40
# I added two idempotent sed commands to the migration CHG wiki article, which reverses the comments on these two lines
<pre>
# and use 'DB_HOST' of 'localhost'
sed -i "s%^#define('DB_HOST', 'localhost');%define('DB_HOST', 'localhost')%" ${vhostDir}/wp-config.php | grep -i DB_HOST
sed -i "s%^define('DB_HOST', 'localhost\:\(.*\)%#define('DB_HOST', 'localhost\:\1%" ${vhostDir}/wp-config.php | grep -i DB_HOST
</pre>
# re-running the plugin activate commands fails with new errors
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # # ACTIVATE NEW PLUGINS
 
activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
PHP Parse error:  syntax error, unexpected identifier "define" in phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1335) : eval()'d code on line 52
PHP Parse error:  syntax error, unexpected identifier "define" in phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1335) : eval()'d code on line 52
PHP Parse error:  syntax error, unexpected identifier "define" in phar:///home/wp/.wp-cli/wp-cli-2.11.0.phar/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1335) : eval()'d code on line 52
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
</pre>
# attempting to load the login page in the wui spits out a similar error to the apache logs
<pre>
[Thu Jan 30 22:16:06.053681 2025] [proxy_fcgi:error] [pid 3058592:tid 3058592] [client 198.54.133.36:0] AH01071: Got error 'PHP message: PHP Parse error:  syntax error, unexpected identifier "define" in /var/www/html/www.openbuildinginstitute.org/wp-config.php on line 54'
</pre>
# oh, it's because I'm missing a semicolon; I fixed it manually and in the wiki sed commands
# now activation works
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
# ACTIVATE NEW PLUGINS
 
activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'activitypub' activated.
Success: Activated 1 of 1 plugins.
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'aurora-heatmap' activated.
Success: Activated 1 of 1 plugins.
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'melapress-login-security' activated.
Success: Activated 1 of 1 plugins.
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
</pre>
# I was then able to login to the wui, but the css was broken
# I was still able to press the "button" to update the db, but then it redirected me to a page with just this crit error
<pre>
There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.
 
Learn more about troubleshooting WordPress.
</pre>
# apache errors complain about master slider again :(
<pre>
[Thu Jan 30 22:21:27.120144 2025] [proxy_fcgi:error] [pid 3071134:tid 3071134] [client 198.54.133.36:0] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught TypeError: ftp_fput(): Argument #1 ($ftp) must be of type FTP\\Connection, null given in /var/www/html/www.openbuildinginstitute.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php:212\nStack trace:\n#0 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php(212): ftp_fput()\n#1 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/masterslider/admin/includes/msp-admin-functions.php(141): WP_Filesystem_FTPext->put_contents()\n#2 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(141): msp_save_custom_styles()\n#3 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(116): Master_Slider_Admin->after_plugin_update()\n#4 [internal function]: Master_Slider_Admin->admin_init()\n#5 /var/www/html/www.openbuildinginstitute.org/htdocs/wp-includes/class-wp-hook.php(324): call_us...', referer: https://www.openbuildinginstitute.org/wp-admin/upgrade.php?step=1&backto=%2Fwp-admin%2Fprofile.php
</pre>
# I logged into the wui on the hetzner2 site
# it looks like both "slider revolution" and "master slider" are installed, so maybe we don't use it?
# yeah, I clicked on "Master Slider" and it says "no slider found" on the list of sliders available; so let's just do as we did on microfactory and just delete it
<pre>
# delete Master Slider plugin (PHP Fatal error:  Uncaught TypeError: ftp_fput)
rm -rf ${docrootDir}/wp-content/plugins/masterslider*
</pre>
# ok, after running that, I can load the wp admin dashboard in the wui
# I then logged-out on both wuis and did a side-by-side comparison
# it's looking decent, but there is a slider on the frontpage that's entirely missing on hetzner3
# actually the frontpage has two sliders. the top one loads fine, but the one below it (which appears to be a revslider div) doesn't appear at all
# I didn't see anything obvious in the js console
# and I didn't see any errors in the apache logs
# ok, in the wp dashboard wui, I entered the "revolution slider" settings page, and it lists two sliders
## the main page one is fine
## there's another named "Buildings Slider" that's the one at-issue
# if I open the "buildings slider" in the wp dashboard, then I see this in the js console
<pre>
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openbuildinginstitute.org/wp-content/uploads/revslider/Buildings/library_7.jpg. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing). Status code: 301.
 
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openbuildinginstitute.org/wp-content/uploads/revslider/Buildings/library_7.jpg. (Reason: CORS request did not succeed). Status code: (null).
 
Slider Revolution has successfully re-requested image to rectify above CORS error.
</pre>
# so I guess it's a CORS issue between 'openbuildinginstitute.org' and 'www.openbuildinginstitute.org'
# I tried fixing it with this change to the nginx config, but it didn't fix the homepage or make that error go away (which only appears in the admin page when editing the slider, anwyay)
<pre>
user@ose:~/sandbox_local/ansible/hetzner3$ git diff
...
diff --git a/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2
index b2b75ec..0e50caf 100644
--- a/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/www.openbuildinginstitute.org.conf.j2
@@ -65,7 +65,14 @@ server {
  proxy_set_header X-Forwarded-Proto https;
  proxy_set_header X-Forwarded-Port 443;
  proxy_set_header Host $host;
-  }
+
+              # handle cors whitelist for naked domain
+              proxy_hide_header Access-Control-Allow-Origin;
+              if ( $http_origin ~ "^https://(www.openbuildinginstitue.org|openbuildinginstitute.org)$" ) {
+                      add_header Access-Control-Allow-Origin $http_origin;
+              }
+
+      }
}
user@ose:~/sandbox_local/ansible/hetzner3$
</pre>
# I wonder if maybe revslider free changed something to only allow one slider per page?
# looks like we're changing 'revslider' from v5.4.7.4 to 6.7.25
# here's the changelog page
# https://www.sliderrevolution.com/documentation/changelog/
## christ, that page doesn't even list versions that old. it starts with "Version 6.0.1 (5th July 2019)"
# I tried editing the page, but it looks empty.
## there's a lower section below the content of the page with lots of input fields
## on the General Settings -> Slider Shortcode input field, it says "[rev_slider alias="Home Slider"]"
## so ^ that explains the top slider, but I can't find anywhere the second slider is referenced
# I tried editing the frontpage on hetzner2, and I found it
## below the actual content block, there's another block named "Page Builder"
## inside the Page Builder block there's several more blocks
## inside the second major block is another few sub blocks, one of which is white and says "PLUGIN SHORTCODES"
## if you hover over the text that says 'PLUGIN SHORTCODES', then several icons appear, one of which is a pencil
## if you click the pencil next to PLUGIN SHORTCODES, then a new modal window opens
## after waiting about 30 seconds, there's a new content box that appears with the contents of '[rev_slider alias="Buildings"]'
# back on hetzner3, this whole "page builder" block is entirely absent
# if I go to the plugins page, then I see a bunch of messages
<pre>
This theme requires the following plugins: Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Meta Box Framework and Safe SVG. The following recommended plugin is currently inactive: WPForms Lite. Begin installing plugins | Activate installed plugin | Dismiss this notice
 
The plugin amr-shortcode-any-widget/amr-shortcode-any-widget.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin be-page-builder/be-page-builder.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin be-themes-one-click-import/init.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist.
</pre>
# ah crap, looks like 'be-page-builder' is missing from our plugins dir
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current # ls -lah /var/tmp/wordpress/plugins | grep -i be-
drwxr-xr-x  7 root      root      4,0K Oct  9  2023 be-gdpr
-rw-r--r--  1 maltfield maltfield  306K Jan  6 01:50 be-gdpr.zip
drwxr-xr-x  4 root      root      4,0K Oct  4  2021 be-portfolio-post
-rw-r--r--  1 maltfield maltfield  56K Jan  6 01:50 be-portfolio-post.zip
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/www.openbuildinginstitute.org_20250130/current #
</pre>
# here's the plugins installed on hetzner2
<pre>
[root@opensourceecology ~]# sudo -u wp -i wp --path="/var/www/html/www.openbuildinginstitute.org/htdocs/" plugin list
...
+------------------------------------------------+----------+--------+---------+
| name                                          | status  | update | version |
+------------------------------------------------+----------+--------+---------+
| akismet                                        | inactive | none  | 4.0    |
| amr-shortcode-any-widget                      | active  | none  | 3.6    |
| be-themes-one-click-import                    | active  | none  | 1.6    |
| be-page-builder                                | active  | none  | 4.6.1  |
| be-portfolio-post                              | active  | none  | 1.1    |
| duplicate-page                                | active  | none  | 2.3    |
| wonderm00ns-simple-facebook-open-graph-tags    | active  | none  | 2.1.2  |
| force-strong-passwords                        | active  | none  | 1.7    |
| google-authenticator                          | active  | none  | 0.48    |
| google-authenticator-encourage-user-activation | active  | none  | 0.2    |
| masterslider                                  | active  | none  | 2.29.0  |
| meta-box-conditional-logic                    | active  | none  | 1.0.8  |
| meta-box-show-hide                            | active  | none  | 0.2.1  |
| meta-box-tabs                                  | active  | none  | 0.1.5  |
| oa-open-graph-for-fb                          | inactive | none  | 1.0.2  |
| open-in-new-window-plugin                      | active  | none  | 2.4    |
| rename-wp-login                                | active  | none  | 2.5.5  |
| revslider                                      | active  | none  | 5.2.5  |
| ssl-insecure-content-fixer                    | active  | none  | 2.5.0  |
| vcaching                                      | active  | none  | 1.6.7  |
| wordpress-importer                            | active  | none  | 0.6.3  |
+------------------------------------------------+----------+--------+---------+
[root@opensourceecology ~]#
</pre>
# it's curious that we're missing many of the plugins that oshine says are required, yet we do require this one that doesn't say it's required
# I did a quick search of the wiki, my notes, and the Brand Exponents website, but I couldn't find anything that mentioned 'be-page-builder'
# I asked them in an email
<pre>
Hello,
 
Can you please tell me where I can download the 'be-page-builder' plugin?
 
Thank you for providing the other plugins; that was very helpful in fixing one of our oshine-based websites.
 
Unfortunately, another site appears to require 'be-page-builder', which I cannot find on wordpress.org
 
* https://wordpress.org/plugin/be-page-builder
 
Can you please tell us where we can download the latest version of 'be-page-builder'?
</pre>
 
=Tue Jan 29, 2025=
 
# today I finished the setup of microfactory and documenting its CHG process on the wiki https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_microfactory_to_hetzner3#Change_Steps
# ...
# moving on, I replaced the failed migration of oswh into making it a static site (since, unlike microfactory, it uses a theme that appears to be broken on the latest version of wordpress, and it's no longer developed)
# I followed the same plan as documented for fef here https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_fef
# I had to fix some hard-coded 'fef' in ^ that wiki article
# after it was done, the frontpage verification looked mostly ok, but the carousel of sponsors at the bottom  were clearly broken (instead of being a dynamic carousel, they just all appear in a vertical column)
# also, some of the images on /be-a-partner/ were broken (well, some are broken on hetnzer2 too, but now there's more broken on hetnzer3)
## https://oswh.opensourceecology.org/wp-content/uploads/2013/02/Scratch-300x166.png
# also the image on this blog post was missing https://oswh.opensourceecology.org/oshw-docjam-how-to-make-a-satellite-jam/
## https://oswh.opensourceecology.org/wp-content/uploads/2013/04/544788_10151353594600753_63894451_n.jpg
# I re-ran the wget, this time writing to a log. But I didn't see any errors
# here's all the files that we have on hetzner3
<pre>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # find htdocs/wp-content/uploads/2013
htdocs/wp-content/uploads/2013
htdocs/wp-content/uploads/2013/02
htdocs/wp-content/uploads/2013/02/FileMarcinted-180x152.jpeg
htdocs/wp-content/uploads/2013/02/CM_sq-300x300.jpeg
htdocs/wp-content/uploads/2013/02/CM_sq-300x300-180x152.jpeg
htdocs/wp-content/uploads/2013/02/BecomeAPartner-175x120.png
htdocs/wp-content/uploads/2013/02/Scratch-175x120.png
htdocs/wp-content/uploads/2013/02/FileMarcinted.jpeg
htdocs/wp-content/uploads/2013/02/8-copia-180x152.jpg
htdocs/wp-content/uploads/2013/02/8-copia-275x257.jpg
htdocs/wp-content/uploads/2013/02/open-source-ecology-175x120.png
htdocs/wp-content/uploads/2013/02/FileMarcinted-275x252.jpeg
htdocs/wp-content/uploads/2013/02/logouishare-175x120.jpeg
htdocs/wp-content/uploads/2013/02/CM_sq-300x300-275x257.jpeg
htdocs/wp-content/uploads/2013/02/8-copia.jpg
htdocs/wp-content/uploads/2013/02/Copia-di-Scratch-175x120.png
htdocs/wp-content/uploads/2013/03
htdocs/wp-content/uploads/2013/03/ODE-175x120-175x120.png
htdocs/wp-content/uploads/2013/03/sarapis-logo-square-175x120.png
htdocs/wp-content/uploads/2013/03/kb2-175x120.jpg
htdocs/wp-content/uploads/2013/03/wi-175x120.jpg
htdocs/wp-content/uploads/2013/03/Alchematter-175x120.png
htdocs/wp-content/uploads/2013/03/Adafruit-175x120.png
htdocs/wp-content/uploads/2013/03/ITP2-175x120.png
htdocs/wp-content/uploads/2013/03/sparkfun-logo.bmp
htdocs/wp-content/uploads/2013/04
htdocs/wp-content/uploads/2013/04/icehubs-175x120.png
root@hetzner3 /var/www/html/oswh.opensourceecology.org #
</pre>
# and here's all the files from the latest wget (which I didn't see errors)
<pre>
[root@opensourceecology wget]# find oswh.opensourceecology.org/wp-content/uploads/
oswh.opensourceecology.org/wp-content/uploads/
oswh.opensourceecology.org/wp-content/uploads/2013
oswh.opensourceecology.org/wp-content/uploads/2013/02
oswh.opensourceecology.org/wp-content/uploads/2013/02/FileMarcinted.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/BecomeAPartner-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/02/open-source-ecology-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/02/8-copia-180x152.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/02/CM_sq-300x300-275x257.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/CM_sq-300x300-180x152.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/FileMarcinted-275x252.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/FileMarcinted-180x152.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/8-copia.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/02/8-copia-275x257.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/02/Copia-di-Scratch-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/02/logouishare-175x120.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/CM_sq-300x300.jpeg
oswh.opensourceecology.org/wp-content/uploads/2013/02/Scratch-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03
oswh.opensourceecology.org/wp-content/uploads/2013/03/sparkfun-logo.bmp
oswh.opensourceecology.org/wp-content/uploads/2013/03/kb2-175x120.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/03/ITP2-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/ODE-175x120-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/Alchematter-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/sarapis-logo-square-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/Adafruit-175x120.png
oswh.opensourceecology.org/wp-content/uploads/2013/03/wi-175x120.jpg
oswh.opensourceecology.org/wp-content/uploads/2013/04
oswh.opensourceecology.org/wp-content/uploads/2013/04/icehubs-175x120.png
[root@opensourceecology wget]#
</pre>
# the number of actual files is way, way higher.
<pre>
[root@opensourceecology wget]# find /var/www/html/oswh.opensourceecology.org/htdocs/wp-content/uploads/ -type f | wc -l
360
[root@opensourceecology wget]#
</pre>
# it's not obvious to me why wget omitted this
# maybe it's because the href desintation link is distinct from the image src?
# anyway, I think the easiest thing to do is to just extract from the tarball of files and put them in-place
<pre>
# STEP 5: Add all uploaded files (fix missing from wget)
tar -xjvf ${backupFileName_files_hetzner2}
rsync -av --progress var/www/html/${vhost_name}/htdocs/wp-content/uploads/ ${vhostDir}/htdocs/wp-content/uploads/
</pre>
# ok, adding ^ that fixed the broken links on the partners and the blog page
# it looks like the header photo is broken because it's trying to load 'opensourcewarehouse.org', not 'oswh.opensourceecology.org'
# I could probably fix this by doing some database find-and-replace CHG before converting it into a static site, but I'm not sure if it's worth the time/effort/money
# the carousal at the bottom of the page is still broken. looks like it's updated by some 'jcarousal' library. I confirmed that I can load that jcarousal javascript file on the new server; it's not missing https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1
# js console on-load has a ton of errors
<pre>
This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. index.html
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png headfooter-logonew.png
Mixed Content: Upgrading insecure display request ‘http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png’ to use ‘https’ headfooter-logonew.png
Loading mixed (insecure) display content “http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png” on a secure page FaviconLoader.sys.mjs:175:20
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1”. index.html:191:110
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1”. index.html:192:101
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1”. index.html:193:115
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1”. index.html:194:93
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1”. index.html:195:105
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1”. index.html:196:102
The resource from “https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1”. index.html:197:111
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1”. index.html:198:116
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1”. index.html:199:126
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1”. index.html:200:123
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1”. index.html:201:113
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1”. index.html:202:118
GET
http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png
NS_BINDING_ABORTED
 
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1”. index.html:203:102
A resource is blocked by OpaqueResponseBlocking, please check browser console for details. DocHackFavicon.png
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1”. index.html:204:102
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png headfooter-logonew.png
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Uncaught ReferenceError: jQuery is not defined
<anonymous> https://oswh.opensourceecology.org/index.html:325
index.html:325:13
Uncaught ReferenceError: jQuery is not defined
<anonymous> https://oswh.opensourceecology.org/index.html:402
index.html:402:21
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” index.html
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Mixed Content: Upgrading insecure display request ‘http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png’ to use ‘https’ headfooter-logonew.png
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” index.html
The resource from “https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0”. index.html:644:85
The resource from “https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” was blocked due to MIME type (“”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1”. index.html:645:81
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://api.flattr.com/js/0.6/load.js?mode=auto index.html:652:26
Uncaught ReferenceError: jQuery is not defined
<anonymous> https://oswh.opensourceecology.org/index.html:715
index.html:715:5
Uncaught ReferenceError: jQuery is not defined
<anonymous> https://oswh.opensourceecology.org/index.html:925
index.html:925:33
Loading failed for the <script> with source “https://api.flattr.com/js/0.6/load.js?mode=auto”. index.html:1:1
</pre>
# here's the same console output when loading the site on hetzner2
<pre>
This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. oswh.opensourceecology.org
Mixed Content: Upgrading insecure display request ‘http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/headfooter-logonew.png’ to use ‘https’ headfooter-logonew.png
Loading mixed (insecure) display content “http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png” on a secure page FaviconLoader.sys.mjs:175:20
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. node.js:416:1
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. jquery-1.7.2.min.js:3:32466
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0” oswh.opensourceecology.org
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. oswh.opensourceecology.org
Blocked loading mixed active content “http://player.vimeo.com/video/58165438” 2 oswh.opensourceecology.org
Loading failed for the <script> with source “https://api.flattr.com/js/0.6/load.js?mode=auto”. oswh.opensourceecology.org:1:1
GET
http://www.opensourcewarehouse.org/wp-content/uploads/2013/02/DocHackFavicon.png
NS_BINDING_ABORTED
 
A resource is blocked by OpaqueResponseBlocking, please check browser console for details. DocHackFavicon.png
</pre>
# so all these MIME type errors are unique to hetzner3
# google suggests it's actually because the file does not exist
# quick check confirms that they're all 404
<pre>
user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“//g' | tr "\n" " "
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1” https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0” https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1” user@disp7253:/tmp$
user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“//g' | tr "\n" " " | xargs firefox
</pre>
# but I think the issue is actually all these weird html-encoded characters at the end
# yep; all of these load fine
<pre>
user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“g' | sed 's/%E2%80%9Dg' | tr "\n" " "
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1 https://oswh.opensourceecology.org/wp-includes/js/imagesloaded.min.js%3Fver=3.2.0 https://oswh.opensourceecology.org/wp-includes/js/wp-embed.min.js%3Fver=4.9.1 user@disp7253:/tmp$
user@disp7253:/tmp$
user@disp7253:/tmp$
user@disp7253:/tmp$ cat urls.txt | grep MIME | awk '{print $4}' | sed 's/“g' | sed 's/%E2%80%9Dg' | tr "\n" " " | xargs firefox
user@disp7253:/tmp$
</pre>
# that's another red herring; if I click on it in the js console, it opens a new tab with the file. All that buffer crap was just added by firefox when I copied it out of the js console
# ah, I think I know the problem: this has some insight https://stackoverflow.com/a/53062522
# so the header 'X-Content-Type-Options: nosniff' makes the client not attempt to guess the MIME type
# our server should be telling the correct mime type because it's a javascript file
# but I think ^ that is determined by the file extension
# ...and the hetzer3 files are *not* using a '.js' extension – because wget is hard-coding these files with their get variable at the time they were scraped
# proof: only '.js' files on hetzner2
<pre>
[root@opensourceecology wget]# find /var/www/html/oswh.opensourceecology.org/ -type f | grep -i jcarousel
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor.old/script/horizontal/jquery.jcarousel.min.js
[root@opensourceecology wget]#
</pre>
# wheras on hetzer3, they don't end in '.js'
<pre>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # find /var/www/html/oswh.opensourceecology.org/ -type f | grep -i jcarousel
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
root@hetzner3 /var/www/html/oswh.opensourceecology.org #
</pre>
# in theory if I just manually rename the files by stripping everything after the '?', the webserver will still return the same file. let's try that
# alright, this renamed all the files
<pre>
js_files=$(find /var/www/html/oswh.opensourceecology.org/ -type f -iname '*\.js\?*')
for f in $js_files; do mv $f $(echo $f | sed 's/\.js\?.*/.js/'); done
</pre>
# here's the result after running that on hetzner3
<pre>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # find /var/www/html/oswh.opensourceecology.org/ -type f -iname '*\.js'
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/contact/contact.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/jscolor/jscolor.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/pirobox/js/pirobox.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/common.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/spiner/spin.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/menu/superfish.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/quickpager/quickpager.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/countdown/jquery.countdown.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-content/themes/Eventor/script/horizontal/jquery.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/js/imagesloaded.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/js/comment-reply.min.js
/var/www/html/oswh.opensourceecology.org/htdocs/wp-includes/js/wp-embed.min.js
root@hetzner3 /var/www/html/oswh.opensourceecology.org #
</pre>
# well that didn't work; the carousal is still broken and I'm still getting a ton of errors about MIME types
# here's a snippet from the actual root index.html file
<pre>
<link rel='stylesheet' id='tiny_css-css'  href='wp-content/themes/Eventor/inc/shortcodes/css/tinymce.css%3Fver=4.9.1.css' type='text/css' media='all' />
<script type='text/javascript' src='wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/menu/superfish.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/common.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/pirobox/js/pirobox.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/contact/contact.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/countdown/jquery.countdown.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/spiner/spin.min.js%3Fver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/jscolor/jscolor.js%3Fver=4.9.1'></script>
</pre>
# I wonder if the issue is the '%3F' instead of '?'
# I edited the file to replaced the '%3F' with '?
<pre>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # sed 's/\.js%3F/.js?/' htdocs/index.html | grep jquery
<script type='text/javascript' src='wp-content/themes/Eventor/script/jquery/jquery-1.7.2.min.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/easing/jquery.easing.1.3.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/elastic/jquery.elastic.source.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/anythingslider/js/jquery.anythingslider.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/animate-colors/jquery.animate-colors.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/countdown/jquery.countdown.js?ver=4.9.1'></script>
<script type='text/javascript' src='wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1'></script>
root@hetzner3 /var/www/html/oswh.opensourceecology.org # sed 's/\.js%3F/.js?/' htdocs/index.html | grep jquery^C
You have mail in /var/mail/root
root@hetzner3 /var/www/html/oswh.opensourceecology.org #
 
root@hetzner3 /var/www/html/oswh.opensourceecology.org # sed -i 's/\.js%3F/.js?/' htdocs/index.html
root@hetzner3 /var/www/html/oswh.opensourceecology.org #
</pre>
# I reloaded the page and, yeah, it's fixed. So wget is breaking things :(
# ah ha! someone else is complaining about this on SO too https://stackoverflow.com/questions/22486047/wget-is-converting-question-marks-to-3f-causing-404s
## they said that adding '--restrict-file-names="ascii" helped
# I tried ^  that, but it still had the same problem; I think we'll just have to clean it up after
# let's try this (for both js and css)
<pre>
find -type f -iname '*\.html' -exec sed -i 's/\.js%3F/.js?/' htdocs/index.html '{}' \;
find -type f -iname '*\.html' -exec sed -i 's/\.css%3F/.css?/' htdocs/index.html '{}' \;
js_files=$(find /var/www/html/oswh.opensourceecology.org/htdocs/ -type f -iname '*\.js\?*')
for f in $js_files; do mv $f $(echo $f | sed 's/\.js\?.*/.js/'); done
css_files=$(find /var/www/html/oswh.opensourceecology.org/htdocs/ -type f -iname '*\.css\?*')
for f in $css_files; do mv $f $(echo $f | sed 's/\.css\?.*/.css/'); done
</pre>
# I updated the process with this on the wiki, re-did the migration on hetzner3, and now it works. great!
# I guess that, while I'm editing these files produced by wget, one more thing I could do is to replaced all instances of 'opensourcewarehouse.org' with 'oswh.opensourceecology.org', which would fix some more things like the broken header image
<pre>
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse.org/oswh.opensourceecology.org/' '{}' \;
</pre>
# this last one is specific to oswh, so I created a new CHG just for it here https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_oswh
# for some reason ^ that command isn't working
## well there's one issue that 'www.opensourcewarehouse.org' becomes 'www.oswh.opensourceecology.org' – which isn't going to work
## and another issue where there's a bunch of orphaned 'opensourcewarehouse.org' strings still :(
<pre>
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/www\.opensourcewarehouse\.org/oswh.opensourceecology.org/' '{}' \;
find ${vhostDir}/htdocs/ -type f -iname '*\.html' -exec sed --in-place=.`date "+%Y%m%d_%H%M%S"` 's/opensourcewarehouse\.org/oswh.opensourceecology.org/' '{}' \;
</pre>
# oh, I had permissions issues. now the header logo is fixed. nice!
# oh wait, but the carousal is broken again :(
# here's the current console messages on refresh
<pre>
This page is in Almost Standards Mode. Page layout may be impacted. For Standards Mode use “<!DOCTYPE html>”. index.html
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT
 
Loading mixed (insecure) display content “http://oswh.opensourceecology.org/wp-content/uploads/2013/02/DocHackFavicon.png%E2%80%9D on a secure page FaviconLoader.sys.mjs:175:20
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT
 
Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. node.js:416:1
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT
 
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/quickpager/quickpager.jquery.js?ver=4.9.1%E2%80%9D. index.html:193:113
GET
https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1
NS_ERROR_CORRUPTED_CONTENT
 
The resource from “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1%E2%80%9D was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff). index.html
Loading failed for the <script> with source “https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/horizontal/jquery.jcarousel.min.js?ver=4.9.1%E2%80%9D. index.html:202:116
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Referrer Policy: Ignoring the less restricted referrer policy “no-referrer-when-downgrade” for the cross-site request: https://api.flattr.com/js/0.6/load.js?mode=auto index.html:652:26
Blocked loading mixed active content “http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0%E2%80%9D index.html
Rewriting old-style YouTube Flash embed (http://www.youtube.com/v/9xGRaPrcvVg?version=3&autohide=1&showinfo=0) to iframe embed (http://www.youtube.com/embed/9xGRaPrcvVg?version=3&autohide=1&showinfo=0). Please update page to use iframe instead of embed/object, if possible. index.html
Uncaught TypeError: jQuery(...).jcarousel is not a function
<anonymous> https://oswh.opensourceecology.org/wp-content/themes/Eventor/script/common.js?ver=4.9.1:117
jQuery 11
common.js:117:27
Blocked loading mixed active content “http://player.vimeo.com/video/58165438%E2%80%9D index.html
Loading failed for the <script> with source “https://api.flattr.com/js/0.6/load.js?mode=auto%E2%80%9D. index.html:1:1
</pre>
# ok, that's actually a MIME red-herring. It's returning 'text/html' because it's a 404 error message.
 
=Tue Jan 28, 2025=
 
# here's TOFU 3/3 (ISP, exit in Ecuador)
<pre>
Ecuador
2025-01-28
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. .
 
 
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
################################################################################## 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
################################################################################## 100.0%
2025-01-28
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
0a8dfec431583aac98eaa4c454a8be0641a54dc93704413c56bc861a8816a563  plugin.json
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
</pre>
# well, a diff of this round of 3TOFUs shows everything matching, except 'woocommerce-gateway-stripe', which changed versions in-between our TOFUs
## it's not a big deal; we still have an older version TOFUd on the server
<pre>
root@hetzner3 /var/tmp/wordpress/plugins # ls | grep -i gate
coingate-for-woocommerce
coingate-for-woocommerce.2.1.1.zip
woocommerce-gateway-stripe
woocommerce-gateway-stripe.9.0.0.zip
root@hetzner3 /var/tmp/wordpress/plugins #
</pre>
# I uploaded all these to hetzner3
# I updated the CHG article to indicate the newer versions
# ...
# ok, now that 3TOFU is out of the way, we're unblocked on microfactory.opensourceecology.org, www.openbuildinginstitute.org, and www.opensourceecology.org, and phplist.opensourceecology.org
# let's start with microfactory
# I went through the install script for microfactory again (but this time it should have copied the oshine themes and dependent plugins)
# this time after login, the website appeared pretty broken still
# but after I logged-in and clicked the db upgrade, the website looked great!
## oh, except only when I'm logged-in. If I open another browser while not logged-in, it's still broken
## also, I get an error after logging-in, when I get redirected to here https://microfactory.opensourceecology.org/wp-admin/profile.php
<pre>
There has been a critical error on this website. Please check your site admin email inbox for instructions. If you continue to have problems, please try the support forums.
 
Learn more about troubleshooting WordPress.
</pre>
# here's the critical error from the apache logs
<pre>
==> microfactory.opensourceecology.org/error.log <==
[Wed Jan 29 00:04:53.421065 2025] [proxy_fcgi:error] [pid 3214134:tid 3214134] [client 146.70.74.100:0] AH01071: Got error 'PHP message: MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg; PHP message: MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_3_6389662388.jpg; PHP message: MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_5_2197896279.jpg; PHP message: PHP Fatal error:  Uncaught TypeError: ftp_fput(): Argument #1 ($ftp) must be of type FTP\\Connection, null given in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php:212\nStack trace:\n#0 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-admin/includes/class-wp-filesystem-ftpext.php(212): ftp_fput()\n#1 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider/admin/includes/msp-admin-functions.php(141): WP_Filesystem_FTPext->put_contents()\n#2 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(141): msp_save_custom_styles()\n#3 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider/admin/class-master-slider-admin.php(116): Master_Slider_Admin->after_plugin_update()\n#4 [internal function]: Master_Slider_Admin->admin_init()\n#5 /var/www/html/microfactory.opensourceecology.org/htdocs/wp-includes/c...'
</pre>
# indeed, the file it wants does not exist
<pre>
root@hetzner3 ~ # ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg
ls: cannot access '/var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg': No such file or directory
root@hetzner3 ~ #
 
root@hetzner3 ~ # ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/
total 8,0K
drwxrwx--- 2 not-apache www-data 4,0K Sep 26  2018 .
drwxrwx--- 3 not-apache www-data 4,0K Sep 26  2018 ..
root@hetzner3 ~ #
</pre>
# curiously, it doesn't exist on the old server either. hmm. maybe this is one of those issues where it was a warning on old php and its a crit error on new php
<pre>
[root@opensourceecology ~]# ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/
total 8.0K
drwxrwx--- 2 apache apache 4.0K Sep 26  2018 .
drwxrwx--- 3 apache apache 4.0K Sep 26  2018 ..
[root@opensourceecology ~]#
</pre>
# I tailed the logs on the old sever as I logged-in there, and I got nothing
# I also grepped the old logs, and I saw nothing similar
<pre>
[root@opensourceecology httpd]# zgrep 'MS_Aq_Resize.process' microfactory.opensourceecology.org/error_log-*.gz
[root@opensourceecology httpd]# zgrep 'imageedit_5_2197896279.jpg' microfactory.opensourceecology.org/error_log-*.gz
[root@opensourceecology httpd]#
</pre>
# I'm wondering if we can just disable this master slider plugin, or if our site actually uses it
# I checked the master slider section of the wp dashboard on the old site, and it showed one entry named 'v1-home' = "v1 Home copy"
# clicking preview just displayed a spinner that stuck forever
# I checked the logs and, yep, now it's spitting out errors about missing files
<pre>
[Wed Jan 29 00:14:05.177783 2025] [:error] [pid 1296] [client 127.0.0.1:40418] MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_1_8284603983.jpg, referer: https://microfactory.opensourceecology.org/wp-admin/admin.php?page=masterslider
[Wed Jan 29 00:14:05.178387 2025] [:error] [pid 1296] [client 127.0.0.1:40418] MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_3_6389662388.jpg, referer: https://microfactory.opensourceecology.org/wp-admin/admin.php?page=masterslider
[Wed Jan 29 00:14:05.178944 2025] [:error] [pid 1296] [client 127.0.0.1:40418] MS_Aq_Resize.process() error: Image file does not exist (or is not an image): /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/uploads/2016/02/imageedit_5_2197896279.jpg, referer: https://microfactory.opensourceecology.org/wp-admin/admin.php?page=masterslider
[Wed Jan 29 00:14:09.005403 2025] [:error] [pid 2552] [client 127.0.0.1:40742] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 1). Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "312"] [id "960012"] [rev "1"] [msg "POST request missing Content-Length Header."] [data "0"] [severity "WARNING"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"] [tag "CAPEC-272"] [hostname "microfactory.opensourceecology.org"] [uri "/wp-cron.php"] [unique_id "Z5ly0cY3Bv263PeWaEUCWwAAAAg"]
</pre>
# so I just disabled the plugin on hetzner3
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # ls -lah /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider
masterslider/    masterslider.zip 
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # rm -rf /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/masterslider*
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
</pre>
# I refreshed the wp dashboard, and now it loads
# at the top, it yells at me saying that it wants Master Slider, as well as other plugins tho
<pre>
This theme recommends the following plugins: Master Slider, Meta Box Framework and Safe SVG.
The following recommended plugin is currently inactive: WPForms Lite.
</pre>
# if I'm not logged-in, the site still looks pretty broken; showing this at the top
<pre>
[tatsu_section bg_color= “rgba(229,238,243,1)” bg_image= “https://microfactory.opensourceecology.org/wp-content/uploads/2018/10/ose-workshop.jpg” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= “top left” bg_size= “cover” bg_animation= “none” padding= ‘{“d”:”80px 0px 120px 0px “,”l”:”0px 0px 60px 0px “,”t”:”125px 0px 125px 0px “,”m”:”120px 0px 120px 0px “}’ margin= ‘{“d”:”0-px 0px 0px 0px “}’ border= “0px 0px 0px 0px” border_color= “” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “1” overlay_color= “rgba(255,255,255,0.63)” full_screen= “1” top_divider= “none” bottom_divider= “none” top_divider_height= ‘{“d”:100}’ top_divider_position= “above” bottom_divider_height= ‘{“d”:”165″}’ bottom_divider_position= “below” top_divider_color= “rgba(255,255,255,1)” bottom_divider_color= “#ffffff” invert_top_divider= “0” invert_bottom_divider= “0” flip_top_divider= “0” flip_bottom_divider= “1” section_id= “home” section_class= “” section_title= “” offset_section= “0” offset_value= “0px” full_screen_header_scheme= “background–light” hide_in= “0” key= “ryhdp1iKX”][tatsu_row full_width= “0” bg_color= “rgba(255,255,255,0)” no_margin_bottom= “0” equal_height_columns= “0” gutter= “medium” column_spacing= “px” fullscreen_cols= “0” swap_cols= “0” padding= ‘{“d”:”100px 0px 0px 0px “}’ row_id= “” row_class= “” hide_in= “0” box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” border_radius= “0” layout= “2/3+1/3” key= “BJx2daJsK7”][tatsu_column bg_color= “” bg_image= “” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= ‘{“d”:”top left”}’ bg_size= ‘{“d”:”cover”}’ padding= ‘{“d”:”0px 50px 0% 0% “,”m”:”0% 0% 0% 0% “}’ custom_margin= “0” margin= ‘{“d”:”0px 0px 0px 0px”}’ border= ‘{“d”:”0px 0px 0px 0px”}’ border_color= “” border_radius= “0” enable_box_shadow= “0” box_shadow_custom= “0 0 15px 0 rgba(198,202,202,0.4)” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “0” overlay_color= “” animate_overlay= “none” link_overlay= “” vertical_align= “middle” column_offset= “0” offset= “0px 0px” z_index= “0” column_parallax= “0” column_width= ‘{“d”:66.67,”l”:66.67,”t”:66.67,”m”:100}’ column_mobile_spacing= “0” animate= “0” animation_type= “fadeIn” animation_delay= “0” image_hover_effect= “none” column_hover_effect= “none” hover_box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” col_id= “” column_class= “” hide_in= “0” layout= “2/3” key= “r1-3OTyjF7”][tatsu_inline_text max_width= ‘{“d”:”100″}’ wrap_alignment= “center” animate= “1” animation_type= “fadeIn” animation_delay= “100” margin= ‘{“d”:”0px 0px 20px 0px “}’ bg_color= “” box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” padding= ‘{“d”:”0px 0px 0px 0px”}’ border_radius= “0” key= “Hyzh_pksKm”][/tatsu_inline_text][tatsu_inline_text max_width= ‘{“d”:”100″}’ wrap_alignment= “center” animate= “1” animation_type= “fadeIn” animation_delay= “100” margin= ‘{“d”:”0px 0px 20px 0px “}’ bg_color= “” box_shadow= “0px 0px 0px 0px rgba(0,0,0,0)” padding= ‘{“d”:”0px 0px 0px 0px”}’ border_radius= “0” key= “r1m3u6ksF7”]
</pre>
# I confirmed that the tatsu plugin is installed and activated
# on the plugin page, we have a bunch of other error messages at the top
<pre>
The plugin colorhub/colorhub.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist.
 
The plugin typehub/typehub.php has been deactivated due to an error: Plugin file does not exist.
 
Some required plugins are missing or inactive.
</pre>
# oh, I realized the issue with the site being broken while not logged-in is because it was cached; let's clear that
<pre>
root@hetzner3 ~ # varnishadm 'ban req.url ~ "."'
 
You have new mail in /var/mail/root
root@hetzner3 ~ #
</pre>
# I did a quick visual check of all the pages, and they look identical on hetzner2 and hetzner3. I'd say this is done!
# so the only thing we need to add is to delete the master slider plugin after install
# I created a new wiki article for the steps for this migration, which will diverge from store https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_microfactory_to_hetzner3
# the versions to change are different now
# here's what we have on hetzner2
<pre>
[root@opensourceecology ~]# sudo -u wp -i wp --path="/var/www/html/microfactory.opensourceecology.org/htdocs/" plugin list
...
+------------------------------------------------+----------+--------+---------+
| name                                          | status  | update | version |
+------------------------------------------------+----------+--------+---------+
| akismet                                        | active  | none  | 4.0.8  |
| be-gdpr                                        | active  | none  | 1.0.0  |
| be-portfolio-post                              | active  | none  | 1.1    |
| colorhub                                      | active  | none  | 1.0.1  |
| duplicate-page                                | active  | none  | 2.7    |
| force-strong-passwords                        | active  | none  | 1.8.0  |
| google-authenticator                          | active  | none  | 0.48    |
| google-authenticator-encourage-user-activation | active  | none  | 0.2    |
| hello                                          | inactive | none  | 1.7    |
| masterslider                                  | active  | none  | 3.2.2  |
| meta-box-conditional-logic                    | active  | none  | 1.3    |
| meta-box-show-hide                            | active  | none  | 1.0.2  |
| meta-box-tabs                                  | active  | none  | 1.0.0  |
| oshine-core                                    | active  | none  | 1.3.4  |
| oshine-modules                                | active  | none  | 2.2.5  |
| rename-wp-login                                | active  | none  | 2.5.5  |
| revslider                                      | active  | none  | 5.4.7.4 |
| ssl-insecure-content-fixer                    | active  | none  | 2.7.0  |
| tatsu                                          | active  | none  | 2.6.8  |
| typehub                                        | active  | none  | 1.2    |
| vcaching                                      | active  | none  | 1.6.9  |
+------------------------------------------------+----------+--------+---------+
[root@opensourceecology ~]#
 
[root@opensourceecology ~]# sudo -u wp -i wp --path="/var/www/html/microfactory.opensourceecology.org/htdocs/" theme list
...
+-----------------+----------+--------+---------+
| name            | status  | update | version |
+-----------------+----------+--------+---------+
| oshin          | active  | none  | 6.5    |
| twentyeleven    | inactive | none  | 2.8    |
| twentyfifteen  | inactive | none  | 2.0    |
| twentyfourteen  | inactive | none  | 2.2    |
| twentyseventeen | inactive | none  | 1.7    |
| twentysixteen  | inactive | none  | 1.5    |
| twentyten      | inactive | none  | 2.5    |
| twentythirteen  | inactive | none  | 2.4    |
| twentytwelve    | inactive | none  | 2.5    |
+-----------------+----------+--------+---------+
[root@opensourceecology ~]#
</pre>
# and here's hetzner3
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # sudo -u wp -i wp --path="${docrootDir}" plugin list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+------------------------------------------------+----------+--------+----------+----------------+-------------+
| name                                          | status  | update | version  | update_version | auto_update |
+------------------------------------------------+----------+--------+----------+----------------+-------------+
| activitypub                                    | active  | none  | 4.6.0    |                | off        |
| akismet                                        | active  | none  | 5.3.5    |                | off        |
| aurora-heatmap                                | active  | none  | 1.7.0    |                | off        |
| be-gdpr                                        | active  | none  | 1.1.6    |                | off        |
| be-portfolio-post                              | active  | none  | 1.1.1    |                | off        |
| bulk-media-register                            | inactive | none  | 1.40    |                | off        |
| advanced-nocaptcha-recaptcha                  | inactive | none  | 7.5.0    |                | off        |
| duplicate-page                                | active  | none  | 4.5      |                | off        |
| enable-media-replace                          | inactive | none  | 4.1.5    |                | off        |
| google-authenticator-encourage-user-activation | active  | none  | 0.2      |                | off        |
| extensions-leaflet-map                        | inactive | none  | 4.4.3    |                | off        |
| google-authenticator                          | active  | none  | 0.54    |                | off        |
| hcaptcha-for-forms-and-more                    | inactive | none  | 4.9.0    |                | off        |
| hello                                          | inactive | none  | 1.7.2    |                | off        |
| include-mastodon-feed                          | inactive | none  | 1.9.9    |                | off        |
| leaflet-map                                    | inactive | none  | 3.4.1    |                | off        |
| melapress-login-security                      | active  | none  | 2.0.1    |                | off        |
| meta-box-conditional-logic                    | active  | none  | 1.6.23  |                | off        |
| meta-box-show-hide                            | active  | none  | 1.3.1    |                | off        |
| meta-box-tabs                                  | active  | none  | 1.1.18  |                | off        |
| oshine-core                                    | active  | none  | 1.6.1    |                | off        |
| oshine-modules                                | active  | none  | 3.3.8    |                | off        |
| raw-html                                      | inactive | none  | 1.6.4    |                | off        |
| regenerate-thumbnails                          | inactive | none  | 3.1.6    |                | off        |
| related-posts-by-taxonomy                      | inactive | none  | 2.7.6    |                | off        |
| revslider                                      | active  | none  | 6.7.25  |                | off        |
| smart-slider-3                                | inactive | none  | 3.5.1.26 |                | off        |
| spam-destroyer                                | inactive | none  | 2.1.4    |                | off        |
| ssl-insecure-content-fixer                    | active  | none  | 2.7.2    |                | off        |
| tatsu                                          | active  | none  | 3.5.3    |                | off        |
| vcaching                                      | active  | none  | 1.8.3    |                | off        |
| woocommerce-multilingual                      | inactive | none  | 5.3.9    |                | off        |
| woocommerce-gateway-stripe                    | inactive | none  | 9.0.0    |                | off        |
| wp-2fa                                        | inactive | none  | 2.8.0    |                | off        |
| wpforms-lite                                  | inactive | none  | 1.9.2.3  |                | off        |
| wpfront-notification-bar                      | inactive | none  | 3.4.2    |                | off        |
| wp-pgp-encrypted-emails                        | inactive | none  | 0.8.0    |                | off        |
| wp-qrcode                                      | inactive | none  | 1.1.1    |                | off        |
| wps-hide-login                                | inactive | none  | 1.9.17.1 |                | off        |
| wordpress-seo                                  | inactive | none  | 24.1    |                | off        |
+------------------------------------------------+----------+--------+----------+----------------+-------------+
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
 
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # sudo -u wp -i wp --path="${docrootDir}" theme list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+-----------------+----------+--------+---------+----------------+-------------+
| name            | status  | update | version | update_version | auto_update |
+-----------------+----------+--------+---------+----------------+-------------+
| oshin          | active  | none  | 7.2.1  |                | off        |
| twentyeleven    | inactive | none  | 4.8    |                | off        |
| twentyfifteen  | inactive | none  | 3.9    |                | off        |
| twentyfourteen  | inactive | none  | 4.1    |                | off        |
| twentyseventeen | inactive | none  | 3.8    |                | off        |
| twentysixteen  | inactive | none  | 3.4    |                | off        |
| twentyten      | inactive | none  | 4.3    |                | off        |
| twentythirteen  | inactive | none  | 4.3    |                | off        |
| twentytwelve    | inactive | none  | 4.4    |                | off        |
+-----------------+----------+--------+---------+----------------+-------------+
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
</pre>
 
 
=Fri Jan 17, 2025=
 
# here's TOFU 2/3 (VPN, exit in Switzerland)
<pre>
Switzerland
2025-01-17
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. .
 
 
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.1.1.zip
######################################################################### 100.0%
2025-01-17
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
16fb41570f8e66b11358c9dfcd754384018af22b0d8c6a19dff50b2ce005a0f5  plugin.json
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
2a958f50e458b900d8cd2d7b980e93e37ca720eebf3c7b4a5f94ed5d9d167079  woocommerce-gateway-stripe.9.1.1.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
</pre>
 
=Sun Jan 05, 2025=
 
# here's TOFU 3/3 (ISP, exit in Ecuador)
<pre>
Ecuador
2025-01-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
INFO: Determining Latest Version of Wordpress Themes
. . . . . . . . . . . .
 
 
WARNING: Failed to download plugin duplicate-page
null
null
 
WARNING: Failed to download plugin hello
Plugin not found.
null
 
WARNING: Failed to download plugin f
closed
This plugin has been closed as of March 25, 2012 and is not available for download. Reason: Unknown.
 
WARNING: Failed to download theme oshin
Theme not found
null
 
https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
-=O=-                                        #    #    #    #               
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
  -=O=-          #    #    #    #                                           
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
  -=O#-  #    #    #                                               
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
##O=#  #                                                                     
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
-=#=-#  #    #                                                               
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
##O#- #                                                                       
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
  -=O=-                            #  # ##
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
-#O=- #    #    #                                                     
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
  #    #    #    #                      -=O=-                       
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
##                                                                -=O=-       
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
##O#- #                                                                       
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
##O#- #                                                                       
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/akismet.5.3.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/classic-editor.1.6.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/contact-form-7.6.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.9.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/jetpack.14.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/shareaholic.9.7.13.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.26.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce.9.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyeleven.4.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfifteen.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfourteen.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentynineteen.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyseventeen.3.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentysixteen.3.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyten.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentythirteen.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwelve.4.4.zip
######################################################################### 100.0%
2025-01-06
33656ccc5752ff7c519d328ac4b9819116aa4e2195b436edc132a9362d1a23f6  activitypub.4.6.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
9e72d6c60c8e3adc9b202bd6ad8659605289cd0921d5f9726dabf24e06a50a16  akismet.5.3.5.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0  black-studio-tinymce-widget.2.7.3.zip
371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213  bouquet.1.2.5.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86  chartbeat.2.0.7.zip
4b2b45b19c61f627ff8730222692a691023dea3435b35b8db95a2418b45ece65  classic-editor.1.6.7.zip
8b1475dfaa0f8a73f8a374286a52f59b6f3d3573e80ea1134d3a527498bdd7ea  coingate-for-woocommerce.2.3.0.zip
445152a678b9e307b52e4e96b4acff9de4744efc25c55b16ec45aec731356268  CologneBlue-REL1_42-007b63c.tar.gz
eb0eb22bdfc8b1cb553274064732c406bb173d7f43e73212c8220986e5f6fb1d  CologneBlue-REL1_43-9134d2e.tar.gz
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
32bd9b76da748f0bda96588ffa50d51604365097e21e930325a5bd9817393240  ConfirmAccount-REL1_43-bb470fd.tar.gz
d3880e7405c6842e164b0f974b9ed55e1a48006f86a2b28ed511128865323d89  contact-form-7.6.0.2.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058  google-authenticator.0.54.zip
9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e  google-authenticator-encourage-user-activation.0.2.zip
c61bab4e64eeff3d519e5a24990f5ba8223651f13aed2e0fb69b904419e62c50  hcaptcha-for-forms-and-more.4.9.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
4d824673ad59e74c5bdfc9c6b6c87341b2d0f0879641b72381612aed30335758  jetpack.14.1.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
08f15bb64ddbdf0813ec5e581515cedae5e8aee9d9a3b0c5e4b2726a5a2ac982  meta-box.5.10.5.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
7ccbc9703a0f194ad3219383de1d12f28e373598ed2448e07145a096d6598a0a  Modern-REL1_42-c8a7124.tar.gz
07e855134780a5f1f9925d391f7eaed6f968f553988237ccc261d7e41ffb5829  Modern-REL1_43-5597681.tar.gz
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
7e1fc73f395bc93f958c8fd5418e0b2acc63dcbaf5acba4c044e67c489e546a0  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
2ee4950e6c2bb47ac2614a02529697a59aee8fb967f094a329d78e9626395f92  safe-svg.2.3.1.zip
97960efc1647253080c82739a47ea8b5239257e05065a3a7f24f5e3b5935e7b7  shareaholic.9.7.13.zip
514d1156304ffff54883b4dff57a789b36983d163a12276127ad9c2399f6614d  smart-slider-3.3.5.1.26.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
fb26419580bbc8f0d35842fd98a1a9bf44851453d20c373fadb8b488272547a4  ssl-insecure-content-fixer.2.7.2.zip
576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311  storefront.4.6.0.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
e33538c5a99ac6c6d083f5e70fdf390c5471247e47a44751b6a7a531b903287f  twentyeleven.4.8.zip
dc3c9337a95ff0619f8f4f2bbe494d0caf445af7e050b93f0de818e799e2a850  twentyfifteen.3.9.zip
91b68629634e181b3b5ba2852f97ae9abdbc4cf3455026264a0d5e323db88907  twentyfourteen.4.1.zip
3b7adce871a143bca30dbfb421a69b005cc53e252c9c9a0e4d768571f6c5be3e  twentynineteen.3.0.zip
166035cbf15c3d618b5d2c11d1aa365b93d597f32735aa6f35887f4828673377  twentyseventeen.3.8.zip
04f14d0210cf72797d29c60132d2de4dc2ea00fffffcb824aa9e95fba385c7d4  twentysixteen.3.4.zip
21cd8f5688fc21493becee40bfb99a57837750f5ef6accca7abce5bcb1c37b46  twentyten.4.3.zip
6016a433cd38023181b00d6e44a14b976bc4963ce5fe97033b6af6046360411f  twentythirteen.4.3.zip
f6a5b6c3c8691f9a75f95d9e6675889a217cb57a8339e37b8aef182a7a4a4eaf  twentytwelve.4.4.zip
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
7b64058a82e2a33d452ae8172ae394dce603c788fd3d8b1cad9c8894a5b4bd30  UserMerge-REL1_43-f9d2664.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c  vcaching.1.8.3.zip
543e34e5dbb2ef74af9ada84767e9118e200bb8a1046a5d49a03c7e462b9a13e  Widgets-REL1_42-17dbd92.tar.gz
7946e4c3c970792a9de655dce79fa68bad2023084bb24e8d01694b4d17329778  Widgets-REL1_43-50da5c6.tar.gz
c38427b9bebd799ff97e68593247dbbb1eef3a3ecd0503c0026c32fdc779de74  woocommerce.9.5.1.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
ce79a4dace0e87caa9314d9b773a9663506556cb78cd19d388522fc5401185bd  wordpress-seo.24.1.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d  wp-super-cache.1.12.4.zip
user@disp8573:/tmp/tmp.cHDv7hw2dy$
</pre>
# ok, we do have some files that didn't match on all 3 TOFUs
## coingate-for-woocommerce wordpress plugin
## ConfirmAccount MediaWiki extension (for v4.2)
## extensions-leaflet-map wordpress plugin
## UserMerge MediaWiki extension (for v4.2)
# the coingate plugin says that it changed from v2.2.0 to v2.3.0, so that makes sense that it would change
# the leaflet-map extension says it changed from v4.4.2 to v4.4.3, so that also makes sense
# the two mediawiki extensions were supposed to be tied to a tag, so I wouldn't expect them to change. But I guess it might make sense if they moved the tags. Not a very good release process, if so.
## we're probably are going to want the v4.3 versions of both the MediaWiki plugins, so this probably isn't an issue
# anyway, I'll keep these ones that were verified and we can re-verify the ones that failed. Here's a follow-up 3TOFU script
<pre>
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
#
#          For more info on 3TOFU (and why this is important), see:
#          * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################
 
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
 
REMOTE_FILES="https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz"
WARNINGS=""
 
# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
 
tmpDir=`mktemp -d`
pushd "${tmpDir}"
 
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
 
# and today's date
date -u +"%Y-%m-%d"
 
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
 
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
 
plugins='extensions-leaflet-map woocommerce-gateway-stripe'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
 
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
 
echo -e "${WARNINGS}"
echo
 
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
 
# checksum
date -u +"%Y-%m-%d"
sha256sum *
</pre>
# here's TOFU 1/3 (Tor, exit in the Netherlands)
<pre>
Congratulations. This browser is configured to use Tor.
2025-01-06
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. .
 
 
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
2025-01-06
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
5c0b2db8a3f0d4f61fa140725ef4bde2553d477209a35ebb0476289afde45fa3  extensions-leaflet-map.4.4.3.zip
902ae1c09846dc522c9688d375b944375566b8e1980c38355a131ff48108b555  plugin.json
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
user@host:/tmp/user/1000/tmp.ddVAgjDjeV$
</pre>
 
=Fri Jan 03, 2025=
 
# here's TOFU 2/3 (VPN, exit in Belgium)
<pre>
Belgium
2025-01-03
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
INFO: Determining Latest Version of Wordpress Themes
. . . . . . . . . . . .
 
 
WARNING: Failed to download plugin duplicate-page
null
null
 
WARNING: Failed to download plugin hello
Plugin not found.
null
 
WARNING: Failed to download plugin f
closed
This plugin has been closed as of March 25, 2012 and is not available for download. Reason: Unknown.
 
WARNING: Failed to download theme oshin
Theme not found
null
 
https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
-#O#- #  #                                                                   
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
  -#O=#  #    #                                                             
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
  -=#=-  #    #    #                                                     
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
-#O#- #  #                                                                   
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
-#O#- #  #                                                                   
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
-#O#- #  #                                                                   
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
  -=O=-                            #  # # #
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
  -=O#-  #    #    #                                                     
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
-=O=-                      #    #    #  #       
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
-=O=-                              ##
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
-#O#- #  #                                                                   
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
-=#=#  #  #                                                                 
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/akismet.5.3.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/classic-editor.1.6.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/contact-form-7.6.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.9.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/jetpack.14.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/shareaholic.9.7.13.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.26.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce.9.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyeleven.4.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfifteen.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfourteen.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentynineteen.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyseventeen.3.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentysixteen.3.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyten.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentythirteen.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwelve.4.4.zip
######################################################################### 100.0%
2025-01-03
33656ccc5752ff7c519d328ac4b9819116aa4e2195b436edc132a9362d1a23f6  activitypub.4.6.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
9e72d6c60c8e3adc9b202bd6ad8659605289cd0921d5f9726dabf24e06a50a16  akismet.5.3.5.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0  black-studio-tinymce-widget.2.7.3.zip
371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213  bouquet.1.2.5.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86  chartbeat.2.0.7.zip
4b2b45b19c61f627ff8730222692a691023dea3435b35b8db95a2418b45ece65  classic-editor.1.6.7.zip
8b1475dfaa0f8a73f8a374286a52f59b6f3d3573e80ea1134d3a527498bdd7ea  coingate-for-woocommerce.2.3.0.zip
445152a678b9e307b52e4e96b4acff9de4744efc25c55b16ec45aec731356268  CologneBlue-REL1_42-007b63c.tar.gz
eb0eb22bdfc8b1cb553274064732c406bb173d7f43e73212c8220986e5f6fb1d  CologneBlue-REL1_43-9134d2e.tar.gz
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  ConfirmAccount-REL1_42-7405319.tar.gz
32bd9b76da748f0bda96588ffa50d51604365097e21e930325a5bd9817393240  ConfirmAccount-REL1_43-bb470fd.tar.gz
d3880e7405c6842e164b0f974b9ed55e1a48006f86a2b28ed511128865323d89  contact-form-7.6.0.2.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
c58461cb4ab04746356a9bd6809da1cf699ffaad7f3be3987c8e6e2cd1830023  extensions-leaflet-map.4.4.2.zip
b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058  google-authenticator.0.54.zip
9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e  google-authenticator-encourage-user-activation.0.2.zip
c61bab4e64eeff3d519e5a24990f5ba8223651f13aed2e0fb69b904419e62c50  hcaptcha-for-forms-and-more.4.9.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
4d824673ad59e74c5bdfc9c6b6c87341b2d0f0879641b72381612aed30335758  jetpack.14.1.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
08f15bb64ddbdf0813ec5e581515cedae5e8aee9d9a3b0c5e4b2726a5a2ac982  meta-box.5.10.5.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
7ccbc9703a0f194ad3219383de1d12f28e373598ed2448e07145a096d6598a0a  Modern-REL1_42-c8a7124.tar.gz
07e855134780a5f1f9925d391f7eaed6f968f553988237ccc261d7e41ffb5829  Modern-REL1_43-5597681.tar.gz
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
1ed7cade320a6be1aec419d5b30d04e71c9e9692165de5ecc6c1df8b75e21d11  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
2ee4950e6c2bb47ac2614a02529697a59aee8fb967f094a329d78e9626395f92  safe-svg.2.3.1.zip
97960efc1647253080c82739a47ea8b5239257e05065a3a7f24f5e3b5935e7b7  shareaholic.9.7.13.zip
514d1156304ffff54883b4dff57a789b36983d163a12276127ad9c2399f6614d  smart-slider-3.3.5.1.26.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
fb26419580bbc8f0d35842fd98a1a9bf44851453d20c373fadb8b488272547a4  ssl-insecure-content-fixer.2.7.2.zip
576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311  storefront.4.6.0.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
e33538c5a99ac6c6d083f5e70fdf390c5471247e47a44751b6a7a531b903287f  twentyeleven.4.8.zip
dc3c9337a95ff0619f8f4f2bbe494d0caf445af7e050b93f0de818e799e2a850  twentyfifteen.3.9.zip
91b68629634e181b3b5ba2852f97ae9abdbc4cf3455026264a0d5e323db88907  twentyfourteen.4.1.zip
3b7adce871a143bca30dbfb421a69b005cc53e252c9c9a0e4d768571f6c5be3e  twentynineteen.3.0.zip
166035cbf15c3d618b5d2c11d1aa365b93d597f32735aa6f35887f4828673377  twentyseventeen.3.8.zip
04f14d0210cf72797d29c60132d2de4dc2ea00fffffcb824aa9e95fba385c7d4  twentysixteen.3.4.zip
21cd8f5688fc21493becee40bfb99a57837750f5ef6accca7abce5bcb1c37b46  twentyten.4.3.zip
6016a433cd38023181b00d6e44a14b976bc4963ce5fe97033b6af6046360411f  twentythirteen.4.3.zip
f6a5b6c3c8691f9a75f95d9e6675889a217cb57a8339e37b8aef182a7a4a4eaf  twentytwelve.4.4.zip
8351c0267c2cd7866ff04c04261f06cd75af9a7130aac848ca43fd047404e229  UserMerge-REL1_42-41759d0.tar.gz
7b64058a82e2a33d452ae8172ae394dce603c788fd3d8b1cad9c8894a5b4bd30  UserMerge-REL1_43-f9d2664.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c  vcaching.1.8.3.zip
543e34e5dbb2ef74af9ada84767e9118e200bb8a1046a5d49a03c7e462b9a13e  Widgets-REL1_42-17dbd92.tar.gz
7946e4c3c970792a9de655dce79fa68bad2023084bb24e8d01694b4d17329778  Widgets-REL1_43-50da5c6.tar.gz
c38427b9bebd799ff97e68593247dbbb1eef3a3ecd0503c0026c32fdc779de74  woocommerce.9.5.1.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
ce79a4dace0e87caa9314d9b773a9663506556cb78cd19d388522fc5401185bd  wordpress-seo.24.1.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d  wp-super-cache.1.12.4.zip
user@disp8342:/tmp/tmp.i0oBc9groR$
</pre>
 
=Wed Jan 01, 2025=
 
# Happy new year!
# This year my major goal is to finish the [[hetzner3]] project
# The hetzner3 project is a migration of all of OSE's websites from the old server (hetzner2 running CentOS) to the new dedicated server (hetzner3 running Debian)
# I finished last year with the following
## forum.opensourceecology.org - I finished copying a snapshot of the forums to hetnzer3; it passed my initial spot checks and looks OK
## store.opensourceecology.org - I finished copying a snapshot of this website to hetnzer3; it's horribly broken, but it's an empty website so that's fine. Marcin said we can drop it entirely, to simplify things
## microfactory.opensourceecology.org - I finished copying a snapshot of this website to hetnzer3; it was pretty broken, and it's blocked until we finish obtaining the oshine theme's required (paid) plugins
## fef.opensourceecology.org - I finished migrating a snapshot of this website to hetnzer3; it had a critical error and theme was >12 years old. We decided to convert it to a static site, which I generated and finished copying to hetzner3; it passed my initial spot checks and looks OK
## oswh.opensourceecology.org - I finished migrating a snapshot of this website to hetnzer3; it uses a paid theme called "eventor". I found the website that sells the theme, but couldn't see any version history. I contacted the author 6 days ago, and they didn't (yet) respond. Marcin said we can convert it to a static site. I'm waiting to hear back from the theme author. If they don't reply in a couple weeks, I'll convert it to a static site.
## seedhome.openbuildinginstitute.org - I finished copying a snapshot of this website to hetnzer3; it's an empty site and looks fine. Marcin said we can drop it entirely, to simplify things.
## www.openbuildinginstitute.org - I haven't begun on this. It's blocked until we finish obtaining the oshine theme's required (paid) plugins
## www.opensourceecology.org - I haven't begun on this. Catarina wants to transition it to the oshine theme.
## phplist.opensourceecology.org - I haven't tried to copy this to hetzner3 yet. I did start a 3TOFU verification of the latest release, but the hash changed after 3TOFU. I created a question on the forums about this, and I'm waiting for a response to clarify what happened.
## wiki.opensourceecology.org - I have copied this to hetzner2, but it's a long upgrade process that I'm still working on.
# ...
# ok, where we left off before was trying figure out the URLs to download these oshine required paid plugins (which are not on wordpress.org)
# I was successfully able to to mitm a wordpress install (using mitmproxy) on a Disposable VM as oshine tried to download its dependent plugins
# I found that I could press 'e' to export a given capture in mitmproxy as a cURL command, but that only worked for one request at a time
# I found that I could limit all the captures shown to *just* those that downloaded a .zip by pressing the 'f' button (for "filter") and then typing:
<pre>
: set view_filter '~t application/zip'
</pre>
# there is a complete list of filters and their definitions (that's how I found that ~t is 'content-type') here https://docs.mitmproxy.org/stable/concepts-filters/
# ok, the oshine wordpress theme downloaded 10 zip files. Here's an export of all those cURL commands
<pre>
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# for file in $(ls -1 | sort -V); do echo "==${file}=="; cat $file; echo; echo; done
==c1==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
 
==c2==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
 
==c3==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
 
==c4==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/oshin-plugins/be-portfolio-post.zip' -H 'Connection: close' https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
 
==c5==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
 
==c6==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
 
==c7==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
 
==c8==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
 
==c9==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
 
==c10==
curl -H 'User-Agent: WordPress/6.1.6; http://localhost' -H 'Accept: */*' --compressed -H 'Referer: https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip' -H 'Connection: close' https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
 
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#
</pre>
# here's just the URLs
<pre>
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# for file in $(ls -1 | sort -V); do cat $file | awk '{print $NF}'; echo; echo; done | sort | uniq
 
https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#
</pre>
# cool, I'm able to download all of them with just a simple wget; this will work for 3TOFU
<pre>
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# wget https://brandexponents.com/oshin-plugins/be-portfolio-post.zip https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
...
--2025-01-01 15:57:44--  https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
Reusing existing connection to brandexponents.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: ‘tatsu.zip’
 
tatsu.zip              [    <=>            ]  15.81M  1.96MB/s    in 7.8s   
 
2025-01-01 15:57:52 (2.03 MB/s) - ‘tatsu.zip’ saved [16579153]
 
FINISHED --2025-01-01 15:57:52--
Total wall clock time: 22s
Downloaded: 10 files, 47M in 19s (2.49 MB/s)
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#
 
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# sha256sum *.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#
 
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl# du -sh *.zip
308K be-gdpr.zip
56K be-portfolio-post.zip
1.6M masterslider.zip
8.0K meta-box-conditional-logic.zip
8.0K meta-box-show-hide.zip
8.0K meta-box-tabs.zip
18M oshine-core.zip
1.7M oshine-modules.zip
11M revslider.zip
16M tatsu.zip
root@disp2713:/var/lib/wordpress/wp-content/plugins/curl#
</pre>
# so the three plugins that are missing from the above list are the three recommended ones that can be downloaded (most easily & safely) from wordpress.org:
## meta-box
## safe-svg
## wpforms-lite
# I used hetzner3:/usr/local/bin/wordpress_3tofu.sh to generate a 3TOFU script
<pre>
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
#
#          For more info on 3TOFU (and why this is important), see:
#          * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:10:08+00:00
################################################################################
 
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
 
REMOTE_FILES=""
WARNINGS=""
 
# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
 
tmpDir=`mktemp -d`
pushd "${tmpDir}"
 
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
 
# and today's date
date -u +"%Y-%m-%d"
 
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
 
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
 
plugins='  activitypub akismet aurora-heatmap bulk-media-register advanced-nocaptcha-recaptcha duplicate-page enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed leaflet-map melapress-login-security raw-html regenerate-thumbnails related-posts-by-taxonomy smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce-multilingual woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wordpress-seo  activitypub akismet aurora-heatmap bulk-media-register advanced-nocaptcha-recaptcha classic-editor contact-form-7 enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed leaflet-map melapress-login-security meta-box raw-html regenerate-thumbnails related-posts-by-taxonomy smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce woocommerce-multilingual coingate-for-woocommerce woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wordpress-seo  activitypub akismet aurora-heatmap bulk-media-register advanced-nocaptcha-recaptcha enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed leaflet-map melapress-login-security raw-html regenerate-thumbnails related-posts-by-taxonomy smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce-multilingual woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wordpress-seo  activitypub akismet aurora-heatmap black-studio-tinymce-widget bulk-media-register advanced-nocaptcha-recaptcha chartbeat enable-media-replace google-authenticator-encourage-user-activation extensions-leaflet-map google-authenticator hcaptcha-for-forms-and-more hello include-mastodon-feed jetpack leaflet-map melapress-login-security raw-html regenerate-thumbnails related-posts-by-taxonomy shareaholic smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce-multilingual woocommerce-gateway-stripe wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wp-super-cache wordpress-seo '
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
 
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
 
themes='  oshin twentyeleven twentyfifteen twentyfourteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve  oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve  twentyeleven twentyfifteen twentyfourteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve  bouquet twentyeleven twentyfifteen twentyfourteen twentyseventeen twentysixteen twentyten twentythirteen '
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
 
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
 
echo -e "${WARNINGS}"
echo
 
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
 
# checksum
date -u +"%Y-%m-%d"
sha256sum *
</pre>
# oh shit, not only is 'wpforms-lite' already in the list -- it's in the list 4 times! I guess I need to update the script to do a de-dupe
# I updated the script
<pre>
root@hetzner3 /usr/local/bin # diff wordpress_3tofu.sh.20250101 wordpress_3tofu.sh
5c5
< # Version: 0.2
---
> # Version: 0.3
11c11
< # Updated: 2024-12-12
---
> # Updated: 2025-01-01
53a54,56
> # remove duplicates and trim whitespace
> plugins="$(echo "${plugins}" | tr " " "\n" | sort | uniq | tr "\n" " " | xargs)"
>
64a68,70
> #
> # remove duplicates and trim whitespace
> themes="$(echo "${themes}" | tr " " "\n" | sort | uniq | tr "\n" " " | xargs)"
root@hetzner3 /usr/local/bin #
</pre>
# ok, this run looks much better
<pre>
root@hetzner3 /usr/local/bin # wordpress_3tofu.sh
 
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
#
#          For more info on 3TOFU (and why this is important), see:
#          * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################
 
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
 
REMOTE_FILES=""
WARNINGS=""
 
# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
 
tmpDir=`mktemp -d`
pushd "${tmpDir}"
 
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
 
# and today's date
date -u +"%Y-%m-%d"
 
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
 
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
 
plugins='activitypub advanced-nocaptcha-recaptcha akismet aurora-heatmap black-studio-tinymce-widget bulk-media-register chartbeat classic-editor coingate-for-woocommerce contact-form-7 duplicate-page enable-media-replace extensions-leaflet-map google-authenticator google-authenticator-encourage-user-activation hcaptcha-for-forms-and-more hello include-mastodon-feed jetpack leaflet-map melapress-login-security meta-box raw-html regenerate-thumbnails related-posts-by-taxonomy shareaholic smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce woocommerce-gateway-stripe woocommerce-multilingual wordpress-seo wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wp-super-cache'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
 
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
 
themes='bouquet oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
 
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
 
echo -e "${WARNINGS}"
echo
 
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
 
# checksum
date -u +"%Y-%m-%d"
sha256sum *
 
You have mail in /var/mail/root
root@hetzner3 /usr/local/bin #
</pre>
# I manually edited the script above to include 'safe-svg' and 'meta-box' ('wpforms-lite' were already present)
# I also added some remote files at the top manually, including the paid plugins and phpList
# I also added the extensions for mediawiki
# I realized that previously I had chosen the extensions for the release of mediawiki v1.42 because it said that was the latest stable version
# in fact, that drop down is wrong. the latest stable LTS version of mediawiki is 1.43. To be safe, I added both.
# as for mediawiki skins, it looks like 1.43 ships with 4 already
<pre>
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # ls -lah /var/tmp/mediawiki/mediawiki-1.43.0/skins/
total 28K
drwxr-xr-x  6 root root  4,0K Dec 29 18:14 .
drwxr-xr-x 14 root root  4,0K Dec 29 18:14 ..
drwxr-xr-x 11 root root  4,0K Dec 29 18:14 MinervaNeue
drwxr-xr-x  7 root root  4,0K Dec 29 18:14 MonoBook
-rw-r--r--  1  501 staff 1,3K Dec  5 15:41 README
drwxr-xr-x  6 root root  4,0K Dec 29 18:14 Timeless
drwxr-xr-x  9 root root  4,0K Dec 29 18:14 Vector
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
</pre>
# we're using these
<pre>
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep skins LocalSettings.php
## The relative URL path to the skins directory
# $wgStylePath        = "$wgScriptPath/skins";
#require_once "$IP/skins/CologneBlue/CologneBlue.php";
#require_once "$IP/skins/Modern/Modern.php";
#require_once "$IP/skins/MonoBook/MonoBook.php";
#require_once "$IP/skins/Vector/Vector.php";
#$wgRightsIcon = "${wgScriptPath}/skins/common/images/gnu-fdl.png";
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
</pre>
# so it appears that we just need to download CologneBlue & Modern.
## https://www.mediawiki.org/wiki/Skin:Cologne_Blue
## https://www.mediawiki.org/wiki/Skin:Modern
# alright, here's the 3TOFU script
<pre>
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
#
#          For more info on 3TOFU (and why this is important), see:
#          * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################
 
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
 
REMOTE_FILES="https://brandexponents.com/oshin-plugins/be-portfolio-post.zip https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz"
WARNINGS=""
 
# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
 
tmpDir=`mktemp -d`
pushd "${tmpDir}"
 
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
 
# and today's date
date -u +"%Y-%m-%d"
 
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
 
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
 
plugins='activitypub advanced-nocaptcha-recaptcha akismet aurora-heatmap black-studio-tinymce-widget bulk-media-register chartbeat classic-editor coingate-for-woocommerce contact-form-7 duplicate-page enable-media-replace extensions-leaflet-map google-authenticator google-authenticator-encourage-user-activation hcaptcha-for-forms-and-more hello include-mastodon-feed jetpack leaflet-map melapress-login-security f raw-html regenerate-thumbnails related-posts-by-taxonomy shareaholic smart-slider-3 spam-destroyer ssl-insecure-content-fixer vcaching woocommerce woocommerce-gateway-stripe woocommerce-multilingual wordpress-seo wp-2fa wpforms-lite wpfront-notification-bar wp-pgp-encrypted-emails wp-qrcode wps-hide-login wp-super-cache safe-svg meta-box'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
 
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
 
themes='bouquet oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
 
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
 
echo -e "${WARNINGS}"
echo
 
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
 
# checksum
date -u +"%Y-%m-%d"
sha256sum *
</pre>
# and here's TOFU 1/3 (Tor, exit in Austria)
<pre>
Congratulations. This browser is configured to use Tor.
2025-01-01
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Plugins
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
INFO: Determining Latest Version of Wordpress Themes
. . . . . . . . . . . .
 
 
WARNING: Failed to download plugin duplicate-page
null
null
 
WARNING: Failed to download plugin hello
Plugin not found.
null
 
WARNING: Failed to download plugin f
closed
This plugin has been closed as of March 25, 2012 and is not available for download. Reason: Unknown.
 
WARNING: Failed to download theme oshin
Theme not found
null
 
https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
-=O=-                  #    #    #    #                                 
https://brandexponents.com/wp/wp-content/uploads/be-gdprzip1166524dfc6ec328/be-gdpr.zip
              -=O=-                            #    #    #    #             
https://brandexponents.com/wp/wp-content/uploads/mastersliderzip37866595e7a5ff66/masterslider.zip
              -=O=-                      #    #    #    #                 
https://brandexponents.com/wp/wp-content/uploads/meta-box-conditional-logiczip162366f26163a3ab4/meta-box-conditional-logic.zip
-=O=-      #    #    #    #                                               
https://brandexponents.com/wp/wp-content/uploads/meta-box-show-hidezip13166f2613ab6f0c/meta-box-show-hide.zip
-=O=-            #    #    #    #                                         
https://brandexponents.com/wp/wp-content/uploads/meta-box-tabszip111866f2610e07388/meta-box-tabs.zip
-=O=-          #    #    #    #                                           
https://brandexponents.com/wp/wp-content/uploads/oshine-corezip161662b43283bd8e/oshine-core.zip
                                  -=O=- #    #    #    #                   
https://brandexponents.com/wp/wp-content/uploads/oshine-moduleszip33866595cdba6de4/oshine-modules.zip
                                  -=O=-                                    ##
https://brandexponents.com/wp/wp-content/uploads/revsliderzip67256769089e02bf8/revslider.zip
    #  #  #    #                                      -=O=-                 
https://brandexponents.com/wp/wp-content/uploads/tatsuzip353674d4ab76b1de/tatsu.zip
                        -=O=-                                            #  ###
https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.tgz
######################################################################### 100.0%
https://netix.dl.sourceforge.net/project/phplist/phplist/3.6.15/phplist-3.6.15.zip
######################################################################### 100.0%
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.zip
-=O=-        #    #    #    #                                               
https://github.com/phpList/phplist3/archive/refs/tags/v3.6.15.tar.gz
-=O=-    #    #    #    #                                                     
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_42-7405319.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_42-17dbd92.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_43-f9d2664.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/ConfirmAccount-REL1_43-bb470fd.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/Widgets-REL1_43-50da5c6.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_42-c8a7124.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_42-007b63c.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/Modern-REL1_43-5597681.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/skins/CologneBlue-REL1_43-9134d2e.tar.gz
######################################################################### 100.0%
https://downloads.wordpress.org/release/wordpress-6.7.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/activitypub.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/akismet.5.3.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/black-studio-tinymce-widget.2.7.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/chartbeat.2.0.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/classic-editor.1.6.7.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/coingate-for-woocommerce.2.2.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/contact-form-7.6.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator.0.54.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.9.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/jetpack.14.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/shareaholic.9.7.13.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.26.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/vcaching.1.8.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce.9.5.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wordpress-seo.24.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/wp-super-cache.1.12.4.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/bouquet.1.2.5.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/storefront.4.6.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyeleven.4.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfifteen.3.9.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyfourteen.4.1.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentynineteen.3.0.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyseventeen.3.8.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentysixteen.3.4.zip
/######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentyten.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentythirteen.4.3.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwelve.4.4.zip
######################################################################### 100.0%
2025-01-01
33656ccc5752ff7c519d328ac4b9819116aa4e2195b436edc132a9362d1a23f6  activitypub.4.6.0.zip
101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9  advanced-nocaptcha-recaptcha.7.5.0.zip
9e72d6c60c8e3adc9b202bd6ad8659605289cd0921d5f9726dabf24e06a50a16  akismet.5.3.5.zip
873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53  aurora-heatmap.1.7.0.zip
68ea2fab097534aef1c9af02f3f97c4ce61d7ecd83da08bcdc02c7c84f6a9a02  be-gdpr.zip
c6c0bdd19bd97f7bdd25e2fd2b503ff929b45ecb9ea372b3bfb8171c51bf7718  be-portfolio-post.zip
206469c81b5b7ec3ab189d3471c996db868a30f5238f7d9165d7541ee66cd8a0  black-studio-tinymce-widget.2.7.3.zip
371916ddd49d1b23fd2b62d571a59734dcae3b2c9d4cbb405999626e2f27f213  bouquet.1.2.5.zip
5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f  bulk-media-register.1.40.zip
7b8f9bba64b316e7e2888c97b9d257a8195e855bfa6c9c74c537fc3eac01ce86  chartbeat.2.0.7.zip
4b2b45b19c61f627ff8730222692a691023dea3435b35b8db95a2418b45ece65  classic-editor.1.6.7.zip
0c3c2b80926f43c5c7ca77fda54e00240af616a541705ae7118af14f38ab48c6  coingate-for-woocommerce.2.2.0.zip
445152a678b9e307b52e4e96b4acff9de4744efc25c55b16ec45aec731356268  CologneBlue-REL1_42-007b63c.tar.gz
eb0eb22bdfc8b1cb553274064732c406bb173d7f43e73212c8220986e5f6fb1d  CologneBlue-REL1_43-9134d2e.tar.gz
f5c873c0e7aecae24a5e209d589db344ee199b0e7fa78906606ec5705b3b8752  ConfirmAccount-REL1_42-7405319.tar.gz
32bd9b76da748f0bda96588ffa50d51604365097e21e930325a5bd9817393240  ConfirmAccount-REL1_43-bb470fd.tar.gz
d3880e7405c6842e164b0f974b9ed55e1a48006f86a2b28ed511128865323d89  contact-form-7.6.0.2.zip
ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0  enable-media-replace.4.1.5.zip
c58461cb4ab04746356a9bd6809da1cf699ffaad7f3be3987c8e6e2cd1830023  extensions-leaflet-map.4.4.2.zip
b520e85b0c2904439b8aaec9c46a94650105899685f11d97e6b5035a568f2058  google-authenticator.0.54.zip
9cd1687d133a4b7870bb58c9a19704aab45bf379b29621fbd4900c5a15fff79e  google-authenticator-encourage-user-activation.0.2.zip
c61bab4e64eeff3d519e5a24990f5ba8223651f13aed2e0fb69b904419e62c50  hcaptcha-for-forms-and-more.4.9.0.zip
bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9  include-mastodon-feed.1.9.9.zip
4d824673ad59e74c5bdfc9c6b6c87341b2d0f0879641b72381612aed30335758  jetpack.14.1.zip
13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217  leaflet-map.3.4.1.zip
7b209670e853d43166b71105104673541934baedb8425a45bc7e7889cbcf8d6e  masterslider.zip
fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5  melapress-login-security.2.0.1.zip
08f15bb64ddbdf0813ec5e581515cedae5e8aee9d9a3b0c5e4b2726a5a2ac982  meta-box.5.10.5.zip
53e46b744d788bb6d0489f923a9f1fdb740a25851a19a5e21ccbdd19af8c58d2  meta-box-conditional-logic.zip
9e349d58ea21b03da63633c3fe8ec9df601129a1210497ac943fa7db83e39ae4  meta-box-show-hide.zip
ae78971c55c22197901fe7ba65c2e7618b9bc96b5f401648b232aefdee129b98  meta-box-tabs.zip
7ccbc9703a0f194ad3219383de1d12f28e373598ed2448e07145a096d6598a0a  Modern-REL1_42-c8a7124.tar.gz
07e855134780a5f1f9925d391f7eaed6f968f553988237ccc261d7e41ffb5829  Modern-REL1_43-5597681.tar.gz
d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c  oshine-core.zip
d515fe1a3905e4c017fa928506b60b49b49eef31f55dababf0d7c33da876a26d  oshine-modules.zip
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.tgz
9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a  phplist-3.6.15.zip
1c045983b84a270a305a84ede3a07941037cc3448e9e2e7a51a2cebcd1044715  plugin.json
f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c  raw-html.1.6.4.zip
757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6  regenerate-thumbnails.3.1.6.zip
4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193  related-posts-by-taxonomy.2.7.6.zip
de0c7cdab7725a156f1fd0e91592137b2cbcbbe120df1efb6366e426032ada23  revslider.zip
2ee4950e6c2bb47ac2614a02529697a59aee8fb967f094a329d78e9626395f92  safe-svg.2.3.1.zip
97960efc1647253080c82739a47ea8b5239257e05065a3a7f24f5e3b5935e7b7  shareaholic.9.7.13.zip
514d1156304ffff54883b4dff57a789b36983d163a12276127ad9c2399f6614d  smart-slider-3.3.5.1.26.zip
41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210  spam-destroyer.2.1.4.zip
fb26419580bbc8f0d35842fd98a1a9bf44851453d20c373fadb8b488272547a4  ssl-insecure-content-fixer.2.7.2.zip
576e3324d4b77840c56d08fa3412bafcfe568c671359eb39e69594e1f8c48311  storefront.4.6.0.zip
1f80fabc560dfcd1dc05d87087e50421b79118633629dd4451d1fae6ce188009  tatsu.zip
e33538c5a99ac6c6d083f5e70fdf390c5471247e47a44751b6a7a531b903287f  twentyeleven.4.8.zip
dc3c9337a95ff0619f8f4f2bbe494d0caf445af7e050b93f0de818e799e2a850  twentyfifteen.3.9.zip
91b68629634e181b3b5ba2852f97ae9abdbc4cf3455026264a0d5e323db88907  twentyfourteen.4.1.zip
3b7adce871a143bca30dbfb421a69b005cc53e252c9c9a0e4d768571f6c5be3e  twentynineteen.3.0.zip
166035cbf15c3d618b5d2c11d1aa365b93d597f32735aa6f35887f4828673377  twentyseventeen.3.8.zip
04f14d0210cf72797d29c60132d2de4dc2ea00fffffcb824aa9e95fba385c7d4  twentysixteen.3.4.zip
21cd8f5688fc21493becee40bfb99a57837750f5ef6accca7abce5bcb1c37b46  twentyten.4.3.zip
6016a433cd38023181b00d6e44a14b976bc4963ce5fe97033b6af6046360411f  twentythirteen.4.3.zip
f6a5b6c3c8691f9a75f95d9e6675889a217cb57a8339e37b8aef182a7a4a4eaf  twentytwelve.4.4.zip
d17b77d02d736493413760ab2651745f4fb705ab569ef8c56aea48dee11ad074  UserMerge-REL1_42-41759d0.tar.gz
7b64058a82e2a33d452ae8172ae394dce603c788fd3d8b1cad9c8894a5b4bd30  UserMerge-REL1_43-f9d2664.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  v3.6.15.zip
02854640d783194c3097c00af4b5e99ae8fa9d9c4359b0a7abf0b5070375fa7c  vcaching.1.8.3.zip
543e34e5dbb2ef74af9ada84767e9118e200bb8a1046a5d49a03c7e462b9a13e  Widgets-REL1_42-17dbd92.tar.gz
7946e4c3c970792a9de655dce79fa68bad2023084bb24e8d01694b4d17329778  Widgets-REL1_43-50da5c6.tar.gz
c38427b9bebd799ff97e68593247dbbb1eef3a3ecd0503c0026c32fdc779de74  woocommerce.9.5.1.zip
aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9  woocommerce-gateway-stripe.9.0.0.zip
fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa  woocommerce-multilingual.5.3.9.zip
75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c  wordpress-6.7.1.zip
ce79a4dace0e87caa9314d9b773a9663506556cb78cd19d388522fc5401185bd  wordpress-seo.24.1.zip
feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38  wp-2fa.2.8.0.zip
130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda  wpforms-lite.1.9.2.3.zip
6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c  wpfront-notification-bar.3.4.2.zip
e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2  wp-pgp-encrypted-emails.0.8.0.zip
e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1  wp-qrcode.1.1.1.zip
bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7  wps-hide-login.1.9.17.1.zip
2ec43525f53953605daca6c3586919c9599ec66a805814bf3bb46751054d807d  wp-super-cache.1.12.4.zip
user@host:/tmp/user/1000/tmp.aQZdXtbcsc$
</pre>
# ok, after 3 days (assuming everything matches), we'll be unblocked on microfactory.opensourceecology.org, www.openbuildinginstitute.org, and www.opensourceecology.org, and phplist.opensourceecology.org
# in the meantime, we can continue with the wiki upgrade process
# ...
# I also finished my logs and hours and sent a bill to OSE for 49 hours in the month of Dec 2024

Revision as of 20:59, 11 April 2025

My work log from the first quarter of the year 2025. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Sun Mar 16, 2025

  1. I sent a follow-up email to Tom to confirm that his new ssh key was
    1. generated following our best-practices as documented on the wiki and https://wiki.opensourceecology.org/wiki/OSE_Server#SSH
    2. probably backed-up on his veracrypt-encrypted usb drive
  2. ...
  3. Here's TOFU 2/3 (VPN, exit in Germany)
Germany
2025-03-16
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Themes 
	. 


https://extdist.wmflabs.org/dist/extensions/DeleteBatch-REL1_43-59ffc3f.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/SmiteSpam-REL1_43-d45b08a.tar.gz
######################################################################### 100.0%
https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz
			-=O=-                         #     #     #    #                   
https://downloads.wordpress.org/release/wordpress-6.7.2.zip
######################################################################### 100.0%
https://downloads.wordpress.org/theme/twentytwentyfive.1.1.zip
######################################################################### 100.0%
2025-03-16
120K	DeleteBatch-REL1_43-59ffc3f.tar.gz
104K	SmiteSpam-REL1_43-d45b08a.tar.gz
7.7M	twentytwentyfive.1.1.zip
336K	v1.8.22.tar.gz
28M	wordpress-6.7.2.zip
2d0080dac51ead0c72f3e2ed06c47357ee8ea09f253dc4804f2a26ced6341b91  DeleteBatch-REL1_43-59ffc3f.tar.gz
8cc71c8aa28084babc0f0f5d5616aea2d4a8f8d309cc9ee3c3a3a1fc013498cf  SmiteSpam-REL1_43-d45b08a.tar.gz
7d588be49c9ad32bf47f83828756a9bb3f49b675a69a9f4daadf5b3daacd1455  twentytwentyfive.1.1.zip
dfaed40766bedcbf1879949e1a60abf816104725d5c416c1e97a4f2edcf9b7c2  v1.8.22.tar.gz
b6acbc116aabdc5a7c9bc4e861647148a1b444c1f2beec5440261b88f02c3a6e  wordpress-6.7.2.zip
user@disp8402:/tmp/tmp.rUpogE2lis$
  1. ...
  2. I tried to load the wiki (on hetzner2), but I got a varnish error

Error 503 Backend fetch failed

Backend fetch failed
Guru Meditation:

XID: 58196320

Varnish cache server
  1. I'm still able to ssh-in
  2. I restarted apache, but--wow--it took maybe 2 minutes for this command to finish
[root@opensourceecology ~]# systemctl restart httpd
[root@opensourceecology ~]#
  1. ...
  2. I updated the OBI CHG wiki article for the migration to hetzter3 to include both [a] a static site *and* [b] a broken wordpress site https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_obi_to_hetzner3
  3. in order to test this, I need a new IP address and I'll need to update the ansible configs with a new vhost that binds to this new IP addres
  4. I tried to login to the hetzner account using the password in the OSE shared keepass, but I kept getting "Invalid credentials."
  5. I launched a new VM in chromium (to ensure it wasn't some b& due to my hardened firefox), and I the same error; including the account being locked
Invalid credentials.

There have been too many login attempts! Please wait for 600 seconds before trying to log in again.
  1. unfortunately, the credentials for the email address for this hetzner account is not listed in our ose shared keepass
  2. I logged-in as my google admin account, reset the password, and added the password to our keepass
  3. I then tried to login to the email account, but google demanded an OTP sent to some phone number.
  4. I sent an email to Marcin asking if he changed the creds, and asking what is the phone number
  5. I found that I could remove the phone number associated with the account from the Google Workspace admin panel
  6. then it said it sent an email with an otp to – the email I'm trying to login into!?
  7. I found a button next to "login challenge" to disable login challenges for 10 minutes
  8. that worked; I'm in.
  9. I discovered that this email account has a *lot* of emails, and many are banking related. So I decided *not* to set it up to foward to the shared operations google group list
  10. I see an email from Feb 19 indicating that the hetzner password changed
  11. I tried to reset the password several times. It never sends the email! Part of the issue is that it requires us to enter both [a] the "client number or login" and [b] the "email address"
    1. the only thing we have entered in our keepass is the email address, which is what we use for login
    2. there's also several distinct login portals for hetzner. One for hetnzer1. One for the cloud. One for the decdicated servers. We have a client ID for hetzner1, but it's alphanumeric. And we have a distinct client ID that was included in the email from hetzner last month about the password change, but that's also alphanumeric. I tried everything I could think-of for the "client number or login", including stripping the alpha characters (so it's truly just a "number"), but the email never came and eventually I got an error indicating that the account is now disabled
Account is disabled.

There have been too many login attempts! Please wait for 600 seconds before trying to log in again.
  1. there's no phone number to call on this page
  2. I found a phone number, but it's a +49. I can only call +1 for free from my Google Voice account :/ https://www.hetzner.com/support-form/
  3. I used the form on the above page to send a cryptographically signed message to hetzner asking them to please send us our "client number" to the email address associated with our account
  4. ...
  5. while I'm blocked and locked-out of the hetzner wui and I'm logged into the Google Workspace, I'll work on creating those accounts
  6. crap, I just realized that the shared keepass db doesn't have an entry for the hcaptcha-specific email account that I created on 2025-02-08
    1. I clearly stated in my log of that day that I added this entry to the keepass db. Is it possible that the file became corrupt somehow? Or maybe I accidentally updated it on hetzner3? https://wiki.opensourceecology.org/wiki/Maltfield_Log/2025_Q1#Sat_Feb_08.2C_2025
  7. no, the file doesn't even exist on hetzner3 yet
root@hetzner3 ~ # ls -lah /etc/keepass
ls: cannot access '/etc/keepass': No such file or directory
root@hetzner3 ~ #
  1. the file is on hetzner2, and says it's last updated today – but for some reason only I have write permissions?
[maltfield@opensourceecology ~]$ ls -lah /etc/keepass/
total 180K
drwxrwx---    2 root      keepass   4.0K Mar 16 22:55 .
drwxr-xr-x. 104 root      root       12K Dec 30 22:33 ..
-rw-r-----    1 maltfield maltfield  45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root      keepass   4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass   4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[maltfield@opensourceecology ~]$
  1. according to the wiki https://wiki.opensourceecology.org/wiki/OSE_Server#Keepass
The passwords.kdbx file should be owned by the user 'root' and the group 'keepass'. It should have the file permissions of 660 (such that it can be read & written by 'root' and users in the 'keepass' group, but not accessible in any way from anyone else).
  1. so I fixed this
[root@opensourceecology keepass]# ls -lah
total 180K
drwxrwx---    2 root      keepass   4.0K Mar 16 22:55 .
drwxr-xr-x. 104 root      root       12K Dec 30 22:33 ..
-rw-r-----    1 maltfield maltfield  45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root      keepass   4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass   4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]# 
[root@opensourceecology keepass]# chown root:keepass passwords.kdbx
[root@opensourceecology keepass]# chmod 0660 passwords.kdbx
[root@opensourceecology keepass]# 
[root@opensourceecology keepass]# ls -lah
total 180K
drwxrwx---    2 root      keepass   4.0K Mar 16 22:55 .
drwxr-xr-x. 104 root      root       12K Dec 30 22:33 ..
-rw-rw----    1 root      keepass    45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root      keepass   4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass   4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]#
  1. I also went ahead and made a backup now
[root@opensourceecology keepass]# cp passwords.kdbx passwords.kdbx.20250316.bak
[root@opensourceecology keepass]# 
[root@opensourceecology keepass]# ls -lah
total 228K
drwxrwx---    2 root      keepass   4.0K Mar 16 23:42 .
drwxr-xr-x. 104 root      root       12K Dec 30 22:33 ..
-rw-rw----    1 root      keepass    45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root      keepass   4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass   4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-r-----    1 root      root       45K Mar 16 23:42 passwords.kdbx.20250316.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]# chown root:keepass passwords.kdbx.20250316.bak 
[root@opensourceecology keepass]# ls -lah
total 228K
drwxrwx---    2 root      keepass   4.0K Mar 16 23:42 .
drwxr-xr-x. 104 root      root       12K Dec 30 22:33 ..
-rw-rw----    1 root      keepass    45K Mar 16 22:55 passwords.kdbx
-rw-rw----    1 root      keepass   4.5K Jul 29  2017 passwords.kdbx.20170728.bak
-rw-rw----    1 root      keepass   4.5K Jul 13  2017 passwords.kdbx.20170804.bak
-rw-rw----    1 root      keepass    33K Aug 20  2019 passwords.kdbx.20190820.bak
-rw-rw----    1 maltfield maltfield  34K Sep  9  2019 passwords.kdbx.20190909.bak
-rw-r-----    1 root      keepass    45K Mar 16 23:42 passwords.kdbx.20250316.bak
-rw-rw----    1 root      keepass    27K May 26  2018 passwords.kdbxs.20180525.bak
[root@opensourceecology keepass]#
  1. my best-guess/hope is that the entry for hcapcha just somehow failed to get added due to a network issue.
  2. I went ahead and reset the password on the hcapctha-specific google account and added it to keepass (again)
  3. ok, omg, wtf. I just realized the ops entry is missing. I *definitely* just added that today! What's happening!?!
  4. I created 3 new entries in keepass
    1. operations-specific email
    2. obi-dreamhost-specific email
    3. hetzner-specific email
  5. I generated random passwords for each, but I didn't set them to be used in the actual accounts yet.
  6. then I saved keepass, closed keepass, and shutdown my ose vm
  7. I restated my ose vm, mounted & unlocked the keepass
  8. the new entries are absent! What's happening!?
  9. I do see an error from the keepass terminal
user@ose:~$ keepassxc
Maximum depth of replacement has been reached. Entry uuid: {REDACTED}
  1. quick google shows this error; not sure if relevant https://github.com/keepassxreboot/keepassxc/issues/1741
  2. omg, actually, the ops and hcaptcha entries are back. So it's like the last save is now opening, but the one before it is not. I added the other two accounts, saved, restarted, and reunlocked
  3. I wonder if somehow there's a local copy that's being put in the mounted dir, which gets written-to during network issues
  4. yeah, on reboot I do see a file before I mount it!
user@ose:~$ ls -lah mnt/ose/keepass/passwords.kdbx 
-rw------- 1 user user 44K Mar 16 18:48 mnt/ose/keepass/passwords.kdbx
user@ose:~$
  1. and, yeah, that local copy has the entries that I had added before. I'm just going to move this out of the way, mount the *real* one on hetnzer2, and confirm my new entries are finally saved
user@ose:~$ mv mnt/ose/keepass/passwords.kdbx passwords.fail.20250316.kdbx
user@ose:~$ 
user@ose:~$ sshfs -p 32415 maltfield@138.201.84.223:/etc/keepass /home/user/mnt/ose/keepass -o IdentityFile=/home/user/.ssh/id_rsa.ose -o uid=1000
Enter passphrase for key '/home/user/.ssh/id_rsa.ose': 
user@ose:~$ ls -lah mnt/ose/keepass/passwords.kdbx
-rw-rw---- 1 user 1005 45K Mar 16 18:57 mnt/ose/keepass/passwords.kdbx
user@ose:~$
  1. ok, all the entries are present again. Now that I'm sure the passwords are durably saved in the shared keepass file on the server, I created the two new accounts (for hetzner and obi dreamhost)
  2. I also added catarina's personal gmail to the ops google group list
  3. I sent a test email to the new obi-specific-dreamhost account email to confirm that Catarina is new getting forwarded those emails
  4. I confirmed that *I* got the email forwarded to me; I'll wait for Catarina's reply

Sat Mar 15, 2025

  1. Marcin generated a new ssh key for me to auth his access
  2. I sent him an email asking him to confirm that the key was
    1. generated following our best-practices as documented on the wiki and https://wiki.opensourceecology.org/wiki/OSE_Server#SSH
    2. probably backed-up on his veracrypt-encrypted usb drive
  3. once he confirms, I'll add it to his authorized_keys file, and then we can confirm access is working
  4. ...
  5. Marcin sent me an email that seemed to suggest that we should delay the content fixes of osemain until after we migrate to hetzner3. I sent an email asking to confirm if that's his intention
  6. ...
  7. Catarina sent me output of `ls -lah ~/.ssh` which showed she has no ssh keypair
  8. I sent her the commands to generate a new keypair and asked if she has a backup solution in-place
  9. ...
  10. Catarina said she found the links to login to her new wiki admin account, but the password didn't work (maybe it expired)
  11. I logged-into the wiki today and tried to reset the password, but I got an error when trying to query for the username https://wiki.opensourceecology.org/index.php?title=Special%3AUserCredentials&target=CatarinaAdmin
No credentials found for this user. Check that the name is spelled correctly.
  1. I checked the list of users in the admin group, and it includes the ones I thought I had deleted!?
  2. According to my notes, I did this already on 2025-02-16 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2025_Q1#Sun_Feb_16.2C_2025
  3. my best-guess is that I accidentally made the change on hetzner3, not hetzner2
  4. again, I edited these accounts, removing the 'administrator' role
    1. https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Audrey+Rampone
    2. https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Elifarley
    3. https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Hart
    4. https://wiki.opensourceecology.org/index.php?title=Special%3AUserRights&user=Will
  5. I then tried to create the "CatarinaAdmin" user, but it said it already exists!
Username entered already in use. Please choose a different name.
  1. It looks like CatarinaAdmin isn't in the admin group yet, though
  2. first I confirmed her email address is correct. Ugh, she has 3x accounts already with the same email address
MariaDB [osewiki_db]> select user_name,user_email from wiki_user where user_name like '%Catarina%';
+---------------+--------------------------+
| user_name     | user_email               |
+---------------+--------------------------+
| Catarina Mota | REDACTED |
| Catarina      | REDACTED |
| Catarinamota  | REDACTED |
| CatarinaAdmin | REDACTED |
+---------------+--------------------------+
4 rows in set (0.00 sec)

MariaDB [osewiki_db]>
  1. I used this special page to send her a new password https://wiki.opensourceecology.org/wiki/Special:PasswordReset
  2. once she's able to confirm access, I'll grant it admin and ask her to setup 2FA.
  3. ...
  4. Catarina responded to my email about wordpress themes, asking me to install all of them on the new osemain site on hetzner3 – including twentytwentyfive
    1. https://wordpress.org/themes/twentytwentyfive/
    2. https://wordpress.org/themes/twentytwentyfour/
    3. https://wordpress.org/themes/generatepress/
    4. https://wordpress.org/themes/astra/
    5. https://wordpress.org/themes/oceanwp/
    6. https://wordpress.org/themes/hestia/
    7. https://wordpress.org/themes/neve/
  5. time for a new 3TOFU; I needed to do one for a few of the mediawiki plugins I suggested for Marcin earlier, anyway
################################################################################
# File:    3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
#          days using 3 distinct networks exiting from 3 distinct countries
# 
#          For more info on 3TOFU (and why this is important), see:
#           * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2025-01-01 21:21:18+00:00
################################################################################

JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --location --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)

REMOTE_FILES="https://extdist.wmflabs.org/dist/extensions/DeleteBatch-REL1_43-59ffc3f.tar.gz https://extdist.wmflabs.org/dist/extensions/SmiteSpam-REL1_43-d45b08a.tar.gz https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz"
WARNINGS=""

# in tails, we must torify
if  "`whoami`" == "amnesia"  ; then
	CURL="/usr/bin/torify ${CURL}"
	PYTHON="/usr/bin/torify ${PYTHON}"
fi

tmpDir=`mktemp -d`
pushd "${tmpDir}"

# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1

# and today's date
date -u +"%Y-%m-%d"

echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")

REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"

themes='twentytwentyfive'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
	echo -n '. '
	json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")

	latest_version=$(echo $json | $JQ -r .version)
	
	if [ "${latest_version}" = "null" ]; then
		error=$(echo $json | $JQ -r .error);
		description=$(echo $json | $JQ -r .description);
		WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
		WARNINGS="${WARNINGS}\n\t$error"
		WARNINGS="${WARNINGS}\n\t$description"
	else
		REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
	fi
	
done
echo

echo -e "${WARNINGS}"
echo

# get the file
for file in ${REMOTE_FILES}; do
	echo "${file}"
	${CURL} --progress-bar -O "${file}"
done

# checksum
date -u +"%Y-%m-%d"
du -sh *
sha256sum *
  1. Here's TOFU 1/3 (Exit in Germany)
Congratulations. This browser is configured to use Tor.
2025-03-16
INFO: Determining Latest Version of Wordpress Core
INFO: Determining Latest Version of Wordpress Themes 
	. 


https://extdist.wmflabs.org/dist/extensions/DeleteBatch-REL1_43-59ffc3f.tar.gz
######################################################################### 100.0%
https://extdist.wmflabs.org/dist/extensions/SmiteSpam-REL1_43-d45b08a.tar.gz
######################################################################### 100.0%
https://github.com/edwardspec/mediawiki-moderation/archive/refs/tags/v1.8.22.tar.gz
   -=O=-               #     #     #     #                                     
https://downloads.wordpress.org/release/wordpress-6.7.2.zip
####################                                                       27.6%Crash Annotation GraphicsCriticalError: |[0][GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt (t=51.5764) [GFX1-]: RenderCompositorSWGL failed mapping default framebuffer, no dt
######################################################################### 100.0%
[1]+  Done                    torbrowser  (wd: ~)
(wd now: /tmp/user/1000/tmp.sHf2RQALl0)
https://downloads.wordpress.org/theme/twentytwentyfive.1.1.zip
######################################################################### 100.0%
2025-03-16
120K	DeleteBatch-REL1_43-59ffc3f.tar.gz
Retrieved from "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2025_Q1&oldid=304922"