Maltfield Log/2024 Q4
My work log from the fourth quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
Contents
- 1 See Also
- 2 Tue Dec 31, 2024
- 3 Mon Dec 30, 2024
- 4 Sun Dec 29, 2024
- 5 Sat Dec 28, 2024
- 6 Fri Dec 27, 2024
- 7 Thr Dec 26, 2024
- 8 Sun Dec 22, 2024
- 9 Fri Dec 20, 2024
- 10 Sat Dec 14, 2024
- 11 Fri Dec 13, 2024
- 12 Thr Dec 12, 2024
- 13 Wed Dec 11, 2024
- 14 Tue Dec 10, 2024
- 15 Mon Oct 07, 2024
- 16 Sun Oct 06, 2024
- 17 Fri Oct 04, 2024
- 18 Thr Oct 03, 2024
- 19 Wed Oct 02, 2024
See Also
Tue Dec 31, 2024
- finally I got an email from the oshine theme support team with a link to download the required plugins
- it's just a link to a google drive. the email was sent
Received: by smtp-relay.sendinblue.com with ESMTP id 9b9d7d38-bd8e-4fc1-9b37-475be98908d8; Tue, 31 December 2024 04:46:17 +0000 (UTC) ... From: "BrandExponents" <support@brandexponents.com>
- the email isn't signed, so it would be trivial for someone malicious to have sent me this link
-------------------- Hi Michael, Please download the plugins from the link below: https://drive.google.com/file/d/1xbs80Rz1hcZOPhl2O_Kh_zvz9jUEymId/view?usp=sharing Warm regards, -- Suman M., Tech Support Executive BrandExponents
- to refresh: here's all the plugins required by the theme
BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu.
- the link above contains a file (plugins.zip) with 8 .zip files inside it
user@host:~/.tb/tor-browser/Browser/Downloads$ unzip plugins.zip Archive: plugins.zip extracting: be-portfolio-post.zip extracting: masterslider.zip extracting: meta-box.zip extracting: meta-box-conditional-logic.zip extracting: meta-box-show-hide.zip extracting: meta-box-tabs.zip extracting: revslider.zip extracting: wpforms-lite.zip user@host:~/.tb/tor-browser/Browser/Downloads$ user@host:~/.tb/tor-browser/Browser/Downloads$ sha256sum * 1dbf8c7b7ab07f964d50efb5cc737dd9cb90ff366e51ab3b2051310dfa87f461 be-portfolio-post.zip 7d7b88a0f60a7c98b67fb97cd6e319bf814c6156504e5f52275052b62310dfb4 masterslider.zip 4af937e0435fa7370b91c256f5b83aa6e41869ae64595ce8b043f8fdf9b34a6f meta-box-conditional-logic.zip 357c4c51dc5e253a3f3b7f59726f94ec4529dfce88b9eb45671fe5ecb409e7a8 meta-box-show-hide.zip 20d0f5f13c23540bba617a232bfd0eee662e2b85022f3a752dad75fc214cd457 meta-box-tabs.zip 5afafc6e01ea7d1c1fb6e3c97b7cc711c39a137244f6a0946557f0bb2a66295e meta-box.zip 345da9e15ba4b618f7b626858a700be697600e39abd9709905b9aad27e5a6491 plugins.zip 1a4e1230e7aac6b136d5af018acaede037b3bfdc4572dd838fac6fef0a2b25c3 revslider.zip 8f1a9e34c3d7f9fae8c9af1d8bfec08989f8175b93a54a8d0f2c4712c525a40f wpforms-lite.zip user@host:~/.tb/tor-browser/Browser/Downloads$
- that doesn't inlude the 3 last ones, but they previously sent us another file (again on google drive) with those
- so we now have all the plugins needed (hopefully without malware; there's no way to authenticate this crap)
- some of the plugins above, though, are superfluous
- meta-box is a free plugin that we can download from wordpress.org https://wordpress.org/plugins/meta-box/
- wpforms-lite is also free and available on wordpress.org https://wordpress.org/plugins/wpforms-lite/
- alright, I downloaded all the other oshine-made plugins from the links they gave me previously
- so here's our shitty TOFU 1/3 (Tor, exit in France)
user@host:~/.tb/tor-browser/Browser/Downloads$ ls be-portfolio-post.zip meta-box-tabs.zip plugins.zip masterslider.zip meta-box.zip revslider.zip meta-box-conditional-logic.zip oshine-core-1.6.1.zip tatsu-3.5.3.zip meta-box-show-hide.zip oshine-modules-3.3.8.zip wpforms-lite.zip user@host:~/.tb/tor-browser/Browser/Downloads$ sha256sum * 1dbf8c7b7ab07f964d50efb5cc737dd9cb90ff366e51ab3b2051310dfa87f461 be-portfolio-post.zip 7d7b88a0f60a7c98b67fb97cd6e319bf814c6156504e5f52275052b62310dfb4 masterslider.zip 4af937e0435fa7370b91c256f5b83aa6e41869ae64595ce8b043f8fdf9b34a6f meta-box-conditional-logic.zip 357c4c51dc5e253a3f3b7f59726f94ec4529dfce88b9eb45671fe5ecb409e7a8 meta-box-show-hide.zip 20d0f5f13c23540bba617a232bfd0eee662e2b85022f3a752dad75fc214cd457 meta-box-tabs.zip 5afafc6e01ea7d1c1fb6e3c97b7cc711c39a137244f6a0946557f0bb2a66295e meta-box.zip d3aba9dd7351476d58e3ffa3c29ccf7d3f0c05736fc688a382e95bca1154034c oshine-core-1.6.1.zip e54903207f51377350276ac50e2be5749848cd3e95513ed85fdfbf652f348e0b oshine-modules-3.3.8.zip 345da9e15ba4b618f7b626858a700be697600e39abd9709905b9aad27e5a6491 plugins.zip 1a4e1230e7aac6b136d5af018acaede037b3bfdc4572dd838fac6fef0a2b25c3 revslider.zip ad769858e414eb983a7fb94dc06e5b3c26958794d359b2224c8def6546d2361e tatsu-3.5.3.zip 8f1a9e34c3d7f9fae8c9af1d8bfec08989f8175b93a54a8d0f2c4712c525a40f wpforms-lite.zip user@host:~/.tb/tor-browser/Browser/Downloads$ du -sh * 56K be-portfolio-post.zip 1.6M masterslider.zip 8.0K meta-box-conditional-logic.zip 8.0K meta-box-show-hide.zip 8.0K meta-box-tabs.zip 1.1M meta-box.zip 18M oshine-core-1.6.1.zip 1.7M oshine-modules-3.3.8.zip 24M plugins.zip 11M revslider.zip 16M tatsu-3.5.3.zip 11M wpforms-lite.zip user@host:~/.tb/tor-browser/Browser/Downloads$
- I don't know what we can do to better authenticate these files
- if any of these files contains a vulnerability, the malicious code would be able to modify any of our other websites (well only the upload dirs, but they could read the db creds and then inject something malicious into the other site's db), and attack our users on any of those websites
- I'm thinking that I should try one wordpress install in a DispVM that downloads these plugins in wordpress directly and diff those from what we just got above
- also, I should see if there's any sort of general scanner for malicious php code – something that can identify things like exec() calls or base32/64 blocks or anything that tries to file_put_contents or file_get_contents from the public internet. I can't possibly read all of these plugins code, but I could probably read through an automated report of code blocks identified as high-risk
- here's a collection of "static code analysis" tools for php https://github.com/guardrailsio/awesome-php-security?tab=readme-ov-file#static-code-analysis
- I went ahead and launched a Debian 12 disposable VM, installed wordpress from apt, and followed this guide https://wiki.debian.org/WordPress
- curiously installing wordpress doesn't install mariadb-server, nor does it configure the db or create an apache vhost; that had to be done manually
- anyway, I copied-in the oshine theme that I had downloaded from themeforest, and activated it on http://localhost
- the theme automatically set itself to "developer mode" because (it says) it realized it's on "localhost"
- I don't know exactly what that means, but I couldn't find anywhere to enter a license key; so I guess that's one difference
- there's a notice at the top that tells me to install the required plugins
This theme requires the following plugins: BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Meta Box Framework, Safe SVG, Slider Revolution and WPForms Lite. Begin installing plugins | Dismiss this notice
- if I click "begin installing plugins", then it brings me to a page listing a ton of plugins
- but if I click "Install" under any of them, then I'm brought to a page asking for my ftp creds. what?
Connection Informatio To perform the requested action, WordPress needs to access your web server. Please enter your FTP credentials to proceed. If you do not remember your credentials, you should contact your web host.
- I thought maybe it was a permissions issues, but that looked fine.
- instead, it looks like I needed to add this to /etc/wordpress/config-localhost.php https://stackoverflow.com/questions/30688431/wordpress-needs-the-ftp-credentials-to-update-plugins#30690783
define('FS_METHOD', 'direct');
- after that I tried to click the "Install" link again. This time it looked like it was working, but then it output some error
Installing Plugin: BE Portfolio Post Type Downloading installation package from https://brandexponents.com/oshin-plugins/be-portfolio-post.zip… Unpacking the package… The package could not be installed. PCLZIP_ERR_BAD_FORMAT (-10) : Invalid archive structure TGMPA v2.6.1 Return to Required Plugins Installer
- ah, the problem was that php-curl wasn't installed. Not only did I have to install it, but I had to restart apache after
sudo apt-get install php-curl sudo systemctl restart apache2
- ugh, I was able to get the URL to the plugin; this is wayy more trustworthy than their google drive account! https://brandexponents.com/oshin-plugins/be-portfolio-post.zip
- here's all of them, one-by-one
https://brandexponents.com/oshin-plugins/be-portfolio-post.zip https://brandexponents.com/thirdparty-plugins/meta-box-conditional-logic.zip https://brandexponents.com/thirdparty-plugins/meta-box-show-hide.zip https://brandexponents.com/thirdparty-plugins/meta-box-tabs.zip https://brandexponents.com/oshin-plugins/oshine-core.zip https://brandexponents.com/be-plugins/oshine-modules.zip https://brandexponents.com/thirdparty-plugins/masterslider.zip https://downloads.wordpress.org/plugin/meta-box.5.10.5.zip https://downloads.wordpress.org/plugin/safe-svg.2.3.1.zip https://brandexponents.com/thirdparty-plugins/revslider.zip https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip https://brandexponents.com/oshin-plugins/be-gdpr.zip
- shit, this one failed with a 404 https://brandexponents.com/be-plugins/oshine-modules.zip
- this one too https://brandexponents.com/be-plugins/tatsu.zip
- I googled trying to find ^ those, and I found this https://www.markeaandrews.com/wordpressfile/Oshine Buyers Package 5.0/Plugins/
Parent Directory - be-page-builder.zip 2016-08-31 04:34 1.6M be-portfolio-post.zip 2016-08-29 22:33 7.3K meta-box-conditional..> 2016-08-29 22:33 8.1K meta-box-show-hide.zip 2016-08-29 22:33 3.2K meta-box-tabs.zip 2016-08-29 22:33 3.9K oshine-modules.zip 2016-12-22 07:23 927K
- I found this site, which claims to be a security review of the oshine theme. It actually marks it down for having included the plugin .zip files in the theme because "plugins are not allowed in themes." – maybe that's why they were removed? https://themecheck.info/score/wordpress-theme-oshin-v6_4_1.html
- unfortunately wordpress unzips the plugins, which makes it hard to use it for our 3TOFU :(
root@disp2713:/var/lib/wordpress/wp-content/plugins# ls akismet index.php meta-box-conditional-logic oshine-core be-gdpr masterslider meta-box-show-hide revslider be-portfolio-post meta-box meta-box-tabs wpforms-lite root@disp2713:/var/lib/wordpress/wp-content/plugins#
- I just created a zip from the dir and got its checksum
root@disp2713:/var/lib/wordpress/wp-content/plugins# zip --recurse-paths revslider.zip revslider ... adding: revslider/public/assets/assets/svg/content/ic_clear_24px.svg (deflated 31%) adding: revslider/public/assets/assets/coloredbg.png (deflated 7%) adding: revslider/public/assets/assets/gridtile.png (deflated 9%) root@disp2713:/var/lib/wordpress/wp-content/plugins# root@disp2713:/var/lib/wordpress/wp-content/plugins# du -sh revslider.zip 8.0M revslider.zip root@disp2713:/var/lib/wordpress/wp-content/plugins# root@disp2713:/var/lib/wordpress/wp-content/plugins# sha256sum revslider.zip 3754090d1573cf7b901a6a29166c4ae54035b60be599357bf34a2b3ccaeaf441 revslider.zip root@disp2713:/var/lib/wordpress/wp-content/plugins#
- unfortunately it differs from what I had just downloaded from Google Drive :(
user@host:~/.tb/tor-browser/Browser/Downloads/tofu1$ sha256sum revslider.zip 1a4e1230e7aac6b136d5af018acaede037b3bfdc4572dd838fac6fef0a2b25c3 revslider.zip user@host:~/.tb/tor-browser/Browser/Downloads/tofu1$
- quick check shows the version of revslider that BrandExponents sent me for download over Google Drive is v6.7.25
user@host:~/.tb/tor-browser/Browser/Downloads/tofu1$ head revslider/revslider.php <?php /* Plugin Name: Slider Revolution Plugin URI: https://www.sliderrevolution.com/?utm_source=admin&utm_medium=button&utm_campaign=srusers&utm_content=info Description: Slider Revolution - More than just a WordPress Slider Author: ThemePunch Text Domain: revslider Domain Path: /languages Version: 6.7.25 Author URI: https://themepunch.com/?utm_source=admin&utm_medium=button&utm_campaign=srusers&utm_content=info user@host:~/.tb/tor-browser/Browser/Downloads/tofu1$
- whereas the one that wordpress just downloaded from their website is v6.5.15
root@disp2713:/var/lib/wordpress/wp-content/plugins# head revslider/revslider.php <?php /* Plugin Name: Slider Revolution Plugin URI: https://www.sliderrevolution.com/ Description: Slider Revolution - Premium responsive slider Author: ThemePunch Text Domain: revslider Domain Path: /languages Version: 6.5.15 Author URI: https://themepunch.com/ root@disp2713:/var/lib/wordpress/wp-content/plugins#
- huh – I would expect that the one on google drive would be more stale than the one they serve to their theme users. wow.
- oh, I found the license input; I found it when a beg notice informed me that I'm running an outdated version of the theme; it sent me to this page, which prompted me for the theme purchase code http://localhost/wp-admin/themes.php?page=be_register#be-welcome
- after entering the license, I was able to update the theme. Then it brought me to a different screen for installing plugins. It began installing, but this time it didn't print out on the screen where it was downloading from. I fired-up wireshark, but I could only see the domain (it was encrypted)
- let's see if I can decrypt it with this https://thelinuxcode.com/decrypt-ssl-tls-wireshark/
export SSLKEYLOGFILE=/home/user/ssl-key.log touch $SSLKEYLOGFILE firefox https://localhost & tail -f $SSLKEYLOGFILE
- oh, I realized that won't matter because the connections are coming from wordpress; not firefox
- I was able to get this working by setting up mitmproxy and configuring wordpress to use it
sudo apt-get install mitmproxy sudo mitmproxy
- in another window, I added the self-signed cert to our whitelist, so wordpress wouldn't thrown an error on trying to connect
mkdir /usr/local/share/ca-certificates/extra cp ~/.mitmproxy/mitmproxy-ca-cert.cer /usr/local/share/ca-certificates/extra/mitmproxy-ca-cert.crt update-ca-certificates
- I tested this with curl, and – sure enough – mitmproxy showed the request; it has a really nice ncurses TUI
# initiate a request; you'll see it pop-up in the terminal with `mitmproxy` running export HTTP_PROXY=http://127.0.0.1:8080 export HTTPS_PROXY=http://127.0.0.1:8080 curl -L https://ddg.gg/?q=ose
- then, to get wordpress to go through the proxy, I added these lines to /etc/wordpress/config-localhost.php
root@disp2713:~# tail /etc/wordpress/config-localhost.php -n3
define('WP_PROXY_HOST', '127.0.0.1');
define('WP_PROXY_PORT', '8080');
?>
root@disp2713:~#
- then I went back to deactivate & delete all the plugins it had installed, and went to the page to install them again
Mon Dec 30, 2024
- yesterday Marcin decided to let go of fef and oswh, and I realized that we can trivially just turn them into static sites
- today I want to actually migrate those static sites from hetzner2 to hetzner3
- let's start with making a CHG page for the fef migration https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_fef
- hmm, I realized that I should set the SITE_DOWN on nginx before taking the backups, but now that backup includes making a static site dump with wget, which requires the site to be online
- fortunately we can take the site down on nginx and then scrape it directly from apache, but that means we have to modify the port from what we did yesterday
- I found that if I use the hostname, it tries to connect to the WAN IP address, which blocks incoming ports to port 8000
[root@opensourceecology fef]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}:8000" "${vhost_name}:8000"
Both --no-clobber and --convert-links were specified, only --convert-links will be used.
--2024-12-30 18:16:22-- http://oswh.opensourceecology.org:8000/
Resolving oswh.opensourceecology.org (oswh.opensourceecology.org)... 2a01:4f8:172:209e::2, 138.201.84.243
Connecting to oswh.opensourceecology.org (oswh.opensourceecology.org)|2a01:4f8:172:209e::2|:8000... failed: Connection refused.
Connecting to oswh.opensourceecology.org (oswh.opensourceecology.org)|138.201.84.243|:8000... failed: Connection refused.
Converted 0 files in 0 seconds.
real 0m0.017s
user 0m0.001s
sys 0m0.002s
[root@opensourceecology fef]#
- the fix is to use 127.0.0.1 and set the host header manually, but somehow this is breaking the recursive pull?
[root@opensourceecology fef]# time nice wget --header "Host: ${vhost_name}" --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "127.0.0.1:8000"
Both --no-clobber and --convert-links were specified, only --convert-links will be used.
--2024-12-30 18:25:45-- http://127.0.0.1:8000/
Connecting to 127.0.0.1:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘127.0.0.1:8000/index.html’
[ <=> ] 21,750 --.-K/s in 0s
2024-12-30 18:25:45 (644 MB/s) - ‘127.0.0.1:8000/index.html’ saved [21750]
FINISHED --2024-12-30 18:25:45--
Total wall clock time: 0.3s
Downloaded: 1 files, 21K in 0s (644 MB/s)
Converting 127.0.0.1:8000/index.html... 1-2
Converted 1 files in 0 seconds.
real 0m0.257s
user 0m0.003s
sys 0m0.000s
[root@opensourceecology fef]#
- I tried many changes to the command, but I think this might be a bug because it isn't even doing a '--convert-links', even though I clearly specified it
- I tried to change /etc/hosts to hard-code fef.opensourceecology.org to 127.0.0.1, but that still won't work because the links it follows are on port 80 (not 8000)
[root@opensourceecology fef]# time nice wget --header "Host: ${vhost_name}" --recursive --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}:8000"
[root@opensourceecology fef]# time nice wget --header "Host: ${vhost_name}" --recursive --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}:8000"
--2024-12-30 20:56:37-- http://fef.opensourceecology.org:8000/
Resolving fef.opensourceecology.org (fef.opensourceecology.org)... 127.0.0.1
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘fef.opensourceecology.org:8000/index.html’
[ <=> ] 21,750 --.-K/s in 0s
2024-12-30 20:56:38 (646 MB/s) - ‘fef.opensourceecology.org:8000/index.html’ saved [21750]
Loading robots.txt; please ignore errors.
--2024-12-30 20:56:38-- http://fef.opensourceecology.org/robots.txt
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
Resolving fef.opensourceecology.org (fef.opensourceecology.org)... 127.0.0.1
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
...
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
--2024-12-30 20:53:42-- http://fef.opensourceecology.org/wp-content/plugins/cyclone-slider-2/templates/thumbnails/script.js?ver=2.10.0
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
--2024-12-30 20:53:42-- http://fef.opensourceecology.org/wp-content/plugins/cyclone-slider-2/js/client.js?ver=2.10.0
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
--2024-12-30 20:53:42-- http://fef.opensourceecology.org/wp-includes/js/wp-embed.min.js?ver=4.9.1
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
FINISHED --2024-12-30 20:53:42--
Total wall clock time: 0.5s
Downloaded: 3 files, 28K in 0s (711 MB/s)
Converting fef.opensourceecology.org:8000/index.html... 2-1
Converted 1 files in 0 seconds.
real 0m0.478s
user 0m0.011s
sys 0m0.007s
[root@opensourceecology fef]#
- all right, this isn't working; I see two alternatives
- restrict nginx only to the server's IP
- figure out some sort of way to put the wordpress site in "read-only" mode
- I discovered a wordpress plugin specifically for generating static sites from its own content (eg for uploading to a "severless" site @ a CDN) https://wordpress.org/plugins/simply-static/
- I also found another tool that we can use to generate a static site, which might not have the issues that wget does https://www.httrack.com/page/2/en/index.html
- it's in the official debian apt repos
- I gave it a local run, and it spat-out one page that just infinitely redirects to itself. Fail.
- the internet says I can use WP_MAINTENANCE_MODE
define(‘WP_MAINTENANCE_MODE’, true);
- ah, nope. That's misinformation. The fucking article that mentioned it was probably ChatGPT outputting bullshit that doesn't actually exist https://wordpress.org/support/topic/wasnt-there-a-maintenance-mode-directive-in-wp-config-php/
- this is real, but not sure if it would help us https://developer.wordpress.org/reference/functions/wp_is_maintenance_mode/
- This SE answer suggests either to modify the wp db user's permission to not have write permission to the tables or to add some new filter that basically MITMs db calls and blocks anything that's not a SELECT https://wordpress.stackexchange.com/questions/243438/configure-wordpress-to-read-from-database-only-never-write
- ugh, ddg for making wordpress read only doesn't have any first-page results talking about locking the db
- here's the docs on LOCK in mysql https://dev.mysql.com/doc/refman/8.4/en/lock-tables.html
- I'm not gonna play on hetzner2; let's try this on hetzner3
MariaDB [fef_db]> show tables;
+-----------------------+
| Tables_in_fef_db |
+-----------------------+
| wp_ahm_events |
| wp_ahm_pages |
| wp_commentmeta |
| wp_comments |
| wp_links |
| wp_options |
| wp_postmeta |
| wp_posts |
| wp_term_relationships |
| wp_term_taxonomy |
| wp_termmeta |
| wp_terms |
| wp_usermeta |
| wp_users |
+-----------------------+
14 rows in set (0,001 sec)
MariaDB [fef_db]> LOCK *;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '*' at line 1
MariaDB [fef_db]> LOCK TABLES *;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '*' at line 1
MariaDB [fef_db]> LOCK TABLES * READ;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '* READ' at line 1
MariaDB [fef_db]> LOCK TABLES READ;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'READ' at line 1
MariaDB [fef_db]> LOCK TABLE READ:
-> ;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'READ:' at line 1
MariaDB [fef_db]> LOCK TABLE fef_db.* READ;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '* READ' at line 1
- wow, the actual command is FLUSH TABLES WITH READ LOCK;
- that was super unclear
- ok, I confirmed it's working for this database
MariaDB [fef_db]> UPDATE wp_users SET user_nicename = "grinch" where user_login = "Maltfield"; ERROR 1223 (HY000): Can't execute the query because you have a conflicting read lock MariaDB [fef_db]>
- shit, that locked all tables in all databases
MariaDB [store_db]> use oswh_db; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed MariaDB [oswh_db]> UPDATE wp_users SET user_nicename = "grinch" where user_login = "Maltfield"; ERROR 1223 (HY000): Can't execute the query because you have a conflicting read lock MariaDB [oswh_db]>
- this doesn't seem very safe. And I'm afraid that it will also break the wordpress static site scrape. let's look into the nginx solution
- first we'll unlock the DBs
MariaDB [oswh_db]> UNLOCK TABLES; Query OK, 0 rows affected (0,000 sec) MariaDB [oswh_db]> UPDATE wp_users SET user_nicename = "grinch" where user_login = "Maltfield"; Query OK, 1 row affected (0,009 sec) Rows matched: 1 Changed: 1 Warnings: 0 MariaDB [oswh_db]>
- ok, I tested adding these lines to the "server{" block of /etc/nginx/sites-enabled/fef.opensourceecology.org, and I confirmed that a local curl on the server works while my attempt to load it in my web browser gets a 403
allow 127.0.0.1; deny all;
- I'm going to test this on prod now
[root@opensourceecology current]# # backup nginx config
[root@opensourceecology current]# cp /etc/nginx/conf.d/fef.opensourceecology.org.conf nginx_fef.opensourceecology.org.${stamp}.conf
[root@opensourceecology current]#
[root@opensourceecology current]# # restrict website to local requests only
[root@opensourceecology current]# grep 'deny' /etc/nginx/conf.d/fef.opensourceecology.org.conf || sed -i 's%^\(\s*\)server_name\(.*\)%\1server_name\2\n\tallow 127.0.0.1;\n\tdeny all;\n%' /etc/nginx/conf.d/fef.opensourceecology.org.conf
[root@opensourceecology current]#
[root@opensourceecology current]# ls
nginx_fef.opensourceecology.org.20241230.conf
[root@opensourceecology current]# diff nginx_fef.opensourceecology.org.20241230.conf /etc/nginx/conf.d/fef.opensourceecology.org.conf
41a42,44
> allow 127.0.0.1;
> deny all;
>
[root@opensourceecology current]# # verify config & reload
[root@opensourceecology current]# nginx -t && systemctl reload nginx
...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
<pre>
# I confirmed that I now can't load the prod fef site (on hetzner2) in my web browser (I get a 403)
# I reverted it
<pre>
[root@opensourceecology current]# cp nginx_fef.opensourceecology.org.20241230.conf /etc/nginx/conf.d/fef.opensourceecology.org.conf
cp: overwrite ‘/etc/nginx/conf.d/fef.opensourceecology.org.conf’? y
[root@opensourceecology current]# nginx -t && systemctl reload nginx nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@opensourceecology current]#
- now let's try the whole thing, with a wget static site dump while it's locked-down from the public internet
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='fef.opensourceecology.org'
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current
# backup nginx config
cp /etc/nginx/conf.d/fef.opensourceecology.org.conf nginx_fef.opensourceecology.org.${stamp}.conf
# restrict website to local requests only
grep 'deny' /etc/nginx/conf.d/fef.opensourceecology.org.conf || sed -i 's%^\(\s*\)server_name\(.*\)%\1server_name\2\n\tallow 127.0.0.1;\n\tdeny all;\n%' /etc/nginx/conf.d/fef.opensourceecology.org.conf
# verify config & reload
nginx -t && systemctl reload nginx
mkdir wget
pushd wget
time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}:8000"
time nice wget --header "Host: ${vhost_name}" --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "127.0.0.1:8000"
- fuck, it's still failing by trying to load it on port 80
[root@opensourceecology wget]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}:8000"
Both --no-clobber and --convert-links were specified, only --convert-links will be used.
--2024-12-30 22:32:24-- http://fef.opensourceecology.org:8000/
Resolving fef.opensourceecology.org (fef.opensourceecology.org)... 127.0.0.1
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:8000... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://fef.opensourceecology.org/ [following]
--2024-12-30 22:32:25-- http://fef.opensourceecology.org/
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
Resolving fef.opensourceecology.org (fef.opensourceecology.org)... 127.0.0.1
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|127.0.0.1|:80... failed: Connection refused.
Converted 0 files in 0 seconds.
real 0m0.211s
user 0m0.000s
sys 0m0.003s
[root@opensourceecology wget]#
- I removed the line from /etc/hosts, and now it fails on port 8080
[root@opensourceecology wget]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}:8000"
Both --no-clobber and --convert-links were specified, only --convert-links will be used.
--2024-12-30 22:33:42-- http://fef.opensourceecology.org:8000/
Resolving fef.opensourceecology.org (fef.opensourceecology.org)... 2a01:4f8:172:209e::2, 138.201.84.243
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|2a01:4f8:172:209e::2|:8000... failed: Connection refused.
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|138.201.84.243|:8000... failed: Connection refused.
Converted 0 files in 0 seconds.
real 0m0.016s
user 0m0.002s
sys 0m0.001s
[root@opensourceecology wget]#
- oh, duh, we're doing that so we don't have to call it by ip address or port
- here's the original; now it's failing because of the source IP isn't 127.0.0.1
[root@opensourceecology wget]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}"
Both --no-clobber and --convert-links were specified, only --convert-links will be used.
--2024-12-30 22:34:53-- http://fef.opensourceecology.org/
Resolving fef.opensourceecology.org (fef.opensourceecology.org)... 2a01:4f8:172:209e::2, 138.201.84.243
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|2a01:4f8:172:209e::2|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://fef.opensourceecology.org/ [following]
--2024-12-30 22:34:53-- https://fef.opensourceecology.org/
Connecting to fef.opensourceecology.org (fef.opensourceecology.org)|2a01:4f8:172:209e::2|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-12-30 22:34:53 ERROR 403: Forbidden.
Converted 0 files in 0 seconds.
real 0m0.024s
user 0m0.006s
sys 0m0.003s
[root@opensourceecology wget]#
- I manually edited the config
server_name fef.opensourceecology.org; allow 127.0.0.1; allow [2a01:4f8:172:209e::2]; allow 138.201.84.243; deny all;
- well, it didn't like that
[root@opensourceecology wget]# nginx -t && systemctl reload nginx nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.openbuildinginstitute.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/ssl.opensourceecology.org.include:11 nginx: [emerg] invalid parameter "[2a01:4f8:172:209e::2]" in /etc/nginx/conf.d/fef.opensourceecology.org.conf:43 nginx: configuration file /etc/nginx/nginx.conf test failed [root@opensourceecology wget]#
- it was happy when I removed the square bracktes
server_name fef.opensourceecology.org; allow 127.0.0.1; allow 2a01:4f8:172:209e::2; allow 138.201.84.243; deny all;
- cool, this time it worked!
[root@opensourceecology wget]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}"
...
Converting fef.opensourceecology.org/wp-content/plugins/wp-simple-galleries/colorbox/themes/theme1/colorbox.css?ver=4.9.1.css... 22-0
Converted 88 files in 0.03 seconds.
real 0m6.786s
user 0m0.109s
sys 0m0.109s
[root@opensourceecology wget]#
[root@opensourceecology wget]# du -sh .
37M .
[root@opensourceecology wget]#
- alright, so this works
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='fef.opensourceecology.org'
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current
# backup nginx config
cp /etc/nginx/conf.d/fef.opensourceecology.org.conf nginx_fef.opensourceecology.org.${stamp}.conf
# restrict website to local requests only
grep 'deny' /etc/nginx/conf.d/fef.opensourceecology.org.conf || sed -i 's%^\(\s*\)server_name\(.*\)%\1server_name\2\n\tallow 127.0.0.1;\n\tallow 2a01:4f8:172:209e::2;\n\tallow 138.201.84.243;\n\tdeny all;\n%' /etc/nginx/conf.d/fef.opensourceecology.org.conf
# verify config & reload
nginx -t && systemctl reload nginx
mkdir wget
pushd wget
time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}"
- I did see some 403 errors in there, and I wonder if it's my nginx rate limiting is causing this
--2024-12-30 22:44:15-- https://fef.opensourceecology.org/wp-content/uploads/2015/08/chicks_7.jpg Reusing existing connection to [fef.opensourceecology.org]:443. HTTP request sent, awaiting response... 403 Forbidden 2024-12-30 22:44:15 ERROR 403: Forbidden. --2024-12-30 22:44:15-- https://fef.opensourceecology.org/wp-content/uploads/2015/08/chicks_7-150x150.jpg Reusing existing connection to [fef.opensourceecology.org]:443. HTTP request sent, awaiting response... 200 OK Length: 32329 (32K) [image/jpeg] Saving to: ‘fef.opensourceecology.org/wp-content/uploads/2015/08/chicks_7-150x150.jpg’ 100%[=======================================================================>] 32,329 --.-K/s in 0s
- after some fiddling with nginx; I realized (when I tailed the mod_security logs) that this is DOS protection coming from apache's mod_evasive
[root@opensourceecology nginx]# tail -f /var/log/httpd/modsec_audit.log ... --27b11513-F-- HTTP/1.1 403 Forbidden Content-Length: 226 Content-Type: text/html; charset=iso-8859-1 --27b11513-E-- --27b11513-H-- Apache-Error: [file "mod_evasive24.c"] [line 248] [level 3] client denied by server configuration: /var/www/html/www.opensourceecology.org/htdocs/wp-json Stopwatch: 1735600807963014 4987 (- - -) Stopwatch2: 1735600807963014 4987; combined=161, p1=116, p2=0, p3=4, p4=25, p5=15, sr=30, sw=1, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.2 (http://www.modsecurity.org/); OWASP_CRS/2.2.9. Server: Apache Engine-Mode: "ENABLED" --27b11513-Z-
- I uncommented the line that whitelisted 127.0.0.1
[root@opensourceecology conf.d]# diff mod_evasive.conf.20241230 mod_evasive.conf 71c71 < #DOSWhitelist 127.0.0.1 --- > DOSWhitelist 127.0.0.1 [root@opensourceecology conf.d]# [root@opensourceecology conf.d]# systemctl reload httpd [root@opensourceecology conf.d]#
- I ran it again, and it made a huge difference
[root@opensourceecology wget]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}"
...
.. nothing to do.
Converting fef.opensourceecology.org/wp-content/plugins/cyclone-slider-2/templates/default/style.css?ver=2.10.0.css... 2-0
Converting fef.opensourceecology.org/wp-content/plugins/wp-simple-galleries/colorbox/themes/theme1/colorbox.css?ver=4.9.1.css... 22-0
Converted 109 files in 0.03 seconds.
real 0m21.372s
user 0m0.148s
sys 0m0.194s
[root@opensourceecology wget]#
[root@opensourceecology wget]#
[root@opensourceecology wget]#
[root@opensourceecology wget]#
[root@opensourceecology wget]# du -sh .
70M .
[root@opensourceecology wget]#
- I ran it again and confirmed there's no 403s. Yay!
[root@opensourceecology wget]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}" &> wget.log
...
[root@opensourceecology wget]# grep 403 wget.log
Length: 413054 (403K) [image/jpeg]
Length: 412969 (403K) [image/jpeg]
[root@opensourceecology wget]#
- I formalized all of this into this section (but I still need to do another full run-through to test it) https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_deprecate_fef#Change_Steps
- alright, I went through the whole process. I had to add a couple missing commands, but now it's good
- I confirmed that the fef static site on hetzner3 loads, and several spot checks suggest it's working fine
- here's the files; it's 1 quarter of a gigabyte. Not nothing, but tolerable as an archive.
root@hetzner3 /var/www/html/fef.opensourceecology.org # ls -lah total 176M d---r-x--- 3 not-apache www-data 4,0K Dec 31 01:38 . d---r-x--- 9 not-apache www-data 4,0K Dec 31 01:37 .. -r-------- 1 not-apache www-data 176M Dec 31 01:38 fef.opensourceecology.org_files.20241231.tar.bz2 d---r-x--- 10 not-apache www-data 4,0K Dec 31 01:28 htdocs -r-------- 1 not-apache www-data 61K Dec 31 01:38 mysqldump_fef.opensourceecology.org.20241231.sql.bz2 -rw-r--r-- 1 not-apache www-data 779 Dec 31 01:38 README.txt root@hetzner3 /var/www/html/fef.opensourceecology.org # ls htdocs/ 2014 'index.html?p=189.html' 'index.html?p=447.html' 'index.html?p=553.html' 'index.html?p=77.html' 2015 'index.html?p=203.html' 'index.html?p=449.html' 'index.html?p=580.html' robots.txt category 'index.html?p=215.html' 'index.html?p=458.html' 'index.html?p=610.html' the-farm-2 index.html 'index.html?p=218.html' 'index.html?p=46.html' 'index.html?p=612.html' the-farm-house 'index.html?p=104.html' 'index.html?p=258.html' 'index.html?p=471.html' 'index.html?p=616.html' wp-content 'index.html?p=119.html' 'index.html?p=264.html' 'index.html?p=475.html' 'index.html?p=618.html' wp-includes 'index.html?p=122.html' 'index.html?p=338.html' 'index.html?p=486.html' 'index.html?p=632.html' wp-json 'index.html?p=129.html' 'index.html?p=402.html' 'index.html?p=510.html' 'index.html?p=642.html' 'xmlrpc.php?rsd' 'index.html?p=143.html' 'index.html?p=411.html' 'index.html?p=511.html' 'index.html?p=64.html' 'index.html?p=160.html' 'index.html?p=424.html' 'index.html?p=512.html' 'index.html?p=663.html' 'index.html?p=169.html' 'index.html?p=438.html' 'index.html?p=542.html' 'index.html?p=674.html' 'index.html?p=173.html' 'index.html?p=43.html' 'index.html?p=546.html' 'index.html?p=67.html' root@hetzner3 /var/www/html/fef.opensourceecology.org # root@hetzner3 /var/www/html/fef.opensourceecology.org # du -sh . 245M . root@hetzner3 /var/www/html/fef.opensourceecology.org #
- ...
- Marcin sent an email saying he also wants:
- not to migrate store.opensourceecolgy.org (it was never setup)
- not to migrate seedhome.openbuildinginstitute.org (it was never setup)
- to convert to a static site microfactory.opensourceecology.org
We want to keep and migrate www.openbuildinginstitute.org which is still in use. The rest can go: We have never actually put any real content on store.opensourceecology.org , nor seed home.openbuildings institute.org <http://seedhome.openbuildinginstitute.org/> - so these definitely do not need to be migrated nor made static. As far as microfactory.open source ecology.org <http://microfactory.opensourceecology.org/> - we can also turn that into a static site. We do not plan to update this site.
- microfactory was another oshine site. I'm not sure if it'll migrate OK or not; it's blocked until the oshine devs give us the required plugins. But we have to do this anyway, because oshine is required by OBI. And Catarina wants to switch OSE to it too.
- anyway, here's our original list
1. forum.opensourceecology.org 2. store.opensourceecology.orgc 3. microfactory.opensourceecology.org 4. fef.opensourceecology.org 5. oswh.opensourceecology.org 6. seedhome.openbuildinginstitute.org 7. www.openbuildinginstitute.org 8. www.opensourceecology.org 9. phplist.opensourceecology.org 10. wiki.opensourceecology.org
- and here's what we have now
1. forum.opensourceecology.org 2. microfactory.opensourceecology.org # can convert to static site, if necessary 3. fef.opensourceecology.org # convert to static site 4. oswh.opensourceecology.org # convert to static site 5. www.openbuildinginstitute.org 6. www.opensourceecology.org 7. phplist.opensourceecology.org 8. wiki.opensourceecology.org
- ...
- continuing with the wiki, I got all the sed commands together to fix LocalSettings.php such that we could run the upgrade
- the upgrade took 30 minutes, and it ended with an error
... log_id=212513
... log_id=212538
Completed migration, updated 212428 row(s) with 2283 new actor(s), 97 error(s)
Beginning migration of log_search
... target_author_id, ls_value=141 ls_log_id=145907
Completed migration, inserted 1 row(s) with 0 new actor(s), 0 error(s)
errors were encountered.
Modifying rev_text_id field of table revision ...done.
Modifying table site_stats ...done.
Populating ar_rev_id.
Populating ar_rev_id...
MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1217): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1528): MessageCache->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1167): WANObjectCache->getWithSetCallback()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1163): BagOStuff->getWithSetCallback()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1106): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1016): MessageCache->getMsgFromNamespace()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(927): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(661): Message->text()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(213): Title::newMainPage()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(118): PopulateArchiveRevId::makeDummyRevisionRow()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(63): PopulateArchiveRevId::checkMysqlAutoIncrementBug()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(45): PopulateArchiveRevId->doDBUpdates()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1377): LoggedUpdateMaintenance->execute()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(512): DatabaseUpdater->populateArchiveRevId()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(475): DatabaseUpdater->runUpdates()
#29 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(181): DatabaseUpdater->doUpdates()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(107): UpdateMediaWiki->execute()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}
real 28m11,950s
user 0m0,153s
sys 0m0,282s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
- is the script idempotent? I tried running it again; this time it exited in 6 seconds
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/update.php"
...
.php: Main slot of revision not found in database. See T212428.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1217): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1528): MessageCache->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1167): WANObjectCache->getWithSetCallback()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1163): BagOStuff->getWithSetCallback()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1106): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1016): MessageCache->getMsgFromNamespace()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(927): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(661): Message->text()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(213): Title::newMainPage()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(118): PopulateArchiveRevId::makeDummyRevisionRow()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(63): PopulateArchiveRevId::checkMysqlAutoIncrementBug()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(45): PopulateArchiveRevId->doDBUpdates()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1377): LoggedUpdateMaintenance->execute()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(512): DatabaseUpdater->populateArchiveRevId()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(475): DatabaseUpdater->runUpdates()
#29 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(181): DatabaseUpdater->doUpdates()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(107): UpdateMediaWiki->execute()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}
real 0m6,596s
user 0m0,006s
sys 0m0,012s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
- I tried the other command that I had to run from the last upgrade attempt
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # time nice sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php"
...
... archive processed up to revision id 302402 of 302632 (3173 rows in 1.3402280807495 seconds)
... archive processed up to revision id 302632 of 302632 (3178 rows in 1.3433260917664 seconds)
Done populating archive table. Processed 3178 rows in 1.343334197998 seconds
Done. Processed 302790 rows in 13.325568199158 seconds
real 0m13,413s
user 0m0,013s
sys 0m0,030s
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
- I was able to load the wiki in the browser, but it has an error again
Sun Dec 29, 2024
- so yesterday I finished the upgrade of our old MediaWiki v1.30.0 to v1.35 on hetzner3
- I quickly realized some errors clicking around on the site, but honestly I'm not sure I should attempt to fix them. probably it makes sense to first try the second upgrade to v1.43 and then try to fix things
- here's some commands for my upgrade plan (not tested yet)
sudo su -
# DECLARE VARIABLES
vhost_name='wiki.opensourceecology.org'
source /root/backups/backup.settings
stamp=$(date "+%Y%m%d_%H%M%S")
chg_dir="/var/tmp/CHG_${stamp}_wiki_1.35-to-1.43"
mkdir -p "${chg_dir}/{pre,post}"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
# Add vhost files
vhost_backup_path="${chg_dir}/pre/${vhost_name}.${stamp}"
mv "${vhostDir}" "${vhost_backup_path}"
mkdir -p ${vhostDir}
rsync -av --progress /var/tmp/mediawiki/mediawiki-1.43.0/ ${docrootDir}/
rsync -av --progress ${vhost_backup_path}/LocalSettings.php ${vhostDir}/
rsync -av --progress ${vhost_backup_path}/htdocs/LocalSettings.php ${docrootDir}/
rsync -av --progress ${vhost_backup_path}/htdocs/images ${docrootDir}/
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
#############
# MediaWiki #
#############
vhost_dir="/var/www/html/wiki.opensourceecology.org"
mw_docroot="${vhost_dir}/htdocs"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/LocalSettings.php"
chmod 0040 "${vhost_dir}/LocalSettings.php"
[ -d "${mw_docroot}/images" ] || mkdir "${mw_docroot}/images"
chown -R www-data:www-data "${mw_docroot}/images"
find "${mw_docroot}/images" -type f -exec chmod 0660 {} \;
find "${mw_docroot}/images" -type d -exec chmod 0770 {} \;
[ -d "${vhost_dir}/cache" ] || mkdir "${vhost_dir}/cache"
chown -R www-data:www-data "${vhost_dir}/cache"
find "${vhost_dir}/cache" -type f -exec chmod 0660 {} \;
find "${vhost_dir}/cache" -type d -exec chmod 0770 {} \;
# RUN UPGRADE
sudo -u www-data php "${docrootDir}/maintenance/update.php"
sudo -u www-data php "${docrootDir}/maintenance/populateContentTables.php"
- not that I realized it was really, really stupid to run the php scripts as root – even if they were 3TOFU'd. We're probably ok because this server isn't publicly accessible, but I'm going to make sure to run this as 'www-data' user in the future
- ah crap, adding the quotes broke the syntax on the mkdir command, so I ended-up with a dir named "{pre,post}" instead of two dirs. That broke the move, which caused the data to get merged. Now I need to startover and do the upgrade to 1.35 again :(
- well, the merge shouldn't be an issue since the 'images/' dir only containts two unimportant files (and the LocalSettings.php files are absent too)
root@hetzner3 /var/tmp/mediawiki # ls -lah mediawiki-1.43.0/images/ total 16K drwxr-xr-x 2 root root 4,0K Dec 29 18:14 . drwxr-xr-x 14 root root 4,0K Dec 29 18:14 .. -rw-r--r-- 1 501 staff 232 Dec 5 15:41 .htaccess -rw-r--r-- 1 501 staff 84 Dec 5 15:41 README root@hetzner3 /var/tmp/mediawiki #
- alright, this worked
root@hetzner3 /var/tmp/mediawiki # mkdir -p ${chg_dir}/{pre,post}
root@hetzner3 /var/tmp/mediawiki #
root@hetzner3 /var/tmp/mediawiki # mv "${vhostDir}" "${vhost_backup_path}"
root@hetzner3 /var/tmp/mediawiki #
root@hetzner3 /var/tmp/mediawiki # echo $vhost_backup_path
/var/tmp/CHG_20241229_182546_wiki_1.35-to-1.43/pre/wiki.opensourceecology.org.20241229_182546
root@hetzner3 /var/tmp/mediawiki #
root@hetzner3 /var/tmp/mediawiki # ls -lah $vhost_backup_path
total 155M
d---r-x--- 4 not-apache www-data 4,0K Dec 29 05:13 .
drwxr-xr-x 3 root root 4,0K Dec 29 18:35 ..
drwxrwx--- 2 www-data www-data 4,0K Dec 29 05:10 cache
drwxr-xr-x 14 root root 4,0K Dec 29 18:14 htdocs
----r----- 1 root root 18K Dec 29 03:41 LocalSettings.20241228.php
----r----- 1 not-apache apache-admins 18K Dec 29 05:12 LocalSettings.php
-rw-r--r-- 1 root root 155M Dec 29 05:03 wiki-error.log
root@hetzner3 /var/tmp/mediawiki #
root@hetzner3 /var/tmp/mediawiki # ls -lah /var/www/html
total 36K
d---r-x--- 8 not-apache www-data 4,0K Dec 29 18:35 .
drwxr-xr-x 3 root root 4,0K Sep 25 02:37 ..
d---r-x--- 3 not-apache www-data 4,0K Dec 26 18:04 fef.opensourceecology.org
d---r-x--- 5 not-apache www-data 4,0K Jul 11 2018 forum.opensourceecology.org
----r----- 1 not-apache www-data 138 Mar 3 2018 .htpasswd
d---r-x--- 3 not-apache www-data 4,0K Dec 26 03:58 microfactory.opensourceecology.org
d---r-x--- 3 not-apache www-data 4,0K Dec 26 20:08 oswh.opensourceecology.org
d---r-x--- 4 not-apache www-data 4,0K Dec 26 20:44 seedhome.openbuildinginstitute.org
d---r-x--- 4 not-apache www-data 4,0K Dec 23 00:25 store.opensourceecology.org
root@hetzner3 /var/tmp/mediawiki #
- ok, first attempt to upgrade failed
root@hetzner3 /var/tmp/mediawiki # sudo -u www-data php "${docrootDir}/maintenance/update.php"
Error: Missing one or more required PHP extensions. Please see
https://www.mediawiki.org/wiki/Manual:Installation_requirements#PHP
for help with installing them.
Please install or enable:
* intl <https://www.php.net/intl>
root@hetzner3 /var/tmp/mediawiki #
- looks like we need 'php-intl' now
root@hetzner3 /var/tmp/mediawiki # apt-cache search php | grep intl php-intl - Internationalisation module for PHP [default] php-symfony-polyfill-intl-grapheme - Symfony polyfill for intl's grapheme_* functions php-symfony-polyfill-intl-icu - Symfony polyfill for intl's ICU-related data and classes php-symfony-polyfill-intl-idn - Symfony polyfill for intl's idn_to_ascii and idn_to_utf8 functions php-symfony-polyfill-intl-messageformatter - Symfony polyfill for intl's MessageFormatter class and related functions php-symfony-polyfill-intl-normalizer - Symfony polyfill for intl's Normalizer class and related functions php-twig-intl-extra - A Twig extension for Intl php8.2-intl - Internationalisation module for PHP php-symfony-intl - limited replacement layer for the PHP extension intl root@hetzner3 /var/tmp/mediawiki #
- next, it doesn't like MonoBook? Looks like it's there
root@hetzner3 /var/tmp/mediawiki # sudo -u www-data php "${docrootDir}/maintenance/update.php"
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
PHP Deprecated: DefaultSettings.php is deprecated and will be removed. Use MainConfigSchema::listDefaultValues() or MainConfigSchema::getDefaultValue() instead. [Called from require_once in /var/www/html/wiki.opensourceecology.org/LocalSettings.php at line 49] in /var/www/html/wiki.opensourceecology.org/htdocs/includes/debug/MWDebug.php on line 385
Error: The MonoBook skin cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(94): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(186): wfLoadSkin()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(83): require_once('...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(308): require_once('...')
#6 {main}
PHP Fatal error: Error Loading extension. Unable to open file /MonoBook/skin.json: filemtime(): stat failed for /MonoBook/skin.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
root@hetzner3 /var/tmp/mediawiki #
root@hetzner3 /var/tmp/mediawiki # ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/skins/
total 28K
d---r-x--- 6 not-apache www-data 4,0K Dec 29 18:14 .
d---r-x--- 14 not-apache www-data 4,0K Dec 29 18:38 ..
d---r-x--- 11 not-apache www-data 4,0K Dec 29 18:14 MinervaNeue
d---r-x--- 7 not-apache www-data 4,0K Dec 29 18:14 MonoBook
----r----- 1 not-apache www-data 1,3K Dec 5 15:41 README
d---r-x--- 6 not-apache www-data 4,0K Dec 29 18:14 Timeless
d---r-x--- 9 not-apache www-data 4,0K Dec 29 18:14 Vector
root@hetzner3 /var/tmp/mediawiki #
- huh, this seems to work fine
root@hetzner3 /var/www/html/wiki.opensourceecology.org # sudo -u www-data echo "<?php echo filemtime('/var/www/html/wiki.opensourceecology.org/htdocs/skins/MonoBook/skin.json'); ?>" | php
1733413878root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- there was very little info about this on the net, but the path starting with '/MonoBook/...' and this thread talking about setting '$wgExtensionDirectory' had me guessing it was a path error
- I fixed it by adding this to the LocalSettings.php config https://www.mediawiki.org/wiki/Manual:$wgStyleDirectory
$wgStyleDirectory = "$IP/skins";
- the docs say that it used to default to "{$IP}/skins" until 1.37, and then it began to default to 'null' -- not sure why
- anyway now we get a similar error about the extensions
root@hetzner3 /var/tmp/mediawiki # sudo -u www-data php "${docrootDir}/maintenance/update.php"
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
Error: The Renameuser extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(313): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(83): require_once('...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(308): require_once('...')
#6 {main}
PHP Fatal error: Error Loading extension. Unable to open file /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Renameuser/extension.json: filemtime(): stat failed for /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Renameuser/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
root@hetzner3 /var/tmp/mediawiki #
- I added this to LocalSettings.php
$wgExtensionsDirectory = "$IP/extensions";
- but we still have error
root@hetzner3 /var/tmp/mediawiki # sudo -u www-data php "${docrootDir}/maintenance/update.php"
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
Error: The Renameuser extension cannot be loaded. Check that all of its files are installed properly.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(57): MediaWiki\Registration\ExtensionRegistry->queue()
#1 /var/www/html/wiki.opensourceecology.org/LocalSettings.php(314): wfLoadExtension()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(220): require_once('...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(83): require_once('...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(308): require_once('...')
#6 {main}
PHP Fatal error: Error Loading extension. Unable to open file /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Renameuser/extension.json: filemtime(): stat failed for /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Renameuser/extension.json in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/MissingExtensionException.php on line 102
root@hetzner3 /var/tmp/mediawiki #
- yeah, there is no 'Renamuser' dir
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls htdocs/extensions/ AbuseFilter ConfirmEdit InputBox MultimediaViewer PdfHandler SecureLinkFixer Thanks CategoryTree DiscussionTools Interwiki Nuke Poem SpamBlacklist TitleBlacklist Cite Echo Linter OATHAuth README SyntaxHighlight_GeSHi VisualEditor CiteThisPage Gadgets LoginNotify PageImages ReplaceText TemplateData WikiEditor CodeEditor ImageMap Math ParserFunctions Scribunto TextExtracts root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- I commented-out that line, and now I get now it finally ran
root@hetzner3 /var/tmp/mediawiki # sudo -u www-data php "${docrootDir}/maintenance/update.php"
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
*******************************************************************************
NOTE: Do not run maintenance scripts directly, use maintenance/run.php instead!
Running scripts directly has been deprecated in MediaWiki 1.40.
It may not work for some (or any) scripts in the future.
*******************************************************************************
MediaWiki 1.43.0 Updater
Your composer.lock file is up to date with current dependencies!
Going to run database updates for osewiki_db-wiki_
Depending on the size of your database this may take a while!
Abort with control-c in the next five seconds (skip this countdown with --quick) ...0
Can not upgrade from versions older than 1.35, please upgrade to that version or later first.
root@hetzner3 /var/tmp/mediawiki #
- but, shit, it says it can't update from versions older than 1.35. Duh, that's why we updated it to 1.35 yesterday! Ugh
- I guess I didn't actually login and confirm the version after the last update
- also, the message yelling us at the top is because apparently we're supposed to use 'run.php' since v1.40, so it's a different command for our first upgrade from our second; this is what we should do, but it gets the same result https://www.mediawiki.org/wiki/Manual:Upgrading
root@hetzner3 /var/tmp/mediawiki # sudo -u www-data php "${docrootDir}/maintenance/run.php" "${docrootDir}/maintenance/update.php"
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 23
MediaWiki 1.43.0 Updater
Your composer.lock file is up to date with current dependencies!
Going to run database updates for osewiki_db-wiki_
Depending on the size of your database this may take a while!
Abort with control-c in the next five seconds (skip this countdown with --quick) ...0
Can not upgrade from versions older than 1.35, please upgrade to that version or later first.
root@hetzner3 /var/tmp/mediawiki #
- well, this the internet says I can verify the version of the wiki by checking the files. that's dumb because we need to check the db
- others say to check Special:Version. Yeah, I could if the site was working; that's also dumb
- I found on SO with a query to check the DB, but it says we're running MedaiWiki v1.24.2? Wtf? https://stackoverflow.com/questions/19074643/determining-mediawiki-version-from-the-database-only
MariaDB [osewiki_db]> select max(ul_key) from wiki_updatelog where ul_key like 'updatelist-%'; +------------------------------+ | max(ul_key) | +------------------------------+ | updatelist-1.24.2-1452650145 | +------------------------------+ 1 row in set (0,001 sec) MariaDB [osewiki_db]>
- well, at this point I guess I have to start back at recreating the site from the hetzner2 transfer, then re-do the first upgrade. But this time I'll be sure to check the DB and the Special:Version pages after the first upgrade before proceeding with the second
- I created a CHG ticket for this whole process https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_wiki_to_hetzner3
- I confirmed that Special:Version on the old wiki does, indeed, say v1.30.0
Installed software Product Version MediaWiki 1.30.0 PHP 5.6.40 (apache2handler) MariaDB 5.5.68-MariaDB ICU 50.1.2 Entry point URLs Entry point URL Article path /wiki/$1 Script path / index.php /index.php api.php /api.php load.php /load.php Installed skins Skin Version License Description Authors Cologne Blue – GPL-2.0+ A lightweight skin with minimal formatting Lee Daniel Crocker and others Modern – GPL-2.0+ A blue/gray theme with sidebar and top bar. Derived from MonoBook River Tarnell and others MonoBook – GPL-2.0+ The classic MediaWiki skin since 2004, named after the black-and-white photo of a book in the page background Gabriel Wicke and others Vector – GPL-2.0+ Modern version of MonoBook with fresh look and many usability improvements Trevor Parscal, Roan Kattouw and others Installed extensions Special pages Extension Version License Description Authors Confirm User Accounts – (4fe25f7) GPL-2.0+ Gives bureaucrats the ability to confirm account requests Aaron Schulz Interwiki 3.1 20160307 GPL-2.0+ Adds a special page to view and edit the interwiki table Stephanie Amanda Stevens, Alexandre Emsenhuber, Robin Pepermans, Siebrand Mazeland, Platonides, Raimond Spekking, Sam Reed, Jack Phoenix, Calimonius the Estrange and others Nuke 1.3.0 GPL-2.0+ Gives administrators the ability to mass delete pages Brion Vibber and Jeroen De Dauw Renameuser – GPL-2.0+ Adds a special page to rename a user (need renameuser right) Ævar Arnfjörð Bjarmason and Aaron Schulz Replace Text 1.2 (4426752) GPL-2.0+ Provides a special page to allow administrators to do a global string find-and-replace on all the content pages of a wiki Yaron Koren, Niklas Laxström and others UserMerge 1.10.1 (4546537) GPL-2.0+ Merges references from one user to another user in the wiki database - will also delete old users following merge. Requires usermerge privileges Tim Laqua, Thomas Gries and Matthew April Parser hooks Extension Version License Description Authors CategoryTree – (850c018) GPL-2.0+ Dynamically navigate the category structure Daniel Kinzler Cite – GPL-2.0+ Adds <ref[ name=id]> and <references/> tags, for citations Ævar Arnfjörð Bjarmason, Andrew Garrett, Brion Vibber, Ed Sanders, Marius Hoch, Steve Sanbeg, Trevor Parscal and others ParserFunctions 1.6.0 GPL-2.0+ Enhance parser with logical functions Tim Starling, Robert Rohde, Ross McClure and Juraj Simlovic Widgets 1.3.0 (fce5acc) GPL-2.0+ Allows wiki administrators to add free-form widgets to the wiki by editing pages within the Widget namespace. Community-contributed widgets can be found on MediaWikiWidgets.org Sergey Chernyshev, Yaron Koren and others Spam prevention Extension Version License Description Authors ConfirmEdit 1.5.0 GPL-2.0+ Provides CAPTCHA techniques to protect against spam and password-guessing Brion Vibber, Florian Schmidt, Sam Reed and others Other Extension Version License Description Authors Gadgets – GPL-2.0+ Lets users select custom CSS and JavaScript gadgets in their preferences Daniel Kinzler and Max Semenik OATHAuth 0.2.2 (bed2e4b) GPL-2.0+ Provides authentication support using HMAC based one-time passwords Ryan Lane ReCaptcha – Installed libraries Library Version License Description Authors composer/semver 1.4.2 MIT Semver library that offers utilities, version constraint parsing and validation. Nils Adermann, Jordi Boggiano and Rob Bast cssjanus/cssjanus 1.2.0 Apache-2.0 Convert CSS stylesheets between left-to-right and right-to-left. firebase/php-jwt 4.0.0 BSD-3-Clause A simple library to encode and decode JSON Web Tokens (JWT) in PHP. Should conform to the current spec. Neuman Vong and Anant Narayanan james-heinrich/getid3 1.9.14 GPL PHP script that extracts useful information from popular multimedia file formats justinrainbow/json-schema 5.2.1 MIT A library to validate a json schema. Bruno Prieto Reis, Justin Rainbow, Igor Wiedler and Robert Schönthal liuggio/statsd-php-client 1.0.18 MIT Statsd (Object Oriented) client library for PHP Giulio De Donato mediawiki/at-ease 1.1.0 GPL-2.0+ Safe replacement to @ for suppressing warnings. Tim Starling and MediaWiki developers monolog/monolog 1.22.1 MIT Sends your logs to files, sockets, inboxes, databases and various web services Jordi Boggiano mustangostang/spyc 0.6.2 MIT A simple YAML loader/dumper class for PHP mustangostang nmred/kafka-php 0.1.5 BSD-3-Clause Kafka client for php oojs/oojs-ui 0.23.0 MIT Provides library of common widgets, layouts, and windows. Timo Tijhof, Bartosz Dziewoński, Ed Sanders, James D. Forrester, Kirsten Menger-Anderson, Rob Moen, Roan Kattouw, Trevor Parscal, Kunal Mehta and Prateek Saxena oyejorge/less.php 1.7.0.14 Apache-2.0 PHP port of the Javascript version of LESS http://lesscss.org (Originally maintained by Josh Schmidt) Matt Agar, Martin Jantošovič and Josh Schmidt pear/console_getopt 1.4.1 BSD-2-Clause More info available on: http://pear.php.net/package/Console_Getopt Greg Beaver, Andrei Zmievski and Stig Bakken pear/mail 1.4.1 BSD-2-Clause Class that provides multiple interfaces for sending emails. Chuck Hagenbuch, Richard Heyes and Aleksander Machniak pear/mail_mime 1.10.1 BSD-3-clause Mail_Mime provides classes to create MIME messages Cipriano Groenendal and Aleksander Machniak pear/mail_mime-decode 1.5.5.2 BSD-2-Clause More info available on: http://pear.php.net/package/Mail_mimeDecode Cipriano Groenendal and Aleksander Machniak pear/net_smtp 1.7.3 PHP-3.01 An implementation of the SMTP protocol Jon Parise and Chuck Hagenbuch pear/net_socket 1.2.1 BSD-2-Clause More info available on: http://pear.php.net/package/Net_Socket Chuck Hagenbuch, Aleksander Machniak and Stig Bakken pear/pear-core-minimal 1.10.3 BSD-3-Clause Minimal set of PEAR core files to be used as composer dependency Christian Weiske pear/pear_exception 1.0.0 BSD-2-Clause The PEAR Exception base class. Helgi Thormar and Greg Beaver pimple/pimple 3.0.2 MIT Pimple, a simple Dependency Injection Container Fabien Potencier psr/log 1.0.2 MIT Common interface for logging libraries PHP-FIG ruflin/elastica 5.1.0 MIT Elasticsearch Client Nicolas Ruflin stil/gd-text 1.0.0 MIT A class drawing multiline and aligned text on pictures. Uses GD extension. symfony/process 3.2.6 MIT Symfony Process Component Fabien Potencier and Symfony Community wikimedia/assert 0.2.2 MIT Provides runtime assertions Daniel Kinzler wikimedia/avro 1.7.7 Apache-2.0 A library for using Apache Avro with PHP. Michael Glaesemann, Andy Wick, Saleem Shafi, A B, Doug Cutting and Tom White wikimedia/base-convert 1.0.1 GPL-2.0+ Convert an arbitrarily-long string from one numeric base to another, optionally zero-padding to a minimum column width. Brion Vibber and Tyler Romeo wikimedia/cdb 1.4.1 GPL-2.0+ Constant Database (CDB) wrapper library for PHP. Provides pure-PHP fallback when dba_* functions are absent. Daniel Kinzler, Tim Starling, Chad Horohoe and Ori Livneh wikimedia/cldr-plural-rule-parser 1.0.0 GPL-2.0+ Evaluates plural rules specified in the CLDR project notation. Tim Starling and Niklas Laxström wikimedia/composer-merge-plugin 1.4.1 MIT Composer plugin to merge multiple composer.json files Bryan Davis wikimedia/css-sanitizer 1.0.2 Apache-2.0 Classes to parse and sanitize CSS Brad Jorsch wikimedia/html-formatter 1.0.1 GPL-2.0+ Performs transformations of HTML by wrapping around libxml2 and working around its countless bugs. MediaWiki contributors wikimedia/ip-set 1.1.0 GPL-2.0+ Efficiently match IP addresses against a set of CIDR specifications. Brandon Black wikimedia/php-session-serializer 1.0.4 GPL-2.0+ Provides methods like PHP's session_encode and session_decode that don't mess with $_SESSION Brad Jorsch wikimedia/purtle 1.0.6 GPL-2.0+ Fast streaming RDF serializer Daniel Kinzler, Thiemo Mättig and Stanislav Malyshev wikimedia/relpath 2.0.0 MIT Compute a relative filepath between two paths. Ori Livneh wikimedia/remex-html 1.0.1 MIT Fast HTML 5 parser Tim Starling wikimedia/running-stat 1.1.0 GPL-2.0+ PHP implementations of online statistical algorithms Ori Livneh wikimedia/scoped-callback 1.0.0 GPL-2.0+ Class for asserting that a callback happens when a dummy object leaves scope Aaron Schulz wikimedia/testing-access-wrapper 1.0.0 GPL-2.0+ A simple helper class to access non-public elements of a class when testing. Adam Roses Wight, Brad Jorsch and Gergő Tisza wikimedia/textcat 1.2.0 LGPL-2.1 PHP port of the TextCat language guesser utility, see http://odur.let.rug.nl/~vannoord/TextCat/. Stanislav Malyshev and Trey Jones wikimedia/timestamp 1.0.0 GPL-2.0+ Creation, parsing, and conversion of timestamps Tyler Romeo wikimedia/utfnormal 1.1.0 GPL-2.0+ Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present Brion Vibber wikimedia/wait-condition-loop 1.0.1 GPL-2.0+ Wait loop that reaches a condition or times out Aaron Schulz wikimedia/wrappedstring 2.2.0 MIT Automatically compact sequentially-outputted strings that share a common prefix / suffix pair. Timo Tijhof zordius/lightncandy 0.23 MIT An extremely fast PHP implementation of handlebars ( http://handlebarsjs.com/ ) and mustache ( http://mustache.github.io/ ). Zordius Chen
- I then ran this on hetzner2, which shows that this db query is useless
[root@opensourceecology ~]# echo "select max(ul_key) from wiki_updatelog where ul_key like 'updatelist-%';" | mysql osewiki_db -uroot -p${mysqlPass}
max(ul_key)
updatelist-1.24.2-1452650145
[root@opensourceecology ~]#
- I think that the only thing I need to change from this process is that I need to change the (time)stamp var, so we get the last backup; no need to make a new one from hetzner2
stamp=$(ls /var/tmp/backups_for_migration_from_hetzner2/ | grep -i wiki | grep -oE "[0-9]{8}")
- ...
- Marcin just emailed me saying that they want to retire fef, but make a backup of it
- I re-visited our forums deprecation CHG to see the commands to use wget to create a static site https://wiki.opensourceecology.org/wiki/CHG-2018-02-04_deprecate_vanilla_forums
- I drafted this to create a fef offline site
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='fef.opensourceecology.org'
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current
mkdir wget
pushd wget
time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "fef.opensourceecology.org" "http://fef.opensourceecology.org"
- I executed it on hetzner2: good news; the whole website is just 32 MB!
[root@opensourceecology wget]# du -sh fef.opensourceecology.org/ 32M fef.opensourceecology.org/ [root@opensourceecology wget]#
- I also downloaded an xml export of the wordpress site. It appears that doesn't include the media files, and it was only 698 kb.
- even if I tell it to export the "media" dir, it doesn't do it.
- anyway, we do have an actual backup of the db and wordpress files already; ;it's only 180 MB
[root@opensourceecology backups_for_migration_to_hetzner2]# du -sh fef.opensourceecology.org_20241226/current/* 179M fef.opensourceecology.org_20241226/current/fef.opensourceecology.org_files.20241226.tar.gz 64K fef.opensourceecology.org_20241226/current/mysqldump_fef.opensourceecology.org.20241226.sql.bz2 [root@opensourceecology backups_for_migration_to_hetzner2]#
- all right, here's what I'm thinking: we keep fef around as a static-site vhost, just like our forums; it will forever look like it does, without having to worry about code rot -- but we won't be able to edit it in the future
- And in the vhost (not docroot), we keep a copy of the export xml, db, and files.
- this is a trade-off where we'll have more data but less time. We should be able to migrate this site much easier, and it will never break again due to code rot
- These backups are lossy, but we should have everything we'd need to re-import it into wordpress in the future, though it would probably never look the same and it would not be trivial (but it will be much easier than just restoring from a static site)
- at the end of the day, the site will require us to lug around 32+180+1 = 213 MB. I think that's fine.
- ...
- Marcin also suggested that we do the same for oswh.opensourceecology.org
- I did the same for oswh
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='oswh.opensourceecology.org'
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current
mkdir wget
pushd wget
time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "${vhost_name}" "${vhost_name}"
- the static site is just 3 MB!
[root@opensourceecology wget]# pwd /var/tmp/backups_for_migration_to_hetzner3/oswh.opensourceecology.org_20241230/current/wget [root@opensourceecology wget]# du -sh * 2.5M oswh.opensourceecology.org [root@opensourceecology wget]#
- the files and db backup is 50 MB
[root@opensourceecology wget]# du -sh /var/tmp/backups_for_migration_to_hetzner2/oswh.opensourceecology.org_20241226/current/* 1.1M /var/tmp/backups_for_migration_to_hetzner2/oswh.opensourceecology.org_20241226/current/mysqldump_oswh.opensourceecology.org.20241226.sql.bz2 48M /var/tmp/backups_for_migration_to_hetzner2/oswh.opensourceecology.org_20241226/current/oswh.opensourceecology.org_files.20241226.tar.gz [root@opensourceecology wget]#
- And the xml export is 11 MB
user@disp6783:~/Downloads$ du -sh opensourcehardwaredocumentationjam.wordpress.2024-12-30.xml 11M opensourcehardwaredocumentationjam.wordpress.2024-12-30.xml user@disp6783:~/Downloads$
- so that means the total oswh would be 3+50+11=64 MB
- I sent an email to Marcin & Catarina with this info
ok, I have good news: the fef and oswh sites are small enough that we can just turn them into static sties. Basically I'll do what we did with the OSE vanilla forums. Pros: + It's nearly negligible extra time to migrate in the future + It isn't generated dynamically, so it wont "rot" in the future + It doesn't require wordpress or themes or plugins Cons: + They're generally bigger in size + You can't edit them (easily) + You can't go from a static site back to a wordpress site again However, to mitigate that last con, I'll also keep three backups: 1. An xml export from wordpress 2. A wordpress DB backup 3. A wordpress file backup I just confirmed that doing this for: * fef.opensourceecology.org requires a total of ~250 MB * oswh.opensourceecology.org requires a total of ~100 MB This is totally reasonable, and I think it's the best happy-medium. Please let me know if you have any questions, and if you'd like me to convert any of your other unused sites into static sites. Thank you, Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net On 12/29/24 22:50, Michael Altfield wrote: > Yeah, making a backup of the data (post/page text and images) is easy. > > The difficult & time-consuming part is trying to keep making it look the > same after the code that renders it rots over time.. > > I'll look into: > > [1] exporting the content from wordpress (which you can import into > wordpress in the future if you ever want restore the site) > > [2] making an html static site copy (like we did with the forums) > > Both are lossy in different ways. I'll let you know how big these are > and if they're practical for the sites we're retiring. > > So far we have the following on the not-migrate list: > > [a] fef.opensourceecology.org > [b] oswh.opensourceecology.org > > Here's the others: > > 2. store.opensourceecology.orgc > 3. microfactory.opensourceecology.org > 6. seedhome.openbuildinginstitute.org > 7. www.openbuildinginstitute.org > > Are there any more you want me to not migrate? > > Michael Altfield > https://www.michaelaltfield.net > PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 > > Note: If you cannot reach me via email, please check to see if I have > changed my email address by visiting my website at https:// > email.michaelaltfield.net > > On 12/29/24 22:13, Marcin Jakubowski wrote: >> You're right about not using unpopular themes and as a practice go to >> only >> wordpress.org/plugins/. >> >> Just had a long discussion with Catarina regarding archival footage from >> the past, which is what the fef site consists of. She has no plans to >> continue the site as it is. It appears that the best way to go forward is >> to simply download the pictures so she has a local backup. She can't find >> the originals from 10 years ago. We have the text from the site. >> What's the >> easiest way to download all the pictures and transfer to us - can you do >> that for us? >> >> Not that I expect that we would want to do this - but we do have a backup >> from last year or whenever - so if we ever wanted to restore the site >> with >> a different theme - we could still do that, correct? >> >> In any case, once we are done with the server migration, let's meet up >> so I >> can get caught up on how everything went, and get a refresher on how to >> access everything if you get hit by a bus. The main point is to migrate >> only - and later think about what additional improvements we may need >> after >> the apprenticeship is running. As long as we have the wiki, main site, >> and >> phplist, we're good for basic operations. >> >> Marcin >> >> MJ
Sat Dec 28, 2024
- here's TOFU 3/3 on phpList and MediaWiki (ISP, exit in Ecuador)
Ecuador 2024-12-28 ######################################################################### 100.0% ######################################################################### 100.0% ######################################################################### 100.0% ######################################################################### 100.0% 2024-12-28 3c457f4960cfcbdf5e1e2fab6b6e3ade006e65a55cd93459e0ecede3d9c23a16 keys.txt 56e0a7fe2ff78e50b9c87b3d589f2d8df5bcab0018e454b2a4a239c8586bd8ee mediawiki-1.43.0.tar.gz cee3d020f25e9b3f66eef214a343be466a1a313db31344ad2a5e87ab0183883e mediawiki-1.43.0.tar.gz.sig 9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a phplist-3.6.14.zip gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa4096/0x73F146FECF9D333C 2014-11-20 [SC] [expired: 2021-06-05] Key fingerprint = F64E BF5F 2099 6AB5 14F1 98A8 73F1 46FE CF9D 333C uid Tim Starling <tstarling@wikimedia.org> sub rsa4096/0x1075249FCCC9CAAF 2014-11-20 [E] [expired: 2021-06-05] pub dsa1024/0xC119E1A64D70938E 2003-11-15 [SCA] Key fingerprint = 4412 76E9 CCD1 5F44 F6D9 7D18 C119 E1A6 4D70 938E uid Brion Vibber <brion@pobox.com> sub elg1024/0x6596FAD2965B3548 2003-11-15 [E] pub dsa1024/0x9B69B3109D3BB7B0 2011-10-24 [SC] Key fingerprint = 1D98 867E 8298 2C8F E0AB C25F 9B69 B310 9D3B B7B0 uid Sam Reed <reedy@wikimedia.org> sub elg2048/0x3BBB95CE2B08BFD2 2011-10-24 [E] pub rsa2048/0x72BC1C5D23107F8A 2014-04-29 [SC] [expires: 2026-04-29] Key fingerprint = 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A uid Chad Horohoe <chad@wikimedia.org> uid keybase.io/demon <demon@keybase.io> sub rsa2048/0x08CF4E7951361C13 2014-04-29 [E] [expires: 2026-04-29] pub rsa4096/0xF6DAD285018FAC02 2014-02-19 [SC] [expired: 2018-10-04] Key fingerprint = 6237 D8D3 ECC1 AE91 8729 296F F6DA D285 018F AC02 uid Tyler Cipriani <tcipriani@wikimedia.org> uid Tyler Cipriani <tyler@tylercipriani.com> uid [jpeg image of size 5098] sub rsa4096/0xB002E1FDEE737D83 2014-02-19 [E] [expired: 2018-10-04] pub rsa3072/0x26752EBB0D9E6218 2021-11-11 [SC] Key fingerprint = 72D2 86F6 F8F0 3C78 F2C5 9C73 2675 2EBB 0D9E 6218 uid Amir Sarabadani <asarabadani@wikimedia.org> sub rsa3072/0x4F889038CE86B378 2021-11-11 [E] pub rsa4096/0x361F943B15C08DD4 2015-05-22 [SC] [expired: 2020-05-20] Key fingerprint = 80D1 13B7 67E3 D519 3672 5679 361F 943B 15C0 8DD4 uid Brian Wolff <bwolff@wikimedia.org> uid Brian Wolff (Bawolff) <bawolff@gmail.com> sub rsa4096/0xBF1629CD074D3DD8 2015-05-22 [E] [expired: 2020-05-20] pub rsa4096/0x131910E01605D9AA 2016-01-08 [SC] [expired: 2020-07-31] Key fingerprint = C83A 8E4D 3C8F EB7C 8A3A 1998 1319 10E0 1605 D9AA uid Mukunda Modell <twentyafterfour@gmail.com> uid Mukunda Modell (WMF) <mmodell@wikimedia.org> uid [jpeg image of size 2928] sub rsa4096/0x5411F23A0C4E5EC1 2018-12-25 [A] [expired: 2020-12-24] sub rsa4096/0x02C99BB8AB1C6DD5 2018-12-25 [E] [expired: 2020-12-24] sub rsa4096/0x60AE06D4875BE862 2018-12-26 [S] [expired: 2019-12-26] pub rsa4096/0xA8F734246D73B586 2024-12-09 [SC] [expires: 2026-12-09] Key fingerprint = E059 C034 E7A4 3058 3C25 2F4A A8F7 3424 6D73 B586 uid atieno njira <pnjira@wikimedia.org> sub rsa4096/0xB65D063E68F2FC8A 2024-12-09 [E] [expires: 2026-12-09] user@disp1764:/tmp/tmp.wJv4Y7wAMf$
- the hashes all match; let's import that new MediaWiki key into our hetzner3 keyring
root@hetzner3 /etc/nginx # cd /var/tmp/mediawiki/ root@hetzner3 /var/tmp/mediawiki # root@hetzner3 /var/tmp/mediawiki # ls keys.txt mediawiki-1.35.0 mediawiki-1.35.0.tar.gz mediawiki-1.35.0.tar.gz.sig root@hetzner3 /var/tmp/mediawiki # root@hetzner3 /var/tmp/mediawiki # rm keys.txt root@hetzner3 /var/tmp/mediawiki # f root@hetzner3 /var/tmp/mediawiki # wget https://www.mediawiki.org/keys/keys.txt --2024-12-28 22:07:27-- https://www.mediawiki.org/keys/keys.txt Resolving www.mediawiki.org (www.mediawiki.org)... 2a02:ec80:300:ed1a::1, 185.15.59.224 Connecting to www.mediawiki.org (www.mediawiki.org)|2a02:ec80:300:ed1a::1|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 59336 (58K) [text/plain] Saving to: ‘keys.txt’ keys.txt 100%[================================================================>] 57,95K --.-KB/s in 0,01s 2024-12-28 22:07:28 (5,02 MB/s) - ‘keys.txt’ saved [59336/59336] root@hetzner3 /var/tmp/mediawiki # root@hetzner3 /var/tmp/mediawiki # sha256sum keys.txt 3c457f4960cfcbdf5e1e2fab6b6e3ade006e65a55cd93459e0ecede3d9c23a16 keys.txt root@hetzner3 /var/tmp/mediawiki #
- the checksum matches our 3TOFU; let's import it
root@hetzner3 /var/tmp/mediawiki # gpg --import keys.txt gpg: key 73F146FECF9D333C: "Tim Starling <tstarling@wikimedia.org>" not changed gpg: key C119E1A64D70938E: 9 signatures not checked due to missing keys gpg: key C119E1A64D70938E: "Brion Vibber <brion@pobox.com>" not changed gpg: key 9B69B3109D3BB7B0: "Sam Reed <reedy@wikimedia.org>" not changed gpg: key 72BC1C5D23107F8A: "Chad Horohoe <chad@wikimedia.org>" not changed gpg: key F6DAD285018FAC02: 12 signatures not checked due to missing keys gpg: key F6DAD285018FAC02: "Tyler Cipriani <tcipriani@wikimedia.org>" not changed gpg: key 26752EBB0D9E6218: "Amir Sarabadani <asarabadani@wikimedia.org>" not changed gpg: key 361F943B15C08DD4: 7 signatures not checked due to missing keys gpg: key 361F943B15C08DD4: "Brian Wolff (Bawolff) <bawolff@gmail.com>" not changed gpg: key 131910E01605D9AA: "Mukunda Modell (WMF) <mmodell@wikimedia.org>" not changed gpg: key A8F734246D73B586: public key "atieno njira <pnjira@wikimedia.org>" imported gpg: Total number processed: 9 gpg: imported: 1 gpg: unchanged: 8 You have mail in /var/mail/root root@hetzner3 /var/tmp/mediawiki # root@hetzner3 /var/tmp/mediawiki # cp keys.txt /home/maltfield/ root@hetzner3 /var/tmp/mediawiki # chown maltfield /home/maltfield/keys.txt root@hetzner3 /var/tmp/mediawiki # root@hetzner3 /var/tmp/mediawiki # logout maltfield@hetzner3:~$ gpg --import keys.txt gpg: key 73F146FECF9D333C: "Tim Starling <tstarling@wikimedia.org>" not changed gpg: key C119E1A64D70938E: 9 signatures not checked due to missing keys gpg: key C119E1A64D70938E: "Brion Vibber <brion@pobox.com>" not changed gpg: key 9B69B3109D3BB7B0: "Sam Reed <reedy@wikimedia.org>" not changed gpg: key 72BC1C5D23107F8A: "Chad Horohoe <chad@wikimedia.org>" not changed gpg: key F6DAD285018FAC02: 12 signatures not checked due to missing keys gpg: key F6DAD285018FAC02: "Tyler Cipriani <tcipriani@wikimedia.org>" not changed gpg: key 26752EBB0D9E6218: "Amir Sarabadani <asarabadani@wikimedia.org>" not changed gpg: key 361F943B15C08DD4: 7 signatures not checked due to missing keys gpg: key 361F943B15C08DD4: "Brian Wolff (Bawolff) <bawolff@gmail.com>" not changed gpg: key 131910E01605D9AA: "Mukunda Modell (WMF) <mmodell@wikimedia.org>" not changed gpg: key A8F734246D73B586: public key "atieno njira <pnjira@wikimedia.org>" imported gpg: Total number processed: 9 gpg: imported: 1 gpg: unchanged: 8 maltfield@hetzner3:~$
- now let's download the latest version of Mediawiki too
root@hetzner3 /var/tmp/mediawiki # wget https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.0.tar.gz https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.0.tar.gz.sig --2024-12-28 22:10:54-- https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.0.tar.gz Resolving releases.wikimedia.org (releases.wikimedia.org)... 2a02:ec80:300:ed1a::1, 185.15.59.224 Connecting to releases.wikimedia.org (releases.wikimedia.org)|2a02:ec80:300:ed1a::1|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 91947888 (88M) [application/x-gzip] Saving to: ‘mediawiki-1.43.0.tar.gz’ mediawiki-1.43.0.tar.gz 100%[================================================================>] 87,69M 28,6MB/s in 3,5s 2024-12-28 22:10:58 (25,1 MB/s) - ‘mediawiki-1.43.0.tar.gz’ saved [91947888/91947888] --2024-12-28 22:10:58-- https://releases.wikimedia.org/mediawiki/1.43/mediawiki-1.43.0.tar.gz.sig Reusing existing connection to [releases.wikimedia.org]:443. HTTP request sent, awaiting response... 200 OK Length: 566 [application/pgp-signature] Saving to: ‘mediawiki-1.43.0.tar.gz.sig’ mediawiki-1.43.0.tar.gz.sig 100%[================================================================>] 566 --.-KB/s in 0s 2024-12-28 22:10:58 (13,7 MB/s) - ‘mediawiki-1.43.0.tar.gz.sig’ saved [566/566] FINISHED --2024-12-28 22:10:58-- Total wall clock time: 4,1s Downloaded: 2 files, 88M in 3,5s (25,1 MB/s) root@hetzner3 /var/tmp/mediawiki #
- I confirmed that the hashes match our 3TOFU *and* the pgp signature is valid; great, this is trustworthy
root@hetzner3 /var/tmp/mediawiki # sha256sum mediawiki-1.43.0.tar.gz* 56e0a7fe2ff78e50b9c87b3d589f2d8df5bcab0018e454b2a4a239c8586bd8ee mediawiki-1.43.0.tar.gz cee3d020f25e9b3f66eef214a343be466a1a313db31344ad2a5e87ab0183883e mediawiki-1.43.0.tar.gz.sig root@hetzner3 /var/tmp/mediawiki # root@hetzner3 /var/tmp/mediawiki # gpg --verify mediawiki-1.43.0.tar.gz.sig gpg: assuming signed data in 'mediawiki-1.43.0.tar.gz' gpg: Signature made 2024-12-20T19:36:38 UTC gpg: using RSA key E059C034E7A430583C252F4AA8F734246D73B586 gpg: Good signature from "atieno njira <pnjira@wikimedia.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: E059 C034 E7A4 3058 3C25 2F4A A8F7 3424 6D73 B586 root@hetzner3 /var/tmp/mediawiki #
- I also downloaded the latest phpList release
root@hetzner3 /var/tmp # wget https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip --2024-12-28 22:13:27-- https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip Resolving altushost-swe.dl.sourceforge.net (altushost-swe.dl.sourceforge.net)... 79.142.76.130 Connecting to altushost-swe.dl.sourceforge.net (altushost-swe.dl.sourceforge.net)|79.142.76.130|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://downloads.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip [following] --2024-12-28 22:13:27-- https://downloads.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 2606:4700::6812:d95, 2606:4700::6812:c95, 104.18.12.149, ... Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|2606:4700::6812:d95|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://kumisystems.dl.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip?viasf=1 [following] --2024-12-28 22:13:27-- https://kumisystems.dl.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip?viasf=1 Resolving kumisystems.dl.sourceforge.net (kumisystems.dl.sourceforge.net)... 2a01:4f8:210:1057::2, 148.251.120.111 Connecting to kumisystems.dl.sourceforge.net (kumisystems.dl.sourceforge.net)|2a01:4f8:210:1057::2|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 29850924 (28M) [application/octet-stream] Saving to: ‘phplist-3.6.14.zip’ phplist-3.6.14.zip 100%[================================================================>] 28,47M 76,2MB/s in 0,4s 2024-12-28 22:13:29 (76,2 MB/s) - ‘phplist-3.6.14.zip’ saved [29850924/29850924] root@hetzner3 /var/tmp #
- and I – oh shit. There's a difference!
root@hetzner3 /var/tmp # sha256sum phplist-3.6.14.zip 938e9bdb64d8c042a04192e1fca42814d906e715ec9c2726756425a1be7e0791 phplist-3.6.14.zip root@hetzner3 /var/tmp #
- I deleted it and downloaded it again. I got the same bad hash. This is crazy. I just did the third TOFU less than an hour ago. Did they change something?!?
- I opened a thread about it in their forums; I'm not touching this release until we get clarification, jesus https://discuss.phplist.org/t/psa-release-hash-changed-publishing-infrastructure-comprimise/9927
- somehow it looks like I was downloading the old version? 3.6.15 is the latest version; not sure how that happened because it says 3.6.15 was released in April. Well that fucks-up our 3TOFU but still this changing hash is a concern.
- I'm going to wait a few days before I touch phpList; let's see what the community has to say
- ...
- meanwhile, we got a response from the oshine support team; they gave me links to their google drive to download their plugins, but said they don't have download links for third party plugins. I asked for the slugs of those plugins
Can you please tell me the slugs of the other required plugins, so I can download them from wordpress.org Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net On 12/27/24 22:13, BrandExponents wrote: > ----------------------------------------------------------- > Hi Michael, > > We can only send you the direct download links of the plugins we own. > The third party supporting plugins for Oshine theme can only be > installed directly from your site's dashboard. > > Here are the download links: > Tatsu > plugin: https://drive.google.com/file/d/1aaSZxC8AumyrFEoAd1bZega9rmI9ZMzW/view?usp=sharing > Oshine > Modules: https://drive.google.com/file/d/1oKJupmosWglpCfYDsRIWGoRV_cxSlrb6/view?usp=sharing > Oshine > Core: https://drive.google.com/file/d/1uGj9BgbMpNqnfJk6wE6goLq42vJwLDrD/view?usp=sharing > [https://drive.google.com/file/d/1uGj9BgbMpNqnfJk6wE6goLq42vJwLDrD/view?usp=sharing] > > Do let me know if you have any queries or concerns. > > Best regards, & Have a nice weekend! > > -- > Suman M., Tech Support Executive
- I'm going to have to do a 3TOFU on these, but first let's figure out what they gave us and what we're still missing
- ugh, you can't even wget these shitty google drive URLs; this is going to be so annoying to verify with 3TOFU
user@host:~$ wget https://drive.google.com/file/d/1aaSZxC8AumyrFEoAd1bZega9rmI9ZMzW/view?usp=sharing
--2024-12-28 22:53:03-- https://drive.google.com/file/d/1aaSZxC8AumyrFEoAd1bZega9rmI9ZMzW/view?usp=sharing
Resolving drive.google.com (drive.google.com)... 142.250.186.174
Connecting to drive.google.com (drive.google.com)|142.250.186.174|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘view?usp=sharing’
view?usp=sharing [ <=> ] 91.42K 231KB/s in 0.4s
2024-12-28 22:53:20 (231 KB/s) - ‘view?usp=sharing’ saved [93616]
user@host:~$
user@host:~$ head -c 2048 view\?usp\=sharing
<!DOCTYPE html><html><head><script nonce="IdpnSdj2tPFuOlj9hU9khQ"> window['_DRIVE_VIEWER_ctiming']={}; </script><meta name="google" content="notranslate"><meta http-equiv="X-UA-Compatible" content="IE=edge;"><style nonce="o_Fo6LpKy3W204sfbq1iYA">@font-face{font-family:'Roboto';font-style:italic;font-weight:400;src:url(fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1Mu51xIIzc.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;src:url(fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmSU5fBBc9.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;src:url(fonts.gstatic.com/s/roboto/v18/KFOmCnqEu92Fr1Mu4mxP.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;src:url(fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmEU9fBBc9.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;src:url(//fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmWUlfBBc9.ttf)format('truetype');}</style><meta name="referrer" content="origin"><title>tatsu-3.5.3.zip - Google Drive</title><script nonce="IdpnSdj2tPFuOlj9hU9khQ">
window['_DRIVE_VIEWER_IVIS'] = document.visibilityState;
</script><meta property="og:title" content="tatsu-3.5.3.zip"><meta property="og:type" content="article"><meta property="og:site_name" content="Google Docs"><meta property="og:url" content="https://drive.google.com/file/d/1aaSZxC8AumyrFEoAd1bZega9rmI9ZMzW/view?usp=sharing&usp=embed_facebook"><link rel="shortcut icon" href="https://ssl.gstatic.com/images/branding/product/1x/drive_2020q4_32dp.png"><script nonce="IdpnSdj2tPFuOlj9hU9khQ"> window['_DRIVE_VIEWER_ctiming']['cls']=performance.now(); </script><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Google+Sans_old:300,400,500,700" nonce="o_Fo6LpKy3W204sfbq1iYA"><link rel="stylesheet" href="https://www.gstatic.com/_/apps-fileview/_/ss/k=apps-fileview.v.ovqSItYnX0g.L.W.O/am=MBg/d=0/rs=AO0039tz5VrQ5sjGTSjn86HXfczh2ZNxIg" data-id="_cl" noncuser@host:~$
- I loaded the first one in Tor Browser and downloaded it manually; here's what it gave me
user@host:~/.tb/tor-browser/Browser/Downloads$ ls tatsu-3.5.3.zip user@host:~/.tb/tor-browser/Browser/Downloads$
- this is weird; there is no style.css file in this plugin's dir
user@host:~/.tb/tor-browser/Browser/Downloads$ unzip tatsu-3.5.3.zip ... user@host:~/.tb/tor-browser/Browser/Downloads$ user@host:~/.tb/tor-browser/Browser/Downloads$ cd tatsu/ user@host:~/.tb/tor-browser/Browser/Downloads/tatsu$ ls admin changelog.txt includes LICENSE.txt README.txt builder css index.php plugin-update-checker tatsu.php changelog.md img languages public uninstall.php user@host:~/.tb/tor-browser/Browser/Downloads/tatsu$
- best I found is the 'tatsu.php' file, but there's no slug given
user@host:~/.tb/tor-browser/Browser/Downloads/tatsu$ head -n15 tatsu.php <?php /** * Plugin Name: Tatsu * Plugin URI: http://www.brandexponents.com * Description: A Powerful and Elegant Live Front End Website Builder for Wordpress. * Version: 3.5.3 * Author: Brand Exponents * Author URI: http://www.brandexponents.com * License: GPL-2.0+ * License URI: http://www.gnu.org/licenses/gpl-2.0.txt * Text Domain: tatsu * Domain Path: /languages */ // If this file is called directly, abort. user@host:~/.tb/tor-browser/Browser/Downloads/tatsu$
- this is frustrating; I don't like ambiguity. let's try the second link; it gave us this 'oshine-modules-3.3.8.zip' file
user@host:~/.tb/tor-browser/Browser/Downloads$ ls oshine-modules-3.3.8.zip tatsu tatsu-3.5.3.zip user@host:~/.tb/tor-browser/Browser/Downloads$
- this one also doesn't have a style.css file. I guess the slug in this case is the folder name and the file name before the .php
- so now we have 'tatsu' and 'oshine-modules'
- the third link get us 'oshine-core'
- reminder: here's the list of plugins that oshine complained we need to install
BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu.
- so I'd say we got the last 3. Now we need to find these:
BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs
- I did a search on wordpress.org for the above plugins, but it's super ambiguous (and dangerous) to try to match these names to other plugins. I could guess, but I have no idea if it's the right plugin.
- For example, there's a plugin just called "Meta Box" – but wtf are these three other plugins?
- I sent another email asking for the slugs
- also, I read in their own documentation that the release that we downloaded from themeforest *should* include these paid plugins for free, but it didn't have the plugins dir.
I'm still waiting for the slugs for the remaining required plugins: 1. BE Portfolio Post Type 2. Meta Box Conditional Logic 3. Meta Box Show Hide 4. Meta Box Tabs These "human readable" names for the plugins are not very useful for finding the actual plugins on wordpress.org/plugins/ Also, your own documentation says that the plugins should have been included in your theme's .zip that I downloaded from themeforest * https://oshine-knowledgebase.brandexponents.com/knowledge-base/install-activate-required-plugins/ > NOTE: When you download the latest theme package from the downloads > of your themeforest account, you will find the latest copy of the > following plugins included in a folder named “plugins” after > extracting the the zip. Why were the plugins abesnt? Can you please fix your latest release on themeforest to include the required plugins, as advertised? Thank you, Michael Altfield
- ...
- regarding oswh, Catarina responded saying the site was originally setup by Simone Cicero, and asked Marcin (CCd) if he wanted to buy another license.
- I don't think we should buy it if it's not actively updated, and the fact that their sales team didn't respond to my email that I sent 3 days ago isn't a great sign.
- Catarina also said that I can try to find some free wp theme for events
- I guess let's wait 2 weeks on oswh to see if Marcin or the Eventor author replies. If they don't, I'll just do some research and find/install a few FOSS event wordpress themes to install and replace this theme
- ...
- ok, back to the wiki
- last time I finished setting up the vhost, but the web browser greeted me with an error telling me to install the mbstring php extension
- I installed `php-mbstring` from apt and added it to ansible
- now when I referesh the web browser, I get a blank page
- error logs show this
[Sun Dec 29 03:30:00.336742 2024] [proxy_fcgi:error] [pid 447104:tid 447104] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Undefined constant "NS_IMAGE" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php:58\nStack trace:\n#0 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once()\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(143): require_once('...')\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(89): require_once('...')\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(44): require('...')\n#4 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 58; PHP message: PHP Fatal error: Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63\nStack trace:\n#0 [internal function]: MediaWiki\\HeaderCallback::callback()\n#1 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63'
- it's complaining about this line in particular
$wgNamespacesWithSubpages[NS_IMAGE] = true;
- but wait, I realized the upgrade guide says I'm supposed to run an upgrade command, which I never did; let's do that.
- oh fuck, well that just fails with the same error lol
root@hetzner3 /var/www/html/wiki.opensourceecology.org # php htdocs/maintenance/update.php
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5
PHP Fatal error: Uncaught Error: Undefined constant "NS_IMAGE" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php:58
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(143): require_once('...')
#2 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(91): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#4 {main}
thrown in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 58
PHP Fatal error: Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63
Stack trace:
#0 [internal function]: MediaWiki\HeaderCallback::callback()
#1 {main}
thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- here's the block that we have in our LocalSettings.php file
# Enable subpages in the main namespace $wgNamespacesWithSubpages[NS_MAIN] = true; $wgNamespacesWithSubpages[NS_TEMPLATE] = true; $wgNamespacesWithSubpages[NS_CATEGORY] = true; $wgNamespacesWithSubpages[NS_MEDIA] = true; $wgNamespacesWithSubpages[NS_IMAGE] = true;
- it looks like this is how we setup subpages in Mediawiki. I found documentation on this setting variable here https://www.mediawiki.org/wiki/Manual:$wgNamespacesWithSubpages#Enabling-for-a-namespace
- but the syntax is a bit different in those docs, it says to use something like this
$wgNamespacesWithSubpages = [ NS_TALK => true, NS_USER => true, NS_USER_TALK => true, NS_PROJECT => true, NS_PROJECT_TALK => true, NS_FILE_TALK => true, NS_MEDIAWIKI => true, NS_MEDIAWIKI_TALK => true, NS_TEMPLATE => true, NS_TEMPLATE_TALK => true, NS_HELP => true, NS_HELP_TALK => true, NS_CATEGORY_TALK => true ];
- so I changed our block to match this syntax
root@hetzner3 /var/www/html/wiki.opensourceecology.org # cp LocalSettings.php LocalSettings.20241228.php root@hetzner3 /var/www/html/wiki.opensourceecology.org # root@hetzner3 /var/www/html/wiki.opensourceecology.org # vim LocalSettings.php root@hetzner3 /var/www/html/wiki.opensourceecology.org # root@hetzner3 /var/www/html/wiki.opensourceecology.org # diff LocalSettings.20241228.php LocalSettings.php 54,58c54,60 < $wgNamespacesWithSubpages[NS_MAIN] = true; < $wgNamespacesWithSubpages[NS_TEMPLATE] = true; < $wgNamespacesWithSubpages[NS_CATEGORY] = true; < $wgNamespacesWithSubpages[NS_MEDIA] = true; < $wgNamespacesWithSubpages[NS_IMAGE] = true; --- > $wgNamespacesWithSubpages = [ > NS_MAIN => true, > NS_TEMPLATE => true, > NS_CATEGORY => true, > NS_MEDIA => true, > NS_IMAGE => true > ]; root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- unfortunately, I got the same error. I ended-up just deleting the line with 'NS_IMAGE'
- now I got a different error
root@hetzner3 /var/www/html/wiki.opensourceecology.org # php htdocs/maintenance/update.php
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5
PHP Warning: require_once(/var/www/html/wiki.opensourceecology.org/htdocs/extensions/CategoryTree/CategoryTree.php): Failed to open stream: No such file or directory in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 62
PHP Fatal error: Uncaught Error: Failed opening required '/var/www/html/wiki.opensourceecology.org/htdocs/extensions/CategoryTree/CategoryTree.php' (include_path='/var/www/html/wiki.opensourceecology.org/htdocs:/var/www/html/wiki.opensourceecology.org/htdocs/includes:/var/www/html/wiki.opensourceecology.org/htdocs/languages:/var/www/html/wiki.opensourceecology.org/htdocs/extensions/OpenID:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/console_getopt:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/mail:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/mail_mime:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/net_smtp:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/net_socket:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/pear-core-minimal/src:/var/www/html/wiki.opensourceecology.org/htdocs/vendor/pear/pear_exception:.:/usr/share/php') in /var/www/html/wiki.opensourceecology.org/LocalSettings.php:62
Stack trace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php(8): require_once()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(143): require_once('...')
#2 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(91): require_once('...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#4 {main}
thrown in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 62
PHP Fatal error: Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63
Stack trace:
#0 [internal function]: MediaWiki\HeaderCallback::callback()
#1 {main}
thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- I had to comment-out the line including the CategroyTree extension
- next error: I had to comment-out the line including the CologneBlue skin
- then the Modern skin, then the MonoBook skin, then the Vector skin
- then I had to comment-out the include for the ConfirmAccount extension, then the ConfirmEdit extension, then the Widgets extension
- while I'm at it, I sent an email to Marcin asking if there's any new extensions he'd want me to install, as that's important to know soon
Hey Marcin, Are there any MediaWiki extensions that you've wanted to add in the past years? I've started working on the upgrade process of the OSE wiki. Part of this includes downloading updates to your extensions, and installing new extensions that you may want. Please let me know if there's any new MediaWiki extensions that you want on hetzner3. Thank you, Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
- next error: I had to comment-out the include of the ParserFunctions extension, and the UserMerge extension
- oh, finally, something new; it says I'm missing some $wgServer var?
root@hetzner3 /var/www/html/wiki.opensourceecology.org # php htdocs/maintenance/update.php PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5 $wgServer must be set in LocalSettings.php. See <a href="https://www.mediawiki.org/wiki/Manual:$wgServer">https://www.mediawiki.org/wiki/Manual:$wgServer</a>.root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- looks like we just had it commented-out for some reason; I'll uncomment it
root@hetzner3 /var/www/html/wiki.opensourceecology.org # grep wgServer LocalSettings.php #$wgServer = "http://opensourceecology.org"; #$wgScriptPath = "$wgServer/w"; global $wgServer; $url = $wgServer . $url; root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- oh, it's doing something now! It flew by too fast for me to catch it all, but here's some snippets
root@hetzner3 /var/www/html/wiki.opensourceecology.org # php htdocs/maintenance/update.php
PHP Warning: Undefined array key "HTTP_USER_AGENT" in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5
MediaWiki 1.35.0 Updater
Your composer.lock file is up to date with current dependencies!
Going to run database updates for osewiki_db-wiki_
Depending on the size of your database this may take a while!
Abort with control-c in the next five seconds (skip this countdown with --quick) ... 0
...have ipb_id field in ipblocks table.
...have ipb_expiry field in ipblocks table.
...already have interwiki table
...indexes seem up to 20031107 standards.
...have rc_type field in recentchanges table.
...index new_name_timestamp already set on recentchanges table.
...have user_real_name field in user table.
...querycache table already exists.
...objectcache table already exists.
...categorylinks table already exists.
...have pagelinks; skipping old links table updates
...il_from OK
...have rc_ip field in recentchanges table.
...index PRIMARY already set on image table.
...have rc_id field in recentchanges table.
...have rc_patrolled field in recentchanges table.
...logging table already exists.
...have user_token field in user table.
...have wl_notificationtimestamp field in watchlist table.
...watchlist talk page rows already present.
...user table does not contain user_emailauthenticationtimestamp field.
...page table already exists.
...have log_params field in logging table.
...logging table has correct log_title encoding.
...have ar_rev_id field in archive table.
...have page_len field in page table.
...revision table does not contain inverse_timestamp field.
...have rev_text_id field in revision table.
...have rev_deleted field in revision table.
...have img_width field in image table.
...have img_metadata field in image table.
...have user_email_token field in user table.
...have ar_text_id field in archive table.
...page_namespace is already a full int (int(11)).
...ar_namespace is already a full int (int(11)).
...rc_namespace is already a full int (int(11)).
...
... 3223
Completed migration, updated 3183 row(s) with 725 new comment(s)
Beginning migration of ipblocks.ipb_reason to ipblocks.ipb_reason_id
... 118
... 282
... 438
... 507
Completed migration, updated 353 row(s) with 12 new comment(s)
Beginning migration of image.img_description to image.img_description_id
... 021SketchPortfolio_UJ_MartinBolton.JPG
... 04_precut2x6x9_1.png
... 080_FlexibleVinylflashing-10x52_5inch-withcut_3.png
...
... rev_id=302843
Completed migration, updated 299612 row(s) with 261 new actor(s), 0 error(s)
Beginning migration of archive.ar_user and archive.ar_user_text to archive.ar_actor
... ar_id=100
... ar_id=200
... ar_id=300
... ar_id=400
... ar_id=500
... ar_id=600
... ar_id=700
... ar_id=800
... ar_id=900
... ar_id=1000
... ar_id=1100
... ar_id=1200
... ar_id=1300
... ar_id=1400
... ar_id=1500
... ar_id=1600
... ar_id=1700
... ar_id=1800
... ar_id=1900
... ar_id=2001
... ar_id=2101
... ar_id=2201
... ar_id=2309
... ar_id=2426
... ar_id=2531
... ar_id=2634
... ar_id=2740
... ar_id=2840
... ar_id=2940
... ar_id=3040
... ar_id=3140
... ar_id=3223
Completed migration, updated 3183 row(s) with 3 new actor(s), 0 error(s)
Beginning migration of ipblocks.ipb_by and ipblocks.ipb_by_text to ipblocks.ipb_by_actor
... ipb_id=118
...
Modifying rev_text_id field of table revision ...done.
Modifying table site_stats ...done.
Populating ar_rev_id.
Populating ar_rev_id...
MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore
.php: Main slot of revision not found in database. See T212428.
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlo
tRecords()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1217): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1528): MessageCache->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1167): WANObjectCache->getWithSetCallback()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1163): BagOStuff->getWithSetCallback()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1106): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1016): MessageCache->getMsgFromNamespace()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(927): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(661): Message->text()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(213): Title::newMainPage()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(118): PopulateArchiveRevId::makeDummyRevisionRow()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/populateArchiveRevId.php(63): PopulateArchiveRevId::checkMysqlAutoIncrementBug()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/includes/LoggedUpdateMaintenance.php(45): PopulateArchiveRevId->doDBUpdates()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(1377): LoggedUpdateMaintenance->execute()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(512): DatabaseUpdater->populateArchiveRevId()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(475): DatabaseUpdater->runUpdates()
#29 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(181): DatabaseUpdater->doUpdates()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(107): UpdateMediaWiki->execute()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(253): require_once('...')
#32 {main}
You have mail in /var/mail/root
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- while that ran, I tried to look into the download URLs for the extensions (eventually needed for 3TOFU)
- it looks like some of these mediawiki extensions are quite – odd. They generate a temp extension zip at the time of download? That's terrible for supply chain security.
- I looked at ConfirmEdit; this is the official page https://www.mediawiki.org/wiki/Extension:ConfirmEdit
- if you click the "download extension" link, it takes you to this page which asks you which version of mediawiki you're using https://www.mediawiki.org/wiki/Special:ExtensionDistributor/ConfirmEdit
- I select v1.42 (our latest version), and click the "continue" button
- that brings me to this page, which says https://www.mediawiki.org/wiki/Special:ExtensionDistributor?extdistname=ConfirmEdit&extdistversion=REL1_42
A snapshot of version 6673c5d of the ConfirmEdit extension for MediaWiki REL1_42 has been created. Your download should start automatically in 5 seconds. The URL for this snapshot is: https://extdist.wmflabs.org/dist/extensions/ConfirmEdit-REL1_42-6673c5d.tar.gz You can use this link to download the extension on any computer, but please do not bookmark it, since its contents will not be updated, and it may be deleted at a later date.
- also, this is a bad example, because ConfirmEdit is one of those shipped with MediaWiki core now.
- from yesterday, here's the extensions we need
Renameuser UserMerge ConfirmAccount Widgets
- wait, shit, no, Renameuser is also part of core now https://www.mediawiki.org/wiki/Manual:Renameuser
- and so is CategoryTree https://www.mediawiki.org/wiki/Extension:CategoryTree
- and so is Parser Functions https://www.mediawiki.org/wiki/Extension:ParserFunctions
- alright, UserMerge is – in fact – an extension https://www.mediawiki.org/wiki/Extension:UserMerge
- same thing as before; it generates a snapshot from the git repo :( https://www.mediawiki.org/wiki/Special:ExtensionDistributor?extdistname=UserMerge&extdistversion=REL1_42
A snapshot of version 41759d0 of the UserMerge extension for MediaWiki REL1_42 has been created. Your download should start automatically in 5 seconds. The URL for this snapshot is: https://extdist.wmflabs.org/dist/extensions/UserMerge-REL1_42-41759d0.tar.gz You can use this link to download the extension on any computer, but please do not bookmark it, since its contents will not be updated, and it may be deleted at a later date.
- alright, well, here's the actual repo https://gerrit.wikimedia.org/g/mediawiki/extensions/UserMerge
- it shows the branches on the left; looks like 'REL1_42' is a branch for MediaWiki v1.42 https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/UserMerge/+/refs/heads/REL1_42
- the latest commit in ^ that branch is 41759d0c61377074d159f7d84130a095822bc7a3, the first 7 characters of which match the filename above '41759d0'
- here's the log for the REL1_42 branch; it shows it was last updated 5 weeks ago. looks like updates are infrequent enough that we might be able to use that tarball for a 3TOFU, even though they say it's ephemeral and shouldn't be bookmarked. I think it'll work https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/UserMerge/+log/refs/heads/REL1_42
- so here's the URLs to the extensions we need
- and here's some releases that we can 3TOFU
- I'm not sure if we should actually be putting these in-place for the intermittent upgrade from MediaWiki v1.30.0 to v1.35
- to be safe, here's the version of the extensions for v1.35
- just kidding, there is no option to download the extension for anything older than v1.39. Well, shit, I guess we'll just have to install the extensions after the second upgrade to v1.42 :shrug:
- ok, the update finally finished. I should have prepended it with `time`, but I think that took like half an hour
- after it ran, I tried loading the website; again I just get a blank white page
- the error logs complain about a call to ini_set; this reminds me of the wordpress bug. guess I have to open one for mediawiki too :/
==> wiki.opensourceecology.org/error.log <==
[Sun Dec 29 04:39:22.406293 2024] [proxy_fcgi:error] [pid 447105:tid 447105] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function ini_set() in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/APCUBagOStuff.php:61\nStack trace:\n#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/ObjectCache.php(308): APCUBagOStuff->__construct()\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php(187): ObjectCache::makeLocalServerCache()\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php(224): ExtensionRegistry->getCache()\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(161): ExtensionRegistry->loadFromQueue()\n#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(89): require_once('...')\n#5 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(44): require('...')\n#6 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/APCUBagOStuff.php on line 61; PHP message: PHP Fatal error: Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63\nStack trace:\n#0 [internal function]: MediaWiki\\HeaderCallback::callback()\n#1 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63'
- in the meantime, I'll ad the same workaround to LocalSettings.php as I added to wordpress
- curiously, this one is unique to MediaWiki
==> wiki.opensourceecology.org/error.log <==
[Sun Dec 29 04:41:57.197364 2024] [proxy_fcgi:error] [pid 447103:tid 447103] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function putenv() in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php:167\nStack trace:\n#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(89): require_once()\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(44): require('...')\n#2 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 167; PHP message: PHP Fatal error: Uncaught Error: Class "WebRequest" not found in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php:63\nStack trace:\n#0 [internal function]: MediaWiki\\HeaderCallback::callback()\n#1 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/HeaderCallback.php on line 63'
- also php_uname; damn, I'm disappointed in Mediawiki
==> wiki.opensourceecology.org/error.log <==
[Sun Dec 29 04:43:59.298957 2024] [proxy_fcgi:error] [pid 447104:tid 447104] [client 127.0.0.1:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function php_uname() in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php:1290\nStack trace:\n#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/db/MWLBFactory.php(104): wfHostname()\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/ServiceWiring.php(306): MWLBFactory::applyDefaultConfig()\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/vendor/wikimedia/services/src/ServiceContainer.php(445): Wikimedia\\Services\\ServiceContainer->{closure}()\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/vendor/wikimedia/services/src/ServiceContainer.php(416): Wikimedia\\Services\\ServiceContainer->createService()\n#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWikiServices.php(679): Wikimedia\\Services\\ServiceContainer->getService()\n#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/exception/MWExceptionHandler.php(135): MediaWiki\\MediaWikiServices->getDBLoadBalancerFactory()\n#...'
- now the LocalSettings.php starts with this
root@hetzner3 /var/www/html/wiki.opensourceecology.org # head -n20 LocalSettings.php
<?php
# fix mediawiki bugs
# * https://core.trac.wordpress.org/ticket/48693
if( ! function_exists('ini_set') ){
function ini_set(){
return;
}
}
if( ! function_exists('putenv') ){
function putenv(){
return;
}
}
if( ! function_exists('php_uname') ){
function php_uname(){
return;
}
}
# the .htaccess file that ships with mediawiki to prevent XSS attacks on IE6
root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- now when I load the site, I get this
<!DOCTYPE html>
<html><head><title>Internal error - Open Source Ecology</title><style>body { font-family: sans-serif; margin: 0; padding: 0.5em 2em; }</style></head><body>
<div class="errorbox mw-content-ltr">[Z3DUd0uDMO6w_6GcLRKPZgAAAAU] 2024-12-29 04:47:51: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"</div>
<!-- Set $wgShowExceptionDetails = true; at the bottom
of LocalSettings.php to show detailed debugging
information. --></body></html>
- I tried tailing wikierror.log, but I didn't get anything useful, so I enabled the exception output as the error above said
- next page load gave me this
[Z3DXAXouL7ks0Tm9gkHDXgAAAAQ] /?nocache=12 MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
Backtrace:
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords()
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1217): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1528): MessageCache->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1167): WANObjectCache->getWithSetCallback()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1163): BagOStuff->getWithSetCallback()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1106): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(1016): MessageCache->getMsgFromNamespace()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(927): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(661): Message->text()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(131): Title::newMainPage()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(151): MediaWiki->parseTitle()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(902): MediaWiki->getTitle()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(543): MediaWiki->main()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(53): MediaWiki->run()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(46): wfIndexMain()
#28 {main}
- the "See T212428." refers to this https://phabricator.wikimedia.org/T212428
- that says to run 'maintenance/populateContentTables.php', so I gave it a try
root@hetzner3 /var/www/html/wiki.opensourceecology.org # php htdocs/maintenance/populateContentTables.php ... ... archive processed up to revision id 302632 of 302632 (3178 rows in 2.0540862083435 seconds) Done populating archive table. Processed 3178 rows in 2.0541031360626 seconds Done. Processed 302790 rows in 16.495725154877 seconds root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- after that, I refreshed the page, and the wiki loads! WooHoo!
- there's a huge banner at the top complaining about missing skins, and the wiki looks a bit fucked-up (probably because of that), but it's loading!
Whoops! The default skin for your wiki, defined in $wgDefaultSkin as Vector, is not available. Your installation seems to include the following skins. See Manual: Skin configuration for information how to enable them and choose the default. monobook / MonoBook (disabled) timeless / Timeless (disabled) vector / Vector (disabled) If you have just installed MediaWiki You probably installed from git, or directly from the source code using some other method. This is expected. Try installing some skins from mediawiki.org's skin directory, by: Downloading the tarball installer, which comes with several skins and extensions. You can copy and paste the skins/ directory from it. Downloading individual skin tarballs from mediawiki.org. Using Git to download skins. Doing this should not interfere with your git repository if you're a MediaWiki developer. If you have just upgraded MediaWiki MediaWiki 1.24 and newer no longer automatically enables installed skins (see Manual: Skin autodiscovery). You can paste the following lines into LocalSettings.php to enable all installed skins: wfLoadSkin( 'MonoBook' ); wfLoadSkin( 'Timeless' ); wfLoadSkin( 'Vector' ); If you have just modified LocalSettings.php Double-check the skin names for typos.
- next I need to figure out the download URLs for the skins and put together a 3TOFU script for both the skins and the extensions
- and also attempt the upgrade from 1.35 to 1.42.
- oh, it looks like some of the skins are included by default; I think the issue is that we were doing a include() when we need to use the wfLoadSkin() function
root@hetzner3 /var/www/html/wiki.opensourceecology.org # ls -lah htdocs/skins total 24K d---r-x--- 5 not-apache www-data 4,0K Dec 26 23:53 . d---r-x--- 14 not-apache www-data 4,0K Dec 28 04:23 .. d---r-x--- 6 not-apache www-data 4,0K Dec 26 23:53 MonoBook ----r----- 1 not-apache www-data 1,3K Sep 24 2020 README d---r-x--- 6 not-apache www-data 4,0K Dec 26 23:53 Timeless d---r-x--- 10 not-apache www-data 4,0K Dec 26 23:53 Vector root@hetzner3 /var/www/html/wiki.opensourceecology.org #
- ok, I fixed the skin issue by adding this to the LocalSettings.php file
# SKINS # ######### ## Default skin: you can change the default skin. Use the internal symbolic ## names, ie 'vector', 'monobook': # $wgDefaultSkin = 'monobook'; $wgDefaultSkin = 'Vector'; wfLoadSkin( 'MonoBook' ); wfLoadSkin( 'Timeless' ); wfLoadSkin( 'Vector' );
- now the wiki frontpage looks sane, except that the GVCS section says this
{{#ifexist:Template:GVCS List/Main Page|/Main Page|}}|}}}
- there's some other things broken; let's do the second upgrade and then we can try to fix them one-by-one
Fri Dec 27, 2024
- Here's TOFU 2/3 (VPN, exit in Germany)
Germany 2024-12-27 ######################################################################### 100.0% ######################################################################### 100.0% ######################################################################### 100.0% ######################################################################### 100.0% 2024-12-27 3c457f4960cfcbdf5e1e2fab6b6e3ade006e65a55cd93459e0ecede3d9c23a16 keys.txt 56e0a7fe2ff78e50b9c87b3d589f2d8df5bcab0018e454b2a4a239c8586bd8ee mediawiki-1.43.0.tar.gz cee3d020f25e9b3f66eef214a343be466a1a313db31344ad2a5e87ab0183883e mediawiki-1.43.0.tar.gz.sig 9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a phplist-3.6.14.zip user@disp4681:/tmp/tmp.5fvnR1Pc6h$ user@disp4681:/tmp/tmp.5fvnR1Pc6h$ gpg --with-fingerprint --with-subkey-fingerprint --keyid-format 0xlong keys.txt gpg: WARNING: no command supplied. Trying to guess what you mean ... pub rsa4096/0x73F146FECF9D333C 2014-11-20 [SC] [expired: 2021-06-05] Key fingerprint = F64E BF5F 2099 6AB5 14F1 98A8 73F1 46FE CF9D 333C uid Tim Starling <tstarling@wikimedia.org> sub rsa4096/0x1075249FCCC9CAAF 2014-11-20 [E] [expired: 2021-06-05] pub dsa1024/0xC119E1A64D70938E 2003-11-15 [SCA] Key fingerprint = 4412 76E9 CCD1 5F44 F6D9 7D18 C119 E1A6 4D70 938E uid Brion Vibber <brion@pobox.com> sub elg1024/0x6596FAD2965B3548 2003-11-15 [E] pub dsa1024/0x9B69B3109D3BB7B0 2011-10-24 [SC] Key fingerprint = 1D98 867E 8298 2C8F E0AB C25F 9B69 B310 9D3B B7B0 uid Sam Reed <reedy@wikimedia.org> sub elg2048/0x3BBB95CE2B08BFD2 2011-10-24 [E] pub rsa2048/0x72BC1C5D23107F8A 2014-04-29 [SC] [expires: 2026-04-29] Key fingerprint = 41B2 ABE8 17AD D3E5 2BDA 946F 72BC 1C5D 2310 7F8A uid Chad Horohoe <chad@wikimedia.org> uid keybase.io/demon <demon@keybase.io> sub rsa2048/0x08CF4E7951361C13 2014-04-29 [E] [expires: 2026-04-29] pub rsa4096/0xF6DAD285018FAC02 2014-02-19 [SC] [expired: 2018-10-04] Key fingerprint = 6237 D8D3 ECC1 AE91 8729 296F F6DA D285 018F AC02 uid Tyler Cipriani <tcipriani@wikimedia.org> uid Tyler Cipriani <tyler@tylercipriani.com> uid [jpeg image of size 5098] sub rsa4096/0xB002E1FDEE737D83 2014-02-19 [E] [expired: 2018-10-04] pub rsa3072/0x26752EBB0D9E6218 2021-11-11 [SC] Key fingerprint = 72D2 86F6 F8F0 3C78 F2C5 9C73 2675 2EBB 0D9E 6218 uid Amir Sarabadani <asarabadani@wikimedia.org> sub rsa3072/0x4F889038CE86B378 2021-11-11 [E] pub rsa4096/0x361F943B15C08DD4 2015-05-22 [SC] [expired: 2020-05-20] Key fingerprint = 80D1 13B7 67E3 D519 3672 5679 361F 943B 15C0 8DD4 uid Brian Wolff <bwolff@wikimedia.org> uid Brian Wolff (Bawolff) <bawolff@gmail.com> sub rsa4096/0xBF1629CD074D3DD8 2015-05-22 [E] [expired: 2020-05-20] pub rsa4096/0x131910E01605D9AA 2016-01-08 [SC] [expired: 2020-07-31] Key fingerprint = C83A 8E4D 3C8F EB7C 8A3A 1998 1319 10E0 1605 D9AA uid Mukunda Modell <twentyafterfour@gmail.com> uid Mukunda Modell (WMF) <mmodell@wikimedia.org> uid [jpeg image of size 2928] sub rsa4096/0x5411F23A0C4E5EC1 2018-12-25 [A] [expired: 2020-12-24] sub rsa4096/0x02C99BB8AB1C6DD5 2018-12-25 [E] [expired: 2020-12-24] sub rsa4096/0x60AE06D4875BE862 2018-12-26 [S] [expired: 2019-12-26] pub rsa4096/0xA8F734246D73B586 2024-12-09 [SC] [expires: 2026-12-09] Key fingerprint = E059 C034 E7A4 3058 3C25 2F4A A8F7 3424 6D73 B586 uid atieno njira <pnjira@wikimedia.org> sub rsa4096/0xB65D063E68F2FC8A 2024-12-09 [E] [expires: 2026-12-09] user@disp4681:/tmp/tmp.5fvnR1Pc6h$
- ...
- I began the process to copy the wiki from hetzner2 to hetzner3
- man, this is a big site; it took over 10 minutes just to create a gzip'd tarball of the site's files
[root@opensourceecology current]# time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}
...
/var/www/html/wiki.opensourceecology.org/cache/l10n_cache-it.cdb
/var/www/html/wiki.opensourceecology.org/cache/l10n_cache-uk.cdb
real 11m37.448s
user 10m48.352s
sys 0m40.263s
[root@opensourceecology current]#
[root@opensourceecology current]# du -sh ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2}
15G /var/tmp/backups_for_migration_to_hetzner2/wiki.opensourceecology.org_20241228/current/wiki.opensourceecology.org_files.20241228.tar.gz
[root@opensourceecology current]#
- curiously the mysqldump fails, something about LOCK TABLES
[root@opensourceecology current]# time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
mysqldump: Got error: 1044: "Access denied for user 'osewiki_user'@'localhost' to database 'osewiki_db'" when using LOCK TABLES
real 0m0.007s
user 0m0.001s
sys 0m0.005s
[root@opensourceecology current]#
- stack exchange says I can bypass this by using the `--single-transaction` flag, but that appears to mean that the dump could be corrupted. The lock is needed to ensure a consistent state https://stackoverflow.com/questions/70376243/mysqldump-got-error-1044-access-denied-for-user-usernamelocalhost-to-dat
· --single-transaction This option sends a START TRANSACTION SQL statement to the server before dumping data. It is useful only with transactional tables such as InnoDB, because then it dumps the consistent state of the database at the time when BEGIN was issued without blocking any applications. When using this option, you should keep in mind that only InnoDB tables are dumped in a consistent state. For example, any MyISAM or MEMORY tables dumped while using this option may still change state. While a --single-transaction dump is in process, to ensure a valid dump file (correct table contents and binary log coordinates), no other connection should use the following statements: ALTER TABLE, CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE. A consistent read is not isolated from those statements, so use of them on a table to be dumped can cause the SELECT that is performed by mysqldump to retrieve the table contents to obtain incorrect contents or fail. The --single-transaction option and the --lock-tables option are mutually exclusive because LOCK TABLES causes any pending transactions to be committed implicitly. This option is not supported for MySQL Cluster tables; the results cannot be guaranteed to be consistent due to the fact that the NDBCLUSTER storage engine supports only the READ_COMMITTED transaction isolation level. You should always use NDB backup and restore instead. To dump large tables, you should combine the --single-transaction option with --quick.
- another site said I could GRANT the LOCK permission to all the db tables to this user to fix this issue https://michaelrigart.be/mysqldump-1044-access-denied-using-lock-tables/
- I tried this (executed in the mysql shell as root)
GRANT SELECT,LOCK TABLES ON osewiki_db.* TO 'osewiki_user'@'localhost';
- well that worked; I wonder if the only reason it didn't throw the same error for the wordpress sites is because the wordpress sites are stupid small (and inactive) by comparison
[root@opensourceecology current]# time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
real 6m53.476s
user 7m4.351s
sys 0m2.092s
[root@opensourceecology current]#
- the upgrade guide on mediawiki actually doesn't say we should copy the files from the latest release to override the old release. Instead it says we should start from the files in the new release, and then just copy a few files/dirs from the old install into it
- https://www.mediawiki.org/wiki/Manual:Upgrading
- LocalSettings.php
- htdocs/LocalSettings.php (since we have a custom one because we want ours outside the docroot)
- htdocs/images/
- htdocs/extensions/
- htdocs/skins/
- I'm going to skip the extensions for now; after the second upgrade, I'll re-visit installing the updated versions of these extensions.
- for now let's just see if I can even do the first upgrade; here's the commands to place the vhost files in-place
# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mkdir -p ${vhostDir}
rsync -av --progress /var/tmp/mediawiki/mediawiki-1.35.0/ ${docrootDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/LocalSettings.php ${vhostDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php ${docrootDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/htdocs/images ${docrootDir}/
- while I'm waiting, I looked into the extensions; here's what we have installed on hetzner2
[root@opensourceecology wiki.opensourceecology.org]# ls -lah htdocs/extensions/ total 104K d---r-x--- 25 not-apache apache 4.0K May 24 2018 . d---r-x--- 17 not-apache apache 4.0K Jul 28 2019 .. d---r-x--- 5 not-apache apache 4.0K May 24 2018 CategoryTree d---r-x--- 6 not-apache apache 4.0K Dec 8 2017 Cite d---r-x--- 4 not-apache apache 4.0K Dec 8 2017 CiteThisPage d---r-x--- 7 not-apache apache 4.0K May 24 2018 ConfirmAccount d---r-x--- 13 not-apache apache 4.0K Dec 8 2017 ConfirmEdit d---r-x--- 6 not-apache apache 4.0K Dec 8 2017 Gadgets d---r-x--- 3 not-apache apache 4.0K Dec 8 2017 ImageMap d---r-x--- 5 not-apache apache 4.0K Dec 8 2017 InputBox d---r-x--- 3 not-apache apache 4.0K Dec 8 2017 Interwiki d---r-x--- 7 not-apache apache 4.0K Dec 8 2017 LocalisationUpdate d---r-x--- 4 not-apache apache 4.0K Dec 8 2017 Nuke d---r-x--- 12 not-apache apache 4.0K May 24 2018 OATHAuth d---r-x--- 4 not-apache apache 4.0K Dec 8 2017 ParserFunctions d---r-x--- 4 not-apache apache 4.0K Dec 8 2017 PdfHandler d---r-x--- 3 not-apache apache 4.0K Dec 8 2017 Poem ----r----- 1 not-apache apache 1.1K Dec 8 2017 README d---r-x--- 5 not-apache apache 4.0K Dec 8 2017 Renameuser d---r-x--- 4 not-apache apache 4.0K May 24 2018 ReplaceText d---r-x--- 6 not-apache apache 4.0K Dec 8 2017 SpamBlacklist d---r-x--- 7 not-apache apache 4.0K Dec 8 2017 SyntaxHighlight_GeSHi d---r-x--- 6 not-apache apache 4.0K Dec 8 2017 TitleBlacklist d---r-x--- 5 not-apache apache 4.0K May 24 2018 UserMerge d---r-x--- 7 not-apache apache 4.0K May 24 2018 Widgets d---r-x--- 5 not-apache apache 4.0K Dec 8 2017 WikiEditor [root@opensourceecology wiki.opensourceecology.org]#
- it looks like we might not be using all of these
[root@opensourceecology wiki.opensourceecology.org]# grep wfLoadExtension LocalSettings.php
wfLoadExtensions([ 'ConfirmEdit', 'ConfirmEdit/ReCaptcha' ]);
wfLoadExtension( 'Cite' );
wfLoadExtension( 'Interwiki' );
wfLoadExtension( 'Gadgets' );
wfLoadExtension( 'ReplaceText' );
wfLoadExtension( 'Renameuser' );
wfLoadExtension( 'UserMerge' );
wfLoadExtension( 'Nuke' );
wfLoadExtension( 'OATHAuth' );
[root@opensourceecology wiki.opensourceecology.org]# grep wfLoadExtension LocalSetti^Chp
[root@opensourceecology wiki.opensourceecology.org]# grep require LocalSettings.php
require_once( "$IP/includes/DefaultSettings.php" );
require_once("{$IP}/extensions/CategoryTree/CategoryTree.php");
require_once "$IP/skins/CologneBlue/CologneBlue.php";
require_once "$IP/skins/Modern/Modern.php";
require_once "$IP/skins/MonoBook/MonoBook.php";
require_once "$IP/skins/Vector/Vector.php";
#require_once( "$IP/extensions/CCAgreement/CCAgreement.php" );
# This extension and directory requires an admin to confirm a user before their account is created
require_once "$IP/extensions/ConfirmAccount/ConfirmAccount.php";
require_once("$IP/extensions/Widgets/Widgets.php");
require_once( "$IP/extensions/ParserFunctions/ParserFunctions.php" );
[root@opensourceecology wiki.opensourceecology.org]#
- also, apparently mediawiki cores ships with a bunch of these (that we don't have to worry about installing)
root@hetzner3 ~ # ls -lah /var/tmp/mediawiki/mediawiki-1.35.0/extensions/ total 124K drwxr-xr-x 30 root root 4,0K Dec 26 23:53 . drwxr-xr-x 14 root root 4,0K Dec 26 23:53 .. drwxr-xr-x 5 root root 4,0K Dec 26 23:53 CategoryTree drwxr-xr-x 6 root root 4,0K Dec 26 23:53 Cite drwxr-xr-x 5 root root 4,0K Dec 26 23:53 CiteThisPage drwxr-xr-x 6 root root 4,0K Dec 26 23:53 CodeEditor drwxr-xr-x 13 root root 4,0K Dec 26 23:53 ConfirmEdit drwxr-xr-x 5 root root 4,0K Dec 26 23:53 Gadgets drwxr-xr-x 6 root root 4,0K Dec 26 23:53 ImageMap drwxr-xr-x 6 root root 4,0K Dec 26 23:53 InputBox drwxr-xr-x 4 root root 4,0K Dec 26 23:53 Interwiki drwxr-xr-x 5 root root 4,0K Dec 26 23:53 LocalisationUpdate drwxr-xr-x 6 root root 4,0K Dec 26 23:53 MultimediaViewer drwxr-xr-x 5 root root 4,0K Dec 26 23:53 Nuke drwxr-xr-x 8 root root 4,0K Dec 26 23:53 OATHAuth drwxr-xr-x 6 root root 4,0K Dec 26 23:53 PageImages drwxr-xr-x 5 root root 4,0K Dec 26 23:53 ParserFunctions drwxr-xr-x 4 root root 4,0K Dec 26 23:53 PdfHandler drwxr-xr-x 5 root root 4,0K Dec 26 23:53 Poem -rw-rw-r-- 1 maltfield maltfield 1,1K Sep 11 2020 README drwxr-xr-x 5 root root 4,0K Dec 26 23:53 Renameuser drwxr-xr-x 6 root root 4,0K Dec 26 23:53 ReplaceText drwxr-xr-x 6 root root 4,0K Dec 26 23:53 Scribunto drwxr-xr-x 6 root root 4,0K Dec 26 23:53 SecureLinkFixer drwxr-xr-x 7 root root 4,0K Dec 26 23:53 SpamBlacklist drwxr-xr-x 8 root root 4,0K Dec 26 23:53 SyntaxHighlight_GeSHi drwxr-xr-x 7 root root 4,0K Dec 26 23:53 TemplateData drwxr-xr-x 5 root root 4,0K Dec 26 23:53 TextExtracts drwxr-xr-x 6 root root 4,0K Dec 26 23:53 TitleBlacklist drwxr-xr-x 11 root root 4,0K Dec 26 23:53 VisualEditor drwxr-xr-x 6 root root 4,0K Dec 26 23:53 WikiEditor root@hetzner3 ~ #
- as far as I can tell, here's the extensions we're using on hetzner2
Cite Interwiki Gadgets ReplaceText Renameuser UserMerge Nuke OATHAuth CategoryTree ConfirmAccount Widgets ParserFunctions
- and if we remove the extensions that already come with core, here's what we need to find the latest versions for
Renameuser UserMerge ConfirmAccount Widgets
- It looks like Renameuser became merged into core in 1.40, so we can scratch that one off https://www.mediawiki.org/wiki/Manual:Renameuser
- UserMerge is still an extension, and it's actively developed https://www.mediawiki.org/wiki/Extension:UserMerge
- ConfirmAccount is still an extension, and it's actively developed https://www.mediawiki.org/wiki/Extension:ConfirmAccount
- Widgets is still an extension, and it's actively developed https://www.mediawiki.org/wiki/Extension:Widgets
- I had some typos in my previous plan's permissions setting at the end; here's the updated one
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='wiki.opensourceecology.org'
dbName='osewiki_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner2/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}
# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
####################
# run on hetzner3 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='wiki.opensourceecology.org'
dbName='osewiki_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
# STEP 1: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"
# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mkdir -p ${vhostDir}
rsync -av --progress /var/tmp/mediawiki/mediawiki-1.35.0/ ${docrootDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/LocalSettings.php ${vhostDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/htdocs/LocalSettings.php ${docrootDir}/
rsync -av --progress var/www/html/wiki.opensourceecology.org/htdocs/images ${docrootDir}/
# UPDATE OLD EXTENSIONS
# TODO
# INSTALLL NEW EXTENSIONS
# TODO
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
#############
# MediaWiki #
#############
vhost_dir="/var/www/html/wiki.opensourceecology.org"
mw_docroot="${vhost_dir}/htdocs"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/LocalSettings.php"
chmod 0040 "${vhost_dir}/LocalSettings.php"
[ -d "${mw_docroot}/images" ] || mkdir "${mw_docroot}/images"
chown -R www-data:www-data "${mw_docroot}/images"
find "${mw_docroot}/images" -type f -exec chmod 0660 {} \;
find "${mw_docroot}/images" -type d -exec chmod 0770 {} \;
[ -d "${vhost_dir}/cache" ] || mkdir "${vhost_dir}/cache"
chown -R www-data:www-data "${vhost_dir}/cache"
find "${vhost_dir}/cache" -type f -exec chmod 0660 {} \;
find "${vhost_dir}/cache" -type d -exec chmod 0770 {} \;
- I pushed the wiki vhost out with ansible, but apache failed to restart
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # systemctl restart apache2 varnish nginx Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details. root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
- looks like I'm missing a file
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current # journalctl --no-pager -u apache2 ... Dec 28 04:58:06 hetzner3 systemd[1]: Starting apache2.service - The Apache HTTP Server... Dec 28 04:58:06 hetzner3 apachectl[43030]: apache2: Syntax error on line 232 of /etc/apache2/apache2.conf: Syntax error on line 22 of /etc/apache2/sites-enabled/wiki.opensourceecology.org.conf: Could not open configuration file /etc/apache2/conf-available/wiki.virtualhost.include: No such file or directory Dec 28 04:58:06 hetzner3 apachectl[43019]: Action 'start' failed. Dec 28 04:58:06 hetzner3 apachectl[43019]: The Apache error log may have more information. Dec 28 04:58:06 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 28 04:58:06 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 28 04:58:06 hetzner3 systemd[1]: Failed to start apache2.service - The Apache HTTP Server. root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/wiki.opensourceecology.org_20241228/current #
- I had to spend some time fixing some issues in the wiki config in ansible https://github.com/OpenSourceEcology/ansible/commit/d070595d4284874d49b735292fc7c9f1dd83a085
- when I finally got apache and nginx configs resolved, loading the new wiki site on hetzner3 gives this error
MediaWiki 1.35 internal error Installing some PHP extensions is required. Required components You are missing a required extension to PHP that MediaWiki requires to run. Please install: mbstring (more information)
- I should be able to install that from apt, and then I can proceed with upgrading the wiki
Thr Dec 26, 2024
- I got a genetic human-generated message back from oshine support, but they didn't answer my question. I replied asking again for the slugs to the required plugins
- I could either sift through their sourcecode to find-out where it downloads its dependency plugins, or wait. In the interest of time, I'm going to wait a bit longer
- ...
- let's proceed with the next wordpress site that's *not* using the oshine theme
- here's the active themes on hetzner2
[root@opensourceecology ~]# wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
[root@opensourceecology ~]# for wordpress_site in $wordpress_sites; do wp_docroot="$(dirname "${wordpress_site}")"; echo $wp_docroot; sudo -u wp -i wp --path="${wp_docroot}" theme list 2>/dev/null; done | grep -Ei '^/var/| active '
/var/www/html/www.openbuildinginstitute.org/htdocs
| oshin | active | available | 4.3.1 |
/var/www/html/oswh.opensourceecology.org/htdocs
| Eventor | active | none | 1.7 |
/var/www/html/d3d.opensourceecology.org/htdocs
/var/www/html/microfactory.opensourceecology.org/htdocs
| oshin | active | none | 6.5 |
/var/www/html/seedhome.openbuildinginstitute.org/htdocs
| twentyseventeen | active | none | 1.4 |
/var/www/html/staging.openbuildinginstitute.org/htdocs
| oshin | active | none | 4.3.1 |
/var/www/html/store.opensourceecology.org/htdocs
| oshin | active | none | 6.6.4.4 |
/var/www/html/fef.opensourceecology.org/htdocs
| simplephotoRes | active | none | 2.0 |
/var/www/html/www.opensourceecology.org/htdocs
| enigmatic | active | none | 3.5 |
/var/www/html/staging.opensourceecology.org/htdocs
| enigmatic | active | none | 3.5 |
/var/www/html/3dp.opensourceecology.org/htdocs
[root@opensourceecology ~]#
- d3d and 3dp didn't have output because they're not a real sites (it was never setup and we're not migrating it; we used microfactory.opensourceecology.org as the vhost instead of d3d/3dp)
- the other sites we have that are *not* currently using oshine are:
- oswh.opensourceecology.org
- seedhome.openbuildinginstitute.org
- there's a couple staging sites, but I don't think I'll migrate these; better to use the staging server
- fef.opensourceecology.org
- www.opensourceecology.org, but this one we're migrating to oshine, so it doesn't count
- again, here's the order of our migrations
1. forum.opensourceecology.org 2. store.opensourceecology.org 3. microfactory.opensourceecology.org 4. fef.opensourceecology.org 5. oswh.opensourceecology.org 6. seedhome.openbuildinginstitute.org 7. www.openbuildinginstitute.org 8. www.opensourceecology.org 9. phplist.opensourceecology.org 10. wiki.opensourceecology.org
- therefore, let's first do fef, then oswh, then seedhome.
- here's the script to migrate fef from hetzner2 to hetzner3
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='fef.opensourceecology.org'
dbName='fef_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner2/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}
# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
####################
# run on hetzner3 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='fef.opensourceecology.org'
dbName='fef_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
# STEP 1: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"
# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mv var/www/html/${vhost_name} ${vhostDir}
# remove '.svn' dirs (we no longer use svn, for security)
find ${docrootDir} -iname '.svn' -exec rm -rf '{}' \;
# add wordpress bug fix
# is the bug fix already present?
if ! $(grep 'https://core.trac.wordpress.org/ticket/48693' ${vhostDir}/wp-config.php) ; then
# the bug fix is absent; add it
backup_filename="wp-config.`date "+%Y%m%d_%H%M%S"`.php"
mv ${vhostDir}/wp-config.php ${vhostDir}/${backup_filename}
cat > ${vhostDir}/wp-config.php <<'EOF'
<?php
# fix wordpress bugs
# * https://core.trac.wordpress.org/ticket/48693
# * https://core.trac.wordpress.org/ticket/62693
if( ! function_exists('ini_set') ){
function ini_set(){
return;
}
}
if( ! function_exists('chmod') ){
function chmod(){
return;
}
}
EOF
tail -n +2 ${vhostDir}/${backup_filename} >> ${vhostDir}/wp-config.php
fi
# verify
ls
vim ${vhostDir}/wp-config.php
# UPDATE CORE
rsync -av --progress /var/tmp/wordpress/core/wordpress/ ${docrootDir}
# UPDATE OLD PLUGINS
for plugin_path in $(find "${docrootDir}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# INSTALLL NEW PLUGINS
new_plugins="activitypub aurora-heatmap melapress-login-security wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"
for plugin in ${new_plugins}; do
plugin_path="${docrootDir}/wp-content/plugins/${plugin}"
source_path="/var/tmp/wordpress/plugins/${plugin}"
if [ -d "${source_path}" ]; then
echo "${plugin}"
rm -rf ${plugin_path};
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# UPDATE/INSTALL THEMES
for theme_path in $(find "${docrootDir}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
# ACTIVATE NEW PLUGINS
activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
- as with yesterday, I then uncommented the fef host in the ansible provision.yml file and re-ran ansible-playbook
- then on the server, I restarted the web services
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current # systemctl restart apache2 varnish nginx root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current #
- then I updated my local comp's /etc/hosts
- I loaded the site, updated the db with the WUI
- I'm able to load the admin dashboard, but when I try to load the frontpage of the site on hetzner3, I just get an error box that says
The theme directory "simplephotoRes" does not exist.
- in the admin dashboard, if I click "Appearance" -> "Themes", I get an error at the top
The active theme is broken. Reverting to the default theme.
- here's the themes dir on hetzner2
[root@opensourceecology current]# ls -lah /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/ total 72K d---r-x--- 17 not-apache apache 4.0K Jan 3 2018 . d---r-x--- 6 not-apache apache 4.0K Oct 3 2018 .. d---r-x--- 4 not-apache apache 4.0K Aug 16 2015 ArtWorksResponsive d---r-x--- 8 not-apache apache 4.0K Aug 16 2015 gk-portfolio d---r-x--- 11 not-apache apache 4.0K Aug 16 2015 gridsby d---r-x--- 4 not-apache apache 4.0K Aug 16 2015 gridthemeresponsive ----r----- 1 not-apache apache 28 Jun 5 2014 index.php d---r-x--- 9 not-apache apache 4.0K Aug 16 2015 portfolio-press d---r-x--- 4 not-apache apache 4.0K Aug 16 2015 simplephotoRes d---r-x--- 7 not-apache apache 4.0K Aug 16 2015 sketch d---r-x--- 7 not-apache apache 4.0K Jan 3 2018 twentyeleven d---r-x--- 7 not-apache apache 4.0K May 7 2015 twentyfifteen d---r-x--- 9 not-apache apache 4.0K May 7 2015 twentyfourteen d---r-x--- 5 not-apache apache 4.0K Jan 3 2018 twentyseventeen d---r-x--- 7 not-apache apache 4.0K Jan 3 2018 twentysixteen d---r-x--- 4 not-apache apache 4.0K Jan 3 2018 twentyten d---r-x--- 8 not-apache apache 4.0K May 7 2015 twentythirteen d---r-x--- 6 not-apache apache 4.0K Jan 3 2018 twentytwelve [root@opensourceecology current]#
- and on hetzner3
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current # ls -lah /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/ total 56K d---r-x--- 13 not-apache www-data 4,0K Dec 26 18:05 . d---r-x--- 7 not-apache www-data 4,0K Dec 26 18:10 .. d---r-x--- 8 not-apache www-data 4,0K May 27 2016 gk-portfolio ----r----- 1 not-apache www-data 28 Jun 5 2014 index.php d---r-x--- 9 not-apache www-data 4,0K Dec 26 2018 portfolio-press d---r-x--- 7 not-apache www-data 4,0K Mar 5 2018 sketch d---r-x--- 7 not-apache www-data 4,0K Jul 16 13:09 twentyeleven d---r-x--- 7 not-apache www-data 4,0K Jul 16 13:28 twentyfifteen d---r-x--- 9 not-apache www-data 4,0K Jul 16 13:23 twentyfourteen d---r-x--- 5 not-apache www-data 4,0K Jul 16 13:29 twentyseventeen d---r-x--- 8 not-apache www-data 4,0K Jul 16 13:29 twentysixteen d---r-x--- 4 not-apache www-data 4,0K Jul 15 17:17 twentyten d---r-x--- 8 not-apache www-data 4,0K Jul 16 13:20 twentythirteen d---r-x--- 8 not-apache www-data 4,0K Jul 16 13:17 twentytwelve root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current #
- so, yeah, that 'simplephotoRes' dir is missing
- worse, I get no results searching for this theme on wordpress.org https://wordpress.org/themes/search/simplephotoRes/
- looks like my notes from Sep show that I was unable to download this theme https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Thr_Sep_26.2C_2024
... portfolio-press https://downloads.wordpress.org/theme/portfolio-press.2.8.0.zip simplephotoRes null Invalid slug provided null sketch https://downloads.wordpress.org/theme/sketch.1.2.4.zip ...
- 0 results on ddg https://duckduckgo.com/?q=%22simplephotoRes%22+%22wordpress%22&t=ftsa&ia=web
- if I login to the old fef site's wp admin dashboard, and go to "Appearance" -> "Themes", it lists the plugin's details:
- Human-readable plugin name = "Simple Photo Responsive"
- Version = 2.0
- Author = Marios Lublinski
- Author website = www.dessign.net
- I went to that website, but it looks like a scam. Lots of AI crap. Maybe the author sold it to some AI scam landing page?
- I checked the style.css file directly on hetzner2, which includes a URL for the theme
[maltfield@opensourceecology simplephotoRes]$ head style.css /* Theme Name: Simple Photo Responsive Theme URI: http://www.dessign.net/simplephototheme/ Description: Simple Photo Theme for WordPress is stylish, customizable, simple, and readable. Perfect for any illustrator, designer and blogger. Version: 2.0 [maltfield@opensourceecology simplephotoRes]$
- unfortunately, that URL simply redirects to some totally unrelated page about a podcast about AI (scam?)
user@disp1974:~$ curl -i http://www.dessign.net/simplephototheme/ HTTP/1.1 301 Moved Permanently Server: nginx Date: Thu, 26 Dec 2024 18:41:17 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://www.dessign.net/simplephototheme/ X-ac: 2.hhn _atomic_ams BYPASS Alt-Svc: h3=":443"; ma=86400 <html> <head><title>301 Moved Permanently</title></head> <body> <center><h1>301 Moved Permanently</h1></center> <hr><center>nginx</center> </body> </html> user@disp1974:~$
- well, I deleted the theme in the above steps because we didn't have an up-to-date theme to replace it.
- the risk is high that a theme that hasn't been updated could just break due to incompatibility with wordpress core, other plugins, or php (we saw this with oshine on our other sites iirc)
- of course, there's an even higher risk that there's an unpatched security vulnerability with running an old theme that's not updated
- I want to know how old this thing is. I searched the whole repo for dates. Looks like it includes a version of jquery that was published in Mar 2012, so this theme is probably over 12 years without updates D:
[root@opensourceecology simplephotoRes]# grep -irE '[0-9]{4}'
Binary file images/logo.png matches
Binary file images/slide-prev.png matches
Binary file images/slide-next.png matches
Binary file images/bg-img.jpg matches
Binary file images/menu-divider.jpg matches
Binary file images/google-plus-icon.png matches
Binary file images/dribbble-icon.png matches
Binary file images/search-icon.jpg matches
Binary file images/twitter-icon.png matches
Binary file images/facebook-icon.png matches
Binary file images/pinterest-icon.png matches
footer.php: © 2012 Simple Photo WordPress. Design by <a href="http://www.dessign.net">Dessign</a></div>
header.php:<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml">
header.php: start_custom_slider('5000');
settings.php: <td><input type="text" name="custom_background_color" class="ss_text" value="<?php echo stripslashes(stripslashes(get_option($shortname.'_custom_background_color',''))); ?>" /> <small>e.g.: #27292a</small></td>
settings.php: <td><input type="text" name="slideshow_timeout" class="ss_text" value="<?php echo stripslashes(stripslashes(get_option($shortname.'_slideshow_timeout',''))); ?>" /> <small>e.g.: 7000</small></td>
style.css:#commentform input[type=submit] { background-color: #161616; color: #fff; border: 1px solid #6E6E6E; padding: 3px 5px; }
js/_notes/dwsync.xml:<file name="jquery-latest.js" server="dessign.net//httpdocs/" local="129896034000000000" remote="129958450800000000" />
js/_notes/dwsync.xml:<file name="scripts.js" server="dessign.net//httpdocs/" local="129900749400000000" remote="129958450800000000" />
js/jquery-latest.js: * Copyright 2011, John Resig
js/jquery-latest.js: * Copyright 2011, The Dojo Foundation
js/jquery-latest.js: * Date: Wed Mar 21 12:46:34 2012 -0700
js/jquery-latest.js: // Prioritize #id over <tag> to avoid XSS via location.hash (#9521)
...
- anyway, let's see if it's at all possible to use this theme on the latest version of wp and php. I'll copy it in manually from the files from hetzer2
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current # tar -xzvf ${backupFileName_files_hetzner2}
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current #
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current # rsync -av --progress var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/simplephotoRes /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/
...
sent 442.547 bytes received 557 bytes 886.208,00 bytes/sec
total size is 440.589 speedup is 0,99
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/fef.opensourceecology.org_20241226/current #
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
- wordpress already deactivated the missing theme, so I had to go-in and reactivate it. As soon as I clicked "Activate", I got a critical error
There has been a critical error on this website. Please check your site admin email inbox for instructions.
- I literally can't even load anything on the site anymore; the whole wp admin dashboard area is broken
- if I tail the logs, then I see why
==> fef.opensourceecology.org/error.log <==
[Thu Dec 26 19:04:48.907704 2024] [proxy_fcgi:error] [pid 1904236:tid 1904282] [client 185.213.155.175:0] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught ArgumentCountError: Too few arguments to function WP_Widget::construct(), 0 passed in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/class-wp-widget-factory.php on line 62 and at least 2 expected in /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/class-wp-widget.php:163\nStack trace:\n#0 /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/class-wp-widget-factory.php(62): WP_Widget->construct()\n#1 /var/www/html/fef.opensourceecology.org/htdocs/wp-includes/widgets.php(123): WP_Widget_Factory->register()\n#2 /var/www/html/fef.opensourceecology.org/htdocs/wp-content/themes/simplephotoRes/functions.php(227): register_widget()\n#3 /var/www/html/fef.opensourceecology.org/htdocs/wp-settings.php(668): include('...')\n#4 /var/www/html/fef.opensourceecology.org/wp-config.php(112): require_once('...')\n#5 /var/www/html/fef.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\n#6 /var/www/html/fef.opensourceecology.org/htdocs/wp-ad...'
- yeah, I'd say this theme is totally not going to work with the latest version of wordpress.
- I sent an email to Marcin & Catarina asking what theme they want to use for fef on hetzner3
Hi Marcin, Hi Catarina, I'm sorry to inform you that you need to find a new theme for your fef website: * https://fef.opensourceecology.org/ The fef website uses the theme 'simplephotoRes' = "Simple Photo Responsive" Unless you can find otherwise, I can't find any information about this theme. The theme doesn't exist on wordpress.org (moreover, it uses an invalid slug name) * https://wordpress.org/themes/simplephotoRes/ The theme's styles.css file says it was developed by "Marios Lublinski". The URL of the theme is = http://www.dessign.net/simplephototheme/. That website (dessign.net) just looks like a landing page full of scam AI products, and the page specific to the theme just redirects to some unrelated podcast about AI. Because I can't find any info about this theme anywhere on the Internet, I don't know exactly when it was last updated, but I did find a version of jquery in it with a version that was released in March 2012, so I think it's safe to say that this theme hasn't been updated in over 12 years. It's generally a bad idea to use themes and plugins that are not popular and not updated, because it could lead to the theme breaking with future versions of wordpress, your other plugins, or future versions of PHP. Not to mention that it could have security vulnerabilities. In fact, I did come across a known security vulnerability in one of the plugins that oshine requires you use (tatsu). I think the only reason you weren't hacked is because I prevent wordpress from being able to edit its own files, and your web server's newly uploaded files cannot execute PHP. This breaks a lot of wordpress admin actions, but it also is probably the reason your site is still online, despite not having been updated in many years. Anyway, I did try to migrate this old theme to your new server, and it does cause a critical error; it appears to be incompatible with the latest version of wordpress. It's not going to work on hetzner3. I highly recommend only using themes that can be downloaded directly from wordpress.org/plugins/, with hundreds of thousands of active installations, and that are actively developed. Please let me know what theme you'd like me to install & activate for fef on hetzner3. Thank you, Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
- ok, well, I'd say fef is stuck now pending their response
- ...
- our next wp site on the list (which doesn't run oshine; which we're also stuck on) is oswh.opensourceecology.org
- here's our script for for our initial creation of oswh from hetzern2 to hetzner3
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='oswh.opensourceecology.org'
dbName='oswh_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner2/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}
# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
####################
# run on hetzner3 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='oswh.opensourceecology.org'
dbName='oswh_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
# STEP 1: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"
# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mv var/www/html/${vhost_name} ${vhostDir}
# remove '.svn' dirs (we no longer use svn, for security)
find ${docrootDir} -iname '.svn' -exec rm -rf '{}' \;
# add wordpress bug fix
# is the bug fix already present?
if ! $(grep 'https://core.trac.wordpress.org/ticket/48693' ${vhostDir}/wp-config.php) ; then
# the bug fix is absent; add it
backup_filename="wp-config.`date "+%Y%m%d_%H%M%S"`.php"
mv ${vhostDir}/wp-config.php ${vhostDir}/${backup_filename}
cat > ${vhostDir}/wp-config.php <<'EOF'
<?php
# fix wordpress bugs
# * https://core.trac.wordpress.org/ticket/48693
# * https://core.trac.wordpress.org/ticket/62693
if( ! function_exists('ini_set') ){
function ini_set(){
return;
}
}
if( ! function_exists('chmod') ){
function chmod(){
return;
}
}
EOF
tail -n +2 ${vhostDir}/${backup_filename} >> ${vhostDir}/wp-config.php
fi
# verify
ls
vim ${vhostDir}/wp-config.php
# UPDATE CORE
rsync -av --progress /var/tmp/wordpress/core/wordpress/ ${docrootDir}
# UPDATE OLD PLUGINS
for plugin_path in $(find "${docrootDir}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# INSTALLL NEW PLUGINS
new_plugins="activitypub aurora-heatmap melapress-login-security wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"
for plugin in ${new_plugins}; do
plugin_path="${docrootDir}/wp-content/plugins/${plugin}"
source_path="/var/tmp/wordpress/plugins/${plugin}"
if [ -d "${source_path}" ]; then
echo "${plugin}"
rm -rf ${plugin_path};
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# UPDATE/INSTALL THEMES
for theme_path in $(find "${docrootDir}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
# ACTIVATE NEW PLUGINS
activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
- as before, I ran ^ that, fixed permissions, pushed with ansible, restarted web services, updated my /etc/hosts, logged-in, and did a db upgrade in the wui
- after that, we have the same issue as fef; theme doesn't exist (this time it's a different theme)
The theme directory "Eventor" does not exist.
- thee doesn't exist on wordpress.org https://wordpress.org/themes/eventor/
- here's the theme info (taken from hetzner2)
[root@opensourceecology themes]# head Eventor/style.css /* Theme Name: Eventor Support URI: http://www.themeskingdom.com/support/ Description: Author: Themeskingdom Author URI: http://www.themeskingdom.com/support/ Version: 1.7 License: GNU General Public License v2.0 License URI: http://www.gnu.org/licenses/gpl-2.0.html Theme URI: http://www.themeskingdom.com/ [root@opensourceecology themes]#
- man, that website is shit. I couldn't find "eventor" in the author's site (probably I could if I clicked a thousand times; there's no way to ctrl+f for it; you have to mouseover an image, and it's javascript so you have to click and click and click to load more to view all of the themes available)
- I ended-up giving up and manually typing this URL-out with trial-and-error, and I found the theme's page https://themeskingdom.com/purchase-options/eventor/
- alright, so it's a paid theme with no public download link
- I sent an email to Marcin & Catarina asking if they had license info
Hi Marcin, Hi Catarina, Do you have any license info for the "Eventor" theme? It looks like oswh uses a paid theme named "eventor" * https://oswh.opensourceecology.org/ I was unable to download the latest version of the Eventor theme from wordpress.org * https://wordpress.org/themes/eventor/ I did manage to find the private website for the theme here: * https://themeskingdom.com/purchase-options/eventor/ Somehow it says that it's $35 for a lifetime plan and $59 for a one-year plan. That doesn't make sense. Anyway, can you please forward me any info I'd need to download the latest version of this theme? Thank you, Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
- I also sent an email to their support, CCing Marcin & Catarina
Hello, Where can I download the latest version of your Eventor theme? We purchased a copy of your Eventor theme years ago, and we need to download the latest version. Please let us know where we can download the latest version of the Eventor theme. Thank you, Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
- allright, so now we're stuck on oswh too
- ...
- our next website (that doesn't use oshine) is seedhome.openbuildinginstitute.org
- here's our script to copy it from hetzner2 to hetzner3
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='seedhome.openbuildinginstitute.org'
dbName='seedhome_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner2/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}
# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
####################
# run on hetzner3 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='seedhome.openbuildinginstitute.org'
dbName='seedhome_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
# STEP 1: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"
# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mv var/www/html/${vhost_name} ${vhostDir}
# remove '.svn' dirs (we no longer use svn, for security)
find ${docrootDir} -iname '.svn' -exec rm -rf '{}' \;
# add wordpress bug fix
# is the bug fix already present?
if ! $(grep 'https://core.trac.wordpress.org/ticket/48693' ${vhostDir}/wp-config.php) ; then
# the bug fix is absent; add it
backup_filename="wp-config.`date "+%Y%m%d_%H%M%S"`.php"
mv ${vhostDir}/wp-config.php ${vhostDir}/${backup_filename}
cat > ${vhostDir}/wp-config.php <<'EOF'
<?php
# fix wordpress bugs
# * https://core.trac.wordpress.org/ticket/48693
# * https://core.trac.wordpress.org/ticket/62693
if( ! function_exists('ini_set') ){
function ini_set(){
return;
}
}
if( ! function_exists('chmod') ){
function chmod(){
return;
}
}
EOF
tail -n +2 ${vhostDir}/${backup_filename} >> ${vhostDir}/wp-config.php
fi
# verify
ls
vim ${vhostDir}/wp-config.php
# UPDATE CORE
rsync -av --progress /var/tmp/wordpress/core/wordpress/ ${docrootDir}
# UPDATE OLD PLUGINS
for plugin_path in $(find "${docrootDir}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# INSTALLL NEW PLUGINS
new_plugins="activitypub aurora-heatmap melapress-login-security wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"
for plugin in ${new_plugins}; do
plugin_path="${docrootDir}/wp-content/plugins/${plugin}"
source_path="/var/tmp/wordpress/plugins/${plugin}"
if [ -d "${source_path}" ]; then
echo "${plugin}"
rm -rf ${plugin_path};
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# UPDATE/INSTALL THEMES
for theme_path in $(find "${docrootDir}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
# ACTIVATE NEW PLUGINS
activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
- as before, I ran ^ that, fixed permissions, pushed with ansible, restarted web services, updated my /etc/hosts, and tried to load it in the browser https://seedhome.openbuildinginstitute.org/
- this time, instead of getting presented with the wp login page, I got presented the OSE forums site; interesting
- nginx logs appear to show the correct vhost
root@hetzner3 /var/log/nginx # tail -f access.log error.log */*log ... ==> seedhome.openbuildinginstitute.org/access.log <== 185.213.155.175 - - [26/Dec/2024:20:57:34 +0000] "GET / HTTP/1.1" 200 151284 "-" "curl/7.88.1" "-"
- apache does say the wrong vhost
root@hetzner3 /var/log/apache2 # tail -f error.log access.log */*log ... ==> forum.opensourceecology.org/access.log <== 185.213.155.175 - - [26/Dec/2024:20:58:51 +0000] "GET / HTTP/1.1" 200 16464 "-" "curl/7.88.1"
- oh, there's no vhost at all for this site in apache; hmm
root@hetzner3 /etc/apache2/sites-enabled # ls -lah total 24K drwxr-xr-x 2 root root 4,0K Dec 26 20:10 . drwxr-xr-x 8 root root 4,0K Dec 26 20:46 .. lrwxrwxrwx 1 root root 61 Sep 25 01:47 00-forum.opensourceecology.org.conf -> /etc/apache2/sites-available/forum.opensourceecology.org.conf lrwxrwxrwx 1 root root 59 Dec 26 18:20 fef.opensourceecology.org.conf -> /etc/apache2/sites-available/fef.opensourceecology.org.conf lrwxrwxrwx 1 root root 68 Dec 26 04:29 microfactory.opensourceecology.org.conf -> /etc/apache2/sites-available/microfactory.opensourceecology.org.conf lrwxrwxrwx 1 root root 60 Dec 26 20:10 oswh.opensourceecology.org.conf -> /etc/apache2/sites-available/oswh.opensourceecology.org.conf lrwxrwxrwx 1 root root 61 Sep 27 04:47 store.opensourceecology.org.conf -> /etc/apache2/sites-available/store.opensourceecology.org.conf root@hetzner3 /etc/apache2/sites-enabled # root@hetzner3 /etc/apache2/sites-enabled # ls -lah ../sites-available/ total 52K drwxr-xr-x 2 root root 4,0K Dec 26 20:10 . drwxr-xr-x 8 root root 4,0K Dec 26 20:46 .. -rw-r--r-- 1 root root 1,3K Jul 7 13:26 000-default.conf -rw-r--r-- 1 root root 868 Oct 4 05:52 000-maintenance.conf -rw-r--r-- 1 root root 870 Oct 4 04:57 000-maintenance.conf.3593326.2024-10-04@05:53:55~ -rw-r--r-- 1 root root 6,1K Jul 18 05:26 default-ssl.conf -rw-r--r-- 1 root root 952 Dec 26 18:19 fef.opensourceecology.org.conf -rw-r--r-- 1 root root 964 Sep 25 01:46 forum.opensourceecology.org.conf -rw-r--r-- 1 root root 1006 Dec 26 04:28 microfactory.opensourceecology.org.conf -rw-r--r-- 1 root root 958 Dec 26 20:09 oswh.opensourceecology.org.conf -rw-r--r-- 1 root root 964 Oct 5 04:19 store.opensourceecology.org.conf -rw-r--r-- 1 root root 1,2K Oct 5 04:18 store.opensourceecology.org.conf.3944920.2024-10-05@04:20:17~ root@hetzner3 /etc/apache2/sites-enabled #
- oh shit, the non-errors that always show up at the end of the ansible runs (restart fails) made me miss the actual error
user@personal:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
...
TASK [maltfield.apache : sites-available/ item .conf] **********************
ok: [hetzner3] => (item=forum.opensourceecology.org)
ok: [hetzner3] => (item=store.opensourceecology.org)
ok: [hetzner3] => (item=microfactory.opensourceecology.org)
ok: [hetzner3] => (item=fef.opensourceecology.org)
ok: [hetzner3] => (item=oswh.opensourceecology.org)
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
failed: [hetzner3] (item=seedhome.openbuildinginstitute.org) => {"ansible_loop_var": "item", "changed": false, "item": "seedhome.openbuildinginstitute.org", "msg": "Could not find or access 'seedhome.openbuildinginstitute.org.conf.j2'\nSearched in:\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.apache/templates/seedhome.openbuildinginstitute.org.conf.j2\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.apache/seedhome.openbuildinginstitute.org.conf.j2\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.apache/tasks/templates/seedhome.openbuildinginstitute.org.conf.j2\n\t/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.apache/tasks/seedhome.openbuildinginstitute.org.conf.j2\n\t/home/user/sandbox_local/ansible/hetzner3/templates/seedhome.openbuildinginstitute.org.conf.j2\n\t/home/user/sandbox_local/ansible/hetzner3/seedhome.openbuildinginstitute.org.conf.j2 on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
PLAY RECAP *********************************************************************
hetzner3 : ok=60 changed=9 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
user@personal:~/sandbox_local/ansible/hetzner3$
- ah, I gave it the wrong tld
user@personal:~/sandbox_local/ansible/hetzner3/roles/maltfield.apache/templates$ ls | grep -i seedhome seedhome.opensourceecology.org.conf.j2 user@personal:~/sandbox_local/ansible/hetzner3/roles/maltfield.apache/templates$
- I renamed the template file
user@personal:~/sandbox_local/ansible/hetzner3/roles/maltfield.apache/templates$ git mv seedhome.opensourceecology.org.conf.j2 seedhome.openbuildinginstitute.org.conf.j2 user@personal:~/sandbox_local/ansible/hetzner3/roles/maltfield.apache/templates$
- the next ansible run put the file in-place
root@hetzner3 /etc/apache2/sites-enabled # ls -lah ../sites-available/ total 56K drwxr-xr-x 2 root root 4,0K Dec 26 21:05 . drwxr-xr-x 8 root root 4,0K Dec 26 21:05 .. -rw-r--r-- 1 root root 1,3K Jul 7 13:26 000-default.conf -rw-r--r-- 1 root root 868 Oct 4 05:52 000-maintenance.conf -rw-r--r-- 1 root root 870 Oct 4 04:57 000-maintenance.conf.3593326.2024-10-04@05:53:55~ -rw-r--r-- 1 root root 6,1K Jul 18 05:26 default-ssl.conf -rw-r--r-- 1 root root 952 Dec 26 18:19 fef.opensourceecology.org.conf -rw-r--r-- 1 root root 964 Sep 25 01:46 forum.opensourceecology.org.conf -rw-r--r-- 1 root root 1006 Dec 26 04:28 microfactory.opensourceecology.org.conf -rw-r--r-- 1 root root 958 Dec 26 20:09 oswh.opensourceecology.org.conf -rw-r--r-- 1 root root 982 Dec 26 21:04 seedhome.openbuildinginstitute.org.conf -rw-r--r-- 1 root root 964 Oct 5 04:19 store.opensourceecology.org.conf -rw-r--r-- 1 root root 1,2K Oct 5 04:18 store.opensourceecology.org.conf.3944920.2024-10-05@04:20:17~ root@hetzner3 /etc/apache2/sites-enabled #
- but the restart failed
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/seedhome.openbuildinginstitute.org_20241226/current # systemctl restart apache2 varnish nginx Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details. root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/seedhome.openbuildinginstitute.org_20241226/current # root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/seedhome.openbuildinginstitute.org_20241226/current # systemctl status apache2 × apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; preset: enabled) Active: failed (Result: exit-code) since Thu 2024-12-26 21:06:00 UTC; 1min 3s ago Duration: 11min 43.849s Docs: https://httpd.apache.org/docs/2.4/ Process: 2767355 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE) CPU: 61ms Dec 26 21:06:00 hetzner3 systemd[1]: Starting apache2.service - The Apache HTTP Server... Dec 26 21:06:00 hetzner3 apachectl[2767369]: AH00112: Warning: DocumentRoot [/var/www/html/seedhome.opensourceecology.org/htdocs] does not > Dec 26 21:06:00 hetzner3 apachectl[2767369]: (2)No such file or directory: AH02291: Cannot access directory '/var/log/apache2/seedhome.open> Dec 26 21:06:00 hetzner3 apachectl[2767369]: AH00014: Configuration check failed Dec 26 21:06:00 hetzner3 apachectl[2767355]: Action 'start' failed. Dec 26 21:06:00 hetzner3 apachectl[2767355]: The Apache error log may have more information. Dec 26 21:06:00 hetzner3 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE Dec 26 21:06:00 hetzner3 systemd[1]: apache2.service: Failed with result 'exit-code'. Dec 26 21:06:00 hetzner3 systemd[1]: Failed to start apache2.service - The Apache HTTP Server.
- oh, the typo was in the file as well; I fixed it
user@personal:~/sandbox_local/ansible/hetzner3$ git diff diff --git a/hetzner3/roles/maltfield.apache/templates/seedhome.openbuildinginstitute.org.conf.j2 b/hetzner3/roles/maltfield.apache/templates/seedhome.openbuildinginstitute.org.conf.j2 index 509cd37..8905bf9 100644 --- a/hetzner3/roles/maltfield.apache/templates/seedhome.openbuildinginstitute.org.conf.j2 +++ b/hetzner3/roles/maltfield.apache/templates/seedhome.openbuildinginstitute.org.conf.j2 @@ -1,7 +1,7 @@ # ansible_managed ################################################################################ -# File: seedhome.opensourceecology.org.conf +# File: seedhome.openbuildinginstitute.org.conf # Version: 0.3 # Purpose: localhost-only-listening, http-only, name-based-vhost for serving # traffic to varnish @@ -11,15 +11,15 @@ ################################################################################ <VirtualHost 127.0.0.1:8000> - ServerName seedhome.opensourceecology.org - DocumentRoot "/var/www/html/seedhome.opensourceecology.org/htdocs" + ServerName seedhome.openbuildinginstitute.org + DocumentRoot "/var/www/html/seedhome.openbuildinginstitute.org/htdocs" - CustomLog "/var/log/apache2/seedhome.opensourceecology.org/access.log" combined - ErrorLog "/var/log/apache2/seedhome.opensourceecology.org/error.log" + CustomLog "/var/log/apache2/seedhome.openbuildinginstitute.org/access.log" combined + ErrorLog "/var/log/apache2/seedhome.openbuildinginstitute.org/error.log" Include 'conf-available/wordpress.virtualhost.include' - <Directory "/var/www/html/seedhome.opensourceecology.org/htdocs"> + <Directory "/var/www/html/seedhome.openbuildinginstitute.org/htdocs"> Include 'conf-available/wordpress.directory.include' Options +FollowSymLinks </Directory> user@personal:~/sandbox_local/ansible/hetzner3$
- I re-ran ansible, and this time the manual restarts worked
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/seedhome.openbuildinginstitute.org_20241226/current # systemctl restart apache2 varnish nginx root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/seedhome.openbuildinginstitute.org_20241226/current #
- I refreshed the page, and – jesus – this is just another wordpress site that we never setup.
- I logged-in to the wp admin dashboard and did the db update
- now, unlike the other sites before, I loaded the site and it works! I mean, it's just a fresh install of a new site with Twenty Seventeen theme active, but at least nothing is broken.
- I finished setting-up the site with the plugins , per the CHG for store https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_store_to_hetzner3#Change_Steps
"Media" -> "Library" Upload https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v3-1blarge.jpg Upload https://wiki.opensourceecology.org/wiki/File:1day.jpg "Settings" -> "ActivityPub" -> "Settings" -> "Enable profiles by type" = "Blog profile only" "Settings" -> "ActivityPub" -> "Settings" -> "Blog-Profile" -> "Change profile ID" = "ose" "Settings" -> "ActivityPub" -> "Settings" -> "Blog-Profile" -> "Change Header Image" = Select "1day.jpg", cropped such the bottom is exactly the bottom of Catarina's white coat "Settings" -> "General" -> "Choose a Site Icon" -> Select "OSE-logo-blueprint-bg-v3-1blarge.jpg", cropped such that there is only a small buffer on the left & right of the text "Login Security" -> "Login Security Policies" -> tick the box that said "enable login security policies" "Login Security" -> "Login Security Policies" -> tick the box that said "Activate password policies" "Login Security" -> "Login Security Policies" -> change "Passwords must be X characters minimum" to "20" "Login Security" -> "Login Security Policies" -> uncheck "Password must contain at least one uppercase and one lowercase character. " "Login Security" -> "Login Security Policies" -> uncheck "Password must contain at least one numeric character (0-9)." "Login Security" -> "Login Security Policies" -> uncheck "Password must contain at least one special character, i.e., a character that is not a letter or a umber, such as ( , ? € ! @ # * etc" "Login Security" -> "Login Security Policies" -> check "Reset password on first login " "Login Security" -> "Login Security Policies" -> check "Do not send password reset links " "Login Security" -> "Login Security Policies" -> check "Activate failed login policies " "Login Security" -> "Login Security Policies" -> change "When a user is locked" from "it can be only unlocked by the administrator" to "unlock it after 60 minutes" "Login Security" -> "Login Security Policies" -> uncheck "Require blocked users to reset password on unblock. " Click the "Save Changes" button "Login Security" -> "Login page hardening" -> in the input form next to "Login page URL", I enter "ose-hidden-login" Click the "Save Changes" button "Settings" -> "Google Authenticator" -> Check every box under "Roles requiring Google Authenticator Enabled" Click "Save Changes" Button
- well I changed a few things, like 's/ose/obi' for activitypub
- anyway, I'd say this site is done. Yay! I think that's our second finished site, after the static site forums X_x
- ...
- ok, well, that does it for all our wordpress sites; all the other ones are stuck in some way
- next-up is phpList
- looks like I never actually did a 3TOFU of phpList?
- I did a 3TOFU of mediawiki and wordpress when I started this project back in August, but phpList is absent https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Sun_Aug_04.2C_2024
- I was hoping that maybe 3TOFU wouldn't be necessary (because their releases were signed), but it looks like I opened an issue about that in Sep of 2023. The lead dev said it's a good idea, but they're still not signing their releases https://github.com/phpList/phplist3/issues/987
- looks like the latest version of phpList is v3.4.7. Currently we're running v3.3.3.
- https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip
- Here's a 3TOFU script for the latest version of phpList
REMOTE_FILES="https://altushost-swe.dl.sourceforge.net/project/phplist/phplist/3.6.14/phplist-3.6.14.zip"
CURL="/usr/bin/curl --retry 5 --retry-all-errors"
PYTHON="/usr/bin/python3"
# in tails, we must torify
if [[ "`whoami`" == "amnesia" ]] ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
# get the file
for file in ${REMOTE_FILES}; do
${CURL} --progress-bar -O ${file}
done
# checksum
date -u +"%Y-%m-%d"
sha256sum *
- here's TOFU 1/3 (Tor, exit in Germany)
Congratulations. This browser is configured to use Tor. 2024-12-26 ######################################################################### 100.0% 2024-12-26 9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a phplist-3.6.14.zip user@host:/tmp/user/1000/tmp.ehfWbAUg29$
- ...
- well, until the 3TOFU on phpList is done, I think all that remains is mediawiki. fortunately, that's one of the most secure packages available; it's signed!
- when I last checked on this in August, the latest LTS of MediaWiki was 1.39 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Sun_Aug_04.2C_2024
- good news: a newer LTS has been released since then = 1.43, which just came-out 5 days ago. Good timing! It's going to be supported until Dec 2027 https://www.mediawiki.org/wiki/Version_lifecycle
- here's the blog post from *2 days ago* about the release https://www.pro.wiki/news/what-new-features-in-mediawiki-1-43-release
- shit, the blog post says you can't upgrade from v1.34 or earlier. To avoid data loss, we have to update to v1.35 first.
- It looks like we're running Mediawiki v1.30.0 on hetzner2
- So we know we need to upgrade to v1.35 first, but the next question is: can we upgrade directly from v1.30.0 to v1.35?
- here's the page that lists all the versions of MediaWiki https://en.wikipedia.org/wiki/MediaWiki_version_history
- unfortunatley, the link to the release notes for v1.35 is broken https://phabricator.wikimedia.org/source/mediawiki/browse/REL1_35/RELEASE-NOTES-1.35
- I guess we'll have to download the release and read the UPGRADE file
wget https://releases.wikimedia.org/mediawiki/1.35/mediawiki-1.35.0.tar.gz
- ok, cool, the notes say that "upgrade from 1.3 or ealier...should generally go smoothly"
user@disp1974:~$ grep -A9 1.3 mediawiki-1.35.0/UPGRADE == Upgrading from 1.3 or earlier == This should generally go smoothly. If you keep your LocalSettings.php, you may need to change the style paths to match the newly rearranged skin modules. Change these lines: $wgStylePath = "$wgScriptPath/stylesheets"; $wgStyleDirectory = "$IP/stylesheets"; $wgLogo = "$wgStylePath/images/wiki.png"; -- Note that the 1.3 beta releases included a potential vulnerability if PHP is configured with register_globals on and the includes directory is served to the Web. For general safety, turn register_globals *off* if you don't _really_ need it for another package. If your hosting provider turns it on and you can't turn it off yourself, send them a kind note explaining that it can expose their servers and their customers to attacks. == Upgrading from 1.2 or earlier == user@disp1974:~$
- the security note sounds a bit alarming, and more-so because I didn't see that we're disabling register_globals on hetzner2 or hetzner3
[root@opensourceecology current]# grep -ir globals /etc/php.ini ; globals: GET, POST, COOKIE, ENV and SERVER. There is a performance penalty ; variables_order directive. It does not mean it will leave the super globals ; http://php.net/auto-globals-jit auto_globals_jit = On [root@opensourceecology current]#
- not to worry, though. it seems that php disabled register globals entirely in PHP 5.4 https://www.a2hosting.com/kb/developer-corner/php/using-php.ini-directives/php-register-globals-directive
- and we're using php 5.6 on hetzner2 already
[root@opensourceecology current]# rpm -qa | grep -i php | grep -i mysql php56w-mysql-5.6.40-1.w7.x86_64 [root@opensourceecology current]#
- alright, so first we need to figure out how to migrate mediawiki from hetzner2 to hetzner3. then upgrade it to v1.35. Then upgrade it to v1.43.
- first, let's download and verify the releases for 1.35 and 1.43
- I know that I've previously 3TOFU'd the MediaWiki release signing key, but it looks like I never imported it into my keying on hetzner3
maltfield@hetzner3:~$ gpg --list-keys /home/maltfield/.gnupg/pubring.kbx ---------------------------------- pub rsa2048 2018-05-31 [SC] 63AF7AA15067C05616FDDD88A3A2E8F226F0BC06 uid [ unknown] WP-CLI Releases <releases@wp-cli.org> sub rsa2048 2018-05-31 [E] maltfield@hetzner3:~$
- here's the 3tofus I did in the past for the keys.txt file
2024-08-04 2e943991a469cb28f4906148b2c3517ab6d5a9285e5342e2312c9f70e643955c keys.txt 2024-08-07 2e943991a469cb28f4906148b2c3517ab6d5a9285e5342e2312c9f70e643955c keys.txt 2024-09-10 2e943991a469cb28f4906148b2c3517ab6d5a9285e5342e2312c9f70e643955c keys.txt
- and here's what I get now
root@hetzner3 ~ # cd /var
=Wed Dec 25, 2024=
# I'm still waiting to hear back on the plugin question for store.opensourceecology.org from Marcin & Catarnna.
# ...
# in the meantime, I'm going to push forward and start working on microfactory.opensourceecology.org
# I haven't even begun to migrate this from hetzner2 to hetzner3 yet, so first I just copied from my CHG ticket for store, changing just a few of the vars https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_store_to_hetzner3#Change_Steps
<pre>
####################
# run on hetzner2 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='microfactory.opensourceecology.org'
dbName='microfactory_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner2/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
# STEP 2: BACKUP DB
mkdir -p ${backupDir_hetzner2}/{current,old}
pushd ${backupDir_hetzner2}/current/
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
time nice mysqldump -u"${dbUser}" -p"${dbPass}" ${dbName} | bzip2 -c > ${backupDir_hetzner2}/current/${backupFileName_db_hetzner2}
# STEP 3: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner2}/current/${backupFileName_files_hetzner2} ${vhostDir}
# STEP 4: COPY TO HETZNER3
ssh -p 32415 maltfield@hetzner3 sudo mkdir -p ${backupDir_hetzner3}/{current,old}
ssh -p 32415 maltfield@hetzner3 sudo mv ${backupDir_hetzner3}/current/* ${backupDir_hetzner3}/old/
rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
####################
# run on hetzner3 #
####################
sudo su -
# DECLARE VARIABLES
vhost_name='microfactory.opensourceecology.org'
dbName='microfactory_db'
dbUser="CHANGEME"
dbPass="CHANGEME"
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner2="/var/tmp/backups_for_migration_to_hetzner3/${vhost_name}_${stamp}"
backupDir_hetzner3="/var/tmp/backups_for_migration_from_hetzner2/${vhost_name}_${stamp}"
backupFileName_db_hetzner2="mysqldump_${vhost_name}.${stamp}.sql.bz2"
backupFileName_files_hetzner2="${vhost_name}_files.${stamp}.tar.gz"
vhostDir="/var/www/html/${vhost_name}"
docrootDir="${vhostDir}/htdocs"
# STEP 1: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner3}/current
cp ${backupFileName_db_hetzner2} ${backupFileName_db_hetzner2}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner2} > db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName}; USE ${dbName};"
time nice mysql ${dbName} -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName}.* TO '${dbUser}'@'localhost' IDENTIFIED BY '${dbPass}'; FLUSH PRIVILEGES;"
# STEP 2: Add vhost files
mv "${vhostDir}" "${backupDir_hetzner3}/old/${vhost_name}.$(date "+%Y%m%d_%H%M%S")"
tar -xzvf ${backupFileName_files_hetzner2}
mv var/www/html/${vhost_name} ${vhostDir}
# remove '.svn' dirs (we no longer use svn, for security)
find ${docrootDir} -iname '.svn' -exec rm -rf '{}' \;
# add wordpress bug fix
# is the bug fix already present?
if [[ ! $(grep 'https://core.trac.wordpress.org/ticket/48693' ${vhostDir}/wp-config.php) ]]; then
# the bug fix is absent; add it
backup_filename="wp-config.`date "+%Y%m%d_%H%M%S"`.php"
mv ${vhostDir}/wp-config.php ${vhostDir}/${backup_filename}
cat > ${vhostDir}/wp-config.php <<'EOF'
<?php
# fix wordpress bugs
# * https://core.trac.wordpress.org/ticket/48693
# * https://core.trac.wordpress.org/ticket/62693
if( ! function_exists('ini_set') ){
function ini_set(){
return;
}
}
if( ! function_exists('chmod') ){
function chmod(){
return;
}
}
EOF
tail -n +2 ${vhostDir}/${backup_filename} >> ${vhostDir}/wp-config.php
fi
# verify
ls
vim ${vhostDir}/wp-config.php
# UPDATE CORE
rsync -av --progress /var/tmp/wordpress/core/wordpress/ ${docrootDir}
# UPDATE OLD PLUGINS
for plugin_path in $(find "${docrootDir}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# INSTALLL NEW PLUGINS
new_plugins="activitypub aurora-heatmap melapress-login-security wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"
for plugin in ${new_plugins}; do
plugin_path="${docrootDir}/wp-content/plugins/${plugin}"
source_path="/var/tmp/wordpress/plugins/${plugin}"
if [ -d "${source_path}" ]; then
echo "${plugin}"
rm -rf ${plugin_path};
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
# UPDATE/INSTALL THEMES
for theme_path in $(find "${docrootDir}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
# SET PERMISSIONS
# first pass, whole site
chown -R not-apache:www-data "/var/www/html"
find "/var/www/html" -type d -exec chmod 0050 {} \;
find "/var/www/html" -type f -exec chmod 0040 {} \;
#############
# WORDPRESS #
#############
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
###########
# phpList #
###########
phplist_sites="$(find /var/www/html -maxdepth 1 -type d -iname *phplist*)"
for vhost_dir in $phplist_sites; do
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}"; done
for dir in ${vhost_dir}; do find "${dir}" -type d -exec chmod 0050 {} \;; done
for dir in ${vhost_dir}; do find "${dir}" -type f -exec chmod 0040 {} \;; done
for dir in ${vhost_dir}; do [ -d "${dir}/public_html/uploadimages" ] || mkdir "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do chown -R not-apache:www-data "${dir}/public_html/uploadimages"; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type f -exec chmod 0660 {} \;; done
for dir in ${vhost_dir}; do find "${dir}/public_html/uploadimages" -type d -exec chmod 0770 {} \;; done
done
# ACTIVATE NEW PLUGINS
activate_plugins="activitypub aurora-heatmap melapress-login-security"
for plugin in ${activate_plugins}; do
sudo -u wp -i wp --path="${docrootDir}" plugin activate ${plugin}
done
- I updated the /etc/hosts and tried to load the login for the new site on hetnzer3, but I got a cert error https://microfactory.opensourceecology.org/wp-login.php
- I checked and the cert seems valid. Actually I don't know how that is, but I'll take it
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
Certificate Name: openbuildinginstitute.org
Serial Number: 37201823a671e8c6da8373cddd4efde6c6a
Key Type: RSA
Domains: www.openbuildinginstitute.org awstats.openbuildinginstitute.org openbuildinginstitute.org seedhome.openbuildinginstitute.org
Expiry Date: 2025-03-11 08:00:42+00:00 (VALID: 75 days)
Certificate Path: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem
Certificate Name: opensourceecology.org
Serial Number: 3ec0988ac3af1baa0909ce9a9f4a6409c21
Key Type: RSA
Domains: fef.opensourceecology.org awstats.opensourceecology.org forum.opensourceecology.org microfactory.opensourceecology.org munin.opensourceecology.org opensourceecology.org oswh.opensourceecology.org phplist.opensourceecology.org staging.opensourceecology.org store.opensourceecology.org wiki.opensourceecology.org www.opensourceecology.org
Expiry Date: 2025-03-11 08:00:47+00:00 (VALID: 75 days)
Certificate Path: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/opensourceecology.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
- I tried to reload nginx, and then I refreshed the page. that made the cert error go away, but now I have a 404 from nginx; I guess I never setup the nginx/varnish/apache vhost configs with ansible?
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # nginx -t 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 2024/12/26 04:08:14 [warn] 1428644#1428644: the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/conf.d/https.opensourceecology.org.include:12 nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # systemctl reload nginx root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
- yeah, it's pretty bare in the nginx configs dir
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # ls -lah /etc/nginx/sites-enabled/ total 20K drwxr-xr-x 2 root root 4,0K Sep 27 04:47 . drwxr-xr-x 8 root root 4,0K Oct 5 03:45 .. -rw-r--r-- 1 root root 659 Sep 24 04:16 00-default.conf -rw-r--r-- 1 root root 1,8K Sep 24 04:16 awstats.opensourceecology.org.conf lrwxrwxrwx 1 root root 59 Sep 24 04:17 forum.opensourceecology.org.conf -> /etc/nginx/sites-available/forum.opensourceecology.org.conf -rw-r--r-- 1 root root 1,5K Sep 24 04:16 munin.opensourceecology.org.conf lrwxrwxrwx 1 root root 59 Sep 27 04:47 store.opensourceecology.org.conf -> /etc/nginx/sites-available/store.opensourceecology.org.conf root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # ls -lah /etc/nginx/sites-available/ total 24K drwxr-xr-x 2 root root 4,0K Oct 5 03:45 . drwxr-xr-x 8 root root 4,0K Oct 5 03:45 .. -rw-r--r-- 1 root root 2,4K Mar 14 2023 default -rw-r--r-- 1 root root 1,6K Sep 24 04:16 forum.opensourceecology.org.conf -rw-r--r-- 1 root root 1,6K Oct 5 03:44 store.opensourceecology.org.conf -rw-r--r-- 1 root root 1,6K Sep 27 04:46 store.opensourceecology.org.conf.3932816.2024-10-05@03:45:29~ root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
- I commented-out the 'microfactory.opensourceecology.org' domain from the ansible provision.yml file, and re-ran `ansible-playbook provision.yml`. It failed with an ssh error :(
user@personal:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.nginx/tasks/main.yml,
line 8, column 3, found a duplicate dict key (command). Using last defined
value only.
[WARNING]: While constructing a mapping from /home/user/sandbox_local/ansible/h
etzner3/roles/maltfield.varnish/tasks/main.yml, line 107, column 3, found a
duplicate dict key (command). Using last defined value only.
PLAY [hetzner3] ****************************************************************
TASK [Gathering Facts] *********************************************************
[WARNING]: Unhandled error in Python interpreter discovery for host hetzner3:
SSH authentication is incorrect
fatal: [hetzner3]: UNREACHABLE! => {"changed": false, "msg": "SSH authentication is incorrect", "unreachable": true}
PLAY RECAP *********************************************************************
hetzner3 : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
user@personal:~/sandbox_local/ansible/hetzner3$
- curiously, it appears that everying's working great with just ssh
user@personal:~/sandbox_local/ansible/hetzner3$ ssh hetzner3 Linux hetzner3 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. maltfield@hetzner3:~$
- oh, I don't think it's using my `~/.ssh/config` file; this fails
user@personal:~/sandbox_local/ansible/hetzner3$ cat hosts hetzner3 ansible_port=32415 ansible_host=144.76.164.201 ansible_user=maltfield user@personal:~/sandbox_local/ansible/hetzner3$ ssh -p32415 maltfield@144.76.164.201 maltfield@144.76.164.201: Permission denied (publickey). user@personal:~/sandbox_local/ansible/hetzner3$
- ok, so the issue was that I had changed to use a distinct ssh auth sock file for OSE, to prevent forwarding of keys between different orgs on the wrong org's servers
- I fixed this by re-creating the ssh-agent socket file.
- Note that I had to wrap it in `eval` otherwise the env variables just get printed, and not actually added to the env
- Also not that I then had to actually do an `ssh hetzner3` to prompt for my password and add the (decyrpted) key to the keyring; ansible still wouldn't work the first time until I did this
user@personal:~/sandbox_local/ansible/hetzner3$ echo $SSH_AUTH_SOCK
/tmp/ssh-XXXXXXw7kxUB/agent.807
user@personal:~/sandbox_local/ansible/hetzner3$
user@personal:~/sandbox_local/ansible/hetzner3$ rm /home/user/cloud/.ssh/identities/ose/ose-agent; eval `ssh-agent -a /home/user/cloud/.ssh/identities/ose/ose-agent`
Agent pid 41866
user@personal:~/sandbox_local/ansible/hetzner3$
user@personal:~/sandbox_local/ansible/hetzner3$ echo $SSH_AUTH_SOCK
/home/user/cloud/.ssh/identities/ose/ose-agent
user@personal:~/sandbox_local/ansible/hetzner3$
user@personal:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.nginx/tasks/main.yml,
line 8, column 3, found a duplicate dict key (command). Using last defined
value only.
[WARNING]: While constructing a mapping from /home/user/sandbox_local/ansible/h
etzner3/roles/maltfield.varnish/tasks/main.yml, line 107, column 3, found a
duplicate dict key (command). Using last defined value only.
PLAY [hetzner3] ****************************************************************
TASK [Gathering Facts] *********************************************************
[WARNING]: Unhandled error in Python interpreter discovery for host hetzner3:
SSH authentication is incorrect
fatal: [hetzner3]: UNREACHABLE! => {"changed": false, "msg": "SSH authentication is incorrect", "unreachable": true}
PLAY RECAP *********************************************************************
hetzner3 : ok=0 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0
user@personal:~/sandbox_local/ansible/hetzner3$
user@personal:~/sandbox_local/ansible/hetzner3$ ssh hetzner3
Enter passphrase for key '/home/user/.ssh/identities/ose/id_rsa.ose':
Enter passphrase for key '/home/user/.ssh/identities/ose/id_rsa.ose':
Linux hetzner3 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
maltfield@hetzner3:~$
logout
Connection to 144.76.164.201 closed.
user@personal:~/sandbox_local/ansible/hetzner3$
user@personal:~/sandbox_local/ansible/hetzner3$ ssh hetzner3
Linux hetzner3 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
maltfield@hetzner3:~$
logout
Connection to 144.76.164.201 closed.
user@personal:~/sandbox_local/ansible/hetzner3$
user@personal:~/sandbox_local/ansible/hetzner3$ ansible-playbook provision.yml
[WARNING]: While constructing a mapping from
/home/user/sandbox_local/ansible/hetzner3/roles/maltfield.nginx/tasks/main.yml,
line 8, column 3, found a duplicate dict key (command). Using last defined
value only.
[WARNING]: While constructing a mapping from /home/user/sandbox_local/ansible/h
etzner3/roles/maltfield.varnish/tasks/main.yml, line 107, column 3, found a
duplicate dict key (command). Using last defined value only.
PLAY [hetzner3] ****************************************************************
TASK [Gathering Facts] *********************************************************
...
- ansible finished but with an error on the restart
- I tried reload the web browser, but I got the same 404 error from apache
- I manually restarted the stack
root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current # systemctl restart apache2 varnish nginx root@hetzner3 /var/tmp/backups_for_migration_from_hetzner2/microfactory.opensourceecology.org_20241226/current #
- then I refreshed the page, and I see a wordpress login page :D https://microfactory.opensourceecology.org/wp-login.php
- I was able to login using the same creds from hetzner2, and then it dumped me to the upgrade page. I clicked the "Update Worpdress Database" button
- I then loaded the frontpage, and you can see it's very broken; there are shortcodes not being resolved
- this is actually a theme that we *do* want to have oshine setup-on, so we should figure out what plugins we need (and how to get them)
- the top of the page yells at us about the plugins we're missing
This theme requires the following plugins: BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Meta Box Framework, Safe SVG and Slider Revolution. The following recommended plugin is currently inactive: WPForms Lite. Begin installing plugins | Activate installed plugin | Dismiss this notice
- so that's 7 plugins that it demands we add that we're missing
- the shortcodes that are broken on the frontpage are for "tatsu", so I kinda want to look into that first
- it sucks that they don't give the slug for these plugins to avoid ambiguity, but I just googled "tatsu" and -- uhh -- the second link is an article from bleeping computer
- as far as I can tell, our hetzner2 server would be vulnerable -- if we didn't explicitly harden our permissions to not allow the webserver to edit its own files, outside of the uploads dir -- which (the uploads dir) doesn't allow php to be executed inside of it
- this is exactly the same reason that we can't simply press the "install" button to install these plugins. I know it's annoying, but clearly it's saved our ass..
- I tried searching wordpress.org/plugins for even the first plugin in their list (BE Portfolio Post Type), but the results were super unclear as to which plugin I need https://wordpress.org/plugins/search/BE+Portfolio+Post+Type/
- I sent an email to oshine ("BrandExponents" <support@brandexponents.com>) asking for them to translate these ambiguous human names to unambiguous plugin slugs
Hi, Can you please tell me: 1. What is the plugin slug 2. What is the download URL for all of the required plugins for the oshine theme (BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu) I've tried searching wordpress.org/plugins for ^ these human-readable plugin names, but it's ambiguous. I'd prefer the actual slugs, so I can make sure I know which plugin to download. PS Your KB is inaccessible; it gives an error http://www.brandexponents.com/oshine-knowledgebase/ Forbidden You don't have permission to access this resource. Thank you,
- otherwise, just clicking-through the links in the header nav bar, I realized that I get a 404 if I click Workshops -> Kansas Community College - Feb 11, 2019
- I do *not* get a 404 when doing the same on the hetzenr2 site
- all of the other pages appeared to load, albeit totally broken (likely to be fixed once we install those required plugins)
Sun Dec 22, 2024
- I tried-out the 'include-mastodon-feed' plugin; it works fine (though it requires javascript)
- I think we should install it on all sites, but leave it deactivated for now
- ...
- I enabled the 'melapress-login-security' plugin
- after enabling it, it redirected me to the page at wp dashboard -> Login Security
- I ticked the box that said "enable login security policies"
- I ticked the box that said "Activate password policies"
- I changed the "Passwords must be X characters minimum" to "20"
- I unchecked "Password must contain at least one uppercase and one lowercase character. "
- I unchecked "Password must contain at least one numeric character (0-9)."
- I unchecked "Password must contain at least one special character, i.e., a character that is not a letter or a number, such as ( , ? € ! @ # * etc"
- I checked "Reset password on first login "
- I checked "Do not send password reset links "
- I checked "Activate failed login policies "
- I changed "When a user is locked" from "it can be only unlocked by the administrator" to "unlock it after 60 minutes"
- I unchecked "Require blocked users to reset password on unblock. "
- I clicked the "Save Changes" button
- I clicked on wp dashboard -> "Login Security" -> "Login page hardening"
- In the input form next to "Login page URL", I entered "ose-hidden-login"
- I clicked the "Save Changes" button
- I think that covers all the plugins that we want to enable on all sites. I sent an email to Marcin & Catarina asking for them to approve my plan
Hey Marcin, Hey Catarina, Please let me know if you approve the following changes to all of your wordpress sites after they're migrated to hetzner3: [1] Install and activate 'melapress-login-security' This plugin replaces 'rename-wp-login' and 'force-strong-passwords', both of which are no longer available. [2] Install and activate 'activitypub' This turns your wordpress site into a decentralized social media site, allowing accounts on the fediverse (eg mastodon, lemmy, threads, flipboard, friendica, peertube, pixelfed, etc) to follow you with ActivityPub. You probably won't notice a difference, but it's a good feature to enable going forward. [3] Install and activate 'aurora-heatmap' This is a freemium plugin that lets you see where users have clicked as they browse your site. It's all locally-stored, privacy-friendly analytics that can be helpful in understanding how your users interact with your website. Moreover, I would also like to install (but not activate) the following plugins (so that you an enable them later, if needed) on all of your wordpress sites: wps-hide-login raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite If there's any additional plugins that you'd like me to make available (even to leave disabled), now is the time to let me know so I can add it to the list and the migration script/steps. Please let me know if you have any questions or concerns about this plan or if it looks good, and I can proceed. Thank you, Michael Altfield https://www.michaelaltfield.net PGP Fingerprint: 0465 E42F 7120 6785 E972 644C FE1B 8449 4E64 0D41 Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
- I found commands to activate the plugins, and I added them to the CHG ticket https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_store_to_hetzner3#Change_Steps
maltfield@hetzner3:~$ docrootDir='/var/www/html/store.opensourceecology.org/htdocs'
sudo -u wp -i wp --path="${docrootDir}" plugin activate activitypub
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Plugin 'activitypub' is already active.
Success: Plugin already activated.
maltfield@hetzner3:~$
maltfield@hetzner3:~$ sudo -u wp -i wp --path="${docrootDir}" plugin deactivate activitypub
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'activitypub' deactivated.
Success: Deactivated 1 of 1 plugins.
maltfield@hetzner3:~$
maltfield@hetzner3:~$ sudo -u wp -i wp --path="${docrootDir}" plugin activate activitypub
PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Plugin 'activitypub' activated.
Success: Activated 1 of 1 plugins.
maltfield@hetzner3:~$
- I went through the entire script process for the migration of the store site https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_store_to_hetzner3#Change_Steps
- I found and fixed several bugs
- I obviously didn't do the actual DNS changes, that's still TODO to add to the article
- I also didn't verify the backups, but that's pretty straight-forward
- ok, I think the only thing blocking migration of store is the adding the DNS steps and getting Marcin/Catarina to approve the process
Fri Dec 20, 2024
- updated logs
- wordpress plugins research
Sat Dec 14, 2024
- I checked-up on the amazon glacier job, and I managed to download its output after it's finished and before it deleted itself!
user@disp4042:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": [
{
"JobId": "Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL",
"Action": "InventoryRetrieval",
"VaultARN": "arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020",
"CreationDate": "2024-12-14T01:59:59.138Z",
"Completed": true,
"StatusCode": "Succeeded",
"StatusMessage": "Succeeded",
"InventorySizeInBytes": 24418,
"CompletionDate": "2024-12-14T05:43:14.278Z",
"InventoryRetrievalParameters": {
"Format": "JSON"
}
}
]
}
user@disp4042:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL" ./output.json
{
"status": 200,
"acceptRanges": "bytes",
"contentType": "application/json"
}
user@disp4042:~$
user@disp4042:~$ du -sh output.json
24K output.json
user@disp4042:~$
- alright, looks like the vault is made up of 65 Archives, which we can now delete
user@disp4042:~$ archive_ids=$(jq .ArchiveList[].ArchiveId < output.json) user@disp4042:~$ user@disp4042:~$ echo $archive_ids | tr " " "\n" | wc -l 65 user@disp4042:~$
- I iterated throgh each of these archives and told AWS to delete them
user@disp4042:~$ for archive_id in ${archive_ids}; do
echo "Deleting Archive: ${archive_id}"
aws glacier delete-archive --archive-id=${archive_id} --vault-name 'deleteMeIn2020' --account-id REDACTED --region us-west-2
done
Deleting Archive: "qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA"
Deleting Archive: "lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw"
Deleting Archive: "fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw"
Deleting Archive: "zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA"
Deleting Archive: "Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw"
Deleting Archive: "P9wIGNBbLaAoz7xGht6Y4k7j33nGgPmg0RQ4sesN2tImQLjFN1dtkooVGrBnQqbPt8YhgvwUXv8eO_N72KRjS3RrZQYvkGxAQ9uPcJ-zaDOG8kII7l4p7UzGfaroO63ZreHItIW4GA"
Deleting Archive: "o-naX0m4kQde-2i-8JZbEESi7r8OlFjIoDjgbQSXT_zt9L_e7qOH3HQ1R7ViQC3i7M0lVLbODsGZm9w9HfI3tHYKb2R1T_WWBwMxFuC_OhYiPX8uepTvvBg2Mg6KysP9H3zNzwGSZw"
Deleting Archive: "mxeiPukWr03RpfDr49IRdJUaJNjIWQM4gdz8S8k3-_1VetpneyWZbwEVKCB1uMTYpPy0L6HZgZP7vJ6b7gz1oeszMnlzZR0-W6Rgt4O0BZ_mwgtGHRKOH0SIpMJHRnePaq9SBR9gew"
Deleting Archive: "TOZBeL9sYVRtzy7gsAC1d930vcOhEBaABsh1ejb6vvad_NVSLu_1v0UvWqwkkf7x_8CCu6_WxolooSClZMhQOA21J_0_HP9GxvPkUvdSOeqmHjuANbIS82IRBOjFT4zFUoZnPhcVUg"
Deleting Archive: "LdlFgzhEnxVsuGMZU4d2c_rfMTGM_3iCvLUZZSpGmmLArCQLs8HxjWLwfDDeKPKEarvSgXOVA-Evy4Ep5WAzESoofG5jdCidL5OispSfHElpPu-60xbmNvQt9neLGZrwa3C_iESGiw"
Deleting Archive: "6GHR8GlRG4EIlkA7O_Ta6BAXN3BQ7HmP0V7TgOp6bOa4cxuIlbHkmCd3I2lUSNwfG1penWOibFvvDhzgcihdmUMtCLepT3rl6HtFR5Lv-ro5mIegCcWQJOUDT0FRfsb7e7IkAze02Q"
Deleting Archive: "lryfyQFE4NbtWg5Q6uTq8Qqyc-y9il9WYe7lHs8H2lzFSBADOJQmCIgp6FxrkiaCcwnSMIReJPWyWcR4UOnurxwONhw8fojEHQTTeOpkf6fgfWBAPP9P6GOZZ0v8d8Jz_-QFVaV6Bw"
Deleting Archive: "O19wuK1PL_Wwf59-fjQuVP2Con0LXLf5Mk9xQA3HDPw4y1ZdwjYdFzmhZdaMUtGX666YKKjJu823l2C6seOTLg1ZbXZVTqQjZTeZGkQdCSRQdxyo3pEPWE2Iqpgb61FCiIETdCANUQ"
Deleting Archive: "6ShVCMDoqdhc4wg84L1bXaq3O2InX-qB9Q9NMRH-xJQ0_TSlIN5b3fysow9-_RuNYc2lK958NrwFiIEa7Q0bVaT9LaZQH8WtoTqnX3DN2xJhb4_KUdu6iUaDdJUoPfsSXtC7xvPb-w"
Deleting Archive: "0M5MSxjrlWJiT0XrncbVBITR__anuTLeOhcq9XvqsX0Q1koa0K0bH-wrZOQO7YsqqPv5Te3AUXPOCzIO6F0g5DQ2tOZq8E_YHX0XmMGjnOfeHIV9m_5GiCQAi3PrUuWM3C4cApTs7A"
Deleting Archive: "fwR6U5jX2T9N4mc14YQNMoA52vXICj-vvgIvYyDO5Qcv-pNeuXarT4gpzIy-XjuuF4KXkp9BXD13AA3hsau9PfW0ypy874m7arznCaMZO8ajm3NIicawZMiHGEikWw82EGY0z4VDIQ"
Deleting Archive: "EZG83EoQ65jxe4ye0-0qszEqRjLE3lAb2Vi7vZ2eYvj1bVJnTc5kvfWgTxl4_w2G1PPk4pn6g2dIsYXosWk3OqWNaWNcYEOHEkNREHycnTpcl0rBkWJoimt9fCKLJCF7FiGavWUMSw"
Deleting Archive: "5xqn4AAJhxnOHLJMfkvGX3Qksj5BTiEyHURILglfH0TPh_GfvbZNHqzdYIW-8sMtJ8OQ1GnnFqAOpty5mMwOSEjaokWkrQhEZK9-q7FBKDXXglAlqQKEJpd2UcTQI47zBEmGRasm-A"
Deleting Archive: "3XL4MENpH6i2Dp6micFWmMR2-qim3D1LQGiyQHME_5_A5jAbepw7WDEJOS2m2gIudSXfCuyclHTqzZYEpr6RwTGIEmYGw1jQ-EDPWYzjGTGDJcwWZEiklTmhLgvezqfyeSnQsdQZtA"
Deleting Archive: "g8RFNrkxynpQ8Yt9y4KyJra09dhxd3fIJxDlaUeDYBe615j7XON8gAdHMAQVerPQ4VF10obcuHnp64-kJFMmkG722hrlp3QBKy262CD4CcSUTSk3m070Mz6q3xySkcPzqRyxDwjtYg"
Deleting Archive: "ktHLXVqR5UxOoXEO5uRNMrIq4Jf2XrA6VmLQ0qgirJUeCler9Zcej90Qyg9bHvhQJPreilT4jwuW08oy7rZD_jnjd_2rcdZ11Y5Zl3V25lSKdRPM-b21o21kaBEr_ihhlIxOmPqJXg"
Deleting Archive: "iUmKTuLdEX3By9oHoqPtJd4KpEQ_2xh5PKV4LPuwBDcXyZmtt4zfq96djdQar1HwYmIh64bXEGqP7kGc0hk0ZtWZc12TtFUL0zohEbKBYr2VFZCQHjmc461TMLskKsOiyd6HbuKUWg"
Deleting Archive: "6gmWP3OdBIdlRuPIbNpJj8AiaR-2Y4FaPTneD6ZwZY2352Wfp6_1YNha4qvO1lapuITAhjdh-GzKY5ybgJag8O4eh8jjtBKuOg3nrjbABpeS7e6Djc-7PEiMKskaeldv5M52gHFUiA"
Deleting Archive: "4Ti7ZVFaexAncEDgak5Evp97aQk45VLA6cix3OCEB1cuGM6akGq2pINO8bzUjhEV8nvpqLLqoa_MSxPWTFl4uQ8sPUCDqG0vayB8PhYHcyNES09BQR9cE2HlR7qfxMDl5Ue946jcCw"
Deleting Archive: "GSWslpTGXPiYW5-gJZ4aLrFQGfDiTifPcqsSbh8CZc6T4K8_udBkSrNV0GNZQB9eLoRrUC5cXYT06FSvZ8kltgM61VUVYOXvO0ox4jYH68_sjHnkUmimk8itpa34hBC_c0zS0ZFRLQ"
Deleting Archive: "3nDMsn_-0igfg6ncwMqx3-UxQLi-ug6LEoBxqyZKsMhd83PPoJk1cqn6QFib2GeyIgJzfCZoTlwrpe9O0_GnrM7u_mUEOsiKTCXP0NadvULehNcUx-2lWQpyRrCiDg5fcBb-f7tY0g"
Deleting Archive: "CnSvT3qmkPPY7exbsftSC-Ci71aqjLjiL1eUa3hYo3OfVkF4s2SQ8n39rH5KaQwo3GTHeJZOVoBTW9vMEf2ufYKc9e_eVAfVcmG-bLgncRQrrV-DlE2hYglzdAalx3H5OXBY8jlD9Q"
Deleting Archive: "WWIYVa-hqJzMS8q1UNNZIKfLx1V3w3lzqpCLWwflYBM7yRocX2CEyFA-aY2EKJt0hRLTshgLXE3L3Sni8bYabDLBrV2Gehgq9reRTRhn8cxoKks4f1NmZwCCTSs6L4bQuJnjjNvOKw"
Deleting Archive: "XQYjqYnyYbKQWIzc1BSWQpn2K8mIoPQQH-bnoje7dB3BGCbzTjbEATGYSV1qJMbeUhiT_b7lwDiZzW1ZEbHVCgMDrWxCswG3eTZxiFdSwym7rELpFh5eC7XQlxuHjHocLY2zbUhYvg"
Deleting Archive: "kn9SKSliFV1eHh_ax1Z9rEWXR5ETF3bhdoy6IuyItI3w63rBgxaNVNk5AFJLpcR2muktNFmsSEp8QucM-B4tMdFD6PtE4K8xPJe_Cvhv3G4e2TPKn3d9HMD5Bx3XjTimGHK6rHnz0A"
Deleting Archive: "4-Rebjng1gztwjx1x5L0Z1uErelURR5vmCUGD3sEW6rBQRUHRjyEQWL22JAm6YPpCoBwIxzVDPyC2NvSofxx2InjmixAUoQsyy3zAgGoW0nSlqNQPfeF1hkRdOCyIDutfMTQ1keEQw"
Deleting Archive: "OVSNJHSIy5f1WRnisLdZ9ElWY4AjdgZwFqk3vDISCtypn5AHVo7wDGOAL76SpF0XzAd-yLgD3fIzf7mvgR4maA_HCANBhIP7Sdvhi7MLMjLnXLoKoHuKayBok_VLNRFfT5XORaTemA"
Deleting Archive: "N1TB1zWhwJq20nTRNcIzVIRL9ms1KnszY0C4XAKhfTgtuWaV1SFWlqaA0xb6NjbX6N3XDisuP0bke-I0G_8RbsFQ_PcRTwRZzNEbr4LOU4WFhLM86s-FjDwjdJHmgyttfMh_1K9RLQ"
Deleting Archive: "wJyG1vWz9IcB8-mnLm9bY3do9KIsxNY9nQ8ClQaOALesN-k3R5GU11p7Q3sVeStelg9IzWvburDcVFdHmJIYHC9RuRbuSZbk_rQvxxrkhtDcviu4i9_hN4SnPHvV3i0hITuiEFGpkA"
Deleting Archive: "hPtzfNk9SSUpI-_KihUEQOb89sbrK3tr0-3au-pe7al_e8qetM7uQEbNTH4_oWPqD2yajF79XPXxi4wkqAcQjoAN4IhnkPVb846wODKTpFXkRs9V8lz6nW0t_GdR2c9uYXf-xM_MpQ"
Deleting Archive: "osvrVQsHYSGCO30f0kO9aneACAA8h80KBmqfBMqDG3RioepW6ndLlNBcSvhfQ2nrcWBwLabIn4A7Rkr7sjbddViPo92viBh4lyZdyDwVcm6Pp1hQv-p2j0vldxYLWpyLDflQ8QRn4A"
Deleting Archive: "OtlG0WN4qd8kIg3xRQvRHoAzICwHRg6S3I8df5r_VRKaUNzJCsnwbO8Z9RiJPAAqqqVqg9I_GKhnt7txvEdUjx5s9hLywWm_OcRm5Lj_rJV_dupUwVlTG8HsdnCIwFseGa1JD5bviw"
Deleting Archive: "2PAyQClvhEMhO-TxdAvV9Qdqa_Lvh4webx9hHIXbVnQQHJxMlhWPikmVpr1zTQRgy23r-WcOouH6gLKQ7WBRSH5yM8q5f8gb0Z2anOAwdR4A9DtxqDIVtI78-7Bs3Bf2b0fYbPQCWw"
Deleting Archive: "Gn7a5jzeimXwa3su0i02OAK2XFmK9faX2WZx77Zq_tOx6j7ihpFEnkuF97Dpo66NgF7M24orh50kMSphvzLex_NbP9tDNoOI8mYG0-7GzOmNSmw9NaZpMLGn9NAVKbxs0byJ3YkquA"
Deleting Archive: "UqxNCpEu1twmhb9qLPxpQXMBv6yLyR37rZ1T_1tQjdl8x0RwukdIoOEGcmpHwdtrJgTA2OrWZ3ZYTncxkXojwWAOROW-wJ4SJANFfxwvGfueFNUSn17qTggcqeE43I5P1xmlxb25wg"
Deleting Archive: "NR3Z9zdD2rW0NG1y3QW735TzykIivP_cnFDMCNX6RcIPh0mRb_6QiC5qy1GrBTIoroorfzaGDIKQ0BY18jbcR3XfEzfcmrZ1FiT1YvQw-c1ag6vT46-noPvmddZ_zyy2O1ItIygI6Q"
Deleting Archive: "3wjWOHj9f48-L180QRkg7onI5CbZcmaaqinUYJZRheCox-hc021rQ3Tl1Houf0s5W-qzk6HVRz3wkilQI_TAi2PXWaFUMibz00DAQfGj9ZQKeSOlxE_3qsIRcmYsYo-TMaU2UsSqNA"
Deleting Archive: "OfCmIMVetV8SxOBYUGFWldcHWFaFuGeLrYYm3A4YrvUU93zBrCLkOoBssToY1QIt_ZGwIueTgyoLTADetpfgswaoou_CwD8xfqss1hQAbQ7CaKW6sQHD-kcw4ii-D1h22lap95AZ4g"
Deleting Archive: "PLs1lsB4c1dV3YaBG1y2SN3OEWmtImJVlz6CA6IknA6y3R8yfQV3FXcLXWC_YpczM6t05xigcynA7m1A6GkuHIyTDOr6-DCOLlEvxDHmFrA4hrzJkl2pLquNWJ9yc-JC83ZV4SkM-Q"
Deleting Archive: "QwTHHmRo-NpqTTe2uy87GgB2MVydnz--3-3Z5u_0gdh5FPxEl2YSyjmJy3CKNDmJaNtrmwLeRF4_GubyZFc-CzlWl6OqZmINkCVSz34wY-k336C8HUOoKm5tPV3riSYaPb7WjjXwNQ"
Deleting Archive: "EmeH9kAWeVAyMa68pIknrJ135ZyXKB8WcjVKGQ58cVQE4Q98SMsX1OerOA4-_Q6epBJ_hgUT7ztFQ5d6PNiPRJ3H8uUIqXG3pkve5MaeA_cqAqvu4apBhU2HgALb1iS3NKy5IRdeUg"
Deleting Archive: "NX074yaGa7FGL_QH5W9-mZ9TVmi0T2428B1lW8aEck6Ydjk3H3W6pgQTisOqE9B7azs1jykJ_IL-fdbkLzhAmrpWNGJBq5hVjfMNSP-Msm976Mf7mnXe6Z6QDkO5PVXaFsNZ1EzNyw"
Deleting Archive: "uHk-GTBb6LVulxOkgs_ZYdF-cvKubUpvdP7hoS9Cqduw8YPInJaHB4LbBHpIxOL1idfYoMm-h4YI_Jq8qN3EnOBHiAjqUEwJAstagfMEvk2E38IlNLu_5J_09E0JM7MZXc4RSEZfNA"
Deleting Archive: "n8UslfWy3wmFYZNYJF3PfuxVoLNORVes-IunJoyzKJDYMNqmkwybrG9KVGoL4sbRspq0Tqmccn87hLGZ_A7kjBB6fvnWuAOjALhNinbDe-RkESPVPWN6464vfCIf3BI3NhK0_nzCNw"
Deleting Archive: "lzlnWYAQWMFp32BM163QS_8kb9wJ_kqaal2XmVb_rXLRDDXhSogYZCanA7oWyi3IdlWECd8R3KT3s50gJo8_kckLtq2uUUjG3Yl1wJuvXQfVh1AwzPOtLlyldqXmDoiVFzw-NrkpIw"
Deleting Archive: "WimEI6ABJtXx6j4jVK4lrVKWbPmI1iPZLErplyB7ChN6MSOH3yMOeANT7L3O6BBI4G17WjSIKE6EN6YgMP9OdgxF4XjHyyGUuWNwqy-nnIETKyp7YrFuuBkSiSloBhZkC6DRqpdrww"
Deleting Archive: "k-Q9oBnWeC3P7zOEN6IMEVFjl3EwPkqi5kbEvEqby4zKEpb_aDj4f88Us1X7QBvG3Pi8GUriEnNlXXlNH5s4-4cBfQryVjY_MOAnSakhgCLXs-srczsWIZvtkkMsh4XFiBpVzYao3w"
Deleting Archive: "Y7_nQQC2uSB7XXEfd_zaKpp_gqGPZ_TQTXDPLmSP8k77n9NImLnTL7apkE6AlJopAkgmPiLOaTgIXc4_mSkUFp5teSOxdPxk19Cvs2fL9S1Yv5U7wihZfrsrwNffyZl289J59G-UBg"
Deleting Archive: "zW6rdGwDojNoD-CjYUf8p_tbX2UMPHedXwUAM4GxNRkO0GoE1Ax5rpGr38LTnzZ_rCX-4F3kdJiAm1ahm-CfAzefUxenayuoS6cg384s5UHbZGsD2QpogBj9EJDDWlzrj8hr8DPC1Q"
Deleting Archive: "W9argz7v3GxZUkItGwILf1QRot8BNbE4kOJVvUrwOGs72KGog0QCGc8PV-3cWUvhfxkCFLuoZE7qJCQmT2Cc_LvaV46hWFFvgs5TFBdIySr2jeil-d8cYR5oN9zAvkYCGuvDlXmgxw"
Deleting Archive: "T01giTZdzpQVhhijB47T97HEtIYDHTG7sVy5mpfUbaxBaGq5fU5C1aKleXpwTKOz7_aTiWAlkeM5rM3Lg_SS3qMI1JBeZR7l8M4W5a4JmFw3MVneRYZIC9JIuTO46F91SIa8JH1vgw"
Deleting Archive: "GPXAjVjuNoyBjKU_Zx5wAxcjtLhsHwHXxKPuKDugGK3-jxNezXUG27MnJ5yDLay6yVJhZ_h3gCwlkd2y2gokIre6CK2wf2Ms3fk_m0BVkGI_Qx1PDRb7RL6P5l7yYeL1HWMhUfLFuw"
Deleting Archive: "o2o3n7hTDoRKwTwD2IN9bHRT7Ox1K1A3ZaauheIlvQyhycJgy4mSqRusieHvYihijY9hqWaIXXDLQMAn6xa55idBgAWkuLe_Px1xNeaae7uy3nNUOb5GgPoDr8YJ9lollj3cKd_iNw"
Deleting Archive: "EM59cEYkyc2dTk6AZjPOWDG4ftxqTxXIM5RAMCgB8xP_wMaawcz8TY8ojij-zF9qve7Ae0grQqxe1R74HLA6Yh3R7UHMueMPThlUhpW_r2atTntGZTOkJVjevCoyCkG3P23wDckMXA"
Deleting Archive: "vgOBLcSW8orHNlOyfPHT071fxpWtCu27wjyHoNx1Lq8V727HYmLX7JZyRXEBpszEYSKIdSU-X1DT1kzDlUeb5amFbcBU3E0s4qSja8fXz769bM89SwSNQ4gWYYgiqUqar6EbJZS2-A"
Deleting Archive: "rmnC1D8xwBN4PI90pLKD-Och9gluScd7c_3tVF9dLOlEPB8Lp_f2Y0m6YwGnmQkpkc43hrPwoaYTzQWOJRMBbhN0vdLc4RT1DRhfCE68HrQM7YzsEYY7ANf4h_lfFAE7mx1JGv16Cg"
Deleting Archive: "DBhGz8QPdNcS4Zp-jLpSOfE4sOk7tiLCQsA_rCJs9nQ1722YiaXeOLSThCvFn9RaUqfRj0UmomPE9_A2bdXbxuNi2Re3Q1uxav8HHR7RkJKdNMzYeTWbbpaDSNGyJZgYVKujqrT8TQ"
Deleting Archive: "8lqDSeF3uiA9PcGb3W4hC1LwUBwT1oPLGColzsnKBq-0RpKZ4aVBMqcpKXlu5oYDGSrM4KjXnEk6ksgRkAtLuimSOWiMKUf-Nq34aDAVG5e1gmRIs8fgE2ghDaa6ToGi3os5-Q48zg"
Deleting Archive: "Q9FdpUiXA32lWVEa1Xdr0vigTynWsUX5nLkzvg6QCP7LsrWpOykIHrzSZIRdSubWKlJkZ5JR6eZgln7DnPV_Wso5JcFjRMP2L0TgpAMPoOSPsrP9uet3pLQPWr_ZP7aWISR7XLjhdg"
Deleting Archive: "e2bM8CyOWaQWtrZXk3OahrVeNpateHCkkyBd4UkBPuaJPz-HNdlnrVMA9M4nZdhqTdNMVfpsLK0HEIWxBT8zVJaHKMRdDWKSs9rb86gxyDjyIX6m4oSlik3I6EC1_ZMFhmpYKlMrPQ"
user@disp4042:~$
- I then logged-into the AWS Console WUI, and it still shows that the "last inventory date" was "August 1, 2018" and still shows a size of 285.3 GB (as of last inventory)
- Perhaps we just need to wait. Probably we'll get a partial charge for December. Hopefully there's $0 charge in January.
- I sent Marcin an email
Hey Marcin, I just initiated deleting the 65 archives that make-up this AWS Glacier vault. Let's never use Glacier again. It was jaw-droppingly difficult to delete this. You can't just delete a vault, you have to first query the AWS API to schedule a job that generates an inventory of the vault. The job takes hours or days to run, and then you need to query the API again to download its result, which is deleted 24 hours after it completes. * https://wiki.opensourceecology.org/wiki/Amazon_Glacier#Delete_from_Glacier After a lot of research and three tries, I finally managed to download the inventory report, which listed 65 archives. Today I iterated through the report and submitted the delete query for all 65 of these archives. I just logged-into the AWS Console WUI, and it still shows that the "last inventory date" was "August 1, 2018" and still shows a size of 285.3 GB (as of last inventory). Please let me know if you're still being charged by Amazon for the month of January 2025. Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org On 10/4/24 15:16, Marcin Jakubowski wrote: > 1. Yes, delete the vault. > > 2. Thanks, good insights - i'll look into those more closely to see what > would fit best. > > MJ
- ...
- I checked munin, and now I see the "process info" charts have legends with all of the processes we've listed. Data for some of the processes is empty, but this is pretty good. The most important is apache & mysql, from my experience
- for some reason the per-process I/O usage chart is 0s for all processes, but we've got loads of other charts on these processes, including: thread count, process count, memory usage, cpu usage, and context switches
- we still only have the 5 mysql charts; I think we've reached a point of diminishing return on that effort. this is good enough.
- I think that crosses off all the TODO items for munin; we have beautiful charts now ☺
- ...
- returning to wordpress...
- Here's TOFU 3/3 (ISP, exit in Ecuador)
Ecuador 2024-12-14 INFO: Determining Latest Version of Wordpress Core INFO: Determining Latest Version of Wordpress Plugins . . . . . . . . . jq: error (at <stdin>:0): Cannot index array with string "1.0.17" . . . . . . . . . . . . . . . . . . . . . WARNING: Failed to download plugin woo-multi-currency null null WARNING: Failed to download plugin woo-multi-currency null null https://downloads.wordpress.org/release/wordpress-6.7.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/activitypub.4.4.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.25.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wordpress-seo.24.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip ######################################################################### 100.0% 2024-12-14 8b1f9a708838b8710b4198da1116689197e0a6134e0a1a5e786500576383034f activitypub.4.4.0.zip 101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9 advanced-nocaptcha-recaptcha.7.5.0.zip 873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53 aurora-heatmap.1.7.0.zip 5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f bulk-media-register.1.40.zip ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0 enable-media-replace.4.1.5.zip 1a53bdcd1ddb160d5807dc17a0f9e474402e22c899b3a9af486c9d5f0d2c4b36 extensions-leaflet-map.4.4.zip 27f1ab1e3f5274335d48d0cadaabdef98284880b0324771890d36a1f562fb44a hcaptcha-for-forms-and-more.4.8.0.zip bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9 include-mastodon-feed.1.9.9.zip 13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217 leaflet-map.3.4.1.zip fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5 melapress-login-security.2.0.1.zip 33f186b028bd2428af17199d06f9c9c2b6d5e5f5a7d8ddc47c23cfc7cff35ff4 plugin.json f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c raw-html.1.6.4.zip 757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6 regenerate-thumbnails.3.1.6.zip 4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193 related-posts-by-taxonomy.2.7.6.zip ebd87841f73bb7946216ae4827a413dcc97fc5094cee2f8ddb6dea7eff356358 smart-slider-3.3.5.1.25.zip 41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210 spam-destroyer.2.1.4.zip aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9 woocommerce-gateway-stripe.9.0.0.zip fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa woocommerce-multilingual.5.3.9.zip 75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c wordpress-6.7.1.zip f9ce7a98840dd4bf490d955320a68ac553c767ba7f0eeae6e4f067be5a927ef3 wordpress-seo.24.0.zip feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38 wp-2fa.2.8.0.zip 130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda wpforms-lite.1.9.2.3.zip 6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c wpfront-notification-bar.3.4.2.zip e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2 wp-pgp-encrypted-emails.0.8.0.zip e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1 wp-qrcode.1.1.1.zip bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7 wps-hide-login.1.9.17.1.zip user@disp1555:/tmp/tmp.Yx1OD1gy5z$
- ok, a diff of these checksums (and the ones from the past 2 days) shows they all match, except plugin.json – which is just metadata (that we won't be using and therefore doesn't matter..as long as we got the same result)
- I rsync'd these 25x 3TOFU'd, wordpress plugin .zip files up to hetzner3 and then copied them into our dir with the other plugins
root@hetzner3 /var/tmp/wordpress/plugins # ls -lah | grep maltfield -rw-r--r-- 1 maltfield maltfield 229K Dec 14 16:51 activitypub.4.4.0.zip -rw-r--r-- 1 maltfield maltfield 244K Dec 14 16:52 advanced-nocaptcha-recaptcha.7.5.0.zip -rw-r--r-- 1 maltfield maltfield 133K Dec 14 16:51 aurora-heatmap.1.7.0.zip -rw-r--r-- 1 maltfield maltfield 50K Dec 14 16:51 bulk-media-register.1.40.zip -rw-r--r-- 1 maltfield maltfield 348K Dec 14 16:52 enable-media-replace.4.1.5.zip -rw-r--r-- 1 maltfield maltfield 2,4M Dec 14 16:52 extensions-leaflet-map.4.4.zip -rw-r--r-- 1 maltfield maltfield 1,4M Dec 14 16:52 hcaptcha-for-forms-and-more.4.8.0.zip -rw-r--r-- 1 maltfield maltfield 13K Dec 14 16:52 include-mastodon-feed.1.9.9.zip -rw-r--r-- 1 maltfield maltfield 72K Dec 14 16:52 leaflet-map.3.4.1.zip -rw-r--r-- 1 maltfield maltfield 415K Dec 14 16:51 melapress-login-security.2.0.1.zip -rw-r--r-- 1 maltfield maltfield 16K Dec 14 16:51 raw-html.1.6.4.zip -rw-r--r-- 1 maltfield maltfield 78K Dec 14 16:52 regenerate-thumbnails.3.1.6.zip -rw-r--r-- 1 maltfield maltfield 92K Dec 14 16:51 related-posts-by-taxonomy.2.7.6.zip -rw-r--r-- 1 maltfield maltfield 3,7M Dec 14 16:51 smart-slider-3.3.5.1.25.zip -rw-r--r-- 1 maltfield maltfield 631K Dec 14 16:51 spam-destroyer.2.1.4.zip -rw-r--r-- 1 maltfield maltfield 1,4M Dec 14 16:51 woocommerce-gateway-stripe.9.0.0.zip -rw-r--r-- 1 maltfield maltfield 5,9M Dec 14 16:52 woocommerce-multilingual.5.3.9.zip -rw-r--r-- 1 maltfield maltfield 28M Dec 14 16:51 wordpress-6.7.1.zip -rw-r--r-- 1 maltfield maltfield 4,0M Dec 14 16:51 wordpress-seo.24.0.zip -rw-r--r-- 1 maltfield maltfield 1008K Dec 14 16:52 wp-2fa.2.8.0.zip -rw-r--r-- 1 maltfield maltfield 11M Dec 14 16:52 wpforms-lite.1.9.2.3.zip -rw-r--r-- 1 maltfield maltfield 805K Dec 14 16:51 wpfront-notification-bar.3.4.2.zip -rw-r--r-- 1 maltfield maltfield 627K Dec 14 16:52 wp-pgp-encrypted-emails.0.8.0.zip -rw-r--r-- 1 maltfield maltfield 246K Dec 14 16:52 wp-qrcode.1.1.1.zip -rw-r--r-- 1 maltfield maltfield 50K Dec 14 16:51 wps-hide-login.1.9.17.1.zip root@hetzner3 /var/tmp/wordpress/plugins #
- I extracted these new ones
root@hetzner3 /var/tmp/wordpress/plugins # new_plugin_files=$(ls -lah | grep maltfield | awk {'print $9}')
root@hetzner3 /var/tmp/wordpress/plugins # echo $new_plugin_files
activitypub.4.4.0.zip advanced-nocaptcha-recaptcha.7.5.0.zip aurora-heatmap.1.7.0.zip bulk-media-register.1.40.zip enable-media-replace.4.1.5.zip extensions-leaflet-map.4.4.zip hcaptcha-for-forms-and-more.4.8.0.zip include-mastodon-feed.1.9.9.zip leaflet-map.3.4.1.zip melapress-login-security.2.0.1.zip raw-html.1.6.4.zip regenerate-thumbnails.3.1.6.zip related-posts-by-taxonomy.2.7.6.zip smart-slider-3.3.5.1.25.zip spam-destroyer.2.1.4.zip woocommerce-gateway-stripe.9.0.0.zip woocommerce-multilingual.5.3.9.zip wordpress-6.7.1.zip wordpress-seo.24.0.zip wp-2fa.2.8.0.zip wpforms-lite.1.9.2.3.zip wpfront-notification-bar.3.4.2.zip wp-pgp-encrypted-emails.0.8.0.zip wp-qrcode.1.1.1.zip wps-hide-login.1.9.17.1.zip
root@hetzner3 /var/tmp/wordpress/plugins # for file in $new_plugin_files; do unzip $file; done
- and I copied these new plugins into the store site, so I could enable them and demo them
root@hetzner3 /var/tmp/wordpress/plugins # for file in $new_plugin_files; do dir=$(echo $file | cut -d. -f1); rsync -av --progress $dir /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/; done root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # ls -lah /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ total 152K d---r-x--- 36 not-apache www-data 4,0K Dec 14 17:36 . d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 .. drwxr-xr-x 7 root root 4,0K Dec 9 14:46 activitypub drwxr-xr-x 7 root root 4,0K Nov 19 14:31 advanced-nocaptcha-recaptcha d---r-x--- 4 not-apache www-data 4,0K Jul 10 22:16 akismet drwxr-xr-x 5 root root 4,0K Mar 25 2024 aurora-heatmap drwxr-xr-x 5 root root 4,0K Nov 5 23:00 bulk-media-register d---r-x--- 3 not-apache www-data 4,0K Sep 27 21:51 classic-editor d---r-x--- 8 not-apache www-data 4,0K Nov 21 2022 coingate-for-woocommerce d---r-x--- 7 not-apache www-data 4,0K Jul 25 08:28 contact-form-7 drwxr-xr-x 10 root root 4,0K Nov 19 12:47 enable-media-replace drwxr-xr-x 11 root root 4,0K Nov 21 20:40 extensions-leaflet-map d---r-x--- 3 not-apache www-data 4,0K Jul 4 2022 google-authenticator d---r-x--- 4 not-apache www-data 4,0K Apr 23 2021 google-authenticator-encourage-user-activation drwxr-xr-x 7 root root 4,0K Dec 1 10:01 hcaptcha-for-forms-and-more ----r----- 1 not-apache www-data 2,3K Apr 9 2019 hello.php drwxr-xr-x 2 root root 4,0K Nov 24 16:52 include-mastodon-feed ----r----- 1 not-apache www-data 28 Apr 9 2019 index.php drwxr-xr-x 6 root root 4,0K Jul 22 02:54 leaflet-map drwxr-xr-x 8 root root 4,0K Dec 10 15:15 melapress-login-security d---r-x--- 8 not-apache www-data 4,0K Sep 27 07:22 meta-box drwxr-xr-x 3 root root 4,0K Nov 11 15:00 raw-html drwxr-xr-x 6 root root 4,0K Aug 14 2023 regenerate-thumbnails drwxr-xr-x 5 root root 4,0K Nov 18 11:47 related-posts-by-taxonomy drwxr-xr-x 4 root root 4,0K Nov 21 07:31 smart-slider-3 drwxr-xr-x 4 root root 4,0K Apr 18 2024 spam-destroyer d---r-x--- 8 not-apache www-data 4,0K Mar 17 2024 ssl-insecure-content-fixer d---r-x--- 4 not-apache www-data 4,0K Oct 21 2019 vcaching d---r-x--- 13 not-apache www-data 4,0K Sep 25 13:56 woocommerce drwxr-xr-x 7 root root 4,0K Dec 12 19:24 woocommerce-gateway-stripe drwxr-xr-x 11 root root 4,0K Dec 9 12:14 woocommerce-multilingual drwxr-xr-x 13 root root 4,0K Sep 24 07:41 wordpress-seo drwxr-xr-x 6 root root 4,0K Nov 19 14:34 wp-2fa drwxr-xr-x 9 root root 4,0K Dec 3 13:12 wpforms-lite drwxr-xr-x 8 root root 4,0K Nov 12 04:39 wpfront-notification-bar drwxr-xr-x 11 root root 4,0K May 25 2021 wp-pgp-encrypted-emails drwxr-xr-x 3 root root 4,0K Jul 27 2023 wp-qrcode drwxr-xr-x 6 root root 4,0K Oct 9 09:23 wps-hide-login root@hetzner3 /var/tmp/wordpress/plugins #
- and I fixed the permissions https://wiki.opensourceecology.org/wiki/Hetzner3#Restore_State_.28snapshot_.26_test.29
- I activated the 'activitypub' plugin. One obvious change is that we'll probably want to just have a single site-wide ActivtyPub user that combines all posts into one feed, as opposed to a different ActivityPub user/feed for each user/author on wordpress. I guess we'd want the later if we actually had a lot of users posting content every month, but we only have a few posts per year, it makes sense to combine it.
- after this change, it says that the endpoint to follow is 'store.opensourceecology.org@store.opensourceecology.org' or 'https://store.opensourceecology.org/@store.opensourceecology.org'
- well that's annoying, but I can change it
- I went to the "Blog-Profile" subsettings and changed it from 'store.opensourceecology.org' to 'OpenSourceEverything' https://store.opensourceecology.org/wp-admin/options-general.php?page=activitypub&tab=blog-profile
- so now the blog-wide ActivityPub endpoint is 'opensourceeverything@store.opensourceecology.org' or 'https://store.opensourceecology.org/@opensourceeverything'
- unfortunately, if you just load 'https://store.opensourceecology.org/@opensourceeverything' in your web browser, it just redirects to the frontpage ('/')
- I imagine that if you search this URL in a mastodon instance, you'd see a feed of posts – but this site isn't public yet (we're hardcoding the IP with /etc/hosts), so I can't test that until after the migration
- ActivityPub lets us setup an avatar and banner photo; I'll just copy what we're doing on our other socials
- Looks like facebook and X use the phto from brick production run from 2012 with Catarina in a white coat doing some angle griding with sparks flying https://wiki.opensourceecology.org/wiki/File:1day.jpg
- and the avatar is the logo from August 2011 https://wiki.opensourceecology.org/wiki/File:Open-source-ecology.png
- curiously the OSE_Logo article on the wiki suggests that a new icon from 2014 should be used, even though it's not what we're using on our socials in 2024 https://wiki.opensourceecology.org/wiki/OSE_Logo#Official_and_Current_Logos
- I'm going to go with the first one from Aug 2011 (what we're using everywhere else)
- shit, no, that file is actually very small. It's only 546 × 345. Maybe that's why we replaced it.
- oh, here's a bigger version of the same. there's 4 versions. some are square. some are rectangular. some have a drop shadow. some don't have a drop shadow. the square ones are 3,780 × 3,780 and the rectangular ones are 3,780 × 2,598
- https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v3-1blarge.jpg
- https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v2-1b-large.jpg
- https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v3-1large.jpg
- https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v2-1large.jpg
- the official mastodon docs say that the avatar is 400x400 px. So we want square. And that's taller than our main logo of 345 px https://docs.joinmastodon.org/user/profile/
- history shows this used to be 120x120 and it was increased to 400x400 in 2017 https://github.com/mastodon/mastodon/issues/3807
- so I'm thinking we should use the square one (without a drop shadow for easier readability as a thumbnail and consistency with facebook branding). That's this https://wiki.opensourceecology.org/wiki/File:OSE-logo-blueprint-bg-v3-1blarge.jpg
- the 1day.jpg banner is 5,184x2,912 px, which should be fine for the banner photo. I'm not going to bother to resize it; any autocropping should do fine, I think
- the media library for 'store' is actually empty, so I uploaded both of these as the first images for the site
- fuck, I got errors
OSE-logo-blueprint-bg-v3-1blarge.jpg The server cannot process the image. This can happen if the server is busy or does not have enough resources to complete the task. Uploading a smaller image may help. Suggested maximum size is 2560 pixels. 1day.jpg The server cannot process the image. This can happen if the server is busy or does not have enough resources to complete the task. Uploading a smaller image may help. Suggested maximum size is 2560 pixels.
- gross, looks like this is another wordpress bug
==> store.opensourceecology.org/error.log <==
[Sat Dec 14 19:52:42.215184 2024] [proxy_fcgi:error] [pid 249572:tid 249594] [client 146.70.184.187:0] AH01071: Got error 'PHP message: PHP Notice: PHP Request Startup: file created in the system's temporary directory in Unknown on line 0; PHP message: PHP Deprecated: strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/functions.php on line 7300; PHP message: PHP Deprecated: str_replace(): Passing null to parameter #3 ($subject) of type array|string is deprecated in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/functions.php on line 2189; PHP message: PHP Fatal error: Uncaught Error: Call to undefined function chmod() in /var/www/html/store.opensourceecology.org/htdocs/wp-admin/includes/file.php:1043\nStack trace:\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-admin/includes/file.php(1105): _wp_handle_upload()\n#1 /var/www/html/store.opensourceecology.org/htdocs/wp-admin/includes/media.php(306): wp_handle_upload()\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-admin/includes/ajax-actions.php(2632): media_handle_upload()\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-admin/async-upload.php(33): wp_ajax_upload_attachment()\n#4 {main}\n thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-admin/includes/file.php on line 1043', referer: https://store.opensourceecology.org/wp-admin/upload.php
- I updated wp-config.php with a hack to fix it like I did with theother bug (on set_ini)
root@hetzner3 /var/www/html/store.opensourceecology.org # diff wp-config.php.20241214 wp-config.php
8a9,13
> if( ! function_exists('chmod') ){
> function chmod(){
> return;
> }
> }
root@hetzner3 /var/www/html/store.opensourceecology.org #
- I tried again, and this time the upload was successful. So, yeah, it's definitely a dumb wordpress bug
- I reported the bug here https://core.trac.wordpress.org/ticket/62693
- And submitted a PR https://github.com/WordPress/wordpress-develop/pull/7352
- anyway, now that the images are available, I set them in the activitypub settings
- the banner photo had an interactive crop of the image; I just made the bottom of it the bottom of Catarina's white coat, such that we can just see the crane's hook in the top. It's not precise, but it's fine for this
- oh, apparently we can't even set the avatar for the blog profile; the plugin just re-uses the "Site Icon" from the general settings https://store.opensourceecology.org/wp-admin/options-general.php
- this site didn't have anything set; so I used the generic OSE logo per above
- I did reduce crop away the width a bit, so it's easier to see on small renderings
- I enabled the "Aurora Heatmap" plugin, and it looks like the default settings should be fine
Fri Dec 13, 2024
- here's TOFU 2/3 (VPN, exit in Sweden) for the new wordpress plugins
Sweden 2024-12-13 INFO: Determining Latest Version of Wordpress Core INFO: Determining Latest Version of Wordpress Plugins . . . . . . . . . jq: error (at <stdin>:0): Cannot index array with string "1.0.17" . . . . . . . . . . . . . . . . . . . . . WARNING: Failed to download plugin woo-multi-currency null null WARNING: Failed to download plugin woo-multi-currency null null https://downloads.wordpress.org/release/wordpress-6.7.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/activitypub.4.4.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.25.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wordpress-seo.24.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip ######################################################################### 100.0% 2024-12-13 8b1f9a708838b8710b4198da1116689197e0a6134e0a1a5e786500576383034f activitypub.4.4.0.zip 101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9 advanced-nocaptcha-recaptcha.7.5.0.zip 873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53 aurora-heatmap.1.7.0.zip 5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f bulk-media-register.1.40.zip ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0 enable-media-replace.4.1.5.zip 1a53bdcd1ddb160d5807dc17a0f9e474402e22c899b3a9af486c9d5f0d2c4b36 extensions-leaflet-map.4.4.zip 27f1ab1e3f5274335d48d0cadaabdef98284880b0324771890d36a1f562fb44a hcaptcha-for-forms-and-more.4.8.0.zip bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9 include-mastodon-feed.1.9.9.zip 13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217 leaflet-map.3.4.1.zip fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5 melapress-login-security.2.0.1.zip db016ec3c115ec20c1f0fba87b48b5eddee3a11f30d573b8a266a01077ee7ee1 plugin.json f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c raw-html.1.6.4.zip 757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6 regenerate-thumbnails.3.1.6.zip 4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193 related-posts-by-taxonomy.2.7.6.zip ebd87841f73bb7946216ae4827a413dcc97fc5094cee2f8ddb6dea7eff356358 smart-slider-3.3.5.1.25.zip 41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210 spam-destroyer.2.1.4.zip aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9 woocommerce-gateway-stripe.9.0.0.zip fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa woocommerce-multilingual.5.3.9.zip 75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c wordpress-6.7.1.zip f9ce7a98840dd4bf490d955320a68ac553c767ba7f0eeae6e4f067be5a927ef3 wordpress-seo.24.0.zip feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38 wp-2fa.2.8.0.zip 130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda wpforms-lite.1.9.2.3.zip 6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c wpfront-notification-bar.3.4.2.zip e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2 wp-pgp-encrypted-emails.0.8.0.zip e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1 wp-qrcode.1.1.1.zip bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7 wps-hide-login.1.9.17.1.zip user@disp7639:/tmp/tmp.FcejZlvblB$
- ...
- I checked munin to see if my changes yesterday have filled the empty charts with data
- 3 out of 5 of the mysql charts now have data. the two that are empty still are "MySQL InnoDB free tablespace" and "MySQL slow queries". It's quite possible that this idle server has no data because – it's idle
- I also wanted to see about adding all the other mysql charts possible, so I made a backup of the munin dir, deleted all the pluings/ symlinks for mysql, and recreated them
root@hetzner3 /usr/share/munin/plugins # tar -czf /var/tmp/munin.20241213.tar.gz /etc/munin/* tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets root@hetzner3 /usr/share/munin/plugins # du -sh /var/tmp/munin.20241213.tar.gz 28K /var/tmp/munin.20241213.tar.gz root@hetzner3 /usr/share/munin/plugins #
- shit, I just accidentally deleted all the mysql plugins
- I was *trying* to delete the symlinks from /etc/munin/plugins. And to be safe, I even created a backup of this dir first. But then I was in the wrong screen session and I didn't realize my pwd
root@hetzner3 /usr/share/munin/plugins # tar -czf /var/tmp/munin.20241213.tar.gz /etc/munin/* tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # du -sh /var/tmp/munin.20241213.tar.gz 28K /var/tmp/munin.20241213.tar.gz root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # rm -f mysql_* root@hetzner3 /usr/share/munin/plugins # ls -lah | grep -i mysql root@hetzner3 /usr/share/munin/plugins #
- fortunately I was able to restore them using apt. Note the package isn't 'munin' but 'munin-plugins-core'
root@hetzner3 /usr/share/munin/plugins # ls -lah | grep -i mysql root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # sudo apt-get -o Dpkg::Options::="--force-confmiss" install --reinstall munin-plugins-core Reading package lists... Done Building dependency tree... Done Reading state information... Done 0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 71 not upgraded. Need to get 0 B/242 kB of archives. After this operation, 0 B of additional disk space will be used. (Reading database ... 66512 files and directories currently installed.) Preparing to unpack .../munin-plugins-core_2.0.73-1_all.deb ... Unpacking munin-plugins-core (2.0.73-1) over (2.0.73-1) ... Setting up munin-plugins-core (2.0.73-1) ... root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # ls -lah | grep -i mysql -rwxr-xr-x 1 root root 43K Mar 21 2023 mysql_ -rwxr-xr-x 1 root root 1,8K Mar 21 2023 mysql_bytes -rwxr-xr-x 1 root root 5,6K Mar 21 2023 mysql_innodb -rwxr-xr-x 1 root root 2,6K Mar 21 2023 mysql_queries -rwxr-xr-x 1 root root 1,5K Mar 21 2023 mysql_slowqueries -rwxr-xr-x 1 root root 1,6K Mar 21 2023 mysql_threads root@hetzner3 /usr/share/munin/plugins # You have new mail in /var/mail/root
- ok, now I *actually* deleted the symlinks and recreated them to be the complete set of possible charts
root@hetzner3 /usr/share/munin/plugins # tar -czf /var/tmp/munin.20241213.tar.gz /etc/munin/* tar: Removing leading `/' from member names tar: Removing leading `/' from hard link targets root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /usr/share/munin/plugins # du -sh /var/tmp/munin.20241213.tar.gz 28K /var/tmp/munin.20241213.tar.gz root@hetzner3 /usr/share/munin/plugins # root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql lrwxrwxrwx 1 root root 31 Sep 25 01:47 mysql_ -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 36 Sep 25 01:47 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes lrwxrwxrwx 1 root root 37 Sep 25 01:47 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb lrwxrwxrwx 1 root root 42 Sep 25 01:47 mysql_isam_space_ -> /usr/share/munin/plugins/mysql_isam_space_ lrwxrwxrwx 1 root root 38 Sep 25 01:47 mysql_queries -> /usr/share/munin/plugins/mysql_queries lrwxrwxrwx 1 root root 42 Sep 25 01:47 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries lrwxrwxrwx 1 root root 38 Sep 25 01:47 mysql_threads -> /usr/share/munin/plugins/mysql_threads root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # rm -f mysql* root@hetzner3 /etc/munin/plugins # for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -sf /usr/share/munin/plugins/mysql_ $i; done root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql lrwxrwxrwx 1 root root 31 Dec 13 20:09 binlog_groupcommit -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 bin_relay_log -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 commands -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 connections -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 files_tables -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_bpool -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_bpool_act -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_insert_buf -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_io -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_io_pend -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_log -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_rows -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_semaphores -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_tnx -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 myisam_indexes -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 network_traffic -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 qcache -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 qcache_mem -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 replication -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 select_types -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 slow -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 sorts -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 table_locks -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 13 20:09 tmp_tables -> /usr/share/munin/plugins/mysql_ root@hetzner3 /etc/munin/plugins #
- I applied and regenerated munin
root@hetzner3 /etc/munin/plugins # service munin-node restart root@hetzner3 /etc/munin/plugins # sudo -u munin /usr/bin/munin-cron root@hetzner3 /etc/munin/plugins #
- well now I only see 1 chart on the munin wui = "MySQL InnoDB free tablespace" :( And it's still empty too
- let's wait and see if the others come back?
- actually, it looks like there's a better way to do this https://www.thesysadmin.rocks/2020/06/24/installing-munin-on-ubuntu-20-04-with-mysql-plugin/
munin-node-configure --shell | grep mysql
- the above command should output a bunch of `ln` commands with exactly what we need to copy & paste & create the mysql symlinks
- note this might just be all the "subcharts" for 'mysql_', but it'll miss all the other 'mysql_*' files (eg 'mysql_queries' links to 'mysql_queries', not 'mysql_' like the rest)
- to get those last ones, we'd want something like
ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins
- after some hours, the whole mysql section in munin disappeared; let's try that again
root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql
lrwxrwxrwx 1 root root 31 Dec 13 20:09 binlog_groupcommit -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 bin_relay_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 commands -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 files_tables -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_bpool -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_bpool_act -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_insert_buf -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_io -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_io_pend -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_rows -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_semaphores -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 innodb_tnx -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 myisam_indexes -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 qcache -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 qcache_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 replication -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 select_types -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 slow -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 sorts -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 table_locks -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec 13 20:09 tmp_tables -> /usr/share/munin/plugins/mysql_
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql | awk '{print $9}'
binlog_groupcommit
bin_relay_log
commands
connections
files_tables
innodb_bpool
innodb_bpool_act
innodb_insert_buf
innodb_io
innodb_io_pend
innodb_log
innodb_rows
innodb_semaphores
innodb_tnx
myisam_indexes
network_traffic
qcache
qcache_mem
replication
select_types
slow
sorts
table_locks
tmp_tables
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # for file in $(ls -lah | grep -i mysql | awk '{print $9}'); do rm -f $file; done
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # ln -s /usr/share/munin/plugins/mysql_* /etc/munin/plugins
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql
lrwxrwxrwx 1 root root 31 Dec 14 01:18 mysql_ -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 36 Dec 14 01:18 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes
lrwxrwxrwx 1 root root 37 Dec 14 01:18 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb
lrwxrwxrwx 1 root root 38 Dec 14 01:18 mysql_queries -> /usr/share/munin/plugins/mysql_queries
lrwxrwxrwx 1 root root 42 Dec 14 01:18 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries
lrwxrwxrwx 1 root root 38 Dec 14 01:18 mysql_threads -> /usr/share/munin/plugins/mysql_threads
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # munin-node-configure --shell 2>&1 | grep mysql
root@hetzner3 /etc/munin/plugins #
root@hetzner3 /etc/munin/plugins # munin-node-configure --suggest 2>&1 | grep mysql
mysql_ | yes | no [DBI connect('mysql;mysql_read_default_file=/etc/mysql/debian.cnf;mysql_connect_timeout=5','munin',...) failed: Access denied for user 'munin'@'localhost' (using password: NO)]
root@hetzner3 /etc/munin/plugins #
- unfortunately `munin-node-configure` failed there at the end (it had no ouptut), so I'm just going to redo what I did before, but this time actually add the missing 'mysql_' prefix to the symlink name
root@hetzner3 /etc/munin/plugins # for i in `/usr/share/munin/plugins/mysql_ suggest`; do ln -sf /usr/share/munin/plugins/mysql_ mysql_$i; done root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # ls -lah | grep -i mysql lrwxrwxrwx 1 root root 31 Dec 14 01:18 mysql_ -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_binlog_groupcommit -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_bin_relay_log -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 36 Dec 14 01:18 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_commands -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_connections -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_files_tables -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 37 Dec 14 01:18 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_bpool -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_bpool_act -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_insert_buf -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_io -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_io_pend -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_log -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_rows -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_semaphores -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_innodb_tnx -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_myisam_indexes -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_network_traffic -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_qcache -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_qcache_mem -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 38 Dec 14 01:18 mysql_queries -> /usr/share/munin/plugins/mysql_queries lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_replication -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_select_types -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_slow -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 42 Dec 14 01:18 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_sorts -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_table_locks -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 38 Dec 14 01:18 mysql_threads -> /usr/share/munin/plugins/mysql_threads lrwxrwxrwx 1 root root 31 Dec 14 01:27 mysql_tmp_tables -> /usr/share/munin/plugins/mysql_ root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # service munin-node restart root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # sudo -u munin /usr/bin/munin-cron root@hetzner3 /etc/munin/plugins #
- well now the UI has a "mysql" section again, but there's only 5 charts.
- As before, 3 out of 5 of the charts have data (but now with a big chunk of the last several hours of data missing)
- for some reason all our new charts are missing too
- I guess I'll wait until tomorrow. If they show up, then I'll want to document how I created these extra charts on 'hetzner3' – or just add that to ansible. If they don't show-up, then forget about it, these 5 charts are probably good enough.
- I confirmed that the two varnish uptime charts are now visible and have data; that's fixed
- I also checked the "process info" charts, which now has data for the "apache2" charts, but the "mysqld" ones are empty. I'm pretty sure this is because the process name is "mariadb" (not "mysqld")
- I filled this out to more processes, so it now reads
[proc] env.procname apache2|mariadbd|nginx|varnishd|varnishlog|varnishncsa|wazuh-db|wazuh-remoted|wazuh-syscheckd|wazuh-analysisd|wazuh-authd|python3|journalctl|systemd|kworker|unattended-upgrade
- I'll have to check in on this tomorrow to see if the charts have updated
- ...
- I wanted to check-in on the glacier inventory job that I kicked-off yesterday, but – as with the one I kicked off a few months ago – it shows no record of it!
user@disp4042:~$ aws glacier get-job-output --account-id OBFUSCATED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json
An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS
user@disp4042:~$ aws glacier list-jobs --account-id OBFUSCATED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": []
}
user@disp4042:~$
- oh wait, that was the job ID from a few months ago. But I get the same result from the one from yesterday
aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN" ./output.json
- jesus christ, SO says that the jobs may expire after 24 hours https://stackoverflow.com/questions/45112105/aws-glacier-job-id-was-not-found
- so the jobs can take many days to run, and they expire within 24 hours. They really do make this as hard as fucking possible to delete vaults! It really should be criminal how hard they make it. This is madness!
- oh, one other comment in ^ that thread worth revisting:
@OrestGulman Are you sure that you have data stored in the Amazon Glacier service? Please note that this is different to storing data in Amazon S3 with a 'Glacier' class
- we actually are trying to delete data in Amazon S3 in the glacier class. I think? Anyway I found the vault in the "Glacier S3" Service of the Console WUI. Could that be adding a complication?
- anyway, for now, I just kicked-off another job
user@disp4042:~$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"location": "/REDACTED/vaults/deleteMeIn2020/jobs/Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL",
"jobId": "Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL"
}
user@disp4042:~$
user@disp4042:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": [
{
"JobId": "Y66F8y-ft3r8ILhMUHth3DbDWwoMZCm0uPXC9R9_dCj74D_0cUwoX5btOTpLh9Vf4eNJS6KPP5JyujUiZ1WG6ciFGgQL",
"Action": "InventoryRetrieval",
"VaultARN": "arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020",
"CreationDate": "2024-12-14T01:59:59.138Z",
"Completed": false,
"StatusCode": "InProgress",
"InventoryRetrievalParameters": {
"Format": "JSON"
}
}
]
}
user@disp4042:~$
Thr Dec 12, 2024
- I wanted to follow-up with the glacier deletion task
- A couple months ago (on Oct 4), I documented that Marcin gave me auth to delete the 285.3 GB 'deleteMeIn2020' bucket from our amazon glacier account, which has been costing us $1.03/mo https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Fri_Oct_04.2C_2024
- It's, like, unbelievably complicated to delete this bucket
- First, before I can delete it, Amazon forces you to generate a recent inventory
- An inventory can only be created via the API, and it takes time for the job to finish
- I last checked on the inventory (that I initiated on Oct 4) on Oct 6, and it wasn't yet ready https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q4#Sun_Oct_06.2C_2024
- ok, here's our commands to get the inventory job's result
sudo apt-get -y install awscli aws configure set aws_access_key_id 'REDACTED' aws configure set aws_secret_access_key 'REDACTED' aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json
- unfortunately, it says the keys that I used (which I extracted from hetzner2:/root/backups/glacierTest.py) don't have permission to see the job's output –- even though those are the keys that I used to initiate the job!
user@disp5233:~$ sudo apt-get -y install awscli Reading package lists... Done Building dependency tree... Done Reading state information... Done awscli is already the newest version (2.9.19-1). The following packages were automatically installed and are no longer required: chromium-common chromium-sandbox libc++1-16 libc++abi1-16 libcommons-compress-java librnp0 libunwind-16 libwpe-1.0-1 libwpebackend-fdo-1.0-1 linux-image-6.1.0-10-amd64 linux-image-6.1.0-11-amd64 linux-image-6.1.0-13-amd64 linux-image-6.1.0-17-amd64 linux-image-6.1.0-18-amd64 linux-image-6.1.0-20-amd64 linux-image-6.1.0-21-amd64 linux-image-6.1.0-22-amd64 linux-image-6.1.0-23-amd64 linux-image-6.1.0-25-amd64 linux-image-6.1.0-26-amd64 Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. user@disp5233:~$ user@disp5233:~$ aws configure set aws_access_key_id 'REDACTED' user@disp5233:~$ aws configure set aws_secret_access_key 'REDACTED' user@disp5233:~$ user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (AccessDeniedException) when calling the GetJobOutput operation: User: arn:aws:iam::REDACTED:user/backup-cron is not authorized to perform: glacier:GetJobOutput on resource: arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020 user@disp5233:~$
- I also found some aws cred file in the root user's home, but it had the exact same creds as the glacierTest.py file :(
[root@opensourceecology ~]# cat /root/.aws/credentials [default] aws_access_key_id = REDACTED aws_secret_access_key = REDACTED [root@opensourceecology ~]#
- I logged into the aws console as my personal 'maltfield' user (using creds in my personal keepass)
- I clicked on my username in the top-right -> "Security Credentials"
- the WUI showed that I have one set of Access Keys that were created "2449 days ago"
- I checked the notes of my keepass and found a set of keys with the same key id
- I reconfigured my aws cli on my local dispVM to use these creds, but I got the same error (though this time for user = 'maltfield')
user@disp5233:~$ aws configure set aws_access_key_id 'REDACTED' user@disp5233:~$ aws configure set aws_secret_access_key 'REDACTED' user@disp5233:~$ user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (AccessDeniedException) when calling the GetJobOutput operation: User: arn:aws:iam::REDACTED:user/maltfield is not authorized to perform: glacier:GetJobOutput on resource: arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020 user@disp5233:~$
- oh, I realized that I was executing the above command literally with 'REDACTED' as the '--account-id' – when I swapped that with the actual numerical account ID, I got the same results for the 'maltfield' user
- ...but when I reverted back to the original creds (for the 'backup-cron' user), I got a different error saying that the job ID was not found. Is it possible that there's a very narrow window where the job needs to be queried? This is impossible!
user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS user@disp5233:~$
- in the WUI, I switched to the s3 Glacier service https://us-west-2.console.aws.amazon.com/glacier/home?region=us-west-2#/vaults
- I clicked-on the 'deleteMeIn2020' bucket, and it still says the last inventory date is "August 1, 2018, 02:41:31 (UTC-05:00)". wtf?
- well a query for all jobs shows no jobs
user@disp5233:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": []
}
user@disp5233:~$
- well, I initiated a new inventory job AGAIN
user@disp5233:~$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"location": "/REDACTED/vaults/deleteMeIn2020/jobs/tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN",
"jobId": "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN"
}
user@disp5233:~$
user@disp5233:~$ aws glacier list-jobs --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"JobList": [
{
"JobId": "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN",
"Action": "InventoryRetrieval",
"VaultARN": "arn:aws:glacier:us-west-2:REDACTED:vaults/deleteMeIn2020",
"CreationDate": "2024-12-12T18:07:24.156Z",
"Completed": false,
"StatusCode": "InProgress",
"InventoryRetrievalParameters": {
"Format": "JSON"
}
}
]
}
user@disp5233:~$
user@disp5233:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN" ./output.json
An error occurred (InvalidParameterValueException) when calling the GetJobOutput operation: The job is not currently available for download: tnLbYFxINicZDRYy06ri1dlxiVX8wVKLMKrQmiyatMuhfs26ggw8o_nMzc2VGpWjF8Z9IDqnXclrdq9B3pFc2X5n99qN
user@disp5233:~$
- I guess now we wait some days and hopefully it doesn't disappear again before we have a chance to check up on it?
- ...
- returning to store.opensourceecology.org, the theme issues are now resolved
- the site is still pretty broken, but it was never really working anyway, so that's as good as it'll get
- what does remain, however, is finding a replacement for our now-unavailable security plugins ' rename-wp-login' and 'force-strong-passwords'
- currently the login is exposed on the normal wp-login.php, which isn't good https://store.opensourceecology.org/wp-login.php
- I recently setup a wordpress website which uses two plugins
- melapress-login-security https://wordpress.org/plugins/melapress-login-security/
- wp-2fa https://wordpress.org/plugins/wp-2fa/
- it looks like I did install wps-hide-login, but I ended up deactivating it because 'melapress-login-security' includes this (as well as forcing strong passwords, feeding two birds with one scone)
- one thing I didn't like so much about the 'wp-2fa' plugin was that it doesn't have a "relaxed" mode.
- Currently we're using a 'google-authenticator' plugin https://wordpress.org/plugins/google-authenticator/
- this plugin hasn't been updated in 2 years, but it has >30,000 active installations, and it appears to be working fine still
- what I really like about it is that it has a checkbox for "Relaxed mode allows for more time drifting on your phone clock (±4 min)."
- this setting is found on the Users -> Profile
- in my experience, 2FA tends to cause security issues with websites due to loss of availability because users constantly get locked out due to time sync issues on their phone. For our purposes, I think the happy-medium between security-and-convenience is found where several codes in the past (and future) few minutes are ok. I think there's very little risk in this, but it's surprising how few TOTP implementations allow this
- oh, actually, it's been years since they've done a release, but the last commit was only 11 months ago, so that's not terrible https://github.com/ivankruchkoff/google-authenticator
- I'm not a huge fan of this plugin otherwise; it's not actively updated. But if I could find an alternative that's equally as lightweight
- I figured I'd go ahead now and open a feature request to add a "relaxed" mode to the alternative 'wp-2fa' that I've already tested, but I couldn't fucking find the forge!
- I opened a support ticket on their wordpress plugin page asking for a link to their vcs forge (eg github) https://wordpress.org/support/topic/where-is-the-vcs-forge/
- anyway, I'll go ahead and 3TOFU all of them, but now I'm leaning towards not using melapress' 'wp-2fa', but yes using melapress' 'melapress-login-security'
- in poking around melapress, I saw they also have a plugin for hCaptcha = 'advanced-nocaptcha-recaptcha' https://wordpress.org/plugins/advanced-nocaptcha-recaptcha/
- I hate reCAPTCHA and cloudflare's often fails, but I've had pretty good experiences with hCaptcha. This supports all of them
- previously I've used 'hcaptcha-for-forms-and-more' and had a good experience, so I'll add both to the 3TOFU https://wordpress.org/plugins/hcaptcha-for-forms-and-more/
- in the site where I used 'hcaptcha-for-forms-and-more' before, they used 'wpforms-lite', so I'll add that too https://wordpress.org/plugins/wpforms-lite/
- looks like OSE is using 'contact-form-7', which is actively developed still https://wordpress.org/plugins/contact-form-7/
- despite the name, I found that 'wpforms-lite' actually is *less* lightweight than 'contact-form-7' in benchmark tests
- contact-form-7 is a bit more popular (10+ million active installs vs 6+ million), but 'wpforms-lite' has a bit better reviews (4.9/5 vs 4/5)
- oh, it looks like contact-form-7 only supports reCAPTCHA whereas wpforms-lite has built-in support for hCaptcha and cf too
- I'll go ahead and add them both to the 3TOFU just incase we need it, but it's probably best to stick to contact-form-7
- let's to a new 3TOFU of some candidate plugins, after which I can demo them and see what we want to use. I'm also going to add some others that we may or may not use
- wps-hide-login
- melapress-login-security
- activitypub
- aurora-heatmap
- raw-html
- related-posts-by-taxonomy
- smart-slider-3
- spam-destroyer
- coinpayments-payment-gateway-for-woocommerce
- woocommerce-gateway-stripe
- wpfront-notification-bar
- wordpress-seo
- wp-pgp-encrypted-emails
- woo-multi-currency
- woocommerce-multilingual
- include-mastodon-feed
- bulk-media-register
- enable-media-replace
- regenerate-thumbnails
- wp-qrcode
- wp-pgp-encrypted-emails
- woo-multi-currency
- woocommerce-multilingual
- include-mastodon-feed
- wp-2fa
- advanced-nocaptcha-recaptcha
- hcaptcha-for-forms-and-more
- leaflet-map
- extensions-leaflet-map
- wpforms-lite
- in the past few months, I recently had to do an upgrade of my personal wordpress site
- to do this securely, I wrote a script that spits-out a script that can be used for 3TOFU of all the themes & plugins installed
user@host:~$ cat /usr/local/bin/wordpress_3tofu.sh
#!/bin/bash
#set -x
################################################################################
# File: wordpress_3tofu.sh
# Version: 0.1
# Purpose: Generates a list of 3TOFU commands to verfiy the latest versions of
# all currently-installed wordpress themes and plugins
# * https://tech.michaelaltfied.net/3tofu
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-28
# Updated: 2024-09-28
################################################################################
################################################################################
# SETTINGS #
################################################################################
################################################################################
# FUNCTIONS #
################################################################################
################################################################################
# MAIN BODY #
################################################################################
#####################
# DECLARE VARIABLES #
#####################
# space-delimited list of URLs for 3TOFU
REMOTE_FILES=''
CURL=$(which curl) || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
########
# CORE #
########
###########
# PLUGINS #
###########
# get list of plugins
plugins=$(sudo -u wp -i wp --path=/var/www/html/wordpress/htdocs --format=csv plugin list | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")
##########
# THEMES #
##########
# get list of themes
themes=$(sudo -u wp -i wp --path=/var/www/html/wordpress/htdocs --format=csv theme list | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")
###################
# OUTPUT COMMANDS #
###################
# HEADER
cat <<EOF
################################################################################
# File: 3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
# days using 3 distinct networks exiting from 3 distinct countries
#
# For more info on 3TOFU (and why this is important), see:
# * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: $(date -u --rfc-3339=seconds)
################################################################################
EOF
cat <<'EOF'
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
REMOTE_FILES=""
WARNINGS=""
# in tails, we must torify
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
EOF
# CORE
cat <<'EOF'
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
EOF
# PLUGINS
echo "plugins='${plugins}'"
cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
EOF
# THEMES
echo "themes='${themes}'"
cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
EOF
# WARNINGS
cat <<'EOF'
echo -e "${WARNINGS}"
echo
EOF
# DOWNLOAD PAYLOADS
cat <<'EOF'
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
EOF
# FINISH
cat <<'EOF'
# checksum
date -u +"%Y-%m-%d"
sha256sum *
EOF
exit 0
user@host:~$
- this script works fine on my personal server, which uses a single wp multisite install, but it needs to be updated for OSE's multiple independent wordpress installs
root@hetzner3 /usr/local/bin # sudo -u wp -i wp --path=/var/www/html/wordpress/htdocs --format=csv plugin list | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " " -bash: -vE: command not found Error: This does not seem to be a WordPress installation. The used path is: /var/www/html/wordpress/htdocs/ Pass --path=`path/to/wordpress` or run `wp core download`. root@hetzner3 /usr/local/bin #
- ignoring the spam of errors, this does work
root@hetzner3 /usr/local/bin # sudo -u wp -i wp --path="/var/www/html/store.opensourceecology.org/htdocs" --format=csv plugin list | grep -vE '^name,' | cut -d, -f1 | tr "\n" " " PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 PHP Warning: wp_update_plugins(): An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/forums/">support forums</a>. (WordPress could not establish a secure connection to WordPress.org. Please contact your server administrator.) in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/functions.php on line 6085 Warning: wp_update_plugins(): An unexpected error occurred. Something may be wrong with WordPress.org or this server’s configuration. If you continue to have problems, please try the <a href="https://wordpress.org/support/forums/">support forums</a>. (WordPress could not establish a secure connection to WordPress.org. Please contact your server administrator.) in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/functions.php on line 6085 akismet classic-editor contact-form-7 google-authenticator-encourage-user-activation google-authenticator hello meta-box ssl-insecure-content-fixer vcaching woocommerce coingate-for-woocommerce root@hetzner3 /usr/local/bin #
- we can throw it in a loop that dynamically gets all of the sites
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
echo $wp_docroot
done
- here's an execution (currently we've restored only 1 wordpress vhost on hetzner3)
root@hetzner3 /usr/local/bin # wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
echo $wp_docroot
done
/var/www/html/store.opensourceecology.org/htdocs
root@hetzner3 /usr/local/bin #
- we can update this loop to get all the plugins for every site as follows
plugins=''
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
echo $wordpress_site;
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
echo $wp_docroot
echo $vhost_dir
plugins="${plugins} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list 2>/dev/null | grep -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done
echo ${plugins}
- here's an execution
root@hetzner3 /usr/local/bin # plugins=''
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
echo $wordpress_site;
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
echo $wp_docroot
echo $vhost_dir
plugins="${plugins} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list 2>/dev/null | grep -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done
echo ${plugins}
/var/www/html/store.opensourceecology.org/htdocs/wp-content
/var/www/html/store.opensourceecology.org/htdocs
/var/www/html/store.opensourceecology.org
akismet classic-editor contact-form-7 google-authenticator-encourage-user-activation google-authenticator hello meta-box ssl-insecure-content-fixer vcaching woocommerce coingate-for-woocommerce
root@hetzner3 /usr/local/bin #
- ok, here's the updated script
maltfield@hetzner3:~$ cat /usr/local/bin/wordpress_3tofu.sh
#!/bin/bash
#set -x
################################################################################
# File: wordpress_3tofu.sh
# Version: 0.2
# Purpose: Generates a list of 3TOFU commands to verfiy the latest versions of
# all currently-installed wordpress themes and plugins
# * https://tech.michaelaltfied.net/3tofu
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-28
# Updated: 2024-12-12
################################################################################
################################################################################
# SETTINGS #
################################################################################
################################################################################
# FUNCTIONS #
################################################################################
################################################################################
# MAIN BODY #
################################################################################
#####################
# DECLARE VARIABLES #
#####################
# space-delimited list of URLs for 3TOFU
REMOTE_FILES=''
CURL=$(which curl) || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
########
# CORE #
########
###########
# PLUGINS #
###########
# get list of plugins
plugins=''
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
plugins="${plugins} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv plugin list 2>/dev/null | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done
##########
# THEMES #
##########
# get list of themes
themes=''
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
themes="${themes} $(sudo -u wp -i wp --path="${wp_docroot}" --format=csv theme list 2>/dev/null | $GREP -vE '^name,' | cut -d, -f1 | tr "\n" " ")"
done
###################
# OUTPUT COMMANDS #
###################
# HEADER
cat <<EOF
################################################################################
# File: 3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
# days using 3 distinct networks exiting from 3 distinct countries
#
# For more info on 3TOFU (and why this is important), see:
# * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: $(date -u --rfc-3339=seconds)
################################################################################
EOF
cat <<'EOF'
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
REMOTE_FILES=""
WARNINGS=""
# in tails, we must torify
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
EOF
# CORE
cat <<'EOF'
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
EOF
# PLUGINS
echo "plugins='${plugins}'"
cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
EOF
# THEMES
echo "themes='${themes}'"
cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
EOF
# WARNINGS
cat <<'EOF'
echo -e "${WARNINGS}"
echo
EOF
# DOWNLOAD PAYLOADS
cat <<'EOF'
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
EOF
# FINISH
cat <<'EOF'
# checksum
date -u +"%Y-%m-%d"
sha256sum *
EOF
exit 0
maltfield@hetzner3:~$
- and an execution shows it's working (again, this script outputs another script that we'll copy-and-paste onto some VM for 3TOFU
maltfield@hetzner3:~$ sudo /usr/local/bin/wordpress_3tofu.sh
################################################################################
# File: 3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
# days using 3 distinct networks exiting from 3 distinct countries
#
# For more info on 3TOFU (and why this is important), see:
# * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-12-12 21:14:30+00:00
################################################################################
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
REMOTE_FILES=""
WARNINGS=""
# in tails, we must torify
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
plugins=' akismet classic-editor contact-form-7 google-authenticator-encourage-user-activation google-authenticator hello meta-box ssl-insecure-content-fixer vcaching woocommerce coingate-for-woocommerce '
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
themes=' oshin storefront twentyeleven twentyfifteen twentyfourteen twentynineteen twentyseventeen twentysixteen twentyten twentythirteen twentytwelve '
echo -ne "INFO: Determining Latest Version of Wordpress Themes \n\t"
for theme in $themes; do
echo -n '. '
json=$($CURL -s "https://api.wordpress.org/themes/info/1.2/?action=theme_information&slug=${theme}")
latest_version=$(echo $json | $JQ -r .version)
if [ "${latest_version}" = "null" ]; then
error=$(echo $json | $JQ -r .error);
description=$(echo $json | $JQ -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download theme ${theme}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} $(echo $json | $JQ -r ".download_link")"
fi
done
echo
echo -e "${WARNINGS}"
echo
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
# checksum
date -u +"%Y-%m-%d"
sha256sum *
maltfield@hetzner3:~$
- ok, so that's great in the future when we need to 3TOFU updates to plugins & themes that are found already-installed on the server, but let's hack it with a manually-defined list of plugins (above) for 3TOFU of these new plugins
#!/bin/bash
#set -x
################################################################################
# File: wordpress_3tofu.sh
# Version: 0.1
# Purpose: Generates a list of 3TOFU commands to verfiy the latest versions of
# all currently-installed wordpress themes and plugins
# * https://tech.michaelaltfied.net/3tofu
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-28
# Updated: 2024-09-28
################################################################################
################################################################################
# SETTINGS #
################################################################################
################################################################################
# FUNCTIONS #
################################################################################
################################################################################
# MAIN BODY #
################################################################################
#####################
# DECLARE VARIABLES #
#####################
# space-delimited list of URLs for 3TOFU
REMOTE_FILES=''
CURL=$(which curl) || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
########
# CORE #
########
###########
# PLUGINS #
###########
# get list of plugins
plugins="wps-hide-login melapress-login-security activitypub aurora-heatmap raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite"
###################
# OUTPUT COMMANDS #
###################
# HEADER
cat <<EOF
################################################################################
# File: 3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
# days using 3 distinct networks exiting from 3 distinct countries
#
# For more info on 3TOFU (and why this is important), see:
# * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: $(date -u --rfc-3339=seconds)
################################################################################
EOF
cat <<'EOF'
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
REMOTE_FILES=""
WARNINGS=""
# in tails, we must torify
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
EOF
# CORE
cat <<'EOF'
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
EOF
# PLUGINS
echo "plugins='${plugins}'"
cat <<'EOF'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
EOF
# WARNINGS
cat <<'EOF'
echo -e "${WARNINGS}"
echo
EOF
# DOWNLOAD PAYLOADS
cat <<'EOF'
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
EOF
# FINISH
cat <<'EOF'
# checksum
date -u +"%Y-%m-%d"
sha256sum *
EOF
exit 0
- I created ^ this script on some DispVM and executed it; it spat-out our 3TOFU script
user@disp1594:~$ vim wordpress_3tofu.sh
user@disp1594:~$
user@disp1594:~$ chmod +x wordpress_3tofu.sh
user@disp1594:~$ ./wordpress_3tofu.sh
################################################################################
# File: 3tofu.sh
# Purpose: Execute these commands on 3 distinct machines (or VMs) on 3 distinct
# days using 3 distinct networks exiting from 3 distinct countries
#
# For more info on 3TOFU (and why this is important), see:
# * https://tech.michaelaltfied.net/3tofu
#
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-12-12 20:45:59+00:00
################################################################################
JQ=$(which jq) || (echo "ERROR: Cannot find 'jq'"; exit 1)
CURL="$(which curl) --retry 5 --retry-all-errors" || (echo "ERROR: Cannot find 'curl'"; exit 1)
GREP=$(which grep) || (echo "ERROR: Cannot find 'grep'"; exit 1)
REMOTE_FILES=""
WARNINGS=""
# in tails, we must torify
if "`whoami`" == "amnesia" ; then
CURL="/usr/bin/torify ${CURL}"
PYTHON="/usr/bin/torify ${PYTHON}"
fi
tmpDir=`mktemp -d`
pushd "${tmpDir}"
# first get some info about our internet connection
${CURL} -s https://ifconfig.co/country | head -n1
${CURL} -s https://check.torproject.org | grep Congratulations | head -n1
# and today's date
date -u +"%Y-%m-%d"
echo "INFO: Determining Latest Version of Wordpress Core"
json=$($CURL -s "https://api.wordpress.org/core/version-check/1.7/")
REMOTE_FILES="${REMOTE_FILES} $(echo "${json}" | $JQ -r '[.offers[]|select(.response=="upgrade")][0].download')"
plugins='wps-hide-login melapress-login-security activitypub aurora-heatmap raw-html related-posts-by-taxonomy smart-slider-3 spam-destroyer coinpayments-payment-gateway-for-woocommerce woocommerce-gateway-stripe wpfront-notification-bar wordpress-seo wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed bulk-media-register enable-media-replace regenerate-thumbnails wp-qrcode wp-pgp-encrypted-emails woo-multi-currency woocommerce-multilingual include-mastodon-feed wp-2fa advanced-nocaptcha-recaptcha hcaptcha-for-forms-and-more leaflet-map extensions-leaflet-map wpforms-lite'
echo -ne "INFO: Determining Latest Version of Wordpress Plugins \n\t"
for plugin in $plugins; do
echo -n '. '
json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/${plugin}.json)
latest_version=$(cat plugin.json | jq -r .version)
url=$(cat plugin.json | jq -r ".versions.\"${latest_version}\"")
if [ "${url}" = "null" ]; then
error=$(cat plugin.json | jq -r .error);
description=$(cat plugin.json | jq -r .description);
WARNINGS="${WARNINGS}\n\nWARNING: Failed to download plugin ${plugin}"
WARNINGS="${WARNINGS}\n\t$error"
WARNINGS="${WARNINGS}\n\t$description"
else
REMOTE_FILES="${REMOTE_FILES} ${url}"
fi
done
echo
echo -e "${WARNINGS}"
echo
# get the file
for file in ${REMOTE_FILES}; do
echo "${file}"
${CURL} --progress-bar -O "${file}"
done
# checksum
date -u +"%Y-%m-%d"
sha256sum *
user@disp1594:~$
- I copied and pasted ^ that script into a whonix dispVM
- here's our TOFU 1/3 (Tor, exit in Poland)
Congratulations. This browser is configured to use Tor. 2024-12-12 INFO: Determining Latest Version of Wordpress Core INFO: Determining Latest Version of Wordpress Plugins . . . . . . . . . jq: error (at <stdin>:0): Cannot index array with string "1.0.17" . . . . . . . . . . . . . . . . . . . . . WARNING: Failed to download plugin woo-multi-currency null null WARNING: Failed to download plugin woo-multi-currency null null https://downloads.wordpress.org/release/wordpress-6.7.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wps-hide-login.1.9.17.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/melapress-login-security.2.0.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/activitypub.4.4.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/aurora-heatmap.1.7.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/raw-html.1.6.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/related-posts-by-taxonomy.2.7.6.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/smart-slider-3.3.5.1.25.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/spam-destroyer.2.1.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-gateway-stripe.9.0.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wpfront-notification-bar.3.4.2.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wordpress-seo.24.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/bulk-media-register.1.40.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/enable-media-replace.4.1.5.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/regenerate-thumbnails.3.1.6.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-qrcode.1.1.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-pgp-encrypted-emails.0.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/woocommerce-multilingual.5.3.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/include-mastodon-feed.1.9.9.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wp-2fa.2.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/advanced-nocaptcha-recaptcha.7.5.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/hcaptcha-for-forms-and-more.4.8.0.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/leaflet-map.3.4.1.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/extensions-leaflet-map.4.4.zip ######################################################################### 100.0% https://downloads.wordpress.org/plugin/wpforms-lite.1.9.2.3.zip ######################################################################### 100.0% 2024-12-12 8b1f9a708838b8710b4198da1116689197e0a6134e0a1a5e786500576383034f activitypub.4.4.0.zip 101f645a8f4becdf0394c27195679fe6d134063fde6bd851dc1d57217db5e0e9 advanced-nocaptcha-recaptcha.7.5.0.zip 873928dd3e940064f5dcac8b74335a9760823147388f472bb755ce5a804eaf53 aurora-heatmap.1.7.0.zip 5dc1fff3c3e664774ea51d52477e28c060e0b6733a47c6fb5db800eba3a4ea0f bulk-media-register.1.40.zip ad98e83a3bce28612025010d5bca77dd2d29f1df539f2667865d6d959f67e3e0 enable-media-replace.4.1.5.zip 1a53bdcd1ddb160d5807dc17a0f9e474402e22c899b3a9af486c9d5f0d2c4b36 extensions-leaflet-map.4.4.zip 27f1ab1e3f5274335d48d0cadaabdef98284880b0324771890d36a1f562fb44a hcaptcha-for-forms-and-more.4.8.0.zip bb0e885969df637767d64d02504d8defb1184db24cd0ade0111ef55ef63c81b9 include-mastodon-feed.1.9.9.zip 13d906d4677dc3da617752fbe9e7540f0bf84128c0fae43598a10b876dac4217 leaflet-map.3.4.1.zip fd1593eefe2fa546926ce0765e7d9944e24c1aca0f9cf2606d3136f4b60cb1b5 melapress-login-security.2.0.1.zip 923f38397284dceda1028a12c01e78bde22e0d0fecfdd8b95e52cfcc04e47342 plugin.json f2cfaf226788dddd8744e723fe1ef53ef0984f956c4fa2678f932f0d8b72116c raw-html.1.6.4.zip 757f29991412ef63a099c4fe77a921d23b51097ddb207dff669fbf24ace6a7d6 regenerate-thumbnails.3.1.6.zip 4f0e6f6505b8eb39b53dd971e8dba8fe98c65a56a7bb24443f4a513c7940f193 related-posts-by-taxonomy.2.7.6.zip ebd87841f73bb7946216ae4827a413dcc97fc5094cee2f8ddb6dea7eff356358 smart-slider-3.3.5.1.25.zip 41bcae0e3cd94b73d7b5761527e68acb9111cb28080dd68f2f83a82cfd87f210 spam-destroyer.2.1.4.zip aa52f9a4c8bbe856fe045e5c76ffedae3573374ee43435de78e1561d8e0169a9 woocommerce-gateway-stripe.9.0.0.zip fbe62fc4ec4b91915024c126d9b86b3798c283f60d95435f3e6e1226ddd722aa woocommerce-multilingual.5.3.9.zip 75f4e9cb71e583ca3f8b19691b5754adb9c981580762137f82443e1eec468f9c wordpress-6.7.1.zip f9ce7a98840dd4bf490d955320a68ac553c767ba7f0eeae6e4f067be5a927ef3 wordpress-seo.24.0.zip feda19ad71ea22abe4dbcff422f6e0e6c8315f26a7d246099967a5eea17b4d38 wp-2fa.2.8.0.zip 130ba1a4f2396a8e183b8ce732c9bc8a3cf6698890f6f216550188e78e082fda wpforms-lite.1.9.2.3.zip 6e1d71809f4421463fc19c5c119c5e49788cd3676b730f7980e3dcd209520a1c wpfront-notification-bar.3.4.2.zip e3cb9db45795a8caed13e00414ce7f43d2bb517a35b88cda98ad91b6871b46e2 wp-pgp-encrypted-emails.0.8.0.zip e50735bcda4e85df1e522fda113ae24fd973f000e75154472544d4bcf51491f1 wp-qrcode.1.1.1.zip bedfe5b456f5a5b3b6d4b29dd6577f6b8492f4594a192678555691e8403a56d7 wps-hide-login.1.9.17.1.zip user@host:/tmp/user/1000/tmp.THquHNCCMu$
- well that looks good, except for the failure of the 'woo-multi-currency' plugin
- looks like the URL for the "latest version" is missing
user@host:~$ json=$(curl -so plugin.json https://api.wordpress.org/plugins/info/1.0/woo-multi-currency.json)
user@host:~$ echo $json
user@host:~$ ls
Desktop Downloads Pictures Public Videos
Documents Music plugin.json Templates
user@host:~$
user@host:~$ cat plugin.json | jq -r .version
2.2.4
user@host:~$
user@host:~$ cat plugin.json | jq -r ".versions.2.2.4"
jq: error: Invalid numeric literal at EOF at line 1, column 6 (while parsing '.2.2.4') at <top-level>, line 1:
.versions.2.2.4
jq: error: syntax error, unexpected LITERAL, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
.versions.2.2.4
jq: 2 compile errors
user@host:~$
user@host:~$ cat plugin.json | jq -r ".versions"
{
"2.1.12": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.12.zip",
"2.1.14": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.14.zip",
"2.1.7": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.7.zip",
"2.1.8": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.8.zip",
"2.1.9": "https://downloads.wordpress.org/plugin/woo-multi-currency.2.1.9.zip"
}
user@host:~$
- whatever, this plugin wasn't so important. Let's just ignore & skip this plugin
- by next week we should have all 3 TOFUs. If they match, I'll go ahead and copy them to hetzner3, figure out which ones we want to use, and then finish off the store.opensourceecology.org migration script CHG steps
- ...
- I still have 1 remaining TODO item on the hetzner3 backups, which was to wait some weeks and then verify that the backups cron/lifecycle rules are working
- well, it's been >11 weeks since I setup backups on hetzner3 https://wiki.opensourceecology.org/wiki/Maltfield_Log/2024_Q3#Sun_Sep_22.2C_2024
- here's what we see in the bucket
root@hetzner3 ~ # sudo rclone lsl b2:ose-server-backups | grep hetzner3 2258493547 2024-12-10 07:29:09.905000000 daily_hetzner3_20241210_072828.tar.gpg 2266009707 2024-12-11 07:58:32.491000000 daily_hetzner3_20241211_075750.tar.gpg 2272696427 2024-12-12 07:25:19.985000000 daily_hetzner3_20241212_072443.tar.gpg 1782579309 2024-10-01 08:05:13.312000000 monthly_hetzner3_20241001_080447.tar.gpg 1986529389 2024-11-01 07:37:07.030000000 monthly_hetzner3_20241101_073631.tar.gpg 2195302509 2024-12-01 07:45:57.990000000 monthly_hetzner3_20241201_074518.tar.gpg 2104289388 2024-11-18 07:23:52.227000000 weekly_hetzner3_20241118_072314.tar.gpg 2153154668 2024-11-25 07:50:49.139000000 weekly_hetzner3_20241125_075009.tar.gpg 2202644588 2024-12-02 08:05:51.395000000 weekly_hetzner3_20241202_080510.tar.gpg 2251448428 2024-12-09 07:31:51.872000000 weekly_hetzner3_20241209_073110.tar.gpg root@hetzner3 ~ #
- so we've got 3 monthlies already. that's great
- we have the past 3 daily backups and nothing more. so that suggests that recent backups are working fine, and the lifecycle rules are too; great
- and we have 4 weeklies, which sounds correct; all looks great!
- ...
- another item I had pending on my TODO list was to verify munin after some time to ensure that the charts were getting populated with data
- obviously this server is basically idle, but any lines at all are fine
- apache looks good
- apache logs were actually empy on hetzner2 for some reason, and there was only two charts: apache processes and apache volume (both empty)
- on hetzner3, we have 3 charts: apache access (accesses per second, mostly zero with a couple spikes ), apache processes (it shows a solid 48-49 idle servers and a solid 100 "free slots" over the past few months), and apache volume (mostly zero with occasional spikes to ~2k-18k bytes per second)
- we have some data for nginx, but not as many charts as we used to have
- on hetzner2, it was called "webserver" whereas on hetzner3 it's called "nginx" for some reason
- on hetzner2, we had 4 charts: nginx requests, nginx requests, nginx status, ngingx status
- oh wait, no, that's the two charts duplicated; it's the same data.
- on hetzner3 we also have these 2 chats. They're near zero, but both have spikes up to ~0.2-1.2 requests per second and 1-9 connections per second
- our varnish charts on hetzner3 are also working
- we have 9 charts on hetzner2 and 8 charts on hetzner3
- looks like we're missing the "uptime" chart on hetzner3. I've read this can be useful just to monitor if varnish has some issue that causes it to restart its child processes, so it would probably be good to add that one, if trivial
- all the other charts have data except the "misbehavoiur" chart is all zeros. I'm guessing that's possibly because there are no misbehaviours. Maybe.
- curiously, this section on hetzner2 used to be called 'varnish4' whereas on hetzner3 it's called "webserver"
- I did notice that all of the mysql charts are empty on hetzner3, so we should probably investigate that too
- the "process info" charts are also all empty, so we should check on that
- well, I confirmed that the varnish_uptime plugin is in-place
root@hetzner3 /etc/munin/plugins # ls -lah | grep -i varnish lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_backend_traffic -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_bad -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_expunge -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_hit_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_memory_usage -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_objects -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_request_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_threads -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_transfer_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_uptime -> /usr/share/munin/plugins/varnish5_ root@hetzner3 /etc/munin/plugins #
- for comparison, here's what we have on hetzner2
[root@opensourceecology plugins]# ls -lah | grep -i varnish -rwxr-xr-x 1 root root 26K Mar 3 2018 varnish4_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_backend_traffic -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_backend_traffic.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_bad -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_bad.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_expunge -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_expunge.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_hit_rate -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_hit_rate.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_memory_usage -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_memory_usage.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_objects -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_objects.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_request_rate -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_request_rate.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_threads -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_threads.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_transfer_rates -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_transfer_rates.bak -> /usr/share/munin/plugins/varnish_ lrwxrwxrwx 1 root root 9 Mar 3 2018 varnish_uptime -> varnish4_ lrwxrwxrwx 1 root root 33 Mar 3 2018 varnish_uptime.bak -> /usr/share/munin/plugins/varnish_ [root@opensourceecology plugins]#
- oh, it looks like the new version that we have on hetzner3 has separated "varnish_uptime" into two distinct uptimes. my guess is that one is the parent and one is the child process
- hetzner2
[root@opensourceecology plugins]# grep -A3 'uptime' varnish_uptime
'uptime' => {
'title' => 'Varnish uptime',
'vlabel' => 'days',
'scale' => 'no',
'values' => {
'uptime' => {
'type' => 'GAUGE',
'cdef' => 'uptime,86400,/'
}
}
},
[root@opensourceecology plugins]#
- hetzner3
root@hetzner3 /etc/munin/plugins # grep -A3 uptime varnish_uptime
'main_uptime' => {
'type' => 'MAIN',
'title' => 'Varnish Child uptime',
'vlabel' => 'days',
'scale' => 'no',
'values' => {
'uptime' => {
'type' => 'GAUGE',
'cdef' => 'uptime,86400,/'
},
}
},
'mgt_uptime' => {
'type' => 'MGT',
'title' => 'Varnish Management uptime',
'vlabel' => 'days',
'scale' => 'no',
'values' => {
'uptime' => {
'type' => 'GAUGE',
'cdef' => 'uptime,86400,/'
},
}
},
You have new mail in /var/mail/root
root@hetzner3 /etc/munin/plugins #
- I updated the symlinks to include these two charts on hetzner3's muni plugins dir
root@hetzner3 /etc/munin/plugins # ls -lah varnish_uptime lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_uptime -> /usr/share/munin/plugins/varnish5_ root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # rm -f varnish_uptime root@hetzner3 /etc/munin/plugins # ln -s /usr/share/munin/plugins/varnish5_ varnish_main_uptime root@hetzner3 /etc/munin/plugins # ln -s /usr/share/munin/plugins/varnish5_ varnish_mgt_uptime root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # ls -lah | grep -i varnish_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_backend_traffic -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_bad -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_expunge -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_hit_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Dec 13 00:03 varnish_main_uptime -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_memory_usage -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Dec 13 00:03 varnish_mgt_uptime -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_objects -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_request_rate -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_threads -> /usr/share/munin/plugins/varnish5_ lrwxrwxrwx 1 root root 34 Sep 25 01:47 varnish_transfer_rate -> /usr/share/munin/plugins/varnish5_ root@hetzner3 /etc/munin/plugins # root@hetzner3 /etc/munin/plugins # service munin-node restart root@hetzner3 /etc/munin/plugins #
- a manual test run looks good
root@hetzner3 /etc/munin/plugins # munin-run varnish_main_uptime uptime.value 6635471 root@hetzner3 /etc/munin/plugins # munin-run varnish_mgt_uptime uptime.value 6635477 root@hetzner3 /etc/munin/plugins #
- I refreshed the munin WUI and, yep, I now see two additional charts. One's named "Varnish Child Uptime" and the other "Varnish Management Uptime". Perfect.
- varnish was the lowest hanging fruit, but the most important missing chart is mysql. Well, all our mysql charts are missing. what gives?
- first, our plugins dirs look very different on both servers
- hetzner2
[root@opensourceecology plugins]# ls -lah | grep -i mysql lrwxrwxrwx 1 root root 31 Oct 8 2019 bin_relay_log -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 commands -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 connections -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 files_tables -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_bpool -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_bpool_act -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_insert_buf -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_io -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_io_pend -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_log -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_rows -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_semaphores -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 innodb_tnx -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 myisam_indexes -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 mysql_ -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 38 Oct 8 2019 mysql_queries -> /usr/share/munin/plugins/mysql_queries lrwxrwxrwx 1 root root 31 Oct 8 2019 network_traffic -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 28 Oct 8 2019 ps_mysqld -> /usr/share/munin/plugins/ps_ lrwxrwxrwx 1 root root 31 Oct 8 2019 qcache -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 qcache_mem -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 replication -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 select_types -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 slow -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 sorts -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 table_locks -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 31 Oct 8 2019 tmp_tables -> /usr/share/munin/plugins/mysql_ [root@opensourceecology plugins]#
- hetzner3
root@hetzner3 /etc/munin/plugins # ls -lah | grep mysql lrwxrwxrwx 1 root root 31 Sep 25 01:47 mysql_ -> /usr/share/munin/plugins/mysql_ lrwxrwxrwx 1 root root 36 Sep 25 01:47 mysql_bytes -> /usr/share/munin/plugins/mysql_bytes lrwxrwxrwx 1 root root 37 Sep 25 01:47 mysql_innodb -> /usr/share/munin/plugins/mysql_innodb lrwxrwxrwx 1 root root 42 Sep 25 01:47 mysql_isam_space_ -> /usr/share/munin/plugins/mysql_isam_space_ lrwxrwxrwx 1 root root 38 Sep 25 01:47 mysql_queries -> /usr/share/munin/plugins/mysql_queries lrwxrwxrwx 1 root root 42 Sep 25 01:47 mysql_slowqueries -> /usr/share/munin/plugins/mysql_slowqueries lrwxrwxrwx 1 root root 38 Sep 25 01:47 mysql_threads -> /usr/share/munin/plugins/mysql_threads root@hetzner3 /etc/munin/plugins #
- there's a lot of different files here, but let's just test the "mysql_queries" one
- hetzner2
[root@opensourceecology plugins]# munin-run mysql_queries delete.value 17433 insert.value 20779 replace.value 206033 select.value 94566392 update.value 39021 cache_hits.value 0 [root@opensourceecology plugins]#
- hetzner3
root@hetzner3 /etc/munin/plugins # munin-run mysql_queries mysqladmin: connect to server at 'localhost' failed error: 'Access denied for user 'munin'@'localhost' (using password: NO)' You have new mail in /var/mail/root root@hetzner3 /etc/munin/plugins #
- ok, it's an auth error
- if I check the munin configs, then it's pretty obvious. I have a password defined for hetzner2 and nothing for hetzner3. I imagine a password might not be necessary if we allow passwordless auth from localhost or something, but I guess I never set that up?
- hetzner2
[root@opensourceecology plugins]# cat ../plugin-conf.d/zzz-ose # ose-specific configs go here per this doc # * http://guide.munin-monitoring.org/en/latest/plugin/use.html#configuring [nginx_wiki.opensourceecology.org_*] env.url https://wiki.opensourceecology.org/nginx_status env.graph_title graph title env.graph_info graph info goes here [nginx_www.opensourceecology.org_*] env.url https://www.opensourceecology.org/nginx_status [mysql*] user root group wheel env.mysqlopts -u munin_user -pREDACTED [multips_memory] env.names varnishd mysqld httpd varnishlog systemd-journal rsyslogd b2 nginx munin munin-node ssh sshd openvpn tuned ossec-analysisd bash vim screen tail gpg gpg2 polkitd tuned [root@opensourceecology plugins]#
- hetzner3
root@hetzner3 /etc/munin/plugins # cat ../plugin-conf.d/zzz-myconf # Ansible managed ################################################################################ # File: zzz-myconf # Version: 0.1 # Purpose: Munin custom config # we set custom munin configs in 'zzz-myconf' per this doc # * http://guide.munin-monitoring.org/en/latest/plugin/use.html#configuring # Author: Michael Altfield <michael@michaelaltfield.net> # Created: 2024-09-14 # Updated: 2024-09-14 ################################################################################ [nginx_*] env.url http://127.0.0.1/nginx_status [apache_*] env.url http://127.0.0.1:%d/server-status?auto env.ports 8000 [mysql*] env.mysqlopts -u munin env.mysqluser munin [multips_memory] env.names mysqld apache2 cache-main ossec-analysisd wazuh-db ossec-syscheckd munin-node munin-html munin-update nginx ssh sshd pickup journalctl trivial-rewrite systemd-journal varnishd wazuh-modulesd unattended-upgrades bash sudo tcpdump varnishlog ossec-remoted systemd-logind su dhclient ossec-authd ossec-execd ossec-logcollector ossec-maild ossec-monitord screen (sd-pam) systemd cleanup qmgr tlsmgr rewrite bounce defer trace verify flush proxymap proxywrite smtp relay showq error retry discard local virtual lmtp anvil scache submission grep tail rsyslogd systemd-udevd dbus-daemon gpg gpg2 cleanMega backup.sh rclone b2 chown chmod tar rm mv python agetty sh qemu-ga gpg-agent awstats.pl openvpn tuned vim polkitd [varnish*] user root env.varnishstat /usr/bin/varnishstat [proc] env.procname mysqld apache2 root@hetzner3 /etc/munin/plugins #
- checking my logs on the wiki, it looks like I first
Wed Dec 11, 2024
- Catarina responded to my mail from yesterday affirming that I should reset the unused store.opensourceecology.org wordpress site to some free wordpress theme for now, and use the license that would otherwise be used for it for www.opensourceecology.org instead
- I updated my /etc/hosts to point to the hetzner3 server for store.opensourceecology.org
- I checked http://store.opensourceecology.org/ in my web browser. it still just returns the generic wordpress critical error message
- I was able to login to the wp admin wui on the hetzner3 store site
- I changed the theme from 'oshine' to 'twenty seventeen'
- now when I load the store site's frontpage, the error has disappeared, but it still says "oshine" at the top and the body is lots of broken shorcodes
- if I go to Settings -> Reading, then I can see that the "Homepage" is set to "Home v37"
- I changed the store hoepage to "Sample Page"
- that's the best page I found. It still has broken tatsu shortcodes in the footer, but at last the word "oshine" doesn't appear anywhere.
- this site is temporary, and these actions are going to get overwritten at the time we actually rsync from live to hetzner3 during the migration, so I need a place to record each of these little changes as part of the migration playbook
- I created a wiki CHG "ticket" for the migration of store.opensourceecology.org from hetzner2 to hetzner3 https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_store_to_hetzner3
- I'm using wp-cli to more easily get the list of themes & plugins versions
- hetzner2
[root@opensourceecology ~]# sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ plugin list ... +------------------------------------------------+----------+--------+---------+ | name | status | update | version | +------------------------------------------------+----------+--------+---------+ | akismet | inactive | none | 4.1.1 | | be-gdpr | active | none | 1.1.2 | | be-portfolio-post | active | none | 1.1 | | classic-editor | inactive | none | 1.4 | | colorhub | active | none | 1.0.5 | | contact-form-7 | active | none | 5.1.1 | | force-strong-passwords | active | none | 1.8.0 | | google-authenticator | active | none | 0.48 | | google-authenticator-encourage-user-activation | active | none | 0.2 | | hello | inactive | none | 1.7.1 | | masterslider | active | none | 3.2.7 | | meta-box | active | none | 4.17.3 | | meta-box-conditional-logic | active | none | 1.6.4 | | meta-box-show-hide | active | none | 1.1.0 | | meta-box-tabs | active | none | 1.1.1 | | oshine-core | active | none | 1.3.7 | | oshine-modules | active | none | 2.2.9 | | redux-vendor-support | active | none | 1.0.1 | | rename-wp-login | active | none | 2.5.5 | | revslider | active | none | 5.4.8.3 | | ssl-insecure-content-fixer | active | none | 2.7.2 | | tatsu | active | none | 2.9.3.3 | | typehub | active | none | 1.4.3 | | vcaching | active | none | 1.6.9 | | woocommerce | active | none | 3.5.7 | | coingate-for-woocommerce | active | none | 1.2.2 | +------------------------------------------------+----------+--------+---------+ [root@opensourceecology ~]#
- hetzner3
+---------------+----------+--------+---------+---------------+---------------+ | name | status | update | version | update_versio | auto_update | | | | | | n | | +---------------+----------+--------+---------+---------------+---------------+ | akismet | inactive | none | 5.3.3 | | off | | classic-edito | inactive | none | 1.6.5 | | off | | r | | | | | | | contact-form- | active | none | 5.9.8 | | off | | 7 | | | | | | | google-authen | active | none | 0.2 | | off | | ticator-encou | | | | | | | rage-user-act | | | | | | | ivation | | | | | | | google-authen | active | none | 0.54 | | off | | ticator | | | | | | | hello | inactive | none | 1.7.1 | | off | | meta-box | active | none | 5.10.2 | | off | | ssl-insecure- | active | none | 2.7.2 | | off | | content-fixer | | | | | | | vcaching | active | none | 1.8.3 | | off | | woocommerce | active | none | 9.3.3 | | off | | coingate-for- | inactive | none | 2.1.1 | | off | | woocommerce | | | | | | +---------------+----------+--------+---------+---------------+---------------+ root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ plugin list <pre> # here's what I got so far <pre> update plugin 'akismet' from v4.1.1 to v5.3.3 uninstall plugin 'be-gdpr' uninstall plugin 'be-portfolio-post' update plugin 'classic-editor' from v1.4 to v1.6.5 uninstall plugin 'colorhub' update plugin 'contact-form-7' from v5.1.1 to v5.9.8 uninstall plugin 'force-strong-passwords' update plugin 'google-authenticator' from v0.48 to 0.54 uninstall plugin 'masterslider' update plugin 'meta-box' from v4.17.3 to v5.10.2 uninstall plugin 'meta-box-conditional-logic' uninstall plugin 'meta-box-show-hide' uninstall plugin 'meta-box-tabs' uninstall plugin 'oshine-core' uninstall plugin 'oshine-modules' uninstall plugin 'redux-vendor-support' uninstall plugin 'rename-wp-login' uninstall plugin 'revslider' uninstall plugin 'tatsu' uninstall plugin 'typehub' update plugin 'vaching' from v1.6.9 to v1.8.3 update plugin 'woocommerce' from v3.5.7 to v9.3.3 update plugin 'coingate-for-woocommerce' from v1.2.2 to v2.1.1
- but I think I'm going to have to add some other plugins. For example we need something to replace the now-defunct 'rename-wp-login' and 'forece-strong-passwords' plugins
- I also got the theme info on hetzner2 & hetzner3 to diff and add to the ticket
- hetzner2
[root@opensourceecology ~]# sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ theme list ... +-----------------+----------+--------+---------+ | name | status | update | version | +-----------------+----------+--------+---------+ | oshin | active | none | 6.6.4.4 | | storefront | inactive | none | 2.4.5 | | twentyeleven | inactive | none | 3.2 | | twentyfifteen | inactive | none | 2.4 | | twentyfourteen | inactive | none | 2.6 | | twentynineteen | inactive | none | 1.3 | | twentyseventeen | inactive | none | 2.1 | | twentysixteen | inactive | none | 1.9 | | twentyten | inactive | none | 2.8 | | twentythirteen | inactive | none | 2.8 | | twentytwelve | inactive | none | 2.9 | +-----------------+----------+--------+---------+ [root@opensourceecology ~]#
- hetzner3
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # sudo -u wp -i wp --path=/var/www/html/store.opensourceecology.org/htdocs/ theme list ... +---------------+----------+--------+---------+---------------+---------------+ | name | status | update | version | update_versio | auto_update | | | | | | n | | +---------------+----------+--------+---------+---------------+---------------+ | oshin | inactive | none | 7.2.1 | | off | | storefront | inactive | none | 4.6.0 | | off | | twentyeleven | inactive | none | 4.7 | | off | | twentyfifteen | inactive | none | 3.8 | | off | | twentyfourtee | inactive | none | 4.0 | | off | | n | | | | | | | twentyninetee | inactive | none | 2.9 | | off | | n | | | | | | | twentysevente | active | none | 3.7 | | off | | en | | | | | | | twentysixteen | inactive | none | 3.3 | | off | | twentyten | inactive | none | 4.2 | | off | | twentythirtee | inactive | none | 4.2 | | off | | n | | | | | | | twentytwelve | inactive | none | 4.3 | | off | +---------------+----------+--------+---------+---------------+---------------+ You have new mail in /var/mail/root root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- I then spent some time updating the CHG script
- I got stuck a bit on the rsync
- first I tried pulling on the hetzner3 from hetzner2, but it didn't like that we have to auth with a password as sudo on hetzner2
root@hetzner3 ~ # rsync -avvv --progress --rsync-path="sudo rsync" -e "ssh -p 32415" maltfield@138.201.84.223:32415${backupDir_hetzner2}/current/* ${backupDir_hetzner3}/current/
opening connection using: ssh -p 32415 -l maltfield 138.201.84.223 "sudo rsync" --server --sender -vvvlogDtpre.iLsfxCIvu . "32415/var/tmp/backups_for_migration_to_hetzner3/store.opensourceecology.org_20241212/current/*" (12 args)
sudo: no tty present and no askpass program specified
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [Receiver=3.2.7]
[Receiver] _exit_cleanup(code=12, file=io.c, line=231): about to call exit(12)
root@hetzner3 ~ #
- instead I decided to push on hetzner2 to hetzner3
- first I updated the hetzner2:/etc/hosts file to hard-code the IP address for "hetzner3", so this will go smother in the future
- ok, this worked
[maltfield@opensourceecology ~]$ rsync -av --progress --rsync-path="sudo rsync" -e "ssh -p 32415" ${backupDir_hetzner2}/current/* maltfield@hetzner3:${backupDir_hetzner3}/current/
sending incremental file list
mysqldump_store.opensourceecology.org.20241211.sql.bz2
1,406,610 100% 109.18MB/s 0:00:00 (xfr#1, to-chk=1/2)
store.opensourceecology.org_files.20241211.tar.gz
184,205,865 100% 92.26MB/s 0:00:01 (xfr#2, to-chk=0/2)
sent 185,658,021 bytes received 54 bytes 53,045,164.29 bytes/sec
total size is 185,612,475 speedup is 1.00
[maltfield@opensourceecology ~]$
Tue Dec 10, 2024
- last week we got an email from the let's encrypt expiry bot saying that our cert is going to expire soon
- we don't normally get these, so it stood-out
Hello, Your certificate (or certificates) for the names listed below will expire in 19 days (on 2024-12-24). Please make sure to renew your certificate before then, or visitors to your web site will encounter errors. We recommend renewing certificates automatically when they have a third of their total lifetime left. For Let's Encrypt's current 90-day certificates, that means renewing 30 days before expiration. See https://letsencrypt.org/docs/integration-guide/ for details. awstats.openbuildinginstitute.org awstats.opensourceecology.org fef.opensourceecology.org forum.opensourceecology.org microfactory.opensourceecology.org munin.opensourceecology.org openbuildinginstitute.org opensourceecology.org oswh.opensourceecology.org phplist.opensourceecology.org seedhome.openbuildinginstitute.org staging.opensourceecology.org store.opensourceecology.org wiki.opensourceecology.org www.openbuildinginstitute.org www.opensourceecology.org For details about when we send these emails, please visit: https://letsencrypt.org/docs/expiration-emails/ In particular, note that this reminder email is still sent if you've obtained a slightly different certificate by adding or removing names. If you've replaced this certificate with a newer one that covers more or fewer names than the list above, you may be able to ignore this message. For any questions or support, please visit: https://community.letsencrypt.org/ Unfortunately, we can't provide support by email. To learn more about the latest technical and organizational updates from Let's Encrypt, sign up for our newsletter: https://letsencrypt.org/opt-in/
- I checked the website in my browser and confirmed that I see a cert that says it's going to expire soon
Not Before Wed, 25 Sep 2024 17:04:09 GMT Not After Tue, 24 Dec 2024 17:04:08 GMT
- It looks like here's the cron file for the cert renewal
[root@opensourceecology log]# cat /etc/cron.d/letsencrypt # once a month, update our letsencrypt cert 20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log [root@opensourceecology log]#
- the latest log entry isn't dated, but the last time it ran, it appears to have decided that Dec 24 was too far away
The following certificates are not due for renewal yet: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem expires on 2024-12-24 (skipped) /etc/letsencrypt/live/opensourceecology.org/fullchain.pem expires on 2024-12-24 (skipped) No renewals were attempted.
- It looks like the last time cron mentions it being executed was in 2024-11-13, which makes sense
[root@opensourceecology log]# grep letsencrypt cron* [root@opensourceecology log]# zgrep letsencrypt cron* cron-20241113.gz:Nov 13 04:20:01 opensourceecology CROND[1103]: (root) CMD (/root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log) [root@opensourceecology log]#
- oh, checking the cron again, it is set to run only on the 13th of every month
- I checked our new server, which has this (installed by debian)
root@hetzner3 ~ # cat /etc/cron.d/certbot # /etc/cron.d/certbot: crontab entries for the certbot package # # Upstream recommends attempting renewal twice a day # # Eventually, this will be an opportunity to validate certificates # haven't been revoked, etc. Renewal will only occur if expiration # is within 30 days. # # Important Note! This cronjob will NOT be executed if you are # running systemd as your init system. If you are running systemd, # the cronjob.timer function takes precedence over this cronjob. For # more details, see the systemd.timer manpage, or use systemctl show # certbot.timer. SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew --no-random-sleep-on-renew root@hetzner3 ~ #
- wow, so certbot says that let's encrypt says that we should check twice per day. then once per month seems ridiculous
- I went ahead and changed it to do once per day
[root@opensourceecology cron.d]# vim letsencrypt [root@opensourceecology cron.d]# [root@opensourceecology cron.d]# cat letsencrypt # once a month, update our letsencrypt cert 20 4 * * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log [root@opensourceecology cron.d]#
- and I gave it a manual run
[root@opensourceecology cron.d]# /root/bin/letsencrypt/renew.sh ... Cert is due for renewal, auto-renewing... Plugins selected: Authenticator webroot, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Renewing an existing certificate for fef.opensourceecology.org and 11 more domains Performing the following challenges: http-01 challenge for awstats.opensourceecology.org http-01 challenge for fef.opensourceecology.org http-01 challenge for forum.opensourceecology.org http-01 challenge for microfactory.opensourceecology.org http-01 challenge for munin.opensourceecology.org http-01 challenge for opensourceecology.org http-01 challenge for oswh.opensourceecology.org http-01 challenge for phplist.opensourceecology.org http-01 challenge for staging.opensourceecology.org http-01 challenge for store.opensourceecology.org http-01 challenge for wiki.opensourceecology.org http-01 challenge for www.opensourceecology.org Using the webroot path /var/www/html/staging.opensourceecology.org/htdocs for all unmatched domains. Waiting for verification... Cleaning up challenges - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - new certificate deployed without reload, fullchain is /etc/letsencrypt/live/opensourceecology.org/fullchain.pem - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations, all renewals succeeded: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (success) /etc/letsencrypt/live/opensourceecology.org/fullchain.pem (success) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Redirecting to /bin/systemctl reload nginx.service [root@opensourceecology cron.d]#
- I confirmed that the cert has been updated
user@disp2766:~$ echo -n | openssl s_client -showcerts -connect opensourceecology.org:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > mycert.pem depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R10 verify return:1 depth=0 CN = fef.opensourceecology.org verify return:1 DONE user@disp2766:~$ openssl x509 -text -in mycert.pem | less user@disp2766:~$ openssl x509 -text -in mycert.pem | grep 'Not ' Not Before: Dec 10 23:19:37 2024 GMT Not After : Mar 10 23:19:36 2025 GMT user@disp2766:~$
- so it looks like nothing was broken here, we just hadn't previously been getting these alerts – and this is the first time we've gotten this alert since I've setup the google group to forward admin emails to members of the google group (incuding marcin and I)
- I wrote Marcin to let him know the issue is not an issue, but I've updated the cron anyway
- ...
- returning to the work of the hetzner3 project on my last work day (2024-10-07), I checked up on hetzner2's use of the oshine plugin, which I thought was in-use in the following sites:
- obi
- osemain
- microfactory
- store
- I confirmed that we are using the 'oshine' theme on obi, microfactory, and store, but osemain is using Enegmatic
- iirc, Catarina asked me to switch osemain to oshine.
- most importantly, we only have two licenses for oshine.
- store was never setup
- microfactory's last blog post was for an event in Feb 2019
- obi's last workshop listing is from 2014, so that's even 10 years old
- I sent an email to Catarina and Marcin asking how they want to allocate the license they *do* have with these sites on the new server
Hey Catarina & Marcin, I think we don't have enough theme licenses for our wordpress sites. When we last visited this in August, I downloaded: [a] two licenses of oshine [b] one license of enigmatic Unfortunately, you have three sites using oshine: [1] openbuildinginstitute.org [2] microfactory.opensourceecology.org [3] store.opensourceecology.org I think you also mentioned that you might want to switch www.opensourceecology.org to Oshine? Well we definitely don't have a license for that. I tried to find some way to copy the license keys from hetzner2 (our old/live prod server), but I couldn't find a way to view the currently-used license keys for each site. And when I install the latest version of the Oshine theme on the new servers, it requests a license key for each site. If you'd like, I can just setup: * openbuildinginstitute.org with oshine #1 * microfactory.opensourceecology.org with oshine #2 * www.opensourceecology.org with enigmatic * change store.opensourceecology.org to use some free theme like twentytwentyfour Or you could buy more licenses and I'll set up the sites with those. Please let me know how you want to allocate your wordpress licenses for your sites. Thank you,
- ah, shortly after I sent that email, I found a third oshine license in my files
- so the only issue is if we still want to use oshine for www.opensourceecology.org – then we'd need to buy a new license or take the one away from store.opensourceecology.org
Update: sorry, I found the third oshine license: 1. Purchased 2016 by "Catarina Mota" 2. Purchased 2018 by "Open Source Ecology" 3. Purchased 2019 by "Catarina Mota" So I think the only issue is that we'd have to either buy a new license for www.opensourceecology.org, if that's a change you'd like to make. Or use the license that's currently in-use by store.opensourceecology.org for www.opensourceecology.org and set store.opensourceecology.org to some free theme. Please let me know how I should proceed in allocating these oshine licenses.
- ...
- anyway, until we hear back about how to handle these oshine licenses, I think we're basically blocked from store.opensourceecology.org. I don't want to associate one of our precious license on the newly installed server's wordpress config if we're not going to use it.
- refresher: here's our site migration order list
1. forum.opensourceecology.org 2. store.opensourceecology.orgc 3. microfactory.opensourceecology.org 4. fef.opensourceecology.org 5. oswh.opensourceecology.org 6. seedhome.openbuildinginstitute.org 7. www.openbuildinginstitute.org 8. www.opensourceecology.org 9. phplist.opensourceecology.org 10. wiki.opensourceecology.org
- well next is microfatory, which is yet-another oshine ambiguity for now
- after that we have fef
- I logged into the fef wp admin wui dashboard https://fef.opensourceecology.org/
- I confirmed that fef is using some theme called "Simple Photo Responsive"
Mon Oct 07, 2024
- I installed the latest version of the 'oshine' theme to the store.opensourceecology.org wordpress site
rsync -av --progress /var/tmp/wordpress/themes/oshin /var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
- ok, after that loading it in the browser still yeilds a blank page, but it's just because it's cached https://store.opensourceecology.org/
- if I just append a bullshit GET variable on the end, then it loads https://store.opensourceecology.org/?nocache=2
- the site is finally loading, but it's all fucked
- I'm not seeing any 403 errors in the network tab of firefox on-load, so I'm pretty sure the issue is just missing plugins
- for example, there's a bunch of shortcodes being displayed raw, like this one at the top
[tatsu_section bg_color= “rgba(29,29,29,1)” bg_image= “http://brandexponents.com/oshine-lite/v37/wp-content/uploads/sites/44/2018/02/home-hero.jpeg” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= “center center” bg_size= “cover” bg_animation= “none” padding= ‘{“d”:”200px 0% 200px 0% “}’ margin= “0px 0px 0px 0px” border= “0px 0px px 0px” border_color= “” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “1” overlay_color= “rgba(0,0,0,0.1)” full_screen= “1” section_id= “” section_class= “” section_title= “” offset_section= “” offset_value= “0” full_screen_header_scheme= “background–dark” hide_in= “0” bg_stretch= “1” key= “REDACTED”]
- ok, if I login to the WUI and go to appearance -> themes, now it does show the oshine theme https://store.opensourceecology.org/wp-admin/themes.php
- I deactivated it and reactivated it, and I got a message at the top
This theme requires the following plugins: BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Safe SVG, Slider Revolution and WPForms Lite. Begin installing plugins | Dismiss this notice
- If I click to customize the theme, it has a tab "Install Plugins"
- If I click the "Install Plugins" tab, it yells at me with big red text
Please provide a valid purchase code of the theme in order to install plugins and import demo
- on another DispVM, I logged into the hetzner2 store.opensourceecology.org
- clicked appearance -> themes -> oshin -> customize
- no, that didn't work
- clicked "Oshine Options" in the left-hand navbar ->
- no, I went through all the settings and couldn't find it there either :(
- I mean, I have the keys already downloaded, but I'd like to keep them consistent. We have two keys and I don't know which was used for store.opensourceecology.org. This is dumb. Why do they make it so hard to find?
- I tried pulling it out of the DB, but I didn't find anything obvious in the options table named "*oshine*"
MariaDB [store_db]> select * from wp_options where option_name like '%oshine%' limit 100;
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
| option_id | option_name | option_value | autoload |
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
| 347 | external_updates-oshine-core | O:8:"stdClass":3:{s:9:"lastCheck";i:1728339766;s:14:"checkedVersion";s:5:"1.3.7";s:6:"update";N;} | no |
| 349 | external_updates-oshine-modules | O:8:"stdClass":3:{s:9:"lastCheck";i:1728339766;s:14:"checkedVersion";s:5:"2.2.9";s:6:"update";N;} | no |
| 352 | oshine_redux_to_colorhub | 1 | yes |
| 355 | oshine_redux_to_typehub | 1 | yes |
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
4 rows in set (0.00 sec)
MariaDB [store_db]>
- I realized it's probably easier to just search the mysqldump file
- got it!
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # grep -ir 'purchase' mysqldump.20240926_072001b.sql | grep -ir code | less
...
193,'be_themes_purchase_data','a:2:{s:8:\"last_tab\";s:0:\"\";s:19:\"theme_purchase_code\";s:36:\"REDACTED\";}','yes'),(194,'be_themes_purchase_data-transients','a:2:{s:14:\"changed_values\";a:0:{}s:9:\"last_save\";i:1471011372;}','yes')
- actually, that still doesn't tell me which server it is
- I think that's the wrong site's db, because I see nothing when I query just the store wordpress db
MariaDB [store_db]> select * from wp_options where option_name like '%be%' and option_value like '%theme_purchase_code%' limit 100; Empty set (0.00 sec) MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase_code%' limit 100; Empty set (0.01 sec) MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase_data%' limit 100; Empty set (0.01 sec) MariaDB [store_db]> select * from wp_options where option_value like '%themes_purchase_data%' limit 100; Empty set (0.01 sec) MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase%' limit 100; Empty set (0.01 sec) MariaDB [store_db]>
- I exported the data from the hetzner2 store theme too, but it wasn't there
user@disp928:~/Downloads$ du -sh * 28K redux_options_be_themes_data_backup_07-10-2024.json user@disp928:~/Downloads$
- fuck it, I'm just going to use these alphabetically
- which sites use this?
[root@opensourceecology hetzner3]# nice find /var/www/html -type d -iname oshin /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/themes/oshin /var/www/html/d3d.opensourceecology.org/htdocs/wp-content/themes/oshine_6.5/Oshine Buyers Package 6.5/oshin /var/www/html/d3d.opensourceecology.org/htdocs/wp-content/themes/oshin /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/themes/oshin /var/www/html/staging.openbuildinginstitute.org/htdocs/wp-content/themes/oshin /var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/oshin /var/www/html/3dp.opensourceecology.org/htdocs/wp-content/themes/oshin [root@opensourceecology hetzner3]#
- d3d and 3dp are both broken with cert errors right now
- right, my notes say these were two sites that marcin abandoned domain naames for. we eventually built microfactory.opensourceecology.org instead
- ugh, power went out
Sun Oct 06, 2024
- I checked on the status of the inventory job of our 'deleteMeIn2020' galcier vault; looks like it's still unavailable. Guess I'll give it a week or so before trying again
user@disp8678:~$ aws configure set aws_access_key_id 'REDACTED" user@disp8678:~$ user@disp8678:~$ aws configure set aws_secret_access_key 'REDACTED' user@disp8678:~$ user@disp8678:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS user@disp8678:~$
...
- I returned to work on fixing the vhost config to permit traffic to wp-config.php temporarily, but I kept getting 429 errors from wordpress.org
- This has been a frustrating, recurring issue for many months. I finally filed a bug report https://meta.trac.wordpress.org/ticket/7792#ticket
Title: Too Many "429 Too Many Requests" Errors (Nginx Misconfiguration causing False-Positives) Since the past ~6 months, I have been frequently unable to access content on wordpress.org If I'm lucky, then when I'm browsing wordpress documentation pages, I'm able to load the main html file with the content, but the website is horribly mis-rendered because many dependent assets don't load (eg css files, images, javascript, etc) due to "429 Too Many Requests" errors. If I'm unlucky, even the main page doesn't load load at all -- due to "429 Too Many Requests". Usually, I start-off being able to load one or more pages, but as I click around the website trying to find the page that I need, I eventually get this error. I am not a bot. I am a human. I'm just trying to load reference documentation as I develop a wordpress plugin. This has been extremely frustrating, and forced me to third party websites and to "guess" php functions, attributes, and return values as I'm developing, reducing my productivity. Since the Snowden revelations of 2013, it's become clear that many at-risk users should not be using the Internet without using privacy-protections like Tor. For security and privacy reasons, I do not access the internet without passing my traffic through Tor or a VPN. To prevent discrimination against at-risk folks, it's important that WordPress servers do not block traffic from shared networks, such as VPNs or Tor exit nodes. It appears that nginx's settings are too strict, and lots of good users are getting caught in the dragnet. Whatever the current nginx config is, please double it to fix these false-positives.
- alright, I updated the apache config and pushed it with ansible
user@personal:~/sandbox_local/ansible/hetzner3$ git diff
diff --git a/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2 b/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
index c0575a3..c413c74 100644
--- a/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
+++ b/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
@@ -2,12 +2,12 @@
################################################################################
# File: security.virtualhost.include
-# Version: 0.2
+# Version: 0.3
# Purpose: File includes some common security-hardening that's intended to be
# Include()d into other vhost files' <VirtualHost> blocks
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-14
-# Updated: 2024-09-24
+# Updated: 2024-10-06
################################################################################
# don't execute any php files inside uploads directories
@@ -56,7 +56,10 @@
# block access to 'wp-login.php' from brute-forcers;
# see wp plugin 'rename-wp-login'
- <LocationMatch ".*wp-login.php">
- Require all denied
- </LocationMatch>
+ # TODO: 2024-10: we need to re-enable this after we find a replacement for the
+ # (now-deprecated) 'rename-wp-login' wordpress plugin
+ # * https://wordpress.org/plugins/rename-wp-login/
+# <LocationMatch ".*wp-login.php">
+# Require all denied
+# </LocationMatch>
user@personal:~/sandbox_local/ansible/hetzner3$
- now I'm able to load the login page, but when I do, I still get 403 errors on a few of the dependent requests
- the mod_security log shows the 403 response for these images, but I'm not sure why it's happening
root@hetzner3 /var/log/apache2 # cat modsec_audit.log ... --69ede32a-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-admin/images/wordpress-logo.svg Stopwatch: 1728256656835561 1333 (- - -) Stopwatch2: 1728256656835561 1333; combined=33, p1=31, p2=0, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --69ede32a-Z-- ... --6a3d866d-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png Stopwatch: 1728256938397718 1105 (- - -) Stopwatch2: 1728256938397718 1105; combined=27, p1=23, p2=0, p3=0, p4=0, p5=3, sr=0, sw=1, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --6a3d866d-Z--
- oh, I wonder if this is it
user@personal:~/sandbox_local/ansible/hetzner3$ grep -irC4 'require all' | less ... roles/maltfield.apache/templates/security.virtualhost.include.j2- <LocationMatch "/images/"> roles/maltfield.apache/templates/security.virtualhost.include.j2- SetHandler ! roles/maltfield.apache/templates/security.virtualhost.include.j2: Require all denied roles/maltfield.apache/templates/security.virtualhost.include.j2- </LocationMatch>
- so one thing we changed from hetzner2 was the logic for preventing php scripts from being executed inside user-uploadable directories.
- in hetzner2, we used mod_php, so this was done with 'php_flag engine off' -- but that doesn't work with php-fpm
# don't execute any php files inside the uploads directory
<LocationMatch "/wp-content/uploads/">
php_flag engine off
</LocationMatch>
<LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Order Deny,Allow
Deny from All
</LocationMatch>
# block dot files, such as svn files from checking out wp core
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
# block access to 'wp-login.php' from brute-forcers; see wp plugin 'rename-wp-login'
<LocationMatch .*wp-login.php>
Deny From All
</LocationMatch>
- I think the idea of "Require all denied" was that it would capture anything that wasn't already sent-off to the php-fpm server proxy. Basically because that should be captured first, and then if apache still sees it, we should deny access to it
- this assumes that the filename ends with php (or our more complex regex above), but that logic doesn't translate for the whole /images/ directory
- alright, I made these changes, which fixed it. basically we want to "SetHandler !" on everything, but only "Require all denied" for the .php files
- Note: I wasn't able to figure out what "SetHandler !" does. Bing/ddg returns no results. And Google just ignores any queries with an exclamation mark in them. It's literally not possible to search-for. But I did find lots of results asking about how to use SetHandler in Apache to point to php-fpm, so my best-guess is that this sets the Handler to 'null' or something, which would overwrite any previous setting that told it to send it to some other cgi proxy or something
# don't execute any php files inside uploads directories
<LocationMatch "/wp-content/uploads/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -21,7 +20,6 @@
<LocationMatch "/uploadimages/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/uploadimages/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -29,7 +27,6 @@
<LocationMatch "/images/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/images/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -38,7 +35,6 @@
# don't execute php files in W3 Total Cache's tmp dir
<LocationMatch "/wp-content/cache/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/wp-content/cache/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -46,17 +42,22 @@
# block dot (hidden) files
<LocationMatch "/\.(?!well\-known)">
+ SetHandler !
Require all denied
</LocationMatch>
# block config files
<LocationMatch "config.php">
+ SetHandler !
Require all denied
</LocationMatch>
# block access to 'wp-login.php' from brute-forcers;
# see wp plugin 'rename-wp-login'
- <LocationMatch ".*wp-login.php">
- Require all denied
- </LocationMatch>
- cool, I was able to login to store.opensourceecology.org on hetzner3 with my old creds now
- the dashboard is littered with alerts:
Action Scheduler: 3 past-due actions found; something may be wrong. Read documentation »
WooCommerce database update required WooCommerce has been updated! To keep things running smoothly, we have to update your database to the newest version. The database update process runs in the background and may take a little while, so please be patient. Advanced users can alternatively update via WP CLI. Update WooCommerce Database Learn more about updates
Geolocation has not been configured. You must enter a valid license key on the MaxMind integration settings page in order to use the geolocation service. If you do not need geolocation for shipping or taxes, you should change the default customer location on the general settings page.
The plugin be-gdpr/be-gdpr.php has been deactivated due to an error: Plugin file does not exist. The plugin be-portfolio-post/be-portfolio-post.php has been deactivated due to an error: Plugin file does not exist. The plugin coingate-for-woocommerce/coingate.php has been deactivated due to an error: Plugin file does not exist. The plugin colorhub/colorhub.php has been deactivated due to an error: Plugin file does not exist. The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist. The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist. The plugin meta-box-conditional-logic/meta-box-conditional-logic.php has been deactivated due to an error: Plugin file does not exist. The plugin meta-box-show-hide/meta-box-show-hide.php has been deactivated due to an error: Plugin file does not exist. The plugin meta-box-tabs/meta-box-tabs.php has been deactivated due to an error: Plugin file does not exist. The plugin oshine-core/oshine-core.php has been deactivated due to an error: Plugin file does not exist. The plugin oshine-modules/oshine-modules.php has been deactivated due to an error: Plugin file does not exist. The plugin redux-vendor-support/redux-vendor-support.php has been deactivated due to an error: Plugin file does not exist. The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist. The plugin revslider/revslider.php has been deactivated due to an error: Plugin file does not exist. The plugin tatsu/tatsu.php has been deactivated due to an error: Plugin file does not exist. The plugin typehub/typehub.php has been deactivated due to an error: Plugin file does not exist.
- I kicked-off the woocommerce db upgrade
- ugh, akismet isn't activated. we have 2,213 comments in the queue
- if I click on 'themes' in the wui, then I get a notice at the top
The active theme is broken. Reverting to the default theme.
- it says that 'oshine' is the active theme
- allright, I downloaded these files before
user@ose:~/tmp/hetzner3$ ls 13757819-enigmatic-responsive-multipurpose-wp-theme-license.txt 28755060-oshine-creative-multipurpose-wordpress-theme-license.txt 47932235-oshine-creative-multipurpose-wordpress-theme-license.txt 52287820-oshine-creative-multipurpose-wordpress-theme-license.txt backup-restore-test themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip user@ose:~/tmp/hetzner3$
- unfortunately, these are paid themes, and I have to coordinate with catarina to get an OTP every time I login, so I can't 3TOFU these :( I'll just have to 1TOFU it
- apparently these two oshine themes have identical contents but different names
user@ose:~/tmp/hetzner3$ sha256sum *.zip ed0628d0e57bb4e44b1af24eb235c6c384433c9ca94806c11b881e16f7f2b74a themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip 7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789 themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip 7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789 themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip user@ose:~/tmp/hetzner3$
- I rsync'd these files up to hetzner3
user@ose:~/tmp/hetzner3$ rsync -av --progress themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip hetzner3:
Enter passphrase for key '/home/user/.ssh/id_rsa':
Enter passphrase for key '/home/user/.ssh/id_rsa':
sending incremental file list
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
10,582,975 100% 318.17kB/s 0:00:32 (xfr#1, to-chk=0/1)
sent 10,585,730 bytes received 35 bytes 201,633.62 bytes/sec
total size is 10,582,975 speedup is 1.00
user@ose:~/tmp/hetzner3$
user@ose:~/tmp/hetzner3$ rsync -av --progress themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip hetzner3:
Enter passphrase for key '/home/user/.ssh/id_rsa':
sending incremental file list
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
11,394,173 100% 996.24kB/s 0:00:11 (xfr#1, to-chk=0/1)
sent 11,397,129 bytes received 35 bytes 303,924.37 bytes/sec
total size is 11,394,173 speedup is 1.00
user@ose:~/tmp/hetzner3$
- I copied them over to our other dir with all the themes
root@hetzner3 /var/tmp/wordpress/themes # ls /home/maltfield/*.zip
/home/maltfield/themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
/home/maltfield/themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
root@hetzner3 /var/tmp/wordpress/themes #
root@hetzner3 /var/tmp/wordpress/themes # rsync -av --progress /home/maltfield/*.zip .
sending incremental file list
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
10.582.975 100% 670,76MB/s 0:00:00 (xfr#1, to-chk=1/2)
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
11.394.173 100% 329,28MB/s 0:00:00 (xfr#2, to-chk=0/2)
sent 21.982.821 bytes received 54 bytes 43.965.750,00 bytes/sec
total size is 21.977.148 speedup is 1,00
root@hetzner3 /var/tmp/wordpress/themes #
root@hetzner3 /var/tmp/wordpress/themes # shred -u /home/maltfield/*.zip
root@hetzner3 /var/tmp/wordpress/themes #
root@hetzner3 /var/tmp/wordpress/themes # chown root:root themeforest-*
root@hetzner3 /var/tmp/wordpress/themes # chmod 0400 themeforest-*
root@hetzner3 /var/tmp/wordpress/themes #
Fri Oct 04, 2024
- Marcin gave me the go-ahead to delete the 'deleteMeIn2020' vault from our AWS Glacier account
1. Yes, delete the vault. 2. Thanks, good insights - i'll look into those more closely to see what would fit best. MJ
- ah ffs, I logged into the amazon WUI, but when I clicked "delete" on the vault, it gave me an error saying I have to delete all the objects in the vault first
This vault is not empty Vaults can be deleted only if there are no archives in the vault as of the last inventory it computed and there have been no writes to the vault since the last inventory. To delete all archives in the vault, use the REST API, the AWS SDK for Java, the AWS SDK for .NET or the AWS CLI.
- apparently this can only be done in the CLI via the API!?! It links to this https://docs.aws.amazon.com/console/glacier/using-aws-sdk
- we do have some 'glacier.py' script on our old server, but it complains about missing module(s)
[root@opensourceecology backups]# ls
backup.old.20180115.sh backup.settings.20221028 glacierRestore sync
backupReport.sh backup.sh glacierTest.py sync.old
backupReport.sh.20221028 backup.sh.20221028 ose-backups-cron.key
backup.settings cleanLocal.pl README.txt
[root@opensourceecology backups]#
[root@opensourceecology backups]# glacier.py
Traceback (most recent call last):
File "/root/bin/glacier.py", line 36, in <module>
import boto.glacier
ImportError: No module named boto.glacier
[root@opensourceecology backups]#
- allright, at least debian has the cli in its repos
user@disp3919:~/Downloads$ apt-cache search awscli awscli - Unified command line interface to Amazon Web Services user@disp3919:~/Downloads$ user@disp3919:~/Downloads$ sudo apt-get install awscli Reading package lists... Done Building dependency tree... Done Reading state information... Done awscli is already the newest version (2.9.19-1). The following packages were automatically installed and are no longer required: librnp0 libwpe-1.0-1 libwpebackend-fdo-1.0-1 linux-image-6.1.0-10-amd64 linux-image-6.1.0-11-amd64 linux-image-6.1.0-13-amd64 linux-image-6.1.0-17-amd64 linux-image-6.1.0-18-amd64 linux-image-6.1.0-20-amd64 linux-image-6.1.0-21-amd64 linux-image-6.1.0-22-amd64 Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. user@disp3919:~/Downloads$ aws
- I was able to auth with some creds I found on hetzner2:/root/backups/glacierTest.py
user@disp3919:~/Downloads$ aws configure set aws_access_key_id 'REDACTED'
user@disp3919:~/Downloads$ aws configure set aws_secret_access_key 'REDACTED'
user@disp3919:~/Downloads$ aws sts get-caller-identity
{
"UserId": "REDACTED",
"Account": "REDACTED",
"Arn": "arn:aws:iam::REDACTED:user/backup-cron"
}
user@disp3919:~/Downloads$
- apparently we now have to create an inventory and then iterate though that inventory to delete all of the objects that it lists https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5
- creating an inventory can take hours or days; let's initiate it now
user@disp3919:~/Downloads$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"location": "/099400651767/vaults/deleteMeIn2020/jobs/ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS",
"jobId": "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS"
}
user@disp3919:~/Downloads$
- I guess now we wait a few days for the job to complete before we can download it, parse it, and then delete all of the objects it identifies per https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5
user@disp3919:~/Downloads$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (InvalidParameterValueException) when calling the GetJobOutput operation: The job is not currently available for download: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS user@disp3919:~/Downloads$
...
- after much debugging, I figured out why store.opensourceecology.org gives different results for a `curl` coming from my laptop vs the server
- I found that the `curl` from my laptop was making it to nginx -> varnish -> apache
- but the logs were mysteriously absent for varnish & apache when I did the curl from the machine itself
- I even did a tcpdump, but I only saw a tiny blip of traffic when doing the command locally
- here's why: the server returns an http -> https redirect to store.opensourceecology.org. When the *server*'s curl command gets that, it does a public DNS lookup and then sends the query to hetzner2!
- I updated the /etc/hosts file to prevent this
root@hetzner3 ~ # cd /etc root@hetzner3 /etc # root@hetzner3 /etc # vim hosts root@hetzner3 /etc # root@hetzner3 /etc # diff hosts.20241004 hosts 2a3,13 > 127.0.0.1 forum.opensourceecology.org > 127.0.0.1 store.opensourceecology.org > 127.0.0.1 microfactory.opensourceecology.org > 127.0.0.1 fef.opensourceecology.org > 127.0.0.1 oswh.opensourceecology.org > 127.0.0.1 seedhome.openbuildinginstitute.org > 127.0.0.1 www.openbuildinginstitute.org > 127.0.0.1 www.opensourceecology.org > 127.0.0.1 phplist.opensourceecology.org > 127.0.0.1 wiki.opensourceecology.org > 3a15 > root@hetzner3 /etc #
- ok, now it's stuck in an infinite redirect. It just keeps going back-and-forth adding and removing the slash at the end
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local5 HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:23:38 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://opensourceecology.org Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:22:29 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://www.opensourceecology.org/ Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:23:39 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://opensourceecology.org Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" ... HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:22:29 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://www.opensourceecology.org/ Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:23:39 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://opensourceecology.org Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" curl: (47) Maximum (50) redirects followed maltfield@hetzner3:~$
- ok, so it looks like it's getting picked-up by the default site
root@hetzner3 /etc/nginx # grep -ir 301 * nginx.conf: return 301 https://$host$request_uri; nginx.conf.1282157.2024-09-28@23:10:52~: return 301 https://$host$request_uri; sites-enabled/00-default.conf: return 301 https://opensourceecology.org; root@hetzner3 /etc/nginx #
- ah shit, yeah, nginx isn't even listening on 127.0.0.1 lol
root@hetzner3 /etc/nginx # less sites-enabled/store.opensourceecology.org.conf
# Ansible managed
################################################################################
# File: store.opensourceecology.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
# Updated: 2024-09-14
################################################################################
server {
access_log /var/log/nginx/store.opensourceecology.org/access.log main;
error_log /var/log/nginx/store.opensourceecology.org/error.log;
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
listen 144.76.164.201:443;
listen [2a01:4f8:200:40d7::2]:443;
server_name store.opensourceecology.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
# root /var/www/html/SITE_DOWN/htdocs/;
# index index.html index.htm;
#
# # force all requests to load exactly this page
# location / {
# try_files $uri /index.html;
# }
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
}
}
- well it is, but this server block is not
root@hetzner3 /etc/nginx # netstat -plan | grep -i 443 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3914728/nginx: mast tcp 0 0 144.76.164.201:4443 0.0.0.0:* LISTEN 3914728/nginx: mast tcp 25 0 144.76.164.201:51710 104.21.40.220:443 CLOSE_WAIT 15751/wazuh-modules tcp 0 0 127.0.0.1:80 127.0.0.1:54436 TIME_WAIT - tcp 0 0 127.0.0.1:54432 127.0.0.1:80 TIME_WAIT - tcp6 0 0 :::443 :::* LISTEN 3914728/nginx: mast tcp6 0 0 2a01:4f8:200:40d7::4443 :::* LISTEN 3914728/nginx: mast tcp6 25 0 2a01:4f8:200:40d7:49016 2606:4700:3033::ac4:443 CLOSE_WAIT 15751/wazuh-modules You have new mail in /var/mail/root root@hetzner3 /etc/nginx #
- yeah, default is -- which is why it's picking it up instead
root@hetzner3 /etc/nginx # grep -ir listen sites-available/forum.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-available/forum.opensourceecology.org.conf: listen 144.76.164.201:443; sites-available/forum.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; sites-available/default: listen 80 default_server; sites-available/default: listen [::]:80 default_server; sites-available/default: # listen 443 ssl default_server; sites-available/default: # listen [::]:443 ssl default_server; sites-available/default:# listen 80; sites-available/default:# listen [::]:80; sites-available/store.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-available/store.opensourceecology.org.conf: listen 144.76.164.201:443; sites-available/store.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; nginx.conf.1282157.2024-09-28@23:10:52~: listen 80; nginx.conf.1282157.2024-09-28@23:10:52~: listen [::]:80; nginx.conf: listen 80; nginx.conf: listen [::]:80; nginx.conf.85740.2024-09-24@04:17:16~:# listen localhost:110; nginx.conf.85740.2024-09-24@04:17:16~:# listen localhost:143; sites-enabled/00-default.conf: listen 443; sites-enabled/00-default.conf: listen [::]:443; sites-enabled/awstats.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-enabled/awstats.opensourceecology.org.conf: listen 144.76.164.201:443; sites-enabled/awstats.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; sites-enabled/awstats.opensourceecology.org.conf: listen 144.76.164.201:4443; sites-enabled/awstats.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:4443; sites-enabled/munin.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-enabled/munin.opensourceecology.org.conf: listen 144.76.164.201:443; sites-enabled/munin.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; sites-enabled/munin.opensourceecology.org.conf: listen 144.76.164.201:4443; sites-enabled/munin.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:4443; root@hetzner3 /etc/nginx #
- this is actually the same as our hetzner2 config
- I updated the nginx config in ansible and pushed it out again
diff --git a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
index f4b62cd..f750651 100644
--- a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
@@ -2,13 +2,13 @@
################################################################################
# File: store.opensourceecology.org.conf
-# Version: 0.2
+# Version: 0.3
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
-# Updated: 2024-09-14
+# Updated: 2024-10-04
################################################################################
server {
@@ -19,6 +19,8 @@ server {
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
+ listen 127.0.0.1:443;
+ listen [::1]:443;
listen {{ ansible_default_ipv4.address }}:443;
listen [{{ ansible_default_ipv6.address }}]:443;
user@personal:~/sandbox_local/ansible/hetzner3$
diff --git a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
index f4b62cd..f750651 100644
--- a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
@@ -2,13 +2,13 @@
################################################################################
# File: store.opensourceecology.org.conf
-# Version: 0.2
+# Version: 0.3
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
-# Updated: 2024-09-14
+# Updated: 2024-10-04
################################################################################
server {
@@ -19,6 +19,8 @@ server {
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
+ listen 127.0.0.1:443;
+ listen [::1]:443;
listen {{ ansible_default_ipv4.address }}:443;
listen [{{ ansible_default_ipv6.address }}]:443;
user@personal:~/sandbox_local/ansible/hetzner3$
- and, well, the good/bad news is that now the curl from the local machine is as equally broken as the curl from my laptop
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local6 HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:46:30 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=local6 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 89 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Sat, 05 Oct 2024 03:46:30 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 98500 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true maltfield@hetzner3:~$
- the 'true' is obviously coming from 'index.html', so my first thought was just to get rid of that file
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # rm index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- but now we're just back to the empty output (again)
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local7 HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:49:48 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=local7 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 94 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Sat, 05 Oct 2024 03:49:49 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2796>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 97 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" maltfield@hetzner3:~$
- varnish logs look fine; it basically just calls the backend
- apahce logs indicate that it did figure out which file to server with php
==> store.opensourceecology.org/error.log <== [Sat Oct 05 03:56:05.631466 2024] [authz_core:debug] [pid 3909393:tid 3909439] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:05.631539 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(123): [client 127.0.0.1:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:05.631557 2024] [proxy:debug] [pid 3909393:tid 3909439] mod_proxy.c(1465): [client 127.0.0.1:0] AH01143: Running scheme fcgi handler (attempt 0) [Sat Oct 05 03:56:05.631571 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(1078): [client 127.0.0.1:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Sat Oct 05 03:56:05.631584 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(1087): [client 127.0.0.1:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:05.631597 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Sat Oct 05 03:56:05.631610 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3242): [client 127.0.0.1:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Sat Oct 05 03:56:05.631624 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3309): [client 127.0.0.1:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Sat Oct 05 03:56:05.631638 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3450): [client 127.0.0.1:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Sat Oct 05 03:56:05.631673 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000) [Sat Oct 05 03:56:06.720816 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [05/Oct/2024:03:56:05 +0000] "GET /index.php?nocache=local10 HTTP/1.1" 301 436 "-" "curl/7.88.1" ==> store.opensourceecology.org/error.log <== [Sat Oct 05 03:56:06.725670 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:06.725738 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:06.725854 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:06.725886 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(123): [client 127.0.0.1:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:06.725895 2024] [proxy:debug] [pid 3909393:tid 3909441] mod_proxy.c(1465): [client 127.0.0.1:0] AH01143: Running scheme fcgi handler (attempt 0) [Sat Oct 05 03:56:06.725901 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(1078): [client 127.0.0.1:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Sat Oct 05 03:56:06.725928 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(1087): [client 127.0.0.1:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:06.725935 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Sat Oct 05 03:56:06.725941 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3242): [client 127.0.0.1:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Sat Oct 05 03:56:06.725950 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3309): [client 127.0.0.1:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Sat Oct 05 03:56:06.725959 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3450): [client 127.0.0.1:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Sat Oct 05 03:56:06.726002 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000) [Sat Oct 05 03:56:07.778759 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [05/Oct/2024:03:56:06 +0000] "GET /?nocache=local10 HTTP/1.1" 200 586 "-" "curl/7.88.1"
- this suggests that it might do this if the theme dir is empty? that would likely apply in our case https://serverfault.com/a/766146
- oh, it *does* load if I try '/wp-admin/'
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/wp-admin/
...
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 05 Oct 2024 04:24:26 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 1516
Connection: keep-alive
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98551
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta name="viewport" content="width=device-width" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="robots" content="noindex,nofollow" />
<title>WordPress › Update</title>
<link rel='stylesheet' id='dashicons-css' href='https://store.opensourceecology.org/wp-includes/css/dashicons.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='https://store.opensourceecology.org/wp-includes/css/buttons.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='https://store.opensourceecology.org/wp-admin/css/forms.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='https://store.opensourceecology.org/wp-admin/css/l10n.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='https://store.opensourceecology.org/wp-admin/css/install.min.css?ver=6.6.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui">
<p id="logo"><a href="https://wordpress.org/">WordPress</a></p>
<h1>Database Update Required</h1>
<p>WordPress has been updated! Next and final step is to update your database to the newest version.</p>
<p>The database update process may take a little while, so please be patient.</p>
<p class="step"><a class="button button-large button-primary" href="upgrade.php?step=1&backto=%2Fwp-admin%2F">Update WordPress Database</a></p>
</body>
</html>
maltfield@hetzner3:~$
- I loaded that in the web browser, and it told me a wordpress database update was needed. I just pressed the button -- it didn't even prompt me to auth
Your WordPress database has been successfully updated!
- I clicked "Continue"
- that redirected me here, and I immediately got '403 forbidden' https://store.opensourceecology.org/wp-login.php?redirect_to=https%3A%2F%2Fstore.opensourceecology.org%2Fwp-admin%2F&reauth=1
- that would be because block access to 'wp-login.php' since we were using a plugin to rename it; we'll have to temp disable that until we replace that (now deprecated) plugin
Thr Oct 03, 2024
- I sent an invoice (AS-0106) to OSE for 67 hours in Sep 2024
...
- continuing to debug store.opensourceecology.org, I see that it's redirecting from '/index.php' to '/'
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2024 04:47:39 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/ X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 131132 98385 Age: 88 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" user@disp3919:~$
- if I try to hit 'index.html', I don't get a redirect -- I just get a 404
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html HTTP/1.1 404 Not Found Server: nginx Date: Fri, 04 Oct 2024 04:48:25 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 131134 Age: 0 Via: 1.1 varnish (Varnish/7.1) user@disp3919:~$
- if I create it
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cp is_hetzner3 index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # ls -lah is_hetzner3 index.html ----r----- 1 root root 5 Oct 4 04:49 index.html ----r----- 1 not-apache www-data 5 Sep 27 04:44 is_hetzner3 root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown not-apache:www-data index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- then it works
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 04:49:51 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 98387 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3919:~$
- hmm, so apache is just refusing to serve 'index.php' files. -- what about 'something.php'?
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # echo "<?php echo 'it works'; ?>" > something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown root:www-data something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chmod 0040 something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- it works, so it's something specific to 'index.php'
user@disp3919:~$ curl -i https://store.opensourceecology.org/something.php HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 04:52:35 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 8 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 98390 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" it worksuser@disp3919:~$
- I'd think it's an issue with the DirectoryIndex, but this looks good
root@hetzner3 /etc/apache2 # grep -ir 'index.php' * conf-available/wordpress.directory.include:# RewriteRule . /index.php [L] mods-available/dir.conf:DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah mods-enabled/dir.conf lrwxrwxrwx 1 root root 26 Sep 25 01:24 mods-enabled/dir.conf -> ../mods-available/dir.conf root@hetzner3 /etc/apache2 #
- I checked the old server, but I didn't see anything that we're missing in the new server
[root@opensourceecology httpd]# grep -ir 'index.php' conf [root@opensourceecology httpd]# grep -ir 'index.php' conf.d conf.d/php.conf:# Add index.php to the list of files that will be served as directory conf.d/php.conf:DirectoryIndex index.php conf.d/00-wiki.opensourceecology.org.conf: Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php conf.d/mod_evasive.conf: # http://security.lss.hr/index.php?page=details&ID=LSS-2005-01-01 [root@opensourceecology httpd]# grep -ir 'index.php' conf.modules.d/ [root@opensourceecology httpd]# grep -ir 'index.php' modsecurity.d/ [root@opensourceecology httpd]#
- I changed wp-config.php to have WP_DEBUG set to 'true', but it didn't print anything extra. It seems like the error is occurring before wordpress
- I set LogLevel of apache.conf to 'debug', and this popped-up
==> forum.opensourceecology.org/access.log <== 127.0.0.1 - - [04/Oct/2024:05:10:02 +0000] "GET /server-status?auto HTTP/1.1" 200 1202 "-" "munin/2.0.73 (libwww-perl/6.68)" ==> forum.opensourceecology.org/error.log <== [Fri Oct 04 05:10:03.564975 2024] [authz_core:debug] [pid 3581402:tid 3581414] mod_authz_core.c(815): [client 127.0.0.1:32934] AH01626: authorization result of Require all denied: denied
- well, that's an unrelated issue with munin, but it seems that the requests to '/server-status' are getting sent to the wrong vhost (forum.opensourceecology.org) and also denied access
- here's the actual output when I do the curl
- first, it outputs this immediately, then it pauses for maybe 10 seconds
==> store.opensourceecology.org/error.log <== [Fri Oct 04 05:11:53.426292 2024] [authz_core:debug] [pid 3581402:tid 3581422] mod_authz_core.c(733): [client 81.17.16.91:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Fri Oct 04 05:11:53.426458 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(123): [client 81.17.16.91:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Fri Oct 04 05:11:53.426496 2024] [proxy:debug] [pid 3581402:tid 3581422] mod_proxy.c(1465): [client 81.17.16.91:0] AH01143: Running scheme fcgi handler (attempt 0) [Fri Oct 04 05:11:53.426517 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1078): [client 81.17.16.91:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Fri Oct 04 05:11:53.426535 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1087): [client 81.17.16.91:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Fri Oct 04 05:11:53.426586 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Fri Oct 04 05:11:53.426612 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3242): [client 81.17.16.91:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Fri Oct 04 05:11:53.426658 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3309): [client 81.17.16.91:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Fri Oct 04 05:11:53.426718 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3450): [client 81.17.16.91:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Fri Oct 04 05:11:53.426793 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)
- after maybe 10 seconds, it outputs this
[Fri Oct 04 05:12:03.646185 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 81.17.16.91 - - [04/Oct/2024:05:11:53 +0000] "GET /index.php?nocache=6 HTTP/1.1" 301 430 "-" "curl/7.88.1"
- so it sounds like maybe this is an issue with the php-fpm config?
- I tried to hit apache through the cli on the server itself and, oh, I get the payload as desired
root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
<script>
//jQuery(document).ready(function(){
// });
</script>
</body>
</html>root@hetzne
You have new mail in /var/mail/root
- I loosened the error reporting settings on php.ini and I got it to spit this out when I curl from my laptop
--2d1eeb03-H--
Apache-Error: [file "mod_authz_core.c"] [line 733] [level 7] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 123] [level 7] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "mod_proxy.c"] [line 1465] [level 7] AH01143: Running scheme fcgi handler (attempt 0)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1078] [level 7] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1087] [level 7] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "proxy_util.c"] [line 3242] [level 7] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
Apache-Error: [file "proxy_util.c"] [line 3309] [level 7] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
Apache-Error: [file "proxy_util.c"] [line 3450] [level 7] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 911] [level 3] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function ini_set() in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php:590\\nStack trace:\\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(82): wp_debug_mode()\\n#1 /var/www/html/store.opensourceecology.org/wp-config.php(105): require_once('...')\\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-blog-header.php(13): require_once('...')\\n#4 /var/www/html/store.opensourceecology.org/htdocs/index.php(17): require('...')\\n#5 {main}\\n thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php on line 590'
Apache-Handler: proxy:unix:/run/php/php8.2-fpm.sock|fcgi://localhost
Stopwatch: 1728019471413382 53626 (- - -)
Stopwatch2: 1728019471413382 53626; combined=41, p1=21, p2=18, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"
- yeah, so this is the wordpress bug that I submitted a PR for last month
- after a brief dialog with the wordpress devs, a workaround until they merge is to define a fake ini_set() function in wp-config.php
- from personal experience, I found it's best to wrap this in a conditional to make sure the function doesn't exist yet
root@hetzner3 /var/www/html/store.opensourceecology.org # cp wp-config.php wp-config.php.20241003
root@hetzner3 /var/www/html/store.opensourceecology.org #
root@hetzner3 /var/www/html/store.opensourceecology.org # vim wp-config.php
root@hetzner3 /var/www/html/store.opensourceecology.org #
root@hetzner3 /var/www/html/store.opensourceecology.org # diff wp-config.php.20241003 wp-config.php
1a2,9
>
> # fix wordpress bug https://core.trac.wordpress.org/ticket/48693
> if( ! function_exists('ini_set') ){
> function ini_set(){
> return;
> }
> }
>
root@hetzner3 /var/www/html/store.opensourceecology.org #
- after that, I'm back to getting blank pages on my curl on my laptop. it's flapping?
- alright, let me see if I can harden php back up again, but with errors actually logging. I'll update something.php to write to the error log
root@hetzner3 /var/www/html/store.opensourceecology.org # cd htdocs/ root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # vim something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cat something.php <?php error_log( "executing something.php" ); echo 'it works'; ?> root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- ok, it's visible
[Fri Oct 04 05:47:06.734936 2024] [proxy_fcgi:error] [pid 3581402:tid 3581452] [client 127.0.0.1:36330] AH01071: Got error 'PHP message: executing something.php'
- first I reduced the apache logs down to 'warn' again. looks good
[Fri Oct 04 05:48:39.732669 2024] [proxy_fcgi:error] [pid 3591906:tid 3591913] [client 127.0.0.1:41102] AH01071: Got error 'PHP message: executing something.php' ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [04/Oct/2024:05:48:39 +0000] "GET /something.php HTTP/1.1" 200 302 "-" "curl/7.88.1"
- I obliterated my manual changes by pushing ansible's apache & php roles
- cool, I confirmed that both curl on my laptop and on the server produce the logs after restarting both apache2 & php8.2-fpm
- for some reason I still get an 'true' on my laptop
user@disp3919:~$ 3919:~$ curl -iL https://store.opensourceecolindex.php?nocache=19 HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2024 05:56:15 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=19 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 98469 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 05:56:16 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 131203 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3919:~$
- but I get the actual html on the local machine
root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
/* ]]> */
</script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-content/themes/oshin/js/script.js?ver=5.0'></script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-includes/js/wp-embed.min.js?ver=5.1.1'></script>
<!-- Option Panel Custom JavaScript -->
<script>
//jQuery(document).ready(function(){
// });
</script>
</body>
</html>root@hetzner3 ~ #
Wed Oct 02, 2024
- Marcin sent me a few emails in the past months asking about OSE's use of Amazon Glacier
- Today he sent a message saying that he got charged $1.03, and isn't sure why
Michael, I'm getting charged $1.03 for Glacier. Can we cancel that? Marcin
- It took me a while to auth
- first I tried to login with my 'maltfield' aws user, but aws rejected my creds (stored in my personal keepass)
- eventually I realized I had to click "Sign in using root user email" -- and then I could auth using the creds stored in the shared keepass
- after logging-in, I went to the "Billing and Cost Management" app https://us-east-1.console.aws.amazon.com/costmanagement/home?region=us-west-2#/home
- on this page, there was a link that said "Last month's total cost: $1.03". Yep, that's all accounted-for. I clicked it.
- the next page showed a joke of a chart with one bar on a bar graph that said "$1.03". And the bar was labeled "Total Cost"
- I had to click on the dropdown menu for "Dimension" and set it to "Service" -- then it listed 4 items
- Glacier - $1.03
- S3 - $0.00
- Tax $0.00
- So I switched over to the "Glacier" app https://us-east-1.console.aws.amazon.com/glacier/home?region=us-east-1
- Curiously, it listed 0 vaults
- but there was a note at the top saying we should use S3 for glaicer, so I clicked over to the "S3" app
We recommend that you use Glacier storage classes in Amazon S3 for archival storage
- here I saw one bucket called "oseserverbackups" in "US West (Oregon) us-west-2"
- the bucket had one 34.0 byte file in it called "test.txt". That's it!
- this file was created July 6, 2018, 19:18:03 (UTC-05:00)
- I downloaded it; it has one line of text
some file destined for s3 this is
- I deleted the 'test.txt' file object from the s3 bucket
- I then deleted the (now empty) 'oseserverbackups' bucket
- unconvinced that that was the issue, I went back to the "glacier" app. This time I cycled through a few of the regions until I got to "us-west-2" -- this time I showed one vault named "deleteMeIn2020"
- I clicked on it, and it said
- this vault was created March 29, 2018, 16:36:06 (UTC-05:00)
- this vault was last inventoried August 1, 2018, 02:41:31 (UTC-05:00)
- this vault is 285.3 GB (as of last inventory)
- well, it's after 2020. So I think we should delete it.
- I sent an email to Marcin asking for a confirmation before I delete it
Hey Marcin, You have a 285.3 GB vault in Amazon Glacier's us-west-2 region. I logged-into your AWS account today and did some digging. I found this vault 285.3 GB vault named 'deleteMeIn2020'. I created this vault in 2018 Q1. It contains a final backup of files from hetzner1. I created it as part of the hetzner2 migration project, thinking that we should delete it in 2020 if we never needed to restore anything from it for 2 years. * https://wiki.opensourceecology.org/wiki/CHG-2018-07-06_hetzner1_deprecation * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q1#Sat_Mar_31.2C_2018 Well, 2020 came and past. Four more years passed. I think you can safely delete the 'deleteMeIn2020' vault. By the way, I also deleted a 53-byte test file from an S3 bucket named 'test.txt' in a bucket in s3 called 'oseserverbackups' in us-west-2. It was the only file in the bucket. I deleted the file and the empty bucket. Would you like me to proceed with deleting the 285.3 GB 'deleteMeIn2020' glacier bucket from your AWS account? Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- meanwhile, I tried to figure out why I couldn't login as 'maltfield', and I realized that, ffs, we don't have IAM setup for our account?? Maybe Marcin deleted it when trying to elimiate costs? IAM is free, though..
- ok, I found my 'maltfield' user under "Security Credentials" -> "Access Management" -> "Users"
- it says my last console sign-in was 424 days ago
- I went to my user's settings, selected the MFA token, and selected "Resync" -- then entered two consecutive OTPs
- I tried to login, and this time it let me in. Well that was annoying.
- I opened cloudtrail and reviewed the latest account events https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?ReadOnly=false
- the most recent event was the 'root' user resyncing the MFA token of the 'matlfield' token
- before that we have two ConsoleLogin for today
- before that 'mjakubowski' user has a MakePayment event (and some other payment related events) on Sep 19
- before that we have a bunch of login & mfa-related entries for Marcin's user on Sep 06, 14, 17, and 19.
- and that's where the log ends; looks like we just get 90 days of logs for free.
...
- hetzner responded to my support inquery about how they handle failed disks
Dear Mr Altfield Unfortunately it's an unmanaged root server monitoring is your responsibility I'm afraid. If you have a problem please open a ticket in your robot account. Please click on "Servers" from the menu on the left and then select the corresponding server. Under the "Support" tab, you can choose "Hard drive is broken". Please follow the instructions. https://docs.hetzner.com/robot/dedicated-server/troubleshooting/serial-numbers-and-information-on-defective-hard-drives/ Our DC is 24/7 available and we exchange broken hardware as soon as possible for free. Hetzner clients can use the Server Monitoring System to monitor their servers and have an email sent to them when the status of one of the monitored services changes: https://docs.hetzner.com/robot/dedicated-server/security/system-monitor/ https://docs.hetzner.com/robot/dedicated-server/raid/software-raid/#email-notification-when-a-drive-in-a-software-raid-fails Please use hetzner-status: https://www.hetzner-status.de/en.html This web page publishes announcements and current fault reports from our datacenters. Would you like to receive email notification of fault reports? Log on as exclusive Hetzner client in your administrations interface. If you have any questions please do not hesitate to contact us. Kind regards Jan Kolb Sales Hetzner Online GmbH Sigmundstrasse 135 90431 Nürnberg Tel: +49 911 234 226-927 Fax: +49 9831 505-3 sales@hetzner.com www.hetzner.com Register Court: Registergericht Ansbach, HRB 6089 CEO: Martin Hetzner, Stephan Konvickova, Günther Müller For the purposes of this communication, we may save some of your personal data. For information on our data privacy policy, please see: www.hetzner.com/datenschutzhinweis 09/29/2024 21:23 - marcin@opensourceecology.org michael@opensourceecology.org wrote: > > > Hi Hetzner, > > Can you please tell us more about the process of disk failure on our new dedicated > server plan (Server Auction #2443019)? > > Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk? > Or do we have to pay a fee? If so, how much? > > And does Hetzner have some system in-place that monitors the hardware for disk > failure? Or do we have to monitor this in software and alert Hetnzer that a disk > is failing? If Hetzner does monitor for disk failure, how does it do it? > > > Thank you, > > Michael Altfield > Senior Technology Advisor > PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B > > Open Source Ecology > www.opensourceecology.org >
- the docs linked-to actually don't mention mdadm, which I setup earlier to monitor and send us email alerts on our disks
- instead, hetzner mentions `smartctl`, which is included in the debian package `smartmontools` -- which wasn't even installed!
root@hetzner3 /etc/mdadm # sudo apt-get install smartmontools ... root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme0n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme1n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED root@hetzner3 /etc/mdadm #
- we can get more information with the `-A` argument
root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme0n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 36 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 3% Data Units Read: 142.729.615 [73,0 TB] Data Units Written: 20.452.874 [10,4 TB] Host Read Commands: 6.862.184.005 Host Write Commands: 876.931.661 Controller Busy Time: 15.948 Power Cycles: 28 Power On Hours: 16.350 Unsafe Shutdowns: 5 Media and Data Integrity Errors: 0 Error Information Log Entries: 159 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 36 Celsius Temperature Sensor 2: 45 Celsius root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme1n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 34 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 3% Data Units Read: 130.064.348 [66,5 TB] Data Units Written: 24.932.683 [12,7 TB] Host Read Commands: 1.276.781.490 Host Write Commands: 879.017.438 Controller Busy Time: 14.879 Power Cycles: 23 Power On Hours: 14.678 Unsafe Shutdowns: 5 Media and Data Integrity Errors: 0 Error Information Log Entries: 149 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 34 Celsius Temperature Sensor 2: 37 Celsius root@hetzner3 /etc/mdadm #
- oh nvm, their third link describes mdadm alerts for monitoring our software raid
- they also said to check /etc/default/mdadm, which I didn't do before
root@hetzner3 /etc/mdadm # cat /etc/default/mdadm # mdadm Debian configuration # # You can run 'dpkg-reconfigure mdadm' to modify the values in this file, if # you want. You can also change the values here and changes will be preserved. # Do note that only the values are preserved; the rest of the file is # rewritten. # # AUTOCHECK: # should mdadm run periodic redundancy checks over your arrays? See # /etc/cron.d/mdadm. AUTOCHECK=true # AUTOSCAN: # should mdadm check once a day for degraded arrays? See # /etc/cron.daily/mdadm. AUTOSCAN=true # START_DAEMON: # should mdadm start the MD monitoring daemon during boot? START_DAEMON=true # DAEMON_OPTIONS: # additional options to pass to the daemon. DAEMON_OPTIONS="--syslog" # VERBOSE: # if this variable is set to true, mdadm will be a little more verbose e.g. # when creating the initramfs. VERBOSE=false root@hetzner3 /etc/mdadm #
- note that "AUTOCHECK" is enabled -- so we're all good here.
...
- ok, back to updating wordpress.
- first, I'm just going to unzip all these (now TOFU-verified) .zip files and make sure there's no zipbombs
root@hetzner3 ~ # cd /var/tmp/wordpress/themes/ root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # ls bouquet.1.2.5.zip sketch.1.2.4.zip twentyfifteen.3.8.zip twentyseventeen.3.7.zip twentythirteen.4.2.zip gk-portfolio.1.5.3.zip storefront.4.6.0.zip twentyfourteen.4.0.zip twentysixteen.3.3.zip twentytwelve.4.3.zip portfolio-press.2.8.0.zip twentyeleven.4.7.zip twentynineteen.2.9.zip twentyten.4.2.zip root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # for file in $(ls *.zip); do unzip $file; done ... root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # ls bouquet portfolio-press.2.8.0.zip twentyeleven twentyfourteen.4.0.zip twentysixteen twentythirteen.4.2.zip bouquet.1.2.5.zip sketch twentyeleven.4.7.zip twentynineteen twentysixteen.3.3.zip twentytwelve gk-portfolio sketch.1.2.4.zip twentyfifteen twentynineteen.2.9.zip twentyten twentytwelve.4.3.zip gk-portfolio.1.5.3.zip storefront twentyfifteen.3.8.zip twentyseventeen twentyten.4.2.zip portfolio-press storefront.4.6.0.zip twentyfourteen twentyseventeen.3.7.zip twentythirteen root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # cd ../plugins/ root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # for file in $(ls *.zip); do unzip $file; done ... root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # ls akismet jetpack vcaching akismet.5.3.3.zip jetpack.13.8.1.zip vcaching.1.8.3.zip black-studio-tinymce-widget meta-box w3-total-cache black-studio-tinymce-widget.2.7.3.zip meta-box.5.10.2.zip w3-total-cache.2.7.6.zip chartbeat ml-slider wonderm00ns-simple-facebook-open-graph-tags chartbeat.2.0.7.zip ml-slider.3.91.0.zip wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip classic-editor open-in-new-window-plugin woocommerce classic-editor.1.6.5.zip open-in-new-window-plugin.3.0.zip woocommerce.9.3.3.zip coingate-for-woocommerce post-types-order wordpress-importer coingate-for-woocommerce.2.1.1.zip post-types-order.2.2.6.zip wordpress-importer.0.8.2.zip contact-form-7 revision-control wordpress-seo contact-form-7.5.9.8.zip revision-control.2.3.2.zip wordpress-seo.23.5.zip duplicate-page shareaholic wpautop-control duplicate-page.4.5.zip shareaholic.9.7.12.zip wpautop-control.1.6.zip duplicate-post share-on-diaspora wp-memory-usage duplicate-post.4.5.zip share-on-diaspora.0.7.9.zip wp-memory-usage.1.2.10.zip google-authenticator shariff wp-optimize google-authenticator.0.54.zip shariff.4.6.14.zip wp-optimize.3.6.0.zip google-authenticator-encourage-user-activation ssl-insecure-content-fixer wp-smushit google-authenticator-encourage-user-activation.0.2.zip ssl-insecure-content-fixer.2.7.2.zip wp-smushit.3.16.6.zip insert-headers-and-footers varnish-http-purge wp-super-cache insert-headers-and-footers.2.2.2.zip varnish-http-purge.5.2.2.zip wp-super-cache.1.12.4.zip root@hetzner3 /var/tmp/wordpress/plugins #
- ok, that looks good. now let's see if we can script copying-over these themes as-needed
- and, to err on the side of caution, I'm going to intentionally delete any theme or plugin dir, even if we don't have one to replace it.
wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
echo "${theme}"
rm -rf ${theme_path};
rsync -av --progress "/var/tmp/wordpress/themes/${theme}/" "${theme_path}/"
done
- after execution, looks like it worked
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/ total 68K d---r-x--- 16 not-apache www-data 4,0K Oct 3 04:02 . d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 .. ----r----- 1 not-apache www-data 28 Jun 5 2014 index.php drwxr-xr-x 2 root root 4,0K Oct 3 04:02 oshin drwxr-xr-x 5 root root 4,0K May 16 08:29 storefront drwxr-xr-x 7 root root 4,0K Jul 16 13:09 twentyeleven drwxr-xr-x 7 root root 4,0K Jul 16 13:28 twentyfifteen drwxr-xr-x 9 root root 4,0K Jul 16 13:23 twentyfourteen drwxr-xr-x 9 root root 4,0K Jul 16 13:30 twentynineteen drwxr-xr-x 5 root root 4,0K Jul 16 13:29 twentyseventeen drwxr-xr-x 8 root root 4,0K Jul 16 13:29 twentysixteen drwxr-xr-x 4 root root 4,0K Jul 15 17:17 twentyten drwxr-xr-x 8 root root 4,0K Jul 16 13:20 twentythirteen drwxr-xr-x 8 root root 4,0K Jul 16 13:17 twentytwelve drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentyfour drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentythree drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentytwo root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- oh, wait, no. it created some silly empty dirs when it didn't have a source to copy-from
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/oshin/ total 8,0K drwxr-xr-x 2 root root 4,0K Oct 3 04:02 . d---r-x--- 16 not-apache www-data 4,0K Oct 3 04:02 .. root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- let's wrap that in a condition. and also disable verbose & progress on rsync, so we can see the whole output
for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
- here's the execution; that's better
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
twentytwelve
twentysixteen
storefront
twentyseventeen
twentyfourteen
twentyeleven
twentytwentythree
oshin
twentytwentyfour
twentythirteen
twentyten
twentyfifteen
twentynineteen
twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 52K
d---r-x--- 12 not-apache www-data 4,0K Oct 3 04:04 .
d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 ..
----r----- 1 not-apache www-data 28 Jun 5 2014 index.php
drwxr-xr-x 5 root root 4,0K May 16 08:29 storefront
drwxr-xr-x 7 root root 4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x 7 root root 4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x 9 root root 4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x 9 root root 4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x 5 root root 4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x 8 root root 4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x 4 root root 4,0K Jul 15 17:17 twentyten
drwxr-xr-x 8 root root 4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x 8 root root 4,0K Jul 16 13:17 twentytwelve
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- now let's do the plugins with this
wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
- I actually messed this up, and I had to restore the original plugins dir from the backup; easy enough
rsync -av --progress /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/
- alright, here's the run
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
meta-box-show-hide
classic-editor
be-portfolio-post
colorhub
ssl-insecure-content-fixer
oshine-core
tatsu
revslider
redux-vendor-support
akismet
rename-wp-login
meta-box-tabs
google-authenticator
coingate-for-woocommerce
be-gdpr
google-authenticator-encourage-user-activation
typehub
meta-box
woocommerce
meta-box-conditional-logic
contact-form-7
vcaching
force-strong-passwords
masterslider
oshine-modules
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah plugins/
total 56K
d---r-x--- 12 1012 48 4,0K Oct 3 04:09 .
d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 ..
drwxr-xr-x 4 root root 4,0K Jul 10 22:16 akismet
drwxr-xr-x 3 root root 4,0K Sep 27 21:51 classic-editor
drwxr-xr-x 8 root root 4,0K Nov 21 2022 coingate-for-woocommerce
drwxr-xr-x 7 root root 4,0K Jul 25 08:28 contact-form-7
drwxr-xr-x 3 root root 4,0K Jul 4 2022 google-authenticator
drwxr-xr-x 4 root root 4,0K Apr 23 2021 google-authenticator-encourage-user-activation
----r----- 1 1012 48 2,3K Apr 9 2019 hello.php
----r----- 1 1012 48 28 Apr 9 2019 index.php
drwxr-xr-x 8 root root 4,0K Sep 27 07:22 meta-box
drwxr-xr-x 8 root root 4,0K Mar 17 2024 ssl-insecure-content-fixer
drwxr-xr-x 4 root root 4,0K Oct 21 2019 vcaching
drwxr-xr-x 13 root root 4,0K Sep 25 13:56 woocommerce
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- with that, I tried wp-cli again, but it gave us an empty plugin list?
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list +------+--------+--------+---------+----------------+-------------+ | name | status | update | version | update_version | auto_update | +------+--------+--------+---------+----------------+-------------+ +------+--------+--------+---------+----------------+-------------+ wp@hetzner3:~$
- oh shoot, I forgot to update permissions. I'll do that now
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
- ok, then I retry wp-cli; it works!
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 +------------------------------------------------+----------+--------+---------+----------------+-------------+ | name | status | update | version | update_version | auto_update | +------------------------------------------------+----------+--------+---------+----------------+-------------+ | akismet | inactive | none | 5.3.3 | | off | | classic-editor | inactive | none | 1.6.5 | | off | | contact-form-7 | active | none | 5.9.8 | | off | | google-authenticator-encourage-user-activation | active | none | 0.2 | | off | | google-authenticator | active | none | 0.54 | | off | | hello | inactive | none | 1.7.1 | | off | | meta-box | active | none | 5.10.2 | | off | | ssl-insecure-content-fixer | active | none | 2.7.2 | | off | | vcaching | active | none | 1.8.3 | | off | | woocommerce | active | none | 9.3.3 | | off | | coingate-for-woocommerce | inactive | none | 2.1.1 | | off | +------------------------------------------------+----------+--------+---------+----------------+-------------+ wp@hetzner3:~$
- unfortunately, I get a blank page when I try to load store.opensourceecology.org in my web browser
- nginx is fine, but the varnish logs show that apache is returning a 403
[Thu Oct 03 04:19:37.076411 2024] [authz_core:error] [pid 3116759:tid 3116768] [client 81.17.16.77:0] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png, referer: https://store.opensourceecology.org/ ==> modsec_audit.log <== --fd8c6d25-A-- [03/Oct/2024:04:19:37.076625 +0000] Zv4bWZVyO5GHCka9cecUKwAAAEE 127.0.0.1 40720 127.0.0.1 8000 --fd8c6d25-B-- GET /wp-includes/images/w-logo-blue-white-bg.png HTTP/1.1 X-Real-IP: 81.17.16.77 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: store.opensourceecology.org User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Accept: image/avif,image/webp,*/* Accept-Language: en-US,en;q=0.5 Referer: https://store.opensourceecology.org/ Sec-Fetch-Dest: image Sec-Fetch-Mode: no-cors Sec-Fetch-Site: same-origin Sec-GPC: 1 Pragma: no-cache Accept-Encoding: gzip hash: #store.opensourceecology.org X-Varnish: 98343 --fd8c6d25-F-- HTTP/1.1 403 Forbidden X-Frame-Options: SAMEORIGIN Content-Length: 199 Content-Type: text/html; charset=iso-8859-1 --fd8c6d25-E-- <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> --fd8c6d25-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png Stopwatch: 1727929177076046 856 (- - -) Stopwatch2: 1727929177076046 856; combined=26, p1=24, p2=0, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --fd8c6d25-Z--
