Maltfield Log/2024 Q4

My work log from the fourth quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

1. Maltfield_Log
2. User:Maltfield
3. Special:Contributions/Maltfield

Wed Oct 02, 2024

1. Marcin sent me a few emails in the past months asking about OSE's use of Amazon Glacier
2. Today he sent a message saying that he got charged $1.03, and isn't sure why

Michael,

I'm getting charged $1.03 for Glacier. Can we cancel that?

Marcin

1. It took me a while to auth
   1. first I tried to login with my 'maltfield' aws user, but aws rejected my creds (stored in my personal keepass)
   2. eventually I realized I had to click "Sign in using root user email" -- and then I could auth using the creds stored in the shared keepass
2. after logging-in, I went to the "Billing and Cost Management" app https://us-east-1.console.aws.amazon.com/costmanagement/home?region=us-west-2#/home
3. on this page, there was a link that said "Last month's total cost: $1.03". Yep, that's all accounted-for. I clicked it.
4. the next page showed a joke of a chart with one bar on a bar graph that said "$1.03". And the bar was labeled "Total Cost"
5. I had to click on the dropdown menu for "Dimension" and set it to "Service" -- then it listed 4 items
   1. Glacier - $1.03
   2. S3 - $0.00
   3. Tax $0.00
6. So I switched over to the "Glacier" app https://us-east-1.console.aws.amazon.com/glacier/home?region=us-east-1
   1. Curiously, it listed 0 vaults
   2. but there was a note at the top saying we should use S3 for glaicer, so I clicked over to the "S3" app

We recommend that you use Glacier storage classes in Amazon S3 for archival storage

1. here I saw one bucket called "oseserverbackups" in "US West (Oregon) us-west-2"
2. the bucket had one 34.0 byte file in it called "test.txt". That's it!
   1. this file was created July 6, 2018, 19:18:03 (UTC-05:00)
   2. I downloaded it; it has one line of text

some file destined for s3 this is

1. 1. I deleted the 'test.txt' file object from the s3 bucket
2. I then deleted the (now empty) 'oseserverbackups' bucket
3. unconvinced that that was the issue, I went back to the "glacier" app. This time I cycled through a few of the regions until I got to "us-west-2" -- this time I showed one vault named "deleteMeIn2020"
4. I clicked on it, and it said
   1. this vault was created March 29, 2018, 16:36:06 (UTC-05:00)
   2. this vault was last inventoried August 1, 2018, 02:41:31 (UTC-05:00)
   3. this vault is 285.3 GB (as of last inventory)
5. well, it's after 2020. So I think we should delete it.
6. I sent an email to Marcin asking for a confirmation before I delete it

Hey Marcin,

You have a 285.3 GB vault in Amazon Glacier's us-west-2 region.

I logged-into your AWS account today and did some digging. I found this vault 285.3 GB vault named 'deleteMeIn2020'. I created this vault in 2018 Q1. It contains a final backup of files from hetzner1. I created it as part of the hetzner2 migration project, thinking that we should delete it in 2020 if we never needed to restore anything from it for 2 years.

 * https://wiki.opensourceecology.org/wiki/CHG-2018-07-06_hetzner1_deprecation

 * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q1#Sat_Mar_31.2C_2018

Well, 2020 came and past. Four more years passed. I think you can safely delete the 'deleteMeIn2020' vault.

By the way, I also deleted a 53-byte test file from an S3 bucket named 'test.txt' in a bucket in s3 called 'oseserverbackups' in us-west-2. It was the only file in the bucket. I deleted the file and the empty bucket.

Would you like me to proceed with deleting the 285.3 GB 'deleteMeIn2020' glacier bucket from your AWS account?


Thank you,

Michael Altfield
Senior Technology Advisor
PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B

Open Source Ecology
www.opensourceecology.org

1. meanwhile, I tried to figure out why I couldn't login as 'maltfield', and I realized that, ffs, we don't have IAM setup for our account?? Maybe Marcin deleted it when trying to elimiate costs? IAM is free, though..
2. ok, I found my 'maltfield' user under "Security Credentials" -> "Access Management" -> "Users"
3. it says my last console sign-in was 424 days ago
4. I went to my user's settings, selected the MFA token, and selected "Resync" -- then entered two consecutive OTPs
5. I tried to login, and this time it let me in. Well that was annoying.
6. I opened cloudtrail and reviewed the latest account events https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?ReadOnly=false
   1. the most recent event was the 'root' user resyncing the MFA token of the 'matlfield' token
   2. before that we have two ConsoleLogin for today
   3. before that 'mjakubowski' user has a MakePayment event (and some other payment related events) on Sep 19
   4. before that we have a bunch of login & mfa-related entries for Marcin's user on Sep 06, 14, 17, and 19.
   5. and that's where the log ends; looks like we just get 90 days of logs for free.

...

1. hetzner responded to my support inquery about how they handle failed disks

Dear Mr Altfield

Unfortunately it's an unmanaged root server monitoring is your responsibility I'm afraid. 


If you have a problem please open a ticket in your robot account. 

Please click on "Servers" from the menu on the left and then select the corresponding server. Under the "Support" tab, you can choose "Hard drive is broken". Please follow the instructions.

https://docs.hetzner.com/robot/dedicated-server/troubleshooting/serial-numbers-and-information-on-defective-hard-drives/


Our DC is 24/7 available and we exchange broken hardware as soon as possible for free.


Hetzner clients can use the Server Monitoring System to monitor their servers and have an email sent to them when the status of one of the monitored services changes: 

https://docs.hetzner.com/robot/dedicated-server/security/system-monitor/

https://docs.hetzner.com/robot/dedicated-server/raid/software-raid/#email-notification-when-a-drive-in-a-software-raid-fails


Please use hetzner-status:

https://www.hetzner-status.de/en.html

This web page publishes announcements and current fault reports from our datacenters. Would you like to receive email notification of fault reports? Log on as exclusive Hetzner client in your administrations interface. 


If you have any questions please do not hesitate to contact us. 

Kind regards

Jan Kolb

Sales

Hetzner Online GmbH
Sigmundstrasse 135
90431 Nürnberg
Tel: +49 911 234 226-927
Fax: +49 9831 505-3
sales@hetzner.com
www.hetzner.com

Register Court: Registergericht Ansbach, HRB 6089
CEO: Martin Hetzner, Stephan Konvickova, Günther Müller

For the purposes of this communication, we may save some 
of your personal data. For information on our data privacy 
policy, please see: www.hetzner.com/datenschutzhinweis


09/29/2024 21:23 - marcin@opensourceecology.org michael@opensourceecology.org wrote:

>
>
> Hi Hetzner,
>
> Can you please tell us more about the process of disk failure on our new dedicated
> server plan (Server Auction #2443019)?
>
> Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk?
> Or do we have to pay a fee? If so, how much?
>
> And does Hetzner have some system in-place that monitors the hardware for disk
> failure? Or do we have to monitor this in software and alert Hetnzer that a disk
> is failing? If Hetzner does monitor for disk failure, how does it do it?
>
>
> Thank you,
>
> Michael Altfield
> Senior Technology Advisor
> PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B
>
> Open Source Ecology
> www.opensourceecology.org
>

1. the docs linked-to actually don't mention mdadm, which I setup earlier to monitor and send us email alerts on our disks
2. instead, hetzner mentions `smartctl`, which is included in the debian package `smartmontools` -- which wasn't even installed!

root@hetzner3 /etc/mdadm # sudo apt-get install smartmontools
...
root@hetzner3 /etc/mdadm # 

root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme1n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

root@hetzner3 /etc/mdadm # 

1. we can get more information with the `-A` argument

root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        36 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    142.729.615 [73,0 TB]
Data Units Written:                 20.452.874 [10,4 TB]
Host Read Commands:                 6.862.184.005
Host Write Commands:                876.931.661
Controller Busy Time:               15.948
Power Cycles:                       28
Power On Hours:                     16.350
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      159
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               36 Celsius
Temperature Sensor 2:               45 Celsius

root@hetzner3 /etc/mdadm # 

root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme1n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    130.064.348 [66,5 TB]
Data Units Written:                 24.932.683 [12,7 TB]
Host Read Commands:                 1.276.781.490
Host Write Commands:                879.017.438
Controller Busy Time:               14.879
Power Cycles:                       23
Power On Hours:                     14.678
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      149
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               34 Celsius
Temperature Sensor 2:               37 Celsius

root@hetzner3 /etc/mdadm # 

1. oh nvm, their third link describes mdadm alerts for monitoring our software raid
2. they also said to check /etc/default/mdadm, which I didn't do before

root@hetzner3 /etc/mdadm # cat /etc/default/mdadm 
# mdadm Debian configuration
#
# You can run 'dpkg-reconfigure mdadm' to modify the values in this file, if
# you want. You can also change the values here and changes will be preserved.
# Do note that only the values are preserved; the rest of the file is
# rewritten.
#

# AUTOCHECK:
#   should mdadm run periodic redundancy checks over your arrays? See
#   /etc/cron.d/mdadm.
AUTOCHECK=true

# AUTOSCAN:
#   should mdadm check once a day for degraded arrays? See
#   /etc/cron.daily/mdadm.
AUTOSCAN=true

# START_DAEMON:
#   should mdadm start the MD monitoring daemon during boot?
START_DAEMON=true

# DAEMON_OPTIONS:
#   additional options to pass to the daemon.
DAEMON_OPTIONS="--syslog"

# VERBOSE:
#   if this variable is set to true, mdadm will be a little more verbose e.g.
#   when creating the initramfs.
VERBOSE=false
root@hetzner3 /etc/mdadm # 

1. note that "AUTOCHECK" is enabled -- so we're all good here.

...

1. ok, back to updating wordpress.
2. first, I'm just going to unzip all these (now TOFU-verified) .zip files and make sure there's no zipbombs

root@hetzner3 ~ # cd /var/tmp/wordpress/themes/
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # ls
bouquet.1.2.5.zip          sketch.1.2.4.zip      twentyfifteen.3.8.zip   twentyseventeen.3.7.zip  twentythirteen.4.2.zip
gk-portfolio.1.5.3.zip     storefront.4.6.0.zip  twentyfourteen.4.0.zip  twentysixteen.3.3.zip    twentytwelve.4.3.zip
portfolio-press.2.8.0.zip  twentyeleven.4.7.zip  twentynineteen.2.9.zip  twentyten.4.2.zip
root@hetzner3 /var/tmp/wordpress/themes #

root@hetzner3 /var/tmp/wordpress/themes # for file in $(ls *.zip); do unzip $file; done
...
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # ls
bouquet                 portfolio-press.2.8.0.zip  twentyeleven           twentyfourteen.4.0.zip   twentysixteen          twentythirteen.4.2.zip
bouquet.1.2.5.zip       sketch                     twentyeleven.4.7.zip   twentynineteen           twentysixteen.3.3.zip  twentytwelve
gk-portfolio            sketch.1.2.4.zip           twentyfifteen          twentynineteen.2.9.zip   twentyten              twentytwelve.4.3.zip
gk-portfolio.1.5.3.zip  storefront                 twentyfifteen.3.8.zip  twentyseventeen          twentyten.4.2.zip
portfolio-press         storefront.4.6.0.zip       twentyfourteen         twentyseventeen.3.7.zip  twentythirteen
root@hetzner3 /var/tmp/wordpress/themes #

root@hetzner3 /var/tmp/wordpress/themes # cd ../plugins/
root@hetzner3 /var/tmp/wordpress/plugins #

root@hetzner3 /var/tmp/wordpress/plugins # for file in $(ls *.zip); do unzip $file; done
...
root@hetzner3 /var/tmp/wordpress/plugins #

root@hetzner3 /var/tmp/wordpress/plugins # ls
akismet                                                 jetpack                               vcaching
akismet.5.3.3.zip                                       jetpack.13.8.1.zip                    vcaching.1.8.3.zip
black-studio-tinymce-widget                             meta-box                              w3-total-cache
black-studio-tinymce-widget.2.7.3.zip                   meta-box.5.10.2.zip                   w3-total-cache.2.7.6.zip
chartbeat                                               ml-slider                             wonderm00ns-simple-facebook-open-graph-tags
chartbeat.2.0.7.zip                                     ml-slider.3.91.0.zip                  wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip
classic-editor                                          open-in-new-window-plugin             woocommerce
classic-editor.1.6.5.zip                                open-in-new-window-plugin.3.0.zip     woocommerce.9.3.3.zip
coingate-for-woocommerce                                post-types-order                      wordpress-importer
coingate-for-woocommerce.2.1.1.zip                      post-types-order.2.2.6.zip            wordpress-importer.0.8.2.zip
contact-form-7                                          revision-control                      wordpress-seo
contact-form-7.5.9.8.zip                                revision-control.2.3.2.zip            wordpress-seo.23.5.zip
duplicate-page                                          shareaholic                           wpautop-control
duplicate-page.4.5.zip                                  shareaholic.9.7.12.zip                wpautop-control.1.6.zip
duplicate-post                                          share-on-diaspora                     wp-memory-usage
duplicate-post.4.5.zip                                  share-on-diaspora.0.7.9.zip           wp-memory-usage.1.2.10.zip
google-authenticator                                    shariff                               wp-optimize
google-authenticator.0.54.zip                           shariff.4.6.14.zip                    wp-optimize.3.6.0.zip
google-authenticator-encourage-user-activation          ssl-insecure-content-fixer            wp-smushit
google-authenticator-encourage-user-activation.0.2.zip  ssl-insecure-content-fixer.2.7.2.zip  wp-smushit.3.16.6.zip
insert-headers-and-footers                              varnish-http-purge                    wp-super-cache
insert-headers-and-footers.2.2.2.zip                    varnish-http-purge.5.2.2.zip          wp-super-cache.1.12.4.zip
root@hetzner3 /var/tmp/wordpress/plugins # 

1. ok, that looks good. now let's see if we can script copying-over these themes as-needed
   1. and, to err on the side of caution, I'm going to intentionally delete any theme or plugin dir, even if we don't have one to replace it.

wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	
	echo "${theme}"
	rm -rf ${theme_path};
	rsync -av --progress "/var/tmp/wordpress/themes/${theme}/" "${theme_path}/"
done

1. after execution, looks like it worked

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 68K
d---r-x--- 16 not-apache www-data 4,0K Oct  3 04:02 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
----r-----  1 not-apache www-data   28 Jun  5  2014 index.php
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 oshin
drwxr-xr-x  5 root       root     4,0K May 16 08:29 storefront
drwxr-xr-x  7 root       root     4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x  7 root       root     4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x  5 root       root     4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x  4 root       root     4,0K Jul 15 17:17 twentyten
drwxr-xr-x  8 root       root     4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:17 twentytwelve
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentyfour
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentythree
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # 

1. oh, wait, no. it created some silly empty dirs when it didn't have a source to copy-from

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/oshin/
total 8,0K
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 .
d---r-x--- 16 not-apache www-data 4,0K Oct  3 04:02 ..
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # 

1. let's wrap that in a condition. and also disable verbose & progress on rsync, so we can see the whole output

for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	source_path="/var/tmp/wordpress/themes/${theme}"
	
	echo "${theme}"
	rm -rf ${theme_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${theme_path}/"
	fi
done

1. here's the execution; that's better

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
        theme=$(basename "${theme_path}")
        source_path="/var/tmp/wordpress/themes/${theme}"
        
        echo "${theme}"
        rm -rf ${theme_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${theme_path}/"
        fi
done
twentytwelve
twentysixteen
storefront
twentyseventeen
twentyfourteen
twentyeleven
twentytwentythree
oshin
twentytwentyfour
twentythirteen
twentyten
twentyfifteen
twentynineteen
twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 52K
d---r-x--- 12 not-apache www-data 4,0K Oct  3 04:04 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
----r-----  1 not-apache www-data   28 Jun  5  2014 index.php
drwxr-xr-x  5 root       root     4,0K May 16 08:29 storefront
drwxr-xr-x  7 root       root     4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x  7 root       root     4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x  5 root       root     4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x  4 root       root     4,0K Jul 15 17:17 twentyten
drwxr-xr-x  8 root       root     4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:17 twentytwelve
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # 

1. now let's do the plugins with this

wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
	plugin=$(basename "${plugin_path}")
	source_path="/var/tmp/wordpress/plugins/${plugin}"
	
	echo "${plugin}"
	rm -rf ${plugin_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${plugin_path}/"
	fi
done

1. I actually messed this up, and I had to restore the original plugins dir from the backup; easy enough

rsync -av --progress /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/

1. alright, here's the run

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
        plugin=$(basename "${plugin_path}")
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        echo "${plugin}"
        rm -rf ${plugin_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done
meta-box-show-hide
classic-editor
be-portfolio-post
colorhub
ssl-insecure-content-fixer
oshine-core
tatsu
revslider
redux-vendor-support
akismet
rename-wp-login
meta-box-tabs
google-authenticator
coingate-for-woocommerce
be-gdpr
google-authenticator-encourage-user-activation
typehub
meta-box
woocommerce
meta-box-conditional-logic
contact-form-7
vcaching
force-strong-passwords
masterslider
oshine-modules
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah plugins/
total 56K
d---r-x--- 12       1012       48 4,0K Oct  3 04:09 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
drwxr-xr-x  4 root       root     4,0K Jul 10 22:16 akismet
drwxr-xr-x  3 root       root     4,0K Sep 27 21:51 classic-editor
drwxr-xr-x  8 root       root     4,0K Nov 21  2022 coingate-for-woocommerce
drwxr-xr-x  7 root       root     4,0K Jul 25 08:28 contact-form-7
drwxr-xr-x  3 root       root     4,0K Jul  4  2022 google-authenticator
drwxr-xr-x  4 root       root     4,0K Apr 23  2021 google-authenticator-encourage-user-activation
----r-----  1       1012       48 2,3K Apr  9  2019 hello.php
----r-----  1       1012       48   28 Apr  9  2019 index.php
drwxr-xr-x  8 root       root     4,0K Sep 27 07:22 meta-box
drwxr-xr-x  8 root       root     4,0K Mar 17  2024 ssl-insecure-content-fixer
drwxr-xr-x  4 root       root     4,0K Oct 21  2019 vcaching
drwxr-xr-x 13 root       root     4,0K Sep 25 13:56 woocommerce
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # 

1. with that, I tried wp-cli again, but it gave us an empty plugin list?

wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
+------+--------+--------+---------+----------------+-------------+
| name | status | update | version | update_version | auto_update |
+------+--------+--------+---------+----------------+-------------+
+------+--------+--------+---------+----------------+-------------+
wp@hetzner3:~$ 

1. oh shoot, I forgot to update permissions. I'll do that now

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do

	wp_docroot="$(dirname "${wordpress_site}")"
	vhost_dir="$(dirname "${wp_docroot}")"

	chown -R not-apache:www-data "${vhost_dir}"
	find "${vhost_dir}" -type d -exec chmod 0050 {} \;
	find "${vhost_dir}" -type f -exec chmod 0040 {} \;

	chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
	chmod 0040 "${vhost_dir}/wp-config.php"

	[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
	find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;

	[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
	find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;

done

1. ok, then I retry wp-cli; it works!

wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+------------------------------------------------+----------+--------+---------+----------------+-------------+
| name                                           | status   | update | version | update_version | auto_update |
+------------------------------------------------+----------+--------+---------+----------------+-------------+
| akismet                                        | inactive | none   | 5.3.3   |                | off         |
| classic-editor                                 | inactive | none   | 1.6.5   |                | off         |
| contact-form-7                                 | active   | none   | 5.9.8   |                | off         |
| google-authenticator-encourage-user-activation | active   | none   | 0.2     |                | off         |
| google-authenticator                           | active   | none   | 0.54    |                | off         |
| hello                                          | inactive | none   | 1.7.1   |                | off         |
| meta-box                                       | active   | none   | 5.10.2  |                | off         |
| ssl-insecure-content-fixer                     | active   | none   | 2.7.2   |                | off         |
| vcaching                                       | active   | none   | 1.8.3   |                | off         |
| woocommerce                                    | active   | none   | 9.3.3   |                | off         |
| coingate-for-woocommerce                       | inactive | none   | 2.1.1   |                | off         |
+------------------------------------------------+----------+--------+---------+----------------+-------------+
wp@hetzner3:~$ 

1. unfortunately, I get a blank page when I try to load store.opensourceecology.org in my web browser
2. nginx is fine, but the varnish logs show that apache is returning a 403

[Thu Oct 03 04:19:37.076411 2024] [authz_core:error] [pid 3116759:tid 3116768] [client 81.17.16.77:0] AH01630: client denied by server configuration: 
/var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png, referer: https://store.opensourceecology.org/

==> modsec_audit.log <==
--fd8c6d25-A--
[03/Oct/2024:04:19:37.076625 +0000] Zv4bWZVyO5GHCka9cecUKwAAAEE 127.0.0.1 40720 127.0.0.1 8000
--fd8c6d25-B--
GET /wp-includes/images/w-logo-blue-white-bg.png HTTP/1.1
X-Real-IP: 81.17.16.77
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: store.opensourceecology.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36
Accept: image/avif,image/webp,*/*
Accept-Language: en-US,en;q=0.5
Referer: https://store.opensourceecology.org/
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
Sec-GPC: 1
Pragma: no-cache
Accept-Encoding: gzip
hash: #store.opensourceecology.org
X-Varnish: 98343

--fd8c6d25-F--
HTTP/1.1 403 Forbidden
X-Frame-Options: SAMEORIGIN
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1

--fd8c6d25-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

--fd8c6d25-H--
Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png
Stopwatch: 1727929177076046 856 (- - -)
Stopwatch2: 1727929177076046 856; combined=26, p1=24, p2=0, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

--fd8c6d25-Z--