Maltfield Log/2024 Q4
My work log from the fourth quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
Contents
See Also
Mon Oct 07, 2024
- I installed the latest version of the 'oshine' theme to the store.opensourceecology.org wordpress site
rsync -av --progress /var/tmp/wordpress/themes/oshin /var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
- ok, after that loading it in the browser still yeilds a blank page, but it's just because it's cached https://store.opensourceecology.org/
- if I just append a bullshit GET variable on the end, then it loads https://store.opensourceecology.org/?nocache=2
- the site is finally loading, but it's all fucked
- I'm not seeing any 403 errors in the network tab of firefox on-load, so I'm pretty sure the issue is just missing plugins
- for example, there's a bunch of shortcodes being displayed raw, like this one at the top
[tatsu_section bg_color= “rgba(29,29,29,1)” bg_image= “http://brandexponents.com/oshine-lite/v37/wp-content/uploads/sites/44/2018/02/home-hero.jpeg” bg_repeat= “no-repeat” bg_attachment= “scroll” bg_position= “center center” bg_size= “cover” bg_animation= “none” padding= ‘{“d”:”200px 0% 200px 0% “}’ margin= “0px 0px 0px 0px” border= “0px 0px px 0px” border_color= “” bg_video= “0” bg_video_mp4_src= “” bg_video_ogg_src= “” bg_video_webm_src= “” bg_overlay= “1” overlay_color= “rgba(0,0,0,0.1)” full_screen= “1” section_id= “” section_class= “” section_title= “” offset_section= “” offset_value= “0” full_screen_header_scheme= “background–dark” hide_in= “0” bg_stretch= “1” key= “REDACTED”]
- ok, if I login to the WUI and go to appearance -> themes, now it does show the oshine theme https://store.opensourceecology.org/wp-admin/themes.php
- I deactivated it and reactivated it, and I got a message at the top
This theme requires the following plugins: BE Portfolio Post Type, Meta Box Conditional Logic, Meta Box Show Hide, Meta Box Tabs, Oshine Core, Oshine Modules and Tatsu. This theme recommends the following plugins: BE GDPR, Master Slider, Safe SVG, Slider Revolution and WPForms Lite. Begin installing plugins | Dismiss this notice
- If I click to customize the theme, it has a tab "Install Plugins"
- If I click the "Install Plugins" tab, it yells at me with big red text
Please provide a valid purchase code of the theme in order to install plugins and import demo
- on another DispVM, I logged into the hetzner2 store.opensourceecology.org
- clicked appearance -> themes -> oshin -> customize
- no, that didn't work
- clicked "Oshine Options" in the left-hand navbar ->
- no, I went through all the settings and couldn't find it there either :(
- I mean, I have the keys already downloaded, but I'd like to keep them consistent. We have two keys and I don't know which was used for store.opensourceecology.org. This is dumb. Why do they make it so hard to find?
- I tried pulling it out of the DB, but I didn't find anything obvious in the options table named "*oshine*"
MariaDB [store_db]> select * from wp_options where option_name like '%oshine%' limit 100;
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
| option_id | option_name | option_value | autoload |
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
| 347 | external_updates-oshine-core | O:8:"stdClass":3:{s:9:"lastCheck";i:1728339766;s:14:"checkedVersion";s:5:"1.3.7";s:6:"update";N;} | no |
| 349 | external_updates-oshine-modules | O:8:"stdClass":3:{s:9:"lastCheck";i:1728339766;s:14:"checkedVersion";s:5:"2.2.9";s:6:"update";N;} | no |
| 352 | oshine_redux_to_colorhub | 1 | yes |
| 355 | oshine_redux_to_typehub | 1 | yes |
+-----------+---------------------------------+---------------------------------------------------------------------------------------------------+----------+
4 rows in set (0.00 sec)
MariaDB [store_db]>
- I realized it's probably easier to just search the mysqldump file
- got it!
root@hetzner3 /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/mysqldump # grep -ir 'purchase' mysqldump.20240926_072001b.sql | grep -ir code | less
...
193,'be_themes_purchase_data','a:2:{s:8:\"last_tab\";s:0:\"\";s:19:\"theme_purchase_code\";s:36:\"REDACTED\";}','yes'),(194,'be_themes_purchase_data-transients','a:2:{s:14:\"changed_values\";a:0:{}s:9:\"last_save\";i:1471011372;}','yes')
- actually, that still doesn't tell me which server it is
- I think that's the wrong site's db, because I see nothing when I query just the store wordpress db
MariaDB [store_db]> select * from wp_options where option_name like '%be%' and option_value like '%theme_purchase_code%' limit 100; Empty set (0.00 sec) MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase_code%' limit 100; Empty set (0.01 sec) MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase_data%' limit 100; Empty set (0.01 sec) MariaDB [store_db]> select * from wp_options where option_value like '%themes_purchase_data%' limit 100; Empty set (0.01 sec) MariaDB [store_db]> select * from wp_options where option_value like '%theme_purchase%' limit 100; Empty set (0.01 sec) MariaDB [store_db]>
- I exported the data from the hetzner2 store theme too, but it wasn't there
user@disp928:~/Downloads$ du -sh * 28K redux_options_be_themes_data_backup_07-10-2024.json user@disp928:~/Downloads$
- fuck it, I'm just going to use these alphabetically
- which sites use this?
[root@opensourceecology hetzner3]# nice find /var/www/html -type d -iname oshin /var/www/html/www.openbuildinginstitute.org/htdocs/wp-content/themes/oshin /var/www/html/d3d.opensourceecology.org/htdocs/wp-content/themes/oshine_6.5/Oshine Buyers Package 6.5/oshin /var/www/html/d3d.opensourceecology.org/htdocs/wp-content/themes/oshin /var/www/html/microfactory.opensourceecology.org/htdocs/wp-content/themes/oshin /var/www/html/staging.openbuildinginstitute.org/htdocs/wp-content/themes/oshin /var/www/html/store.opensourceecology.org/htdocs/wp-content/themes/oshin /var/www/html/3dp.opensourceecology.org/htdocs/wp-content/themes/oshin [root@opensourceecology hetzner3]#
- d3d and 3dp are both broken with cert errors right now
- right, my notes say these were two sites that marcin abandoned domain naames for. we eventually built microfactory.opensourceecology.org instead
- ugh, power went out
Sun Oct 06, 2024
- I checked on the status of the inventory job of our 'deleteMeIn2020' galcier vault; looks like it's still unavailable. Guess I'll give it a week or so before trying again
user@disp8678:~$ aws configure set aws_access_key_id 'REDACTED" user@disp8678:~$ user@disp8678:~$ aws configure set aws_secret_access_key 'REDACTED' user@disp8678:~$ user@disp8678:~$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (ResourceNotFoundException) when calling the GetJobOutput operation: The job ID was not found: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS user@disp8678:~$
...
- I returned to work on fixing the vhost config to permit traffic to wp-config.php temporarily, but I kept getting 429 errors from wordpress.org
- This has been a frustrating, recurring issue for many months. I finally filed a bug report https://meta.trac.wordpress.org/ticket/7792#ticket
Title: Too Many "429 Too Many Requests" Errors (Nginx Misconfiguration causing False-Positives) Since the past ~6 months, I have been frequently unable to access content on wordpress.org If I'm lucky, then when I'm browsing wordpress documentation pages, I'm able to load the main html file with the content, but the website is horribly mis-rendered because many dependent assets don't load (eg css files, images, javascript, etc) due to "429 Too Many Requests" errors. If I'm unlucky, even the main page doesn't load load at all -- due to "429 Too Many Requests". Usually, I start-off being able to load one or more pages, but as I click around the website trying to find the page that I need, I eventually get this error. I am not a bot. I am a human. I'm just trying to load reference documentation as I develop a wordpress plugin. This has been extremely frustrating, and forced me to third party websites and to "guess" php functions, attributes, and return values as I'm developing, reducing my productivity. Since the Snowden revelations of 2013, it's become clear that many at-risk users should not be using the Internet without using privacy-protections like Tor. For security and privacy reasons, I do not access the internet without passing my traffic through Tor or a VPN. To prevent discrimination against at-risk folks, it's important that WordPress servers do not block traffic from shared networks, such as VPNs or Tor exit nodes. It appears that nginx's settings are too strict, and lots of good users are getting caught in the dragnet. Whatever the current nginx config is, please double it to fix these false-positives.
- alright, I updated the apache config and pushed it with ansible
user@personal:~/sandbox_local/ansible/hetzner3$ git diff
diff --git a/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2 b/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
index c0575a3..c413c74 100644
--- a/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
+++ b/hetzner3/roles/maltfield.apache/templates/security.virtualhost.include.j2
@@ -2,12 +2,12 @@
################################################################################
# File: security.virtualhost.include
-# Version: 0.2
+# Version: 0.3
# Purpose: File includes some common security-hardening that's intended to be
# Include()d into other vhost files' <VirtualHost> blocks
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2024-09-14
-# Updated: 2024-09-24
+# Updated: 2024-10-06
################################################################################
# don't execute any php files inside uploads directories
@@ -56,7 +56,10 @@
# block access to 'wp-login.php' from brute-forcers;
# see wp plugin 'rename-wp-login'
- <LocationMatch ".*wp-login.php">
- Require all denied
- </LocationMatch>
+ # TODO: 2024-10: we need to re-enable this after we find a replacement for the
+ # (now-deprecated) 'rename-wp-login' wordpress plugin
+ # * https://wordpress.org/plugins/rename-wp-login/
+# <LocationMatch ".*wp-login.php">
+# Require all denied
+# </LocationMatch>
user@personal:~/sandbox_local/ansible/hetzner3$
- now I'm able to load the login page, but when I do, I still get 403 errors on a few of the dependent requests
- the mod_security log shows the 403 response for these images, but I'm not sure why it's happening
root@hetzner3 /var/log/apache2 # cat modsec_audit.log ... --69ede32a-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-admin/images/wordpress-logo.svg Stopwatch: 1728256656835561 1333 (- - -) Stopwatch2: 1728256656835561 1333; combined=33, p1=31, p2=0, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --69ede32a-Z-- ... --6a3d866d-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png Stopwatch: 1728256938397718 1105 (- - -) Stopwatch2: 1728256938397718 1105; combined=27, p1=23, p2=0, p3=0, p4=0, p5=3, sr=0, sw=1, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --6a3d866d-Z--
- oh, I wonder if this is it
user@personal:~/sandbox_local/ansible/hetzner3$ grep -irC4 'require all' | less ... roles/maltfield.apache/templates/security.virtualhost.include.j2- <LocationMatch "/images/"> roles/maltfield.apache/templates/security.virtualhost.include.j2- SetHandler ! roles/maltfield.apache/templates/security.virtualhost.include.j2: Require all denied roles/maltfield.apache/templates/security.virtualhost.include.j2- </LocationMatch>
- so one thing we changed from hetzner2 was the logic for preventing php scripts from being executed inside user-uploadable directories.
- in hetzner2, we used mod_php, so this was done with 'php_flag engine off' -- but that doesn't work with php-fpm
# don't execute any php files inside the uploads directory
<LocationMatch "/wp-content/uploads/">
php_flag engine off
</LocationMatch>
<LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Order Deny,Allow
Deny from All
</LocationMatch>
# block dot files, such as svn files from checking out wp core
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
# block access to 'wp-login.php' from brute-forcers; see wp plugin 'rename-wp-login'
<LocationMatch .*wp-login.php>
Deny From All
</LocationMatch>
- I think the idea of "Require all denied" was that it would capture anything that wasn't already sent-off to the php-fpm server proxy. Basically because that should be captured first, and then if apache still sees it, we should deny access to it
- this assumes that the filename ends with php (or our more complex regex above), but that logic doesn't translate for the whole /images/ directory
- alright, I made these changes, which fixed it. basically we want to "SetHandler !" on everything, but only "Require all denied" for the .php files
- Note: I wasn't able to figure out what "SetHandler !" does. Bing/ddg returns no results. And Google just ignores any queries with an exclamation mark in them. It's literally not possible to search-for. But I did find lots of results asking about how to use SetHandler in Apache to point to php-fpm, so my best-guess is that this sets the Handler to 'null' or something, which would overwrite any previous setting that told it to send it to some other cgi proxy or something
# don't execute any php files inside uploads directories
<LocationMatch "/wp-content/uploads/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -21,7 +20,6 @@
<LocationMatch "/uploadimages/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/uploadimages/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -29,7 +27,6 @@
<LocationMatch "/images/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/images/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -38,7 +35,6 @@
# don't execute php files in W3 Total Cache's tmp dir
<LocationMatch "/wp-content/cache/">
SetHandler !
- Require all denied
</LocationMatch>
<LocationMatch "/wp-content/cache/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$">
Require all denied
@@ -46,17 +42,22 @@
# block dot (hidden) files
<LocationMatch "/\.(?!well\-known)">
+ SetHandler !
Require all denied
</LocationMatch>
# block config files
<LocationMatch "config.php">
+ SetHandler !
Require all denied
</LocationMatch>
# block access to 'wp-login.php' from brute-forcers;
# see wp plugin 'rename-wp-login'
- <LocationMatch ".*wp-login.php">
- Require all denied
- </LocationMatch>
- cool, I was able to login to store.opensourceecology.org on hetzner3 with my old creds now
- the dashboard is littered with alerts:
Action Scheduler: 3 past-due actions found; something may be wrong. Read documentation »
WooCommerce database update required WooCommerce has been updated! To keep things running smoothly, we have to update your database to the newest version. The database update process runs in the background and may take a little while, so please be patient. Advanced users can alternatively update via WP CLI. Update WooCommerce Database Learn more about updates
Geolocation has not been configured. You must enter a valid license key on the MaxMind integration settings page in order to use the geolocation service. If you do not need geolocation for shipping or taxes, you should change the default customer location on the general settings page.
The plugin be-gdpr/be-gdpr.php has been deactivated due to an error: Plugin file does not exist. The plugin be-portfolio-post/be-portfolio-post.php has been deactivated due to an error: Plugin file does not exist. The plugin coingate-for-woocommerce/coingate.php has been deactivated due to an error: Plugin file does not exist. The plugin colorhub/colorhub.php has been deactivated due to an error: Plugin file does not exist. The plugin force-strong-passwords/slt-force-strong-passwords.php has been deactivated due to an error: Plugin file does not exist. The plugin masterslider/masterslider.php has been deactivated due to an error: Plugin file does not exist. The plugin meta-box-conditional-logic/meta-box-conditional-logic.php has been deactivated due to an error: Plugin file does not exist. The plugin meta-box-show-hide/meta-box-show-hide.php has been deactivated due to an error: Plugin file does not exist. The plugin meta-box-tabs/meta-box-tabs.php has been deactivated due to an error: Plugin file does not exist. The plugin oshine-core/oshine-core.php has been deactivated due to an error: Plugin file does not exist. The plugin oshine-modules/oshine-modules.php has been deactivated due to an error: Plugin file does not exist. The plugin redux-vendor-support/redux-vendor-support.php has been deactivated due to an error: Plugin file does not exist. The plugin rename-wp-login/rename-wp-login.php has been deactivated due to an error: Plugin file does not exist. The plugin revslider/revslider.php has been deactivated due to an error: Plugin file does not exist. The plugin tatsu/tatsu.php has been deactivated due to an error: Plugin file does not exist. The plugin typehub/typehub.php has been deactivated due to an error: Plugin file does not exist.
- I kicked-off the woocommerce db upgrade
- ugh, akismet isn't activated. we have 2,213 comments in the queue
- if I click on 'themes' in the wui, then I get a notice at the top
The active theme is broken. Reverting to the default theme.
- it says that 'oshine' is the active theme
- allright, I downloaded these files before
user@ose:~/tmp/hetzner3$ ls 13757819-enigmatic-responsive-multipurpose-wp-theme-license.txt 28755060-oshine-creative-multipurpose-wordpress-theme-license.txt 47932235-oshine-creative-multipurpose-wordpress-theme-license.txt 52287820-oshine-creative-multipurpose-wordpress-theme-license.txt backup-restore-test themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip user@ose:~/tmp/hetzner3$
- unfortunately, these are paid themes, and I have to coordinate with catarina to get an OTP every time I login, so I can't 3TOFU these :( I'll just have to 1TOFU it
- apparently these two oshine themes have identical contents but different names
user@ose:~/tmp/hetzner3$ sha256sum *.zip ed0628d0e57bb4e44b1af24eb235c6c384433c9ca94806c11b881e16f7f2b74a themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip 7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789 themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip 7506d6759ff1ee3f66d6135176537f12067ce86f2d5ba045c125f20df6240789 themeforest-4EaAhtH1-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip user@ose:~/tmp/hetzner3$
- I rsync'd these files up to hetzner3
user@ose:~/tmp/hetzner3$ rsync -av --progress themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip hetzner3:
Enter passphrase for key '/home/user/.ssh/id_rsa':
Enter passphrase for key '/home/user/.ssh/id_rsa':
sending incremental file list
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
10,582,975 100% 318.17kB/s 0:00:32 (xfr#1, to-chk=0/1)
sent 10,585,730 bytes received 35 bytes 201,633.62 bytes/sec
total size is 10,582,975 speedup is 1.00
user@ose:~/tmp/hetzner3$
user@ose:~/tmp/hetzner3$ rsync -av --progress themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip hetzner3:
Enter passphrase for key '/home/user/.ssh/id_rsa':
sending incremental file list
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
11,394,173 100% 996.24kB/s 0:00:11 (xfr#1, to-chk=0/1)
sent 11,397,129 bytes received 35 bytes 303,924.37 bytes/sec
total size is 11,394,173 speedup is 1.00
user@ose:~/tmp/hetzner3$
- I copied them over to our other dir with all the themes
root@hetzner3 /var/tmp/wordpress/themes # ls /home/maltfield/*.zip
/home/maltfield/themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
/home/maltfield/themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
root@hetzner3 /var/tmp/wordpress/themes #
root@hetzner3 /var/tmp/wordpress/themes # rsync -av --progress /home/maltfield/*.zip .
sending incremental file list
themeforest-2XwUOcbo-enigmatic-responsive-multipurpose-wp-theme-wordpress-theme.zip
10.582.975 100% 670,76MB/s 0:00:00 (xfr#1, to-chk=1/2)
themeforest-3JjZqZRr-oshine-creative-multipurpose-wordpress-theme-wordpress-theme.zip
11.394.173 100% 329,28MB/s 0:00:00 (xfr#2, to-chk=0/2)
sent 21.982.821 bytes received 54 bytes 43.965.750,00 bytes/sec
total size is 21.977.148 speedup is 1,00
root@hetzner3 /var/tmp/wordpress/themes #
root@hetzner3 /var/tmp/wordpress/themes # shred -u /home/maltfield/*.zip
root@hetzner3 /var/tmp/wordpress/themes #
root@hetzner3 /var/tmp/wordpress/themes # chown root:root themeforest-*
root@hetzner3 /var/tmp/wordpress/themes # chmod 0400 themeforest-*
root@hetzner3 /var/tmp/wordpress/themes #
Fri Oct 04, 2024
- Marcin gave me the go-ahead to delete the 'deleteMeIn2020' vault from our AWS Glacier account
1. Yes, delete the vault. 2. Thanks, good insights - i'll look into those more closely to see what would fit best. MJ
- ah ffs, I logged into the amazon WUI, but when I clicked "delete" on the vault, it gave me an error saying I have to delete all the objects in the vault first
This vault is not empty Vaults can be deleted only if there are no archives in the vault as of the last inventory it computed and there have been no writes to the vault since the last inventory. To delete all archives in the vault, use the REST API, the AWS SDK for Java, the AWS SDK for .NET or the AWS CLI.
- apparently this can only be done in the CLI via the API!?! It links to this https://docs.aws.amazon.com/console/glacier/using-aws-sdk
- we do have some 'glacier.py' script on our old server, but it complains about missing module(s)
[root@opensourceecology backups]# ls
backup.old.20180115.sh backup.settings.20221028 glacierRestore sync
backupReport.sh backup.sh glacierTest.py sync.old
backupReport.sh.20221028 backup.sh.20221028 ose-backups-cron.key
backup.settings cleanLocal.pl README.txt
[root@opensourceecology backups]#
[root@opensourceecology backups]# glacier.py
Traceback (most recent call last):
File "/root/bin/glacier.py", line 36, in <module>
import boto.glacier
ImportError: No module named boto.glacier
[root@opensourceecology backups]#
- allright, at least debian has the cli in its repos
user@disp3919:~/Downloads$ apt-cache search awscli awscli - Unified command line interface to Amazon Web Services user@disp3919:~/Downloads$ user@disp3919:~/Downloads$ sudo apt-get install awscli Reading package lists... Done Building dependency tree... Done Reading state information... Done awscli is already the newest version (2.9.19-1). The following packages were automatically installed and are no longer required: librnp0 libwpe-1.0-1 libwpebackend-fdo-1.0-1 linux-image-6.1.0-10-amd64 linux-image-6.1.0-11-amd64 linux-image-6.1.0-13-amd64 linux-image-6.1.0-17-amd64 linux-image-6.1.0-18-amd64 linux-image-6.1.0-20-amd64 linux-image-6.1.0-21-amd64 linux-image-6.1.0-22-amd64 Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded. user@disp3919:~/Downloads$ aws
- I was able to auth with some creds I found on hetzner2:/root/backups/glacierTest.py
user@disp3919:~/Downloads$ aws configure set aws_access_key_id 'REDACTED'
user@disp3919:~/Downloads$ aws configure set aws_secret_access_key 'REDACTED'
user@disp3919:~/Downloads$ aws sts get-caller-identity
{
"UserId": "REDACTED",
"Account": "REDACTED",
"Arn": "arn:aws:iam::REDACTED:user/backup-cron"
}
user@disp3919:~/Downloads$
- apparently we now have to create an inventory and then iterate though that inventory to delete all of the objects that it lists https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5
- creating an inventory can take hours or days; let's initiate it now
user@disp3919:~/Downloads$ aws glacier initiate-job --job-parameters '{"Type": "inventory-retrieval"}' --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020
{
"location": "/099400651767/vaults/deleteMeIn2020/jobs/ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS",
"jobId": "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS"
}
user@disp3919:~/Downloads$
- I guess now we wait a few days for the job to complete before we can download it, parse it, and then delete all of the objects it identifies per https://gist.github.com/veuncent/ac21ae8131f24d3971a621fac0d95be5
user@disp3919:~/Downloads$ aws glacier get-job-output --account-id REDACTED --region us-west-2 --vault-name deleteMeIn2020 --job-id "ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS" ./output.json An error occurred (InvalidParameterValueException) when calling the GetJobOutput operation: The job is not currently available for download: ucc6VDVVygGXS3EnMRVtzyqDpunVE81S91S_mUHuFL7-bfeMgVr6SxsVB3-_8g1Fs_NMdr_kV0rFCd_JFZU17EbUYXoS user@disp3919:~/Downloads$
...
- after much debugging, I figured out why store.opensourceecology.org gives different results for a `curl` coming from my laptop vs the server
- I found that the `curl` from my laptop was making it to nginx -> varnish -> apache
- but the logs were mysteriously absent for varnish & apache when I did the curl from the machine itself
- I even did a tcpdump, but I only saw a tiny blip of traffic when doing the command locally
- here's why: the server returns an http -> https redirect to store.opensourceecology.org. When the *server*'s curl command gets that, it does a public DNS lookup and then sends the query to hetzner2!
- I updated the /etc/hosts file to prevent this
root@hetzner3 ~ # cd /etc root@hetzner3 /etc # root@hetzner3 /etc # vim hosts root@hetzner3 /etc # root@hetzner3 /etc # diff hosts.20241004 hosts 2a3,13 > 127.0.0.1 forum.opensourceecology.org > 127.0.0.1 store.opensourceecology.org > 127.0.0.1 microfactory.opensourceecology.org > 127.0.0.1 fef.opensourceecology.org > 127.0.0.1 oswh.opensourceecology.org > 127.0.0.1 seedhome.openbuildinginstitute.org > 127.0.0.1 www.openbuildinginstitute.org > 127.0.0.1 www.opensourceecology.org > 127.0.0.1 phplist.opensourceecology.org > 127.0.0.1 wiki.opensourceecology.org > 3a15 > root@hetzner3 /etc #
- ok, now it's stuck in an infinite redirect. It just keeps going back-and-forth adding and removing the slash at the end
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local5 HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:23:38 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://opensourceecology.org Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:22:29 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://www.opensourceecology.org/ Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:23:39 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://opensourceecology.org Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" ... HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:22:29 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://www.opensourceecology.org/ Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:23:39 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive Location: https://opensourceecology.org Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" curl: (47) Maximum (50) redirects followed maltfield@hetzner3:~$
- ok, so it looks like it's getting picked-up by the default site
root@hetzner3 /etc/nginx # grep -ir 301 * nginx.conf: return 301 https://$host$request_uri; nginx.conf.1282157.2024-09-28@23:10:52~: return 301 https://$host$request_uri; sites-enabled/00-default.conf: return 301 https://opensourceecology.org; root@hetzner3 /etc/nginx #
- ah shit, yeah, nginx isn't even listening on 127.0.0.1 lol
root@hetzner3 /etc/nginx # less sites-enabled/store.opensourceecology.org.conf
# Ansible managed
################################################################################
# File: store.opensourceecology.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
# Updated: 2024-09-14
################################################################################
server {
access_log /var/log/nginx/store.opensourceecology.org/access.log main;
error_log /var/log/nginx/store.opensourceecology.org/error.log;
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
listen 144.76.164.201:443;
listen [2a01:4f8:200:40d7::2]:443;
server_name store.opensourceecology.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
# root /var/www/html/SITE_DOWN/htdocs/;
# index index.html index.htm;
#
# # force all requests to load exactly this page
# location / {
# try_files $uri /index.html;
# }
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
}
}
- well it is, but this server block is not
root@hetzner3 /etc/nginx # netstat -plan | grep -i 443 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 3914728/nginx: mast tcp 0 0 144.76.164.201:4443 0.0.0.0:* LISTEN 3914728/nginx: mast tcp 25 0 144.76.164.201:51710 104.21.40.220:443 CLOSE_WAIT 15751/wazuh-modules tcp 0 0 127.0.0.1:80 127.0.0.1:54436 TIME_WAIT - tcp 0 0 127.0.0.1:54432 127.0.0.1:80 TIME_WAIT - tcp6 0 0 :::443 :::* LISTEN 3914728/nginx: mast tcp6 0 0 2a01:4f8:200:40d7::4443 :::* LISTEN 3914728/nginx: mast tcp6 25 0 2a01:4f8:200:40d7:49016 2606:4700:3033::ac4:443 CLOSE_WAIT 15751/wazuh-modules You have new mail in /var/mail/root root@hetzner3 /etc/nginx #
- yeah, default is -- which is why it's picking it up instead
root@hetzner3 /etc/nginx # grep -ir listen sites-available/forum.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-available/forum.opensourceecology.org.conf: listen 144.76.164.201:443; sites-available/forum.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; sites-available/default: listen 80 default_server; sites-available/default: listen [::]:80 default_server; sites-available/default: # listen 443 ssl default_server; sites-available/default: # listen [::]:443 ssl default_server; sites-available/default:# listen 80; sites-available/default:# listen [::]:80; sites-available/store.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-available/store.opensourceecology.org.conf: listen 144.76.164.201:443; sites-available/store.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; nginx.conf.1282157.2024-09-28@23:10:52~: listen 80; nginx.conf.1282157.2024-09-28@23:10:52~: listen [::]:80; nginx.conf: listen 80; nginx.conf: listen [::]:80; nginx.conf.85740.2024-09-24@04:17:16~:# listen localhost:110; nginx.conf.85740.2024-09-24@04:17:16~:# listen localhost:143; sites-enabled/00-default.conf: listen 443; sites-enabled/00-default.conf: listen [::]:443; sites-enabled/awstats.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-enabled/awstats.opensourceecology.org.conf: listen 144.76.164.201:443; sites-enabled/awstats.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; sites-enabled/awstats.opensourceecology.org.conf: listen 144.76.164.201:4443; sites-enabled/awstats.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:4443; sites-enabled/munin.opensourceecology.org.conf:# Purpose: Internet-listening web server for truncating https, basic DOS sites-enabled/munin.opensourceecology.org.conf: listen 144.76.164.201:443; sites-enabled/munin.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:443; sites-enabled/munin.opensourceecology.org.conf: listen 144.76.164.201:4443; sites-enabled/munin.opensourceecology.org.conf: listen [2a01:4f8:200:40d7::2]:4443; root@hetzner3 /etc/nginx #
- this is actually the same as our hetzner2 config
- I updated the nginx config in ansible and pushed it out again
diff --git a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
index f4b62cd..f750651 100644
--- a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
@@ -2,13 +2,13 @@
################################################################################
# File: store.opensourceecology.org.conf
-# Version: 0.2
+# Version: 0.3
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
-# Updated: 2024-09-14
+# Updated: 2024-10-04
################################################################################
server {
@@ -19,6 +19,8 @@ server {
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
+ listen 127.0.0.1:443;
+ listen [::1]:443;
listen {{ ansible_default_ipv4.address }}:443;
listen [{{ ansible_default_ipv6.address }}]:443;
user@personal:~/sandbox_local/ansible/hetzner3$
diff --git a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2 b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
index f4b62cd..f750651 100644
--- a/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
+++ b/hetzner3/roles/maltfield.nginx/templates/store.opensourceecology.org.conf.j2
@@ -2,13 +2,13 @@
################################################################################
# File: store.opensourceecology.org.conf
-# Version: 0.2
+# Version: 0.3
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@michaelaltfield.net>
# Created: 2019-04-09
-# Updated: 2024-09-14
+# Updated: 2024-10-04
################################################################################
server {
@@ -19,6 +19,8 @@ server {
include conf.d/secure.include;
include conf.d/https.opensourceecology.org.include;
+ listen 127.0.0.1:443;
+ listen [::1]:443;
listen {{ ansible_default_ipv4.address }}:443;
listen [{{ ansible_default_ipv6.address }}]:443;
user@personal:~/sandbox_local/ansible/hetzner3$
- and, well, the good/bad news is that now the curl from the local machine is as equally broken as the curl from my laptop
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local6 HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:46:30 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=local6 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 89 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Sat, 05 Oct 2024 03:46:30 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 98500 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true maltfield@hetzner3:~$
- the 'true' is obviously coming from 'index.html', so my first thought was just to get rid of that file
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # rm index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- but now we're just back to the empty output (again)
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/index.php?nocache=local7 HTTP/1.1 301 Moved Permanently Server: nginx Date: Sat, 05 Oct 2024 03:49:48 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=local7 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 94 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Sat, 05 Oct 2024 03:49:49 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/", <https://store.opensourceecology.org/wp-json/wp/v2/pages/2796>; rel="alternate"; title="JSON"; type="application/json", <https://store.opensourceecology.org/>; rel=shortlink X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 97 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" maltfield@hetzner3:~$
- varnish logs look fine; it basically just calls the backend
- apahce logs indicate that it did figure out which file to server with php
==> store.opensourceecology.org/error.log <== [Sat Oct 05 03:56:05.631466 2024] [authz_core:debug] [pid 3909393:tid 3909439] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:05.631539 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(123): [client 127.0.0.1:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:05.631557 2024] [proxy:debug] [pid 3909393:tid 3909439] mod_proxy.c(1465): [client 127.0.0.1:0] AH01143: Running scheme fcgi handler (attempt 0) [Sat Oct 05 03:56:05.631571 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(1078): [client 127.0.0.1:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Sat Oct 05 03:56:05.631584 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909439] mod_proxy_fcgi.c(1087): [client 127.0.0.1:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:05.631597 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Sat Oct 05 03:56:05.631610 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3242): [client 127.0.0.1:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Sat Oct 05 03:56:05.631624 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3309): [client 127.0.0.1:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Sat Oct 05 03:56:05.631638 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3450): [client 127.0.0.1:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Sat Oct 05 03:56:05.631673 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000) [Sat Oct 05 03:56:06.720816 2024] [proxy:debug] [pid 3909393:tid 3909439] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [05/Oct/2024:03:56:05 +0000] "GET /index.php?nocache=local10 HTTP/1.1" 301 436 "-" "curl/7.88.1" ==> store.opensourceecology.org/error.log <== [Sat Oct 05 03:56:06.725670 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:06.725738 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:06.725854 2024] [authz_core:debug] [pid 3909393:tid 3909441] mod_authz_core.c(733): [client 127.0.0.1:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Sat Oct 05 03:56:06.725886 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(123): [client 127.0.0.1:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:06.725895 2024] [proxy:debug] [pid 3909393:tid 3909441] mod_proxy.c(1465): [client 127.0.0.1:0] AH01143: Running scheme fcgi handler (attempt 0) [Sat Oct 05 03:56:06.725901 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(1078): [client 127.0.0.1:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Sat Oct 05 03:56:06.725928 2024] [proxy_fcgi:debug] [pid 3909393:tid 3909441] mod_proxy_fcgi.c(1087): [client 127.0.0.1:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Sat Oct 05 03:56:06.725935 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Sat Oct 05 03:56:06.725941 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3242): [client 127.0.0.1:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Sat Oct 05 03:56:06.725950 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3309): [client 127.0.0.1:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Sat Oct 05 03:56:06.725959 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3450): [client 127.0.0.1:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Sat Oct 05 03:56:06.726002 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000) [Sat Oct 05 03:56:07.778759 2024] [proxy:debug] [pid 3909393:tid 3909441] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [05/Oct/2024:03:56:06 +0000] "GET /?nocache=local10 HTTP/1.1" 200 586 "-" "curl/7.88.1"
- this suggests that it might do this if the theme dir is empty? that would likely apply in our case https://serverfault.com/a/766146
- oh, it *does* load if I try '/wp-admin/'
maltfield@hetzner3:~$ curl -iLkH 'Host: store.opensourceecology.org' https://localhost/wp-admin/
...
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 05 Oct 2024 04:24:26 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 1516
Connection: keep-alive
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98551
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta name="viewport" content="width=device-width" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="robots" content="noindex,nofollow" />
<title>WordPress › Update</title>
<link rel='stylesheet' id='dashicons-css' href='https://store.opensourceecology.org/wp-includes/css/dashicons.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='buttons-css' href='https://store.opensourceecology.org/wp-includes/css/buttons.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='forms-css' href='https://store.opensourceecology.org/wp-admin/css/forms.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='l10n-css' href='https://store.opensourceecology.org/wp-admin/css/l10n.min.css?ver=6.6.1' type='text/css' media='all' />
<link rel='stylesheet' id='install-css' href='https://store.opensourceecology.org/wp-admin/css/install.min.css?ver=6.6.1' type='text/css' media='all' />
</head>
<body class="wp-core-ui">
<p id="logo"><a href="https://wordpress.org/">WordPress</a></p>
<h1>Database Update Required</h1>
<p>WordPress has been updated! Next and final step is to update your database to the newest version.</p>
<p>The database update process may take a little while, so please be patient.</p>
<p class="step"><a class="button button-large button-primary" href="upgrade.php?step=1&backto=%2Fwp-admin%2F">Update WordPress Database</a></p>
</body>
</html>
maltfield@hetzner3:~$
- I loaded that in the web browser, and it told me a wordpress database update was needed. I just pressed the button -- it didn't even prompt me to auth
Your WordPress database has been successfully updated!
- I clicked "Continue"
- that redirected me here, and I immediately got '403 forbidden' https://store.opensourceecology.org/wp-login.php?redirect_to=https%3A%2F%2Fstore.opensourceecology.org%2Fwp-admin%2F&reauth=1
- that would be because block access to 'wp-login.php' since we were using a plugin to rename it; we'll have to temp disable that until we replace that (now deprecated) plugin
Thr Oct 03, 2024
- I sent an invoice (AS-0106) to OSE for 67 hours in Sep 2024
...
- continuing to debug store.opensourceecology.org, I see that it's redirecting from '/index.php' to '/'
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2024 04:47:39 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/ X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 131132 98385 Age: 88 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" user@disp3919:~$
- if I try to hit 'index.html', I don't get a redirect -- I just get a 404
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html HTTP/1.1 404 Not Found Server: nginx Date: Fri, 04 Oct 2024 04:48:25 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 131134 Age: 0 Via: 1.1 varnish (Varnish/7.1) user@disp3919:~$
- if I create it
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cp is_hetzner3 index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # ls -lah is_hetzner3 index.html ----r----- 1 root root 5 Oct 4 04:49 index.html ----r----- 1 not-apache www-data 5 Sep 27 04:44 is_hetzner3 root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown not-apache:www-data index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- then it works
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 04:49:51 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 98387 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3919:~$
- hmm, so apache is just refusing to serve 'index.php' files. -- what about 'something.php'?
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # echo "<?php echo 'it works'; ?>" > something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown root:www-data something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chmod 0040 something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- it works, so it's something specific to 'index.php'
user@disp3919:~$ curl -i https://store.opensourceecology.org/something.php HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 04:52:35 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 8 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 98390 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" it worksuser@disp3919:~$
- I'd think it's an issue with the DirectoryIndex, but this looks good
root@hetzner3 /etc/apache2 # grep -ir 'index.php' * conf-available/wordpress.directory.include:# RewriteRule . /index.php [L] mods-available/dir.conf:DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah mods-enabled/dir.conf lrwxrwxrwx 1 root root 26 Sep 25 01:24 mods-enabled/dir.conf -> ../mods-available/dir.conf root@hetzner3 /etc/apache2 #
- I checked the old server, but I didn't see anything that we're missing in the new server
[root@opensourceecology httpd]# grep -ir 'index.php' conf [root@opensourceecology httpd]# grep -ir 'index.php' conf.d conf.d/php.conf:# Add index.php to the list of files that will be served as directory conf.d/php.conf:DirectoryIndex index.php conf.d/00-wiki.opensourceecology.org.conf: Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php conf.d/mod_evasive.conf: # http://security.lss.hr/index.php?page=details&ID=LSS-2005-01-01 [root@opensourceecology httpd]# grep -ir 'index.php' conf.modules.d/ [root@opensourceecology httpd]# grep -ir 'index.php' modsecurity.d/ [root@opensourceecology httpd]#
- I changed wp-config.php to have WP_DEBUG set to 'true', but it didn't print anything extra. It seems like the error is occurring before wordpress
- I set LogLevel of apache.conf to 'debug', and this popped-up
==> forum.opensourceecology.org/access.log <== 127.0.0.1 - - [04/Oct/2024:05:10:02 +0000] "GET /server-status?auto HTTP/1.1" 200 1202 "-" "munin/2.0.73 (libwww-perl/6.68)" ==> forum.opensourceecology.org/error.log <== [Fri Oct 04 05:10:03.564975 2024] [authz_core:debug] [pid 3581402:tid 3581414] mod_authz_core.c(815): [client 127.0.0.1:32934] AH01626: authorization result of Require all denied: denied
- well, that's an unrelated issue with munin, but it seems that the requests to '/server-status' are getting sent to the wrong vhost (forum.opensourceecology.org) and also denied access
- here's the actual output when I do the curl
- first, it outputs this immediately, then it pauses for maybe 10 seconds
==> store.opensourceecology.org/error.log <== [Fri Oct 04 05:11:53.426292 2024] [authz_core:debug] [pid 3581402:tid 3581422] mod_authz_core.c(733): [client 81.17.16.91:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Fri Oct 04 05:11:53.426458 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(123): [client 81.17.16.91:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Fri Oct 04 05:11:53.426496 2024] [proxy:debug] [pid 3581402:tid 3581422] mod_proxy.c(1465): [client 81.17.16.91:0] AH01143: Running scheme fcgi handler (attempt 0) [Fri Oct 04 05:11:53.426517 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1078): [client 81.17.16.91:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Fri Oct 04 05:11:53.426535 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1087): [client 81.17.16.91:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Fri Oct 04 05:11:53.426586 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Fri Oct 04 05:11:53.426612 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3242): [client 81.17.16.91:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Fri Oct 04 05:11:53.426658 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3309): [client 81.17.16.91:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Fri Oct 04 05:11:53.426718 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3450): [client 81.17.16.91:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Fri Oct 04 05:11:53.426793 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)
- after maybe 10 seconds, it outputs this
[Fri Oct 04 05:12:03.646185 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 81.17.16.91 - - [04/Oct/2024:05:11:53 +0000] "GET /index.php?nocache=6 HTTP/1.1" 301 430 "-" "curl/7.88.1"
- so it sounds like maybe this is an issue with the php-fpm config?
- I tried to hit apache through the cli on the server itself and, oh, I get the payload as desired
root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
<script>
//jQuery(document).ready(function(){
// });
</script>
</body>
</html>root@hetzne
You have new mail in /var/mail/root
- I loosened the error reporting settings on php.ini and I got it to spit this out when I curl from my laptop
--2d1eeb03-H--
Apache-Error: [file "mod_authz_core.c"] [line 733] [level 7] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 123] [level 7] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "mod_proxy.c"] [line 1465] [level 7] AH01143: Running scheme fcgi handler (attempt 0)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1078] [level 7] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1087] [level 7] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "proxy_util.c"] [line 3242] [level 7] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
Apache-Error: [file "proxy_util.c"] [line 3309] [level 7] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
Apache-Error: [file "proxy_util.c"] [line 3450] [level 7] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 911] [level 3] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function ini_set() in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php:590\\nStack trace:\\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(82): wp_debug_mode()\\n#1 /var/www/html/store.opensourceecology.org/wp-config.php(105): require_once('...')\\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-blog-header.php(13): require_once('...')\\n#4 /var/www/html/store.opensourceecology.org/htdocs/index.php(17): require('...')\\n#5 {main}\\n thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php on line 590'
Apache-Handler: proxy:unix:/run/php/php8.2-fpm.sock|fcgi://localhost
Stopwatch: 1728019471413382 53626 (- - -)
Stopwatch2: 1728019471413382 53626; combined=41, p1=21, p2=18, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"
- yeah, so this is the wordpress bug that I submitted a PR for last month
- after a brief dialog with the wordpress devs, a workaround until they merge is to define a fake ini_set() function in wp-config.php
- from personal experience, I found it's best to wrap this in a conditional to make sure the function doesn't exist yet
root@hetzner3 /var/www/html/store.opensourceecology.org # cp wp-config.php wp-config.php.20241003
root@hetzner3 /var/www/html/store.opensourceecology.org #
root@hetzner3 /var/www/html/store.opensourceecology.org # vim wp-config.php
root@hetzner3 /var/www/html/store.opensourceecology.org #
root@hetzner3 /var/www/html/store.opensourceecology.org # diff wp-config.php.20241003 wp-config.php
1a2,9
>
> # fix wordpress bug https://core.trac.wordpress.org/ticket/48693
> if( ! function_exists('ini_set') ){
> function ini_set(){
> return;
> }
> }
>
root@hetzner3 /var/www/html/store.opensourceecology.org #
- after that, I'm back to getting blank pages on my curl on my laptop. it's flapping?
- alright, let me see if I can harden php back up again, but with errors actually logging. I'll update something.php to write to the error log
root@hetzner3 /var/www/html/store.opensourceecology.org # cd htdocs/ root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # vim something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cat something.php <?php error_log( "executing something.php" ); echo 'it works'; ?> root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- ok, it's visible
[Fri Oct 04 05:47:06.734936 2024] [proxy_fcgi:error] [pid 3581402:tid 3581452] [client 127.0.0.1:36330] AH01071: Got error 'PHP message: executing something.php'
- first I reduced the apache logs down to 'warn' again. looks good
[Fri Oct 04 05:48:39.732669 2024] [proxy_fcgi:error] [pid 3591906:tid 3591913] [client 127.0.0.1:41102] AH01071: Got error 'PHP message: executing something.php' ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [04/Oct/2024:05:48:39 +0000] "GET /something.php HTTP/1.1" 200 302 "-" "curl/7.88.1"
- I obliterated my manual changes by pushing ansible's apache & php roles
- cool, I confirmed that both curl on my laptop and on the server produce the logs after restarting both apache2 & php8.2-fpm
- for some reason I still get an 'true' on my laptop
user@disp3919:~$ 3919:~$ curl -iL https://store.opensourceecolindex.php?nocache=19 HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2024 05:56:15 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=19 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 98469 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 05:56:16 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 131203 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3919:~$
- but I get the actual html on the local machine
root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
/* ]]> */
</script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-content/themes/oshin/js/script.js?ver=5.0'></script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-includes/js/wp-embed.min.js?ver=5.1.1'></script>
<!-- Option Panel Custom JavaScript -->
<script>
//jQuery(document).ready(function(){
// });
</script>
</body>
</html>root@hetzner3 ~ #
Wed Oct 02, 2024
- Marcin sent me a few emails in the past months asking about OSE's use of Amazon Glacier
- Today he sent a message saying that he got charged $1.03, and isn't sure why
Michael, I'm getting charged $1.03 for Glacier. Can we cancel that? Marcin
- It took me a while to auth
- first I tried to login with my 'maltfield' aws user, but aws rejected my creds (stored in my personal keepass)
- eventually I realized I had to click "Sign in using root user email" -- and then I could auth using the creds stored in the shared keepass
- after logging-in, I went to the "Billing and Cost Management" app https://us-east-1.console.aws.amazon.com/costmanagement/home?region=us-west-2#/home
- on this page, there was a link that said "Last month's total cost: $1.03". Yep, that's all accounted-for. I clicked it.
- the next page showed a joke of a chart with one bar on a bar graph that said "$1.03". And the bar was labeled "Total Cost"
- I had to click on the dropdown menu for "Dimension" and set it to "Service" -- then it listed 4 items
- Glacier - $1.03
- S3 - $0.00
- Tax $0.00
- So I switched over to the "Glacier" app https://us-east-1.console.aws.amazon.com/glacier/home?region=us-east-1
- Curiously, it listed 0 vaults
- but there was a note at the top saying we should use S3 for glaicer, so I clicked over to the "S3" app
We recommend that you use Glacier storage classes in Amazon S3 for archival storage
- here I saw one bucket called "oseserverbackups" in "US West (Oregon) us-west-2"
- the bucket had one 34.0 byte file in it called "test.txt". That's it!
- this file was created July 6, 2018, 19:18:03 (UTC-05:00)
- I downloaded it; it has one line of text
some file destined for s3 this is
- I deleted the 'test.txt' file object from the s3 bucket
- I then deleted the (now empty) 'oseserverbackups' bucket
- unconvinced that that was the issue, I went back to the "glacier" app. This time I cycled through a few of the regions until I got to "us-west-2" -- this time I showed one vault named "deleteMeIn2020"
- I clicked on it, and it said
- this vault was created March 29, 2018, 16:36:06 (UTC-05:00)
- this vault was last inventoried August 1, 2018, 02:41:31 (UTC-05:00)
- this vault is 285.3 GB (as of last inventory)
- well, it's after 2020. So I think we should delete it.
- I sent an email to Marcin asking for a confirmation before I delete it
Hey Marcin, You have a 285.3 GB vault in Amazon Glacier's us-west-2 region. I logged-into your AWS account today and did some digging. I found this vault 285.3 GB vault named 'deleteMeIn2020'. I created this vault in 2018 Q1. It contains a final backup of files from hetzner1. I created it as part of the hetzner2 migration project, thinking that we should delete it in 2020 if we never needed to restore anything from it for 2 years. * https://wiki.opensourceecology.org/wiki/CHG-2018-07-06_hetzner1_deprecation * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q1#Sat_Mar_31.2C_2018 Well, 2020 came and past. Four more years passed. I think you can safely delete the 'deleteMeIn2020' vault. By the way, I also deleted a 53-byte test file from an S3 bucket named 'test.txt' in a bucket in s3 called 'oseserverbackups' in us-west-2. It was the only file in the bucket. I deleted the file and the empty bucket. Would you like me to proceed with deleting the 285.3 GB 'deleteMeIn2020' glacier bucket from your AWS account? Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- meanwhile, I tried to figure out why I couldn't login as 'maltfield', and I realized that, ffs, we don't have IAM setup for our account?? Maybe Marcin deleted it when trying to elimiate costs? IAM is free, though..
- ok, I found my 'maltfield' user under "Security Credentials" -> "Access Management" -> "Users"
- it says my last console sign-in was 424 days ago
- I went to my user's settings, selected the MFA token, and selected "Resync" -- then entered two consecutive OTPs
- I tried to login, and this time it let me in. Well that was annoying.
- I opened cloudtrail and reviewed the latest account events https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?ReadOnly=false
- the most recent event was the 'root' user resyncing the MFA token of the 'matlfield' token
- before that we have two ConsoleLogin for today
- before that 'mjakubowski' user has a MakePayment event (and some other payment related events) on Sep 19
- before that we have a bunch of login & mfa-related entries for Marcin's user on Sep 06, 14, 17, and 19.
- and that's where the log ends; looks like we just get 90 days of logs for free.
...
- hetzner responded to my support inquery about how they handle failed disks
Dear Mr Altfield Unfortunately it's an unmanaged root server monitoring is your responsibility I'm afraid. If you have a problem please open a ticket in your robot account. Please click on "Servers" from the menu on the left and then select the corresponding server. Under the "Support" tab, you can choose "Hard drive is broken". Please follow the instructions. https://docs.hetzner.com/robot/dedicated-server/troubleshooting/serial-numbers-and-information-on-defective-hard-drives/ Our DC is 24/7 available and we exchange broken hardware as soon as possible for free. Hetzner clients can use the Server Monitoring System to monitor their servers and have an email sent to them when the status of one of the monitored services changes: https://docs.hetzner.com/robot/dedicated-server/security/system-monitor/ https://docs.hetzner.com/robot/dedicated-server/raid/software-raid/#email-notification-when-a-drive-in-a-software-raid-fails Please use hetzner-status: https://www.hetzner-status.de/en.html This web page publishes announcements and current fault reports from our datacenters. Would you like to receive email notification of fault reports? Log on as exclusive Hetzner client in your administrations interface. If you have any questions please do not hesitate to contact us. Kind regards Jan Kolb Sales Hetzner Online GmbH Sigmundstrasse 135 90431 Nürnberg Tel: +49 911 234 226-927 Fax: +49 9831 505-3 sales@hetzner.com www.hetzner.com Register Court: Registergericht Ansbach, HRB 6089 CEO: Martin Hetzner, Stephan Konvickova, Günther Müller For the purposes of this communication, we may save some of your personal data. For information on our data privacy policy, please see: www.hetzner.com/datenschutzhinweis 09/29/2024 21:23 - marcin@opensourceecology.org michael@opensourceecology.org wrote: > > > Hi Hetzner, > > Can you please tell us more about the process of disk failure on our new dedicated > server plan (Server Auction #2443019)? > > Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk? > Or do we have to pay a fee? If so, how much? > > And does Hetzner have some system in-place that monitors the hardware for disk > failure? Or do we have to monitor this in software and alert Hetnzer that a disk > is failing? If Hetzner does monitor for disk failure, how does it do it? > > > Thank you, > > Michael Altfield > Senior Technology Advisor > PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B > > Open Source Ecology > www.opensourceecology.org >
- the docs linked-to actually don't mention mdadm, which I setup earlier to monitor and send us email alerts on our disks
- instead, hetzner mentions `smartctl`, which is included in the debian package `smartmontools` -- which wasn't even installed!
root@hetzner3 /etc/mdadm # sudo apt-get install smartmontools ... root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme0n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme1n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED root@hetzner3 /etc/mdadm #
- we can get more information with the `-A` argument
root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme0n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 36 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 3% Data Units Read: 142.729.615 [73,0 TB] Data Units Written: 20.452.874 [10,4 TB] Host Read Commands: 6.862.184.005 Host Write Commands: 876.931.661 Controller Busy Time: 15.948 Power Cycles: 28 Power On Hours: 16.350 Unsafe Shutdowns: 5 Media and Data Integrity Errors: 0 Error Information Log Entries: 159 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 36 Celsius Temperature Sensor 2: 45 Celsius root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme1n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 34 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 3% Data Units Read: 130.064.348 [66,5 TB] Data Units Written: 24.932.683 [12,7 TB] Host Read Commands: 1.276.781.490 Host Write Commands: 879.017.438 Controller Busy Time: 14.879 Power Cycles: 23 Power On Hours: 14.678 Unsafe Shutdowns: 5 Media and Data Integrity Errors: 0 Error Information Log Entries: 149 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 34 Celsius Temperature Sensor 2: 37 Celsius root@hetzner3 /etc/mdadm #
- oh nvm, their third link describes mdadm alerts for monitoring our software raid
- they also said to check /etc/default/mdadm, which I didn't do before
root@hetzner3 /etc/mdadm # cat /etc/default/mdadm # mdadm Debian configuration # # You can run 'dpkg-reconfigure mdadm' to modify the values in this file, if # you want. You can also change the values here and changes will be preserved. # Do note that only the values are preserved; the rest of the file is # rewritten. # # AUTOCHECK: # should mdadm run periodic redundancy checks over your arrays? See # /etc/cron.d/mdadm. AUTOCHECK=true # AUTOSCAN: # should mdadm check once a day for degraded arrays? See # /etc/cron.daily/mdadm. AUTOSCAN=true # START_DAEMON: # should mdadm start the MD monitoring daemon during boot? START_DAEMON=true # DAEMON_OPTIONS: # additional options to pass to the daemon. DAEMON_OPTIONS="--syslog" # VERBOSE: # if this variable is set to true, mdadm will be a little more verbose e.g. # when creating the initramfs. VERBOSE=false root@hetzner3 /etc/mdadm #
- note that "AUTOCHECK" is enabled -- so we're all good here.
...
- ok, back to updating wordpress.
- first, I'm just going to unzip all these (now TOFU-verified) .zip files and make sure there's no zipbombs
root@hetzner3 ~ # cd /var/tmp/wordpress/themes/ root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # ls bouquet.1.2.5.zip sketch.1.2.4.zip twentyfifteen.3.8.zip twentyseventeen.3.7.zip twentythirteen.4.2.zip gk-portfolio.1.5.3.zip storefront.4.6.0.zip twentyfourteen.4.0.zip twentysixteen.3.3.zip twentytwelve.4.3.zip portfolio-press.2.8.0.zip twentyeleven.4.7.zip twentynineteen.2.9.zip twentyten.4.2.zip root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # for file in $(ls *.zip); do unzip $file; done ... root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # ls bouquet portfolio-press.2.8.0.zip twentyeleven twentyfourteen.4.0.zip twentysixteen twentythirteen.4.2.zip bouquet.1.2.5.zip sketch twentyeleven.4.7.zip twentynineteen twentysixteen.3.3.zip twentytwelve gk-portfolio sketch.1.2.4.zip twentyfifteen twentynineteen.2.9.zip twentyten twentytwelve.4.3.zip gk-portfolio.1.5.3.zip storefront twentyfifteen.3.8.zip twentyseventeen twentyten.4.2.zip portfolio-press storefront.4.6.0.zip twentyfourteen twentyseventeen.3.7.zip twentythirteen root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # cd ../plugins/ root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # for file in $(ls *.zip); do unzip $file; done ... root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # ls akismet jetpack vcaching akismet.5.3.3.zip jetpack.13.8.1.zip vcaching.1.8.3.zip black-studio-tinymce-widget meta-box w3-total-cache black-studio-tinymce-widget.2.7.3.zip meta-box.5.10.2.zip w3-total-cache.2.7.6.zip chartbeat ml-slider wonderm00ns-simple-facebook-open-graph-tags chartbeat.2.0.7.zip ml-slider.3.91.0.zip wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip classic-editor open-in-new-window-plugin woocommerce classic-editor.1.6.5.zip open-in-new-window-plugin.3.0.zip woocommerce.9.3.3.zip coingate-for-woocommerce post-types-order wordpress-importer coingate-for-woocommerce.2.1.1.zip post-types-order.2.2.6.zip wordpress-importer.0.8.2.zip contact-form-7 revision-control wordpress-seo contact-form-7.5.9.8.zip revision-control.2.3.2.zip wordpress-seo.23.5.zip duplicate-page shareaholic wpautop-control duplicate-page.4.5.zip shareaholic.9.7.12.zip wpautop-control.1.6.zip duplicate-post share-on-diaspora wp-memory-usage duplicate-post.4.5.zip share-on-diaspora.0.7.9.zip wp-memory-usage.1.2.10.zip google-authenticator shariff wp-optimize google-authenticator.0.54.zip shariff.4.6.14.zip wp-optimize.3.6.0.zip google-authenticator-encourage-user-activation ssl-insecure-content-fixer wp-smushit google-authenticator-encourage-user-activation.0.2.zip ssl-insecure-content-fixer.2.7.2.zip wp-smushit.3.16.6.zip insert-headers-and-footers varnish-http-purge wp-super-cache insert-headers-and-footers.2.2.2.zip varnish-http-purge.5.2.2.zip wp-super-cache.1.12.4.zip root@hetzner3 /var/tmp/wordpress/plugins #
- ok, that looks good. now let's see if we can script copying-over these themes as-needed
- and, to err on the side of caution, I'm going to intentionally delete any theme or plugin dir, even if we don't have one to replace it.
wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
echo "${theme}"
rm -rf ${theme_path};
rsync -av --progress "/var/tmp/wordpress/themes/${theme}/" "${theme_path}/"
done
- after execution, looks like it worked
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/ total 68K d---r-x--- 16 not-apache www-data 4,0K Oct 3 04:02 . d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 .. ----r----- 1 not-apache www-data 28 Jun 5 2014 index.php drwxr-xr-x 2 root root 4,0K Oct 3 04:02 oshin drwxr-xr-x 5 root root 4,0K May 16 08:29 storefront drwxr-xr-x 7 root root 4,0K Jul 16 13:09 twentyeleven drwxr-xr-x 7 root root 4,0K Jul 16 13:28 twentyfifteen drwxr-xr-x 9 root root 4,0K Jul 16 13:23 twentyfourteen drwxr-xr-x 9 root root 4,0K Jul 16 13:30 twentynineteen drwxr-xr-x 5 root root 4,0K Jul 16 13:29 twentyseventeen drwxr-xr-x 8 root root 4,0K Jul 16 13:29 twentysixteen drwxr-xr-x 4 root root 4,0K Jul 15 17:17 twentyten drwxr-xr-x 8 root root 4,0K Jul 16 13:20 twentythirteen drwxr-xr-x 8 root root 4,0K Jul 16 13:17 twentytwelve drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentyfour drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentythree drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentytwo root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- oh, wait, no. it created some silly empty dirs when it didn't have a source to copy-from
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/oshin/ total 8,0K drwxr-xr-x 2 root root 4,0K Oct 3 04:02 . d---r-x--- 16 not-apache www-data 4,0K Oct 3 04:02 .. root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- let's wrap that in a condition. and also disable verbose & progress on rsync, so we can see the whole output
for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
- here's the execution; that's better
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
twentytwelve
twentysixteen
storefront
twentyseventeen
twentyfourteen
twentyeleven
twentytwentythree
oshin
twentytwentyfour
twentythirteen
twentyten
twentyfifteen
twentynineteen
twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 52K
d---r-x--- 12 not-apache www-data 4,0K Oct 3 04:04 .
d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 ..
----r----- 1 not-apache www-data 28 Jun 5 2014 index.php
drwxr-xr-x 5 root root 4,0K May 16 08:29 storefront
drwxr-xr-x 7 root root 4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x 7 root root 4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x 9 root root 4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x 9 root root 4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x 5 root root 4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x 8 root root 4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x 4 root root 4,0K Jul 15 17:17 twentyten
drwxr-xr-x 8 root root 4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x 8 root root 4,0K Jul 16 13:17 twentytwelve
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- now let's do the plugins with this
wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
- I actually messed this up, and I had to restore the original plugins dir from the backup; easy enough
rsync -av --progress /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/
- alright, here's the run
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
meta-box-show-hide
classic-editor
be-portfolio-post
colorhub
ssl-insecure-content-fixer
oshine-core
tatsu
revslider
redux-vendor-support
akismet
rename-wp-login
meta-box-tabs
google-authenticator
coingate-for-woocommerce
be-gdpr
google-authenticator-encourage-user-activation
typehub
meta-box
woocommerce
meta-box-conditional-logic
contact-form-7
vcaching
force-strong-passwords
masterslider
oshine-modules
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah plugins/
total 56K
d---r-x--- 12 1012 48 4,0K Oct 3 04:09 .
d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 ..
drwxr-xr-x 4 root root 4,0K Jul 10 22:16 akismet
drwxr-xr-x 3 root root 4,0K Sep 27 21:51 classic-editor
drwxr-xr-x 8 root root 4,0K Nov 21 2022 coingate-for-woocommerce
drwxr-xr-x 7 root root 4,0K Jul 25 08:28 contact-form-7
drwxr-xr-x 3 root root 4,0K Jul 4 2022 google-authenticator
drwxr-xr-x 4 root root 4,0K Apr 23 2021 google-authenticator-encourage-user-activation
----r----- 1 1012 48 2,3K Apr 9 2019 hello.php
----r----- 1 1012 48 28 Apr 9 2019 index.php
drwxr-xr-x 8 root root 4,0K Sep 27 07:22 meta-box
drwxr-xr-x 8 root root 4,0K Mar 17 2024 ssl-insecure-content-fixer
drwxr-xr-x 4 root root 4,0K Oct 21 2019 vcaching
drwxr-xr-x 13 root root 4,0K Sep 25 13:56 woocommerce
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- with that, I tried wp-cli again, but it gave us an empty plugin list?
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list +------+--------+--------+---------+----------------+-------------+ | name | status | update | version | update_version | auto_update | +------+--------+--------+---------+----------------+-------------+ +------+--------+--------+---------+----------------+-------------+ wp@hetzner3:~$
- oh shoot, I forgot to update permissions. I'll do that now
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
- ok, then I retry wp-cli; it works!
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 +------------------------------------------------+----------+--------+---------+----------------+-------------+ | name | status | update | version | update_version | auto_update | +------------------------------------------------+----------+--------+---------+----------------+-------------+ | akismet | inactive | none | 5.3.3 | | off | | classic-editor | inactive | none | 1.6.5 | | off | | contact-form-7 | active | none | 5.9.8 | | off | | google-authenticator-encourage-user-activation | active | none | 0.2 | | off | | google-authenticator | active | none | 0.54 | | off | | hello | inactive | none | 1.7.1 | | off | | meta-box | active | none | 5.10.2 | | off | | ssl-insecure-content-fixer | active | none | 2.7.2 | | off | | vcaching | active | none | 1.8.3 | | off | | woocommerce | active | none | 9.3.3 | | off | | coingate-for-woocommerce | inactive | none | 2.1.1 | | off | +------------------------------------------------+----------+--------+---------+----------------+-------------+ wp@hetzner3:~$
- unfortunately, I get a blank page when I try to load store.opensourceecology.org in my web browser
- nginx is fine, but the varnish logs show that apache is returning a 403
[Thu Oct 03 04:19:37.076411 2024] [authz_core:error] [pid 3116759:tid 3116768] [client 81.17.16.77:0] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png, referer: https://store.opensourceecology.org/ ==> modsec_audit.log <== --fd8c6d25-A-- [03/Oct/2024:04:19:37.076625 +0000] Zv4bWZVyO5GHCka9cecUKwAAAEE 127.0.0.1 40720 127.0.0.1 8000 --fd8c6d25-B-- GET /wp-includes/images/w-logo-blue-white-bg.png HTTP/1.1 X-Real-IP: 81.17.16.77 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: store.opensourceecology.org User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Accept: image/avif,image/webp,*/* Accept-Language: en-US,en;q=0.5 Referer: https://store.opensourceecology.org/ Sec-Fetch-Dest: image Sec-Fetch-Mode: no-cors Sec-Fetch-Site: same-origin Sec-GPC: 1 Pragma: no-cache Accept-Encoding: gzip hash: #store.opensourceecology.org X-Varnish: 98343 --fd8c6d25-F-- HTTP/1.1 403 Forbidden X-Frame-Options: SAMEORIGIN Content-Length: 199 Content-Type: text/html; charset=iso-8859-1 --fd8c6d25-E-- <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> --fd8c6d25-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png Stopwatch: 1727929177076046 856 (- - -) Stopwatch2: 1727929177076046 856; combined=26, p1=24, p2=0, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --fd8c6d25-Z--
