Revision as of 21:06, 31 October 2024

My work log from the fourth quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

Thr Oct 03, 2024

I sent an invoice (AS-0106) to OSE for 67 hours in Sep 2024

...

continuing to debug store.opensourceecology.org, I see that it's redirecting from '/index.php' to '/'

user@disp3919:~$ curl -i https://store.opensourceecology.org/index.php
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 04 Oct 2024 04:47:39 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: https://store.opensourceecology.org/
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 131132 98385
Age: 88
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

user@disp3919:~$

if I try to hit 'index.html', I don't get a redirect -- I just get a 404

user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html
HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 04 Oct 2024 04:48:25 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/"
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 131134
Age: 0
Via: 1.1 varnish (Varnish/7.1)

user@disp3919:~$

if I create it

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cp is_hetzner3 index.html
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # ls -lah is_hetzner3 index.html 
----r----- 1 root       root     5 Oct  4 04:49 index.html
----r----- 1 not-apache www-data 5 Sep 27 04:44 is_hetzner3
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown not-apache:www-data index.html 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

then it works

user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Oct 2024 04:49:51 GMT
Content-Type: text/html
Content-Length: 5
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT
ETag: "5-6239f651921da"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
Pragma: public
Cache-Control: public, max-age=300
X-Varnish: 98387
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

true
user@disp3919:~$

hmm, so apache is just refusing to serve 'index.php' files. -- what about 'something.php'?

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # echo "<?php echo 'it works'; ?>" > something.php
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown root:www-data something.php 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chmod 0040 something.php 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

it works, so it's something specific to 'index.php'

user@disp3919:~$ curl -i https://store.opensourceecology.org/something.php
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Oct 2024 04:52:35 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98390
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

it worksuser@disp3919:~$

I'd think it's an issue with the DirectoryIndex, but this looks good

root@hetzner3 /etc/apache2 # grep -ir 'index.php' *
conf-available/wordpress.directory.include:#            RewriteRule . /index.php [L]
mods-available/dir.conf:DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm
root@hetzner3 /etc/apache2 # 

root@hetzner3 /etc/apache2 # ls -lah mods-enabled/dir.conf 
lrwxrwxrwx 1 root root 26 Sep 25 01:24 mods-enabled/dir.conf -> ../mods-available/dir.conf
root@hetzner3 /etc/apache2 #

I checked the old server, but I didn't see anything that we're missing in the new server

[root@opensourceecology httpd]# grep -ir 'index.php' conf
[root@opensourceecology httpd]# grep -ir 'index.php' conf.d
conf.d/php.conf:# Add index.php to the list of files that will be served as directory
conf.d/php.conf:DirectoryIndex index.php
conf.d/00-wiki.opensourceecology.org.conf:	Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php
conf.d/mod_evasive.conf:    #   http://security.lss.hr/index.php?page=details&ID=LSS-2005-01-01
[root@opensourceecology httpd]# grep -ir 'index.php' conf.modules.d/
[root@opensourceecology httpd]# grep -ir 'index.php' modsecurity.d/
[root@opensourceecology httpd]#

I changed wp-config.php to have WP_DEBUG set to 'true', but it didn't print anything extra. It seems like the error is occurring before wordpress
I set LogLevel of apache.conf to 'debug', and this popped-up

==> forum.opensourceecology.org/access.log <==
127.0.0.1 - - [04/Oct/2024:05:10:02 +0000] "GET /server-status?auto HTTP/1.1" 200 1202 "-" "munin/2.0.73 (libwww-perl/6.68)"

==> forum.opensourceecology.org/error.log <==
[Fri Oct 04 05:10:03.564975 2024] [authz_core:debug] [pid 3581402:tid 3581414] mod_authz_core.c(815): [client 127.0.0.1:32934] AH01626: authorization result of Require all denied: denied

well, that's an unrelated issue with munin, but it seems that the requests to '/server-status' are getting sent to the wrong vhost (forum.opensourceecology.org) and also denied access
here's the actual output when I do the curl
1. first, it outputs this immediately, then it pauses for maybe 10 seconds

==> store.opensourceecology.org/error.log <==
[Fri Oct 04 05:11:53.426292 2024] [authz_core:debug] [pid 3581402:tid 3581422] mod_authz_core.c(733): [client 81.17.16.91:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
[Fri Oct 04 05:11:53.426458 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(123): [client 81.17.16.91:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Fri Oct 04 05:11:53.426496 2024] [proxy:debug] [pid 3581402:tid 3581422] mod_proxy.c(1465): [client 81.17.16.91:0] AH01143: Running scheme fcgi handler (attempt 0)
[Fri Oct 04 05:11:53.426517 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1078): [client 81.17.16.91:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
[Fri Oct 04 05:11:53.426535 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1087): [client 81.17.16.91:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
[Fri Oct 04 05:11:53.426586 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80)
[Fri Oct 04 05:11:53.426612 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3242): [client 81.17.16.91:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
[Fri Oct 04 05:11:53.426658 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3309): [client 81.17.16.91:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
[Fri Oct 04 05:11:53.426718 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3450): [client 81.17.16.91:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
[Fri Oct 04 05:11:53.426793 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)

1. after maybe 10 seconds, it outputs this

[Fri Oct 04 05:12:03.646185 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80)

==> store.opensourceecology.org/access.log <==
81.17.16.91 - - [04/Oct/2024:05:11:53 +0000] "GET /index.php?nocache=6 HTTP/1.1" 301 430 "-" "curl/7.88.1"

so it sounds like maybe this is an issue with the php-fpm config?
I tried to hit apache through the cli on the server itself and, oh, I get the payload as desired

root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
<script>
        //jQuery(document).ready(function(){
                        // });
</script>
</body>
</html>root@hetzne
You have new mail in /var/mail/root

I loosened the error reporting settings on php.ini and I got it to spit this out when I curl from my laptop

--2d1eeb03-H--
Apache-Error: [file "mod_authz_core.c"] [line 733] [level 7] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 123] [level 7] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "mod_proxy.c"] [line 1465] [level 7] AH01143: Running scheme fcgi handler (attempt 0)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1078] [level 7] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1087] [level 7] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "proxy_util.c"] [line 3242] [level 7] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
Apache-Error: [file "proxy_util.c"] [line 3309] [level 7] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
Apache-Error: [file "proxy_util.c"] [line 3450] [level 7] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 911] [level 3] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function ini_set() in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php:590\\nStack trace:\\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(82): wp_debug_mode()\\n#1 /var/www/html/store.opensourceecology.org/wp-config.php(105): require_once('...')\\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-blog-header.php(13): require_once('...')\\n#4 /var/www/html/store.opensourceecology.org/htdocs/index.php(17): require('...')\\n#5 {main}\\n  thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php on line 590'
Apache-Handler: proxy:unix:/run/php/php8.2-fpm.sock|fcgi://localhost
Stopwatch: 1728019471413382 53626 (- - -)
Stopwatch2: 1728019471413382 53626; combined=41, p1=21, p2=18, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

yeah, so this is the wordpress bug that I submitted a PR for last month
after a brief dialog with the wordpress devs, a workaround until they merge is to define a fake ini_set() function in wp-config.php
1. from personal experience, I found it's best to wrap this in a conditional to make sure the function doesn't exist yet

root@hetzner3 /var/www/html/store.opensourceecology.org # cp wp-config.php wp-config.php.20241003
root@hetzner3 /var/www/html/store.opensourceecology.org # 

root@hetzner3 /var/www/html/store.opensourceecology.org # vim wp-config.php
root@hetzner3 /var/www/html/store.opensourceecology.org # 

root@hetzner3 /var/www/html/store.opensourceecology.org # diff wp-config.php.20241003 wp-config.php
1a2,9
> 
> # fix wordpress bug https://core.trac.wordpress.org/ticket/48693
> if( ! function_exists('ini_set') ){
>       function ini_set(){
>               return;
>       }
> }
> 
root@hetzner3 /var/www/html/store.opensourceecology.org #

after that, I'm back to getting blank pages on my curl on my laptop. it's flapping?
alright, let me see if I can harden php back up again, but with errors actually logging. I'll update something.php to write to the error log

root@hetzner3 /var/www/html/store.opensourceecology.org # cd htdocs/
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # vim something.php 
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cat something.php 
<?php
error_log( "executing something.php" );
echo 'it works';
?>
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #

ok, it's visible

[Fri Oct 04 05:47:06.734936 2024] [proxy_fcgi:error] [pid 3581402:tid 3581452] [client 127.0.0.1:36330] AH01071: Got error 'PHP message: executing something.php'

first I reduced the apache logs down to 'warn' again. looks good


[Fri Oct 04 05:48:39.732669 2024] [proxy_fcgi:error] [pid 3591906:tid 3591913] [client 127.0.0.1:41102] AH01071: Got error 'PHP message: executing something.php'

==> store.opensourceecology.org/access.log <==
127.0.0.1 - - [04/Oct/2024:05:48:39 +0000] "GET /something.php HTTP/1.1" 200 302 "-" "curl/7.88.1"

I obliterated my manual changes by pushing ansible's apache & php roles
cool, I confirmed that both curl on my laptop and on the server produce the logs after restarting both apache2 & php8.2-fpm
for some reason I still get an 'true' on my laptop

user@disp3919:~$ 3919:~$ curl -iL https://store.opensourceecolindex.php?nocache=19
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Fri, 04 Oct 2024 05:56:15 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Redirect-By: WordPress
X-Frame-Options: SAMEORIGIN
Location: https://store.opensourceecology.org/?nocache=19
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
X-Varnish: 98469
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Oct 2024 05:56:16 GMT
Content-Type: text/html
Content-Length: 5
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT
ETag: "5-6239f651921da"
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Referrer-Policy: no-referrer-when-downgrade
Pragma: public
Cache-Control: public, max-age=300
X-Varnish: 131203
Age: 0
Via: 1.1 varnish (Varnish/7.1)
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report"

true
user@disp3919:~$

but I get the actual html on the local machine

root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
/* ]]> */
</script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-content/themes/oshin/js/script.js?ver=5.0'></script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-includes/js/wp-embed.min.js?ver=5.1.1'></script>
<!-- Option Panel Custom JavaScript -->
<script>
        //jQuery(document).ready(function(){
                        // });
</script>
</body>
</html>root@hetzner3 ~ #

Wed Oct 02, 2024

Marcin sent me a few emails in the past months asking about OSE's use of Amazon Glacier
Today he sent a message saying that he got charged $1.03, and isn't sure why

Michael,

I'm getting charged $1.03 for Glacier. Can we cancel that?

Marcin

It took me a while to auth
1. first I tried to login with my 'maltfield' aws user, but aws rejected my creds (stored in my personal keepass)
2. eventually I realized I had to click "Sign in using root user email" -- and then I could auth using the creds stored in the shared keepass
after logging-in, I went to the "Billing and Cost Management" app https://us-east-1.console.aws.amazon.com/costmanagement/home?region=us-west-2#/home
on this page, there was a link that said "Last month's total cost: $1.03". Yep, that's all accounted-for. I clicked it.
the next page showed a joke of a chart with one bar on a bar graph that said "$1.03". And the bar was labeled "Total Cost"
I had to click on the dropdown menu for "Dimension" and set it to "Service" -- then it listed 4 items
1. Glacier - $1.03
2. S3 - $0.00
3. Tax $0.00
So I switched over to the "Glacier" app https://us-east-1.console.aws.amazon.com/glacier/home?region=us-east-1
1. Curiously, it listed 0 vaults
2. but there was a note at the top saying we should use S3 for glaicer, so I clicked over to the "S3" app

We recommend that you use Glacier storage classes in Amazon S3 for archival storage

here I saw one bucket called "oseserverbackups" in "US West (Oregon) us-west-2"
the bucket had one 34.0 byte file in it called "test.txt". That's it!
1. this file was created July 6, 2018, 19:18:03 (UTC-05:00)
2. I downloaded it; it has one line of text

some file destined for s3 this is

1. I deleted the 'test.txt' file object from the s3 bucket
I then deleted the (now empty) 'oseserverbackups' bucket
unconvinced that that was the issue, I went back to the "glacier" app. This time I cycled through a few of the regions until I got to "us-west-2" -- this time I showed one vault named "deleteMeIn2020"
I clicked on it, and it said
1. this vault was created March 29, 2018, 16:36:06 (UTC-05:00)
2. this vault was last inventoried August 1, 2018, 02:41:31 (UTC-05:00)
3. this vault is 285.3 GB (as of last inventory)
well, it's after 2020. So I think we should delete it.
I sent an email to Marcin asking for a confirmation before I delete it

Hey Marcin,

You have a 285.3 GB vault in Amazon Glacier's us-west-2 region.

I logged-into your AWS account today and did some digging. I found this vault 285.3 GB vault named 'deleteMeIn2020'. I created this vault in 2018 Q1. It contains a final backup of files from hetzner1. I created it as part of the hetzner2 migration project, thinking that we should delete it in 2020 if we never needed to restore anything from it for 2 years.

 * https://wiki.opensourceecology.org/wiki/CHG-2018-07-06_hetzner1_deprecation

 * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q1#Sat_Mar_31.2C_2018

Well, 2020 came and past. Four more years passed. I think you can safely delete the 'deleteMeIn2020' vault.

By the way, I also deleted a 53-byte test file from an S3 bucket named 'test.txt' in a bucket in s3 called 'oseserverbackups' in us-west-2. It was the only file in the bucket. I deleted the file and the empty bucket.

Would you like me to proceed with deleting the 285.3 GB 'deleteMeIn2020' glacier bucket from your AWS account?


Thank you,

Michael Altfield
Senior Technology Advisor
PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B

Open Source Ecology
www.opensourceecology.org

meanwhile, I tried to figure out why I couldn't login as 'maltfield', and I realized that, ffs, we don't have IAM setup for our account?? Maybe Marcin deleted it when trying to elimiate costs? IAM is free, though..
ok, I found my 'maltfield' user under "Security Credentials" -> "Access Management" -> "Users"
it says my last console sign-in was 424 days ago
I went to my user's settings, selected the MFA token, and selected "Resync" -- then entered two consecutive OTPs
I tried to login, and this time it let me in. Well that was annoying.
I opened cloudtrail and reviewed the latest account events https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?ReadOnly=false
1. the most recent event was the 'root' user resyncing the MFA token of the 'matlfield' token
2. before that we have two ConsoleLogin for today
3. before that 'mjakubowski' user has a MakePayment event (and some other payment related events) on Sep 19
4. before that we have a bunch of login & mfa-related entries for Marcin's user on Sep 06, 14, 17, and 19.
5. and that's where the log ends; looks like we just get 90 days of logs for free.

...

hetzner responded to my support inquery about how they handle failed disks

Dear Mr Altfield

Unfortunately it's an unmanaged root server monitoring is your responsibility I'm afraid. 

If you have a problem please open a ticket in your robot account. 

Please click on "Servers" from the menu on the left and then select the corresponding server. Under the "Support" tab, you can choose "Hard drive is broken". Please follow the instructions.

https://docs.hetzner.com/robot/dedicated-server/troubleshooting/serial-numbers-and-information-on-defective-hard-drives/

Our DC is 24/7 available and we exchange broken hardware as soon as possible for free.

Hetzner clients can use the Server Monitoring System to monitor their servers and have an email sent to them when the status of one of the monitored services changes: 

https://docs.hetzner.com/robot/dedicated-server/security/system-monitor/

https://docs.hetzner.com/robot/dedicated-server/raid/software-raid/#email-notification-when-a-drive-in-a-software-raid-fails

Please use hetzner-status:

https://www.hetzner-status.de/en.html

This web page publishes announcements and current fault reports from our datacenters. Would you like to receive email notification of fault reports? Log on as exclusive Hetzner client in your administrations interface. 

If you have any questions please do not hesitate to contact us. 

Kind regards

Jan Kolb

Sales

Hetzner Online GmbH
Sigmundstrasse 135
90431 Nürnberg
Tel: +49 911 234 226-927
Fax: +49 9831 505-3
sales@hetzner.com
www.hetzner.com

Register Court: Registergericht Ansbach, HRB 6089
CEO: Martin Hetzner, Stephan Konvickova, Günther Müller

For the purposes of this communication, we may save some 
of your personal data. For information on our data privacy 
policy, please see: www.hetzner.com/datenschutzhinweis

09/29/2024 21:23 - marcin@opensourceecology.org michael@opensourceecology.org wrote:

>
>
> Hi Hetzner,
>
> Can you please tell us more about the process of disk failure on our new dedicated
> server plan (Server Auction #2443019)?
>
> Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk?
> Or do we have to pay a fee? If so, how much?
>
> And does Hetzner have some system in-place that monitors the hardware for disk
> failure? Or do we have to monitor this in software and alert Hetnzer that a disk
> is failing? If Hetzner does monitor for disk failure, how does it do it?
>
>
> Thank you,
>
> Michael Altfield
> Senior Technology Advisor
> PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7  70D2 AA3E DF71 60E2 D97B
>
> Open Source Ecology
> www.opensourceecology.org
>

the docs linked-to actually don't mention mdadm, which I setup earlier to monitor and send us email alerts on our disks
instead, hetzner mentions `smartctl`, which is included in the debian package `smartmontools` -- which wasn't even installed!

root@hetzner3 /etc/mdadm # sudo apt-get install smartmontools
...
root@hetzner3 /etc/mdadm # 

root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme1n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

root@hetzner3 /etc/mdadm #

we can get more information with the `-A` argument

root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme0n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        36 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    142.729.615 [73,0 TB]
Data Units Written:                 20.452.874 [10,4 TB]
Host Read Commands:                 6.862.184.005
Host Write Commands:                876.931.661
Controller Busy Time:               15.948
Power Cycles:                       28
Power On Hours:                     16.350
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      159
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               36 Celsius
Temperature Sensor 2:               45 Celsius

root@hetzner3 /etc/mdadm # 

root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme1n1
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF SMART DATA SECTION ===
SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        34 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    130.064.348 [66,5 TB]
Data Units Written:                 24.932.683 [12,7 TB]
Host Read Commands:                 1.276.781.490
Host Write Commands:                879.017.438
Controller Busy Time:               14.879
Power Cycles:                       23
Power On Hours:                     14.678
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      149
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               34 Celsius
Temperature Sensor 2:               37 Celsius

root@hetzner3 /etc/mdadm #

oh nvm, their third link describes mdadm alerts for monitoring our software raid
they also said to check /etc/default/mdadm, which I didn't do before

root@hetzner3 /etc/mdadm # cat /etc/default/mdadm 
# mdadm Debian configuration
#
# You can run 'dpkg-reconfigure mdadm' to modify the values in this file, if
# you want. You can also change the values here and changes will be preserved.
# Do note that only the values are preserved; the rest of the file is
# rewritten.
#

# AUTOCHECK:
#   should mdadm run periodic redundancy checks over your arrays? See
#   /etc/cron.d/mdadm.
AUTOCHECK=true

# AUTOSCAN:
#   should mdadm check once a day for degraded arrays? See
#   /etc/cron.daily/mdadm.
AUTOSCAN=true

# START_DAEMON:
#   should mdadm start the MD monitoring daemon during boot?
START_DAEMON=true

# DAEMON_OPTIONS:
#   additional options to pass to the daemon.
DAEMON_OPTIONS="--syslog"

# VERBOSE:
#   if this variable is set to true, mdadm will be a little more verbose e.g.
#   when creating the initramfs.
VERBOSE=false
root@hetzner3 /etc/mdadm #

note that "AUTOCHECK" is enabled -- so we're all good here.

...

ok, back to updating wordpress.
first, I'm just going to unzip all these (now TOFU-verified) .zip files and make sure there's no zipbombs

root@hetzner3 ~ # cd /var/tmp/wordpress/themes/
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # ls
bouquet.1.2.5.zip          sketch.1.2.4.zip      twentyfifteen.3.8.zip   twentyseventeen.3.7.zip  twentythirteen.4.2.zip
gk-portfolio.1.5.3.zip     storefront.4.6.0.zip  twentyfourteen.4.0.zip  twentysixteen.3.3.zip    twentytwelve.4.3.zip
portfolio-press.2.8.0.zip  twentyeleven.4.7.zip  twentynineteen.2.9.zip  twentyten.4.2.zip
root@hetzner3 /var/tmp/wordpress/themes #

root@hetzner3 /var/tmp/wordpress/themes # for file in $(ls *.zip); do unzip $file; done
...
root@hetzner3 /var/tmp/wordpress/themes # 

root@hetzner3 /var/tmp/wordpress/themes # ls
bouquet                 portfolio-press.2.8.0.zip  twentyeleven           twentyfourteen.4.0.zip   twentysixteen          twentythirteen.4.2.zip
bouquet.1.2.5.zip       sketch                     twentyeleven.4.7.zip   twentynineteen           twentysixteen.3.3.zip  twentytwelve
gk-portfolio            sketch.1.2.4.zip           twentyfifteen          twentynineteen.2.9.zip   twentyten              twentytwelve.4.3.zip
gk-portfolio.1.5.3.zip  storefront                 twentyfifteen.3.8.zip  twentyseventeen          twentyten.4.2.zip
portfolio-press         storefront.4.6.0.zip       twentyfourteen         twentyseventeen.3.7.zip  twentythirteen
root@hetzner3 /var/tmp/wordpress/themes #

root@hetzner3 /var/tmp/wordpress/themes # cd ../plugins/
root@hetzner3 /var/tmp/wordpress/plugins #

root@hetzner3 /var/tmp/wordpress/plugins # for file in $(ls *.zip); do unzip $file; done
...
root@hetzner3 /var/tmp/wordpress/plugins #

root@hetzner3 /var/tmp/wordpress/plugins # ls
akismet                                                 jetpack                               vcaching
akismet.5.3.3.zip                                       jetpack.13.8.1.zip                    vcaching.1.8.3.zip
black-studio-tinymce-widget                             meta-box                              w3-total-cache
black-studio-tinymce-widget.2.7.3.zip                   meta-box.5.10.2.zip                   w3-total-cache.2.7.6.zip
chartbeat                                               ml-slider                             wonderm00ns-simple-facebook-open-graph-tags
chartbeat.2.0.7.zip                                     ml-slider.3.91.0.zip                  wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip
classic-editor                                          open-in-new-window-plugin             woocommerce
classic-editor.1.6.5.zip                                open-in-new-window-plugin.3.0.zip     woocommerce.9.3.3.zip
coingate-for-woocommerce                                post-types-order                      wordpress-importer
coingate-for-woocommerce.2.1.1.zip                      post-types-order.2.2.6.zip            wordpress-importer.0.8.2.zip
contact-form-7                                          revision-control                      wordpress-seo
contact-form-7.5.9.8.zip                                revision-control.2.3.2.zip            wordpress-seo.23.5.zip
duplicate-page                                          shareaholic                           wpautop-control
duplicate-page.4.5.zip                                  shareaholic.9.7.12.zip                wpautop-control.1.6.zip
duplicate-post                                          share-on-diaspora                     wp-memory-usage
duplicate-post.4.5.zip                                  share-on-diaspora.0.7.9.zip           wp-memory-usage.1.2.10.zip
google-authenticator                                    shariff                               wp-optimize
google-authenticator.0.54.zip                           shariff.4.6.14.zip                    wp-optimize.3.6.0.zip
google-authenticator-encourage-user-activation          ssl-insecure-content-fixer            wp-smushit
google-authenticator-encourage-user-activation.0.2.zip  ssl-insecure-content-fixer.2.7.2.zip  wp-smushit.3.16.6.zip
insert-headers-and-footers                              varnish-http-purge                    wp-super-cache
insert-headers-and-footers.2.2.2.zip                    varnish-http-purge.5.2.2.zip          wp-super-cache.1.12.4.zip
root@hetzner3 /var/tmp/wordpress/plugins #

ok, that looks good. now let's see if we can script copying-over these themes as-needed
1. and, to err on the side of caution, I'm going to intentionally delete any theme or plugin dir, even if we don't have one to replace it.

wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	
	echo "${theme}"
	rm -rf ${theme_path};
	rsync -av --progress "/var/tmp/wordpress/themes/${theme}/" "${theme_path}/"
done

after execution, looks like it worked

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 68K
d---r-x--- 16 not-apache www-data 4,0K Oct  3 04:02 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
----r-----  1 not-apache www-data   28 Jun  5  2014 index.php
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 oshin
drwxr-xr-x  5 root       root     4,0K May 16 08:29 storefront
drwxr-xr-x  7 root       root     4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x  7 root       root     4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x  5 root       root     4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x  4 root       root     4,0K Jul 15 17:17 twentyten
drwxr-xr-x  8 root       root     4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:17 twentytwelve
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentyfour
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentythree
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

oh, wait, no. it created some silly empty dirs when it didn't have a source to copy-from

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/oshin/
total 8,0K
drwxr-xr-x  2 root       root     4,0K Oct  3 04:02 .
d---r-x--- 16 not-apache www-data 4,0K Oct  3 04:02 ..
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

let's wrap that in a condition. and also disable verbose & progress on rsync, so we can see the whole output

for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
	theme=$(basename "${theme_path}")
	source_path="/var/tmp/wordpress/themes/${theme}"
	
	echo "${theme}"
	rm -rf ${theme_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${theme_path}/"
	fi
done

here's the execution; that's better

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
        theme=$(basename "${theme_path}")
        source_path="/var/tmp/wordpress/themes/${theme}"
        
        echo "${theme}"
        rm -rf ${theme_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${theme_path}/"
        fi
done
twentytwelve
twentysixteen
storefront
twentyseventeen
twentyfourteen
twentyeleven
twentytwentythree
oshin
twentytwentyfour
twentythirteen
twentyten
twentyfifteen
twentynineteen
twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 52K
d---r-x--- 12 not-apache www-data 4,0K Oct  3 04:04 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
----r-----  1 not-apache www-data   28 Jun  5  2014 index.php
drwxr-xr-x  5 root       root     4,0K May 16 08:29 storefront
drwxr-xr-x  7 root       root     4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x  7 root       root     4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x  9 root       root     4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x  5 root       root     4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x  4 root       root     4,0K Jul 15 17:17 twentyten
drwxr-xr-x  8 root       root     4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x  8 root       root     4,0K Jul 16 13:17 twentytwelve
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

now let's do the plugins with this

wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
	plugin=$(basename "${plugin_path}")
	source_path="/var/tmp/wordpress/plugins/${plugin}"
	
	echo "${plugin}"
	rm -rf ${plugin_path};
	if [ -d "${source_path}" ]; then
		rsync -a ${source_path}/ "${plugin_path}/"
	fi
done

I actually messed this up, and I had to restore the original plugins dir from the backup; easy enough

rsync -av --progress /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/

alright, here's the run

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"

for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
        plugin=$(basename "${plugin_path}")
        source_path="/var/tmp/wordpress/plugins/${plugin}"
        
        echo "${plugin}"
        rm -rf ${plugin_path};
        if [ -d "${source_path}" ]; then
                rsync -a ${source_path}/ "${plugin_path}/"
        fi
done
meta-box-show-hide
classic-editor
be-portfolio-post
colorhub
ssl-insecure-content-fixer
oshine-core
tatsu
revslider
redux-vendor-support
akismet
rename-wp-login
meta-box-tabs
google-authenticator
coingate-for-woocommerce
be-gdpr
google-authenticator-encourage-user-activation
typehub
meta-box
woocommerce
meta-box-conditional-logic
contact-form-7
vcaching
force-strong-passwords
masterslider
oshine-modules
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah plugins/
total 56K
d---r-x--- 12       1012       48 4,0K Oct  3 04:09 .
d---r-x---  7 not-apache www-data 4,0K Jul 23 15:15 ..
drwxr-xr-x  4 root       root     4,0K Jul 10 22:16 akismet
drwxr-xr-x  3 root       root     4,0K Sep 27 21:51 classic-editor
drwxr-xr-x  8 root       root     4,0K Nov 21  2022 coingate-for-woocommerce
drwxr-xr-x  7 root       root     4,0K Jul 25 08:28 contact-form-7
drwxr-xr-x  3 root       root     4,0K Jul  4  2022 google-authenticator
drwxr-xr-x  4 root       root     4,0K Apr 23  2021 google-authenticator-encourage-user-activation
----r-----  1       1012       48 2,3K Apr  9  2019 hello.php
----r-----  1       1012       48   28 Apr  9  2019 index.php
drwxr-xr-x  8 root       root     4,0K Sep 27 07:22 meta-box
drwxr-xr-x  8 root       root     4,0K Mar 17  2024 ssl-insecure-content-fixer
drwxr-xr-x  4 root       root     4,0K Oct 21  2019 vcaching
drwxr-xr-x 13 root       root     4,0K Sep 25 13:56 woocommerce
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #

with that, I tried wp-cli again, but it gave us an empty plugin list?

wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
+------+--------+--------+---------+----------------+-------------+
| name | status | update | version | update_version | auto_update |
+------+--------+--------+---------+----------------+-------------+
+------+--------+--------+---------+----------------+-------------+
wp@hetzner3:~$

oh shoot, I forgot to update permissions. I'll do that now

wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"

for wordpress_site in $wordpress_sites; do

	wp_docroot="$(dirname "${wordpress_site}")"
	vhost_dir="$(dirname "${wp_docroot}")"

	chown -R not-apache:www-data "${vhost_dir}"
	find "${vhost_dir}" -type d -exec chmod 0050 {} \;
	find "${vhost_dir}" -type f -exec chmod 0040 {} \;

	chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
	chmod 0040 "${vhost_dir}/wp-config.php"

	[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
	find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;

	[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
	chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
	find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
	find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;

done

ok, then I retry wp-cli; it works!

wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list
PHP Warning:  Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196
+------------------------------------------------+----------+--------+---------+----------------+-------------+
| name                                           | status   | update | version | update_version | auto_update |
+------------------------------------------------+----------+--------+---------+----------------+-------------+
| akismet                                        | inactive | none   | 5.3.3   |                | off         |
| classic-editor                                 | inactive | none   | 1.6.5   |                | off         |
| contact-form-7                                 | active   | none   | 5.9.8   |                | off         |
| google-authenticator-encourage-user-activation | active   | none   | 0.2     |                | off         |
| google-authenticator                           | active   | none   | 0.54    |                | off         |
| hello                                          | inactive | none   | 1.7.1   |                | off         |
| meta-box                                       | active   | none   | 5.10.2  |                | off         |
| ssl-insecure-content-fixer                     | active   | none   | 2.7.2   |                | off         |
| vcaching                                       | active   | none   | 1.8.3   |                | off         |
| woocommerce                                    | active   | none   | 9.3.3   |                | off         |
| coingate-for-woocommerce                       | inactive | none   | 2.1.1   |                | off         |
+------------------------------------------------+----------+--------+---------+----------------+-------------+
wp@hetzner3:~$

unfortunately, I get a blank page when I try to load store.opensourceecology.org in my web browser
nginx is fine, but the varnish logs show that apache is returning a 403

[Thu Oct 03 04:19:37.076411 2024] [authz_core:error] [pid 3116759:tid 3116768] [client 81.17.16.77:0] AH01630: client denied by server configuration: 
/var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png, referer: https://store.opensourceecology.org/

==> modsec_audit.log <==
--fd8c6d25-A--
[03/Oct/2024:04:19:37.076625 +0000] Zv4bWZVyO5GHCka9cecUKwAAAEE 127.0.0.1 40720 127.0.0.1 8000
--fd8c6d25-B--
GET /wp-includes/images/w-logo-blue-white-bg.png HTTP/1.1
X-Real-IP: 81.17.16.77
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: store.opensourceecology.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36
Accept: image/avif,image/webp,*/*
Accept-Language: en-US,en;q=0.5
Referer: https://store.opensourceecology.org/
Sec-Fetch-Dest: image
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
Sec-GPC: 1
Pragma: no-cache
Accept-Encoding: gzip
hash: #store.opensourceecology.org
X-Varnish: 98343

--fd8c6d25-F--
HTTP/1.1 403 Forbidden
X-Frame-Options: SAMEORIGIN
Content-Length: 199
Content-Type: text/html; charset=iso-8859-1

--fd8c6d25-E--
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
</body></html>

--fd8c6d25-H--
Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png
Stopwatch: 1727929177076046 856 (- - -)
Stopwatch2: 1727929177076046 856; combined=26, p1=24, p2=0, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"

--fd8c6d25-Z--

Maltfield Log/2024 Q4: Difference between revisions

Revision as of 21:06, 31 October 2024

Contents

See Also

Thr Oct 03, 2024

Wed Oct 02, 2024

Navigation menu

Maltfield Log/2024 Q4: Difference between revisions

Revision as of 21:06, 31 October 2024

See Also

Thr Oct 03, 2024

Wed Oct 02, 2024

Navigation menu

Search