Maltfield Log/2024 Q4
Jump to navigation
Jump to search
My work log from the fourth quarter of the year 2024. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
See Also
Thr Oct 03, 2024
- I sent an invoice (AS-0106) to OSE for 67 hours in Sep 2024
...
- continuing to debug store.opensourceecology.org, I see that it's redirecting from '/index.php' to '/'
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.php HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2024 04:47:39 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/ X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 131132 98385 Age: 88 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" user@disp3919:~$
- if I try to hit 'index.html', I don't get a redirect -- I just get a 404
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html HTTP/1.1 404 Not Found Server: nginx Date: Fri, 04 Oct 2024 04:48:25 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://store.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 131134 Age: 0 Via: 1.1 varnish (Varnish/7.1) user@disp3919:~$
- if I create it
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cp is_hetzner3 index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # ls -lah is_hetzner3 index.html ----r----- 1 root root 5 Oct 4 04:49 index.html ----r----- 1 not-apache www-data 5 Sep 27 04:44 is_hetzner3 root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown not-apache:www-data index.html root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- then it works
user@disp3919:~$ curl -i https://store.opensourceecology.org/index.html HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 04:49:51 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 98387 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3919:~$
- hmm, so apache is just refusing to serve 'index.php' files. -- what about 'something.php'?
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # echo "<?php echo 'it works'; ?>" > something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chown root:www-data something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # chmod 0040 something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- it works, so it's something specific to 'index.php'
user@disp3919:~$ curl -i https://store.opensourceecology.org/something.php HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 04:52:35 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 8 Connection: keep-alive X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 98390 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" it worksuser@disp3919:~$
- I'd think it's an issue with the DirectoryIndex, but this looks good
root@hetzner3 /etc/apache2 # grep -ir 'index.php' * conf-available/wordpress.directory.include:# RewriteRule . /index.php [L] mods-available/dir.conf:DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm root@hetzner3 /etc/apache2 # root@hetzner3 /etc/apache2 # ls -lah mods-enabled/dir.conf lrwxrwxrwx 1 root root 26 Sep 25 01:24 mods-enabled/dir.conf -> ../mods-available/dir.conf root@hetzner3 /etc/apache2 #
- I checked the old server, but I didn't see anything that we're missing in the new server
[root@opensourceecology httpd]# grep -ir 'index.php' conf [root@opensourceecology httpd]# grep -ir 'index.php' conf.d conf.d/php.conf:# Add index.php to the list of files that will be served as directory conf.d/php.conf:DirectoryIndex index.php conf.d/00-wiki.opensourceecology.org.conf: Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php conf.d/mod_evasive.conf: # http://security.lss.hr/index.php?page=details&ID=LSS-2005-01-01 [root@opensourceecology httpd]# grep -ir 'index.php' conf.modules.d/ [root@opensourceecology httpd]# grep -ir 'index.php' modsecurity.d/ [root@opensourceecology httpd]#
- I changed wp-config.php to have WP_DEBUG set to 'true', but it didn't print anything extra. It seems like the error is occurring before wordpress
- I set LogLevel of apache.conf to 'debug', and this popped-up
==> forum.opensourceecology.org/access.log <== 127.0.0.1 - - [04/Oct/2024:05:10:02 +0000] "GET /server-status?auto HTTP/1.1" 200 1202 "-" "munin/2.0.73 (libwww-perl/6.68)" ==> forum.opensourceecology.org/error.log <== [Fri Oct 04 05:10:03.564975 2024] [authz_core:debug] [pid 3581402:tid 3581414] mod_authz_core.c(815): [client 127.0.0.1:32934] AH01626: authorization result of Require all denied: denied
- well, that's an unrelated issue with munin, but it seems that the requests to '/server-status' are getting sent to the wrong vhost (forum.opensourceecology.org) and also denied access
- here's the actual output when I do the curl
- first, it outputs this immediately, then it pauses for maybe 10 seconds
==> store.opensourceecology.org/error.log <== [Fri Oct 04 05:11:53.426292 2024] [authz_core:debug] [pid 3581402:tid 3581422] mod_authz_core.c(733): [client 81.17.16.91:0] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods) [Fri Oct 04 05:11:53.426458 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(123): [client 81.17.16.91:0] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Fri Oct 04 05:11:53.426496 2024] [proxy:debug] [pid 3581402:tid 3581422] mod_proxy.c(1465): [client 81.17.16.91:0] AH01143: Running scheme fcgi handler (attempt 0) [Fri Oct 04 05:11:53.426517 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1078): [client 81.17.16.91:0] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0 [Fri Oct 04 05:11:53.426535 2024] [proxy_fcgi:debug] [pid 3581402:tid 3581422] mod_proxy_fcgi.c(1087): [client 81.17.16.91:0] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php [Fri Oct 04 05:11:53.426586 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2797): AH00942: FCGI: has acquired connection for (*:80) [Fri Oct 04 05:11:53.426612 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3242): [client 81.17.16.91:0] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000 [Fri Oct 04 05:11:53.426658 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3309): [client 81.17.16.91:0] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000) [Fri Oct 04 05:11:53.426718 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3450): [client 81.17.16.91:0] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000) [Fri Oct 04 05:11:53.426793 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(3832): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.2-fpm.sock (localhost:8000)
- after maybe 10 seconds, it outputs this
[Fri Oct 04 05:12:03.646185 2024] [proxy:debug] [pid 3581402:tid 3581422] proxy_util.c(2813): AH00943: FCGI: has released connection for (*:80) ==> store.opensourceecology.org/access.log <== 81.17.16.91 - - [04/Oct/2024:05:11:53 +0000] "GET /index.php?nocache=6 HTTP/1.1" 301 430 "-" "curl/7.88.1"
- so it sounds like maybe this is an issue with the php-fpm config?
- I tried to hit apache through the cli on the server itself and, oh, I get the payload as desired
root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
<script>
//jQuery(document).ready(function(){
// });
</script>
</body>
</html>root@hetzne
You have new mail in /var/mail/root
- I loosened the error reporting settings on php.ini and I got it to spit this out when I curl from my laptop
--2d1eeb03-H--
Apache-Error: [file "mod_authz_core.c"] [line 733] [level 7] AH01625: authorization result of <RequireAny>: granted (directive limited to other methods)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 123] [level 7] AH01060: set r->filename to proxy:fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "mod_proxy.c"] [line 1465] [level 7] AH01143: Running scheme fcgi handler (attempt 0)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1078] [level 7] AH01076: url: fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php proxyname: (null) proxyport: 0
Apache-Error: [file "mod_proxy_fcgi.c"] [line 1087] [level 7] AH01078: serving URL fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php
Apache-Error: [file "proxy_util.c"] [line 3242] [level 7] AH00944: connecting fcgi://localhost/var/www/html/store.opensourceecology.org/htdocs/index.php to localhost:8000
Apache-Error: [file "proxy_util.c"] [line 3309] [level 7] AH02545: fcgi: has determined UDS as /run/php/php8.2-fpm.sock (for localhost:8000)
Apache-Error: [file "proxy_util.c"] [line 3450] [level 7] AH00947: connecting /var/www/html/store.opensourceecology.org/htdocs/index.php to /run/php/php8.2-fpm.sock:0 (localhost:8000)
Apache-Error: [file "mod_proxy_fcgi.c"] [line 911] [level 3] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to undefined function ini_set() in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php:590\\nStack trace:\\n#0 /var/www/html/store.opensourceecology.org/htdocs/wp-settings.php(82): wp_debug_mode()\\n#1 /var/www/html/store.opensourceecology.org/wp-config.php(105): require_once('...')\\n#2 /var/www/html/store.opensourceecology.org/htdocs/wp-load.php(55): require_once('...')\\n#3 /var/www/html/store.opensourceecology.org/htdocs/wp-blog-header.php(13): require_once('...')\\n#4 /var/www/html/store.opensourceecology.org/htdocs/index.php(17): require('...')\\n#5 {main}\\n thrown in /var/www/html/store.opensourceecology.org/htdocs/wp-includes/load.php on line 590'
Apache-Handler: proxy:unix:/run/php/php8.2-fpm.sock|fcgi://localhost
Stopwatch: 1728019471413382 53626 (- - -)
Stopwatch2: 1728019471413382 53626; combined=41, p1=21, p2=18, p3=1, p4=0, p5=1, sr=0, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/).
Server: Apache
Engine-Mode: "ENABLED"
- yeah, so this is the wordpress bug that I submitted a PR for last month
- after a brief dialog with the wordpress devs, a workaround until they merge is to define a fake ini_set() function in wp-config.php
- from personal experience, I found it's best to wrap this in a conditional to make sure the function doesn't exist yet
root@hetzner3 /var/www/html/store.opensourceecology.org # cp wp-config.php wp-config.php.20241003
root@hetzner3 /var/www/html/store.opensourceecology.org #
root@hetzner3 /var/www/html/store.opensourceecology.org # vim wp-config.php
root@hetzner3 /var/www/html/store.opensourceecology.org #
root@hetzner3 /var/www/html/store.opensourceecology.org # diff wp-config.php.20241003 wp-config.php
1a2,9
>
> # fix wordpress bug https://core.trac.wordpress.org/ticket/48693
> if( ! function_exists('ini_set') ){
> function ini_set(){
> return;
> }
> }
>
root@hetzner3 /var/www/html/store.opensourceecology.org #
- after that, I'm back to getting blank pages on my curl on my laptop. it's flapping?
- alright, let me see if I can harden php back up again, but with errors actually logging. I'll update something.php to write to the error log
root@hetzner3 /var/www/html/store.opensourceecology.org # cd htdocs/ root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # vim something.php root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs # cat something.php <?php error_log( "executing something.php" ); echo 'it works'; ?> root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs #
- ok, it's visible
[Fri Oct 04 05:47:06.734936 2024] [proxy_fcgi:error] [pid 3581402:tid 3581452] [client 127.0.0.1:36330] AH01071: Got error 'PHP message: executing something.php'
- first I reduced the apache logs down to 'warn' again. looks good
[Fri Oct 04 05:48:39.732669 2024] [proxy_fcgi:error] [pid 3591906:tid 3591913] [client 127.0.0.1:41102] AH01071: Got error 'PHP message: executing something.php' ==> store.opensourceecology.org/access.log <== 127.0.0.1 - - [04/Oct/2024:05:48:39 +0000] "GET /something.php HTTP/1.1" 200 302 "-" "curl/7.88.1"
- I obliterated my manual changes by pushing ansible's apache & php roles
- cool, I confirmed that both curl on my laptop and on the server produce the logs after restarting both apache2 & php8.2-fpm
- for some reason I still get an 'true' on my laptop
user@disp3919:~$ 3919:~$ curl -iL https://store.opensourceecolindex.php?nocache=19 HTTP/1.1 301 Moved Permanently Server: nginx Date: Fri, 04 Oct 2024 05:56:15 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Redirect-By: WordPress X-Frame-Options: SAMEORIGIN Location: https://store.opensourceecology.org/?nocache=19 X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade X-Varnish: 98469 Age: 0 Via: 1.1 varnish (Varnish/7.1) Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" HTTP/1.1 200 OK Server: nginx Date: Fri, 04 Oct 2024 05:56:16 GMT Content-Type: text/html Content-Length: 5 Connection: keep-alive X-Frame-Options: SAMEORIGIN Last-Modified: Fri, 04 Oct 2024 04:49:23 GMT ETag: "5-6239f651921da" X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block X-Frame-Options: deny Referrer-Policy: no-referrer-when-downgrade Pragma: public Cache-Control: public, max-age=300 X-Varnish: 131203 Age: 0 Via: 1.1 varnish (Varnish/7.1) Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http://opensourceecology.org/hpkp-report" true user@disp3919:~$
- but I get the actual html on the local machine
root@hetzner3 ~ # curl -iLH 'Host: store.opensourceecology.org' 127.0.0.1:8000/index.php
...
/* ]]> */
</script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-content/themes/oshin/js/script.js?ver=5.0'></script>
<script type='text/javascript' src='https://store.opensourceecology.org/wp-includes/js/wp-embed.min.js?ver=5.1.1'></script>
<!-- Option Panel Custom JavaScript -->
<script>
//jQuery(document).ready(function(){
// });
</script>
</body>
</html>root@hetzner3 ~ #
Wed Oct 02, 2024
- Marcin sent me a few emails in the past months asking about OSE's use of Amazon Glacier
- Today he sent a message saying that he got charged $1.03, and isn't sure why
Michael, I'm getting charged $1.03 for Glacier. Can we cancel that? Marcin
- It took me a while to auth
- first I tried to login with my 'maltfield' aws user, but aws rejected my creds (stored in my personal keepass)
- eventually I realized I had to click "Sign in using root user email" -- and then I could auth using the creds stored in the shared keepass
- after logging-in, I went to the "Billing and Cost Management" app https://us-east-1.console.aws.amazon.com/costmanagement/home?region=us-west-2#/home
- on this page, there was a link that said "Last month's total cost: $1.03". Yep, that's all accounted-for. I clicked it.
- the next page showed a joke of a chart with one bar on a bar graph that said "$1.03". And the bar was labeled "Total Cost"
- I had to click on the dropdown menu for "Dimension" and set it to "Service" -- then it listed 4 items
- Glacier - $1.03
- S3 - $0.00
- Tax $0.00
- So I switched over to the "Glacier" app https://us-east-1.console.aws.amazon.com/glacier/home?region=us-east-1
- Curiously, it listed 0 vaults
- but there was a note at the top saying we should use S3 for glaicer, so I clicked over to the "S3" app
We recommend that you use Glacier storage classes in Amazon S3 for archival storage
- here I saw one bucket called "oseserverbackups" in "US West (Oregon) us-west-2"
- the bucket had one 34.0 byte file in it called "test.txt". That's it!
- this file was created July 6, 2018, 19:18:03 (UTC-05:00)
- I downloaded it; it has one line of text
some file destined for s3 this is
- I deleted the 'test.txt' file object from the s3 bucket
- I then deleted the (now empty) 'oseserverbackups' bucket
- unconvinced that that was the issue, I went back to the "glacier" app. This time I cycled through a few of the regions until I got to "us-west-2" -- this time I showed one vault named "deleteMeIn2020"
- I clicked on it, and it said
- this vault was created March 29, 2018, 16:36:06 (UTC-05:00)
- this vault was last inventoried August 1, 2018, 02:41:31 (UTC-05:00)
- this vault is 285.3 GB (as of last inventory)
- well, it's after 2020. So I think we should delete it.
- I sent an email to Marcin asking for a confirmation before I delete it
Hey Marcin, You have a 285.3 GB vault in Amazon Glacier's us-west-2 region. I logged-into your AWS account today and did some digging. I found this vault 285.3 GB vault named 'deleteMeIn2020'. I created this vault in 2018 Q1. It contains a final backup of files from hetzner1. I created it as part of the hetzner2 migration project, thinking that we should delete it in 2020 if we never needed to restore anything from it for 2 years. * https://wiki.opensourceecology.org/wiki/CHG-2018-07-06_hetzner1_deprecation * https://wiki.opensourceecology.org/wiki/Maltfield_Log/2018_Q1#Sat_Mar_31.2C_2018 Well, 2020 came and past. Four more years passed. I think you can safely delete the 'deleteMeIn2020' vault. By the way, I also deleted a 53-byte test file from an S3 bucket named 'test.txt' in a bucket in s3 called 'oseserverbackups' in us-west-2. It was the only file in the bucket. I deleted the file and the empty bucket. Would you like me to proceed with deleting the 285.3 GB 'deleteMeIn2020' glacier bucket from your AWS account? Thank you, Michael Altfield Senior Technology Advisor PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B Open Source Ecology www.opensourceecology.org
- meanwhile, I tried to figure out why I couldn't login as 'maltfield', and I realized that, ffs, we don't have IAM setup for our account?? Maybe Marcin deleted it when trying to elimiate costs? IAM is free, though..
- ok, I found my 'maltfield' user under "Security Credentials" -> "Access Management" -> "Users"
- it says my last console sign-in was 424 days ago
- I went to my user's settings, selected the MFA token, and selected "Resync" -- then entered two consecutive OTPs
- I tried to login, and this time it let me in. Well that was annoying.
- I opened cloudtrail and reviewed the latest account events https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?ReadOnly=false
- the most recent event was the 'root' user resyncing the MFA token of the 'matlfield' token
- before that we have two ConsoleLogin for today
- before that 'mjakubowski' user has a MakePayment event (and some other payment related events) on Sep 19
- before that we have a bunch of login & mfa-related entries for Marcin's user on Sep 06, 14, 17, and 19.
- and that's where the log ends; looks like we just get 90 days of logs for free.
...
- hetzner responded to my support inquery about how they handle failed disks
Dear Mr Altfield Unfortunately it's an unmanaged root server monitoring is your responsibility I'm afraid. If you have a problem please open a ticket in your robot account. Please click on "Servers" from the menu on the left and then select the corresponding server. Under the "Support" tab, you can choose "Hard drive is broken". Please follow the instructions. https://docs.hetzner.com/robot/dedicated-server/troubleshooting/serial-numbers-and-information-on-defective-hard-drives/ Our DC is 24/7 available and we exchange broken hardware as soon as possible for free. Hetzner clients can use the Server Monitoring System to monitor their servers and have an email sent to them when the status of one of the monitored services changes: https://docs.hetzner.com/robot/dedicated-server/security/system-monitor/ https://docs.hetzner.com/robot/dedicated-server/raid/software-raid/#email-notification-when-a-drive-in-a-software-raid-fails Please use hetzner-status: https://www.hetzner-status.de/en.html This web page publishes announcements and current fault reports from our datacenters. Would you like to receive email notification of fault reports? Log on as exclusive Hetzner client in your administrations interface. If you have any questions please do not hesitate to contact us. Kind regards Jan Kolb Sales Hetzner Online GmbH Sigmundstrasse 135 90431 Nürnberg Tel: +49 911 234 226-927 Fax: +49 9831 505-3 sales@hetzner.com www.hetzner.com Register Court: Registergericht Ansbach, HRB 6089 CEO: Martin Hetzner, Stephan Konvickova, Günther Müller For the purposes of this communication, we may save some of your personal data. For information on our data privacy policy, please see: www.hetzner.com/datenschutzhinweis 09/29/2024 21:23 - marcin@opensourceecology.org michael@opensourceecology.org wrote: > > > Hi Hetzner, > > Can you please tell us more about the process of disk failure on our new dedicated > server plan (Server Auction #2443019)? > > Specifically, if a disk fails, does Hetzner cover the cost of replacing the disk? > Or do we have to pay a fee? If so, how much? > > And does Hetzner have some system in-place that monitors the hardware for disk > failure? Or do we have to monitor this in software and alert Hetnzer that a disk > is failing? If Hetzner does monitor for disk failure, how does it do it? > > > Thank you, > > Michael Altfield > Senior Technology Advisor > PGP Fingerprint: 8A4B 0AF8 162F 3B6A 79B7 70D2 AA3E DF71 60E2 D97B > > Open Source Ecology > www.opensourceecology.org >
- the docs linked-to actually don't mention mdadm, which I setup earlier to monitor and send us email alerts on our disks
- instead, hetzner mentions `smartctl`, which is included in the debian package `smartmontools` -- which wasn't even installed!
root@hetzner3 /etc/mdadm # sudo apt-get install smartmontools ... root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme0n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED root@hetzner3 /etc/mdadm # smartctl -H /dev/nvme1n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED root@hetzner3 /etc/mdadm #
- we can get more information with the `-A` argument
root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme0n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 36 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 3% Data Units Read: 142.729.615 [73,0 TB] Data Units Written: 20.452.874 [10,4 TB] Host Read Commands: 6.862.184.005 Host Write Commands: 876.931.661 Controller Busy Time: 15.948 Power Cycles: 28 Power On Hours: 16.350 Unsafe Shutdowns: 5 Media and Data Integrity Errors: 0 Error Information Log Entries: 159 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 36 Celsius Temperature Sensor 2: 45 Celsius root@hetzner3 /etc/mdadm # root@hetzner3 /etc/mdadm # smartctl -A /dev/nvme1n1 smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-21-amd64] (local build) Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 34 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 3% Data Units Read: 130.064.348 [66,5 TB] Data Units Written: 24.932.683 [12,7 TB] Host Read Commands: 1.276.781.490 Host Write Commands: 879.017.438 Controller Busy Time: 14.879 Power Cycles: 23 Power On Hours: 14.678 Unsafe Shutdowns: 5 Media and Data Integrity Errors: 0 Error Information Log Entries: 149 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 34 Celsius Temperature Sensor 2: 37 Celsius root@hetzner3 /etc/mdadm #
- oh nvm, their third link describes mdadm alerts for monitoring our software raid
- they also said to check /etc/default/mdadm, which I didn't do before
root@hetzner3 /etc/mdadm # cat /etc/default/mdadm # mdadm Debian configuration # # You can run 'dpkg-reconfigure mdadm' to modify the values in this file, if # you want. You can also change the values here and changes will be preserved. # Do note that only the values are preserved; the rest of the file is # rewritten. # # AUTOCHECK: # should mdadm run periodic redundancy checks over your arrays? See # /etc/cron.d/mdadm. AUTOCHECK=true # AUTOSCAN: # should mdadm check once a day for degraded arrays? See # /etc/cron.daily/mdadm. AUTOSCAN=true # START_DAEMON: # should mdadm start the MD monitoring daemon during boot? START_DAEMON=true # DAEMON_OPTIONS: # additional options to pass to the daemon. DAEMON_OPTIONS="--syslog" # VERBOSE: # if this variable is set to true, mdadm will be a little more verbose e.g. # when creating the initramfs. VERBOSE=false root@hetzner3 /etc/mdadm #
- note that "AUTOCHECK" is enabled -- so we're all good here.
...
- ok, back to updating wordpress.
- first, I'm just going to unzip all these (now TOFU-verified) .zip files and make sure there's no zipbombs
root@hetzner3 ~ # cd /var/tmp/wordpress/themes/ root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # ls bouquet.1.2.5.zip sketch.1.2.4.zip twentyfifteen.3.8.zip twentyseventeen.3.7.zip twentythirteen.4.2.zip gk-portfolio.1.5.3.zip storefront.4.6.0.zip twentyfourteen.4.0.zip twentysixteen.3.3.zip twentytwelve.4.3.zip portfolio-press.2.8.0.zip twentyeleven.4.7.zip twentynineteen.2.9.zip twentyten.4.2.zip root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # for file in $(ls *.zip); do unzip $file; done ... root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # ls bouquet portfolio-press.2.8.0.zip twentyeleven twentyfourteen.4.0.zip twentysixteen twentythirteen.4.2.zip bouquet.1.2.5.zip sketch twentyeleven.4.7.zip twentynineteen twentysixteen.3.3.zip twentytwelve gk-portfolio sketch.1.2.4.zip twentyfifteen twentynineteen.2.9.zip twentyten twentytwelve.4.3.zip gk-portfolio.1.5.3.zip storefront twentyfifteen.3.8.zip twentyseventeen twentyten.4.2.zip portfolio-press storefront.4.6.0.zip twentyfourteen twentyseventeen.3.7.zip twentythirteen root@hetzner3 /var/tmp/wordpress/themes # root@hetzner3 /var/tmp/wordpress/themes # cd ../plugins/ root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # for file in $(ls *.zip); do unzip $file; done ... root@hetzner3 /var/tmp/wordpress/plugins # root@hetzner3 /var/tmp/wordpress/plugins # ls akismet jetpack vcaching akismet.5.3.3.zip jetpack.13.8.1.zip vcaching.1.8.3.zip black-studio-tinymce-widget meta-box w3-total-cache black-studio-tinymce-widget.2.7.3.zip meta-box.5.10.2.zip w3-total-cache.2.7.6.zip chartbeat ml-slider wonderm00ns-simple-facebook-open-graph-tags chartbeat.2.0.7.zip ml-slider.3.91.0.zip wonderm00ns-simple-facebook-open-graph-tags.3.3.3.zip classic-editor open-in-new-window-plugin woocommerce classic-editor.1.6.5.zip open-in-new-window-plugin.3.0.zip woocommerce.9.3.3.zip coingate-for-woocommerce post-types-order wordpress-importer coingate-for-woocommerce.2.1.1.zip post-types-order.2.2.6.zip wordpress-importer.0.8.2.zip contact-form-7 revision-control wordpress-seo contact-form-7.5.9.8.zip revision-control.2.3.2.zip wordpress-seo.23.5.zip duplicate-page shareaholic wpautop-control duplicate-page.4.5.zip shareaholic.9.7.12.zip wpautop-control.1.6.zip duplicate-post share-on-diaspora wp-memory-usage duplicate-post.4.5.zip share-on-diaspora.0.7.9.zip wp-memory-usage.1.2.10.zip google-authenticator shariff wp-optimize google-authenticator.0.54.zip shariff.4.6.14.zip wp-optimize.3.6.0.zip google-authenticator-encourage-user-activation ssl-insecure-content-fixer wp-smushit google-authenticator-encourage-user-activation.0.2.zip ssl-insecure-content-fixer.2.7.2.zip wp-smushit.3.16.6.zip insert-headers-and-footers varnish-http-purge wp-super-cache insert-headers-and-footers.2.2.2.zip varnish-http-purge.5.2.2.zip wp-super-cache.1.12.4.zip root@hetzner3 /var/tmp/wordpress/plugins #
- ok, that looks good. now let's see if we can script copying-over these themes as-needed
- and, to err on the side of caution, I'm going to intentionally delete any theme or plugin dir, even if we don't have one to replace it.
wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
echo "${theme}"
rm -rf ${theme_path};
rsync -av --progress "/var/tmp/wordpress/themes/${theme}/" "${theme_path}/"
done
- after execution, looks like it worked
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/ total 68K d---r-x--- 16 not-apache www-data 4,0K Oct 3 04:02 . d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 .. ----r----- 1 not-apache www-data 28 Jun 5 2014 index.php drwxr-xr-x 2 root root 4,0K Oct 3 04:02 oshin drwxr-xr-x 5 root root 4,0K May 16 08:29 storefront drwxr-xr-x 7 root root 4,0K Jul 16 13:09 twentyeleven drwxr-xr-x 7 root root 4,0K Jul 16 13:28 twentyfifteen drwxr-xr-x 9 root root 4,0K Jul 16 13:23 twentyfourteen drwxr-xr-x 9 root root 4,0K Jul 16 13:30 twentynineteen drwxr-xr-x 5 root root 4,0K Jul 16 13:29 twentyseventeen drwxr-xr-x 8 root root 4,0K Jul 16 13:29 twentysixteen drwxr-xr-x 4 root root 4,0K Jul 15 17:17 twentyten drwxr-xr-x 8 root root 4,0K Jul 16 13:20 twentythirteen drwxr-xr-x 8 root root 4,0K Jul 16 13:17 twentytwelve drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentyfour drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentythree drwxr-xr-x 2 root root 4,0K Oct 3 04:02 twentytwentytwo root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- oh, wait, no. it created some silly empty dirs when it didn't have a source to copy-from
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/oshin/ total 8,0K drwxr-xr-x 2 root root 4,0K Oct 3 04:02 . d---r-x--- 16 not-apache www-data 4,0K Oct 3 04:02 .. root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- let's wrap that in a condition. and also disable verbose & progress on rsync, so we can see the whole output
for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
- here's the execution; that's better
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # for theme_path in $(find "${wp_docroot}/wp-content/themes" -mindepth 1 -maxdepth 1 -type d); do
theme=$(basename "${theme_path}")
source_path="/var/tmp/wordpress/themes/${theme}"
echo "${theme}"
rm -rf ${theme_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${theme_path}/"
fi
done
twentytwelve
twentysixteen
storefront
twentyseventeen
twentyfourteen
twentyeleven
twentytwentythree
oshin
twentytwentyfour
twentythirteen
twentyten
twentyfifteen
twentynineteen
twentytwentytwo
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah themes/
total 52K
d---r-x--- 12 not-apache www-data 4,0K Oct 3 04:04 .
d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 ..
----r----- 1 not-apache www-data 28 Jun 5 2014 index.php
drwxr-xr-x 5 root root 4,0K May 16 08:29 storefront
drwxr-xr-x 7 root root 4,0K Jul 16 13:09 twentyeleven
drwxr-xr-x 7 root root 4,0K Jul 16 13:28 twentyfifteen
drwxr-xr-x 9 root root 4,0K Jul 16 13:23 twentyfourteen
drwxr-xr-x 9 root root 4,0K Jul 16 13:30 twentynineteen
drwxr-xr-x 5 root root 4,0K Jul 16 13:29 twentyseventeen
drwxr-xr-x 8 root root 4,0K Jul 16 13:29 twentysixteen
drwxr-xr-x 4 root root 4,0K Jul 15 17:17 twentyten
drwxr-xr-x 8 root root 4,0K Jul 16 13:20 twentythirteen
drwxr-xr-x 8 root root 4,0K Jul 16 13:17 twentytwelve
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- now let's do the plugins with this
wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
- I actually messed this up, and I had to restore the original plugins dir from the backup; easy enough
rsync -av --progress /var/tmp/hetzner2-www-20240926/root/backups/sync/daily_hetzner2_20240926_072001/www/var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/ /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/
- alright, here's the run
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # wp_docroot="/var/www/html/store.opensourceecology.org/htdocs"
for plugin_path in $(find "${wp_docroot}/wp-content/plugins" -mindepth 1 -maxdepth 1 -type d); do
plugin=$(basename "${plugin_path}")
source_path="/var/tmp/wordpress/plugins/${plugin}"
echo "${plugin}"
rm -rf ${plugin_path};
if [ -d "${source_path}" ]; then
rsync -a ${source_path}/ "${plugin_path}/"
fi
done
meta-box-show-hide
classic-editor
be-portfolio-post
colorhub
ssl-insecure-content-fixer
oshine-core
tatsu
revslider
redux-vendor-support
akismet
rename-wp-login
meta-box-tabs
google-authenticator
coingate-for-woocommerce
be-gdpr
google-authenticator-encourage-user-activation
typehub
meta-box
woocommerce
meta-box-conditional-logic
contact-form-7
vcaching
force-strong-passwords
masterslider
oshine-modules
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content # ls -lah plugins/
total 56K
d---r-x--- 12 1012 48 4,0K Oct 3 04:09 .
d---r-x--- 7 not-apache www-data 4,0K Jul 23 15:15 ..
drwxr-xr-x 4 root root 4,0K Jul 10 22:16 akismet
drwxr-xr-x 3 root root 4,0K Sep 27 21:51 classic-editor
drwxr-xr-x 8 root root 4,0K Nov 21 2022 coingate-for-woocommerce
drwxr-xr-x 7 root root 4,0K Jul 25 08:28 contact-form-7
drwxr-xr-x 3 root root 4,0K Jul 4 2022 google-authenticator
drwxr-xr-x 4 root root 4,0K Apr 23 2021 google-authenticator-encourage-user-activation
----r----- 1 1012 48 2,3K Apr 9 2019 hello.php
----r----- 1 1012 48 28 Apr 9 2019 index.php
drwxr-xr-x 8 root root 4,0K Sep 27 07:22 meta-box
drwxr-xr-x 8 root root 4,0K Mar 17 2024 ssl-insecure-content-fixer
drwxr-xr-x 4 root root 4,0K Oct 21 2019 vcaching
drwxr-xr-x 13 root root 4,0K Sep 25 13:56 woocommerce
root@hetzner3 /var/www/html/store.opensourceecology.org/htdocs/wp-content #
- with that, I tried wp-cli again, but it gave us an empty plugin list?
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list +------+--------+--------+---------+----------------+-------------+ | name | status | update | version | update_version | auto_update | +------+--------+--------+---------+----------------+-------------+ +------+--------+--------+---------+----------------+-------------+ wp@hetzner3:~$
- oh shoot, I forgot to update permissions. I'll do that now
wordpress_sites="$(find /var/www/html -type d -wholename *htdocs/wp-content)"
for wordpress_site in $wordpress_sites; do
wp_docroot="$(dirname "${wordpress_site}")"
vhost_dir="$(dirname "${wp_docroot}")"
chown -R not-apache:www-data "${vhost_dir}"
find "${vhost_dir}" -type d -exec chmod 0050 {} \;
find "${vhost_dir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhost_dir}/wp-config.php"
chmod 0040 "${vhost_dir}/wp-config.php"
[ -d "${wp_docroot}/wp-content/uploads" ] || mkdir "${wp_docroot}/wp-content/uploads"
chown -R not-apache:www-data "${wp_docroot}/wp-content/uploads"
find "${wp_docroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
[ -d "${wp_docroot}/wp-content/tmp" ] || mkdir "${wp_docroot}/wp-content/tmp"
chown -R not-apache:www-data "${wp_docroot}/wp-content/tmp"
find "${wp_docroot}/wp-content/tmp" -type f -exec chmod 0660 {} \;
find "${wp_docroot}/wp-content/tmp" -type d -exec chmod 0770 {} \;
done
- ok, then I retry wp-cli; it works!
wp@hetzner3:~$ wp --path=/var/www/html/store.opensourceecology.org/htdocs plugin list PHP Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 Warning: Undefined array key "HTTP_HOST" in /var/www/html/store.opensourceecology.org/htdocs/wp-content/plugins/vcaching/vcaching.php on line 196 +------------------------------------------------+----------+--------+---------+----------------+-------------+ | name | status | update | version | update_version | auto_update | +------------------------------------------------+----------+--------+---------+----------------+-------------+ | akismet | inactive | none | 5.3.3 | | off | | classic-editor | inactive | none | 1.6.5 | | off | | contact-form-7 | active | none | 5.9.8 | | off | | google-authenticator-encourage-user-activation | active | none | 0.2 | | off | | google-authenticator | active | none | 0.54 | | off | | hello | inactive | none | 1.7.1 | | off | | meta-box | active | none | 5.10.2 | | off | | ssl-insecure-content-fixer | active | none | 2.7.2 | | off | | vcaching | active | none | 1.8.3 | | off | | woocommerce | active | none | 9.3.3 | | off | | coingate-for-woocommerce | inactive | none | 2.1.1 | | off | +------------------------------------------------+----------+--------+---------+----------------+-------------+ wp@hetzner3:~$
- unfortunately, I get a blank page when I try to load store.opensourceecology.org in my web browser
- nginx is fine, but the varnish logs show that apache is returning a 403
[Thu Oct 03 04:19:37.076411 2024] [authz_core:error] [pid 3116759:tid 3116768] [client 81.17.16.77:0] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png, referer: https://store.opensourceecology.org/ ==> modsec_audit.log <== --fd8c6d25-A-- [03/Oct/2024:04:19:37.076625 +0000] Zv4bWZVyO5GHCka9cecUKwAAAEE 127.0.0.1 40720 127.0.0.1 8000 --fd8c6d25-B-- GET /wp-includes/images/w-logo-blue-white-bg.png HTTP/1.1 X-Real-IP: 81.17.16.77 X-Forwarded-Proto: https X-Forwarded-Port: 443 Host: store.opensourceecology.org User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Accept: image/avif,image/webp,*/* Accept-Language: en-US,en;q=0.5 Referer: https://store.opensourceecology.org/ Sec-Fetch-Dest: image Sec-Fetch-Mode: no-cors Sec-Fetch-Site: same-origin Sec-GPC: 1 Pragma: no-cache Accept-Encoding: gzip hash: #store.opensourceecology.org X-Varnish: 98343 --fd8c6d25-F-- HTTP/1.1 403 Forbidden X-Frame-Options: SAMEORIGIN Content-Length: 199 Content-Type: text/html; charset=iso-8859-1 --fd8c6d25-E-- <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access this resource.</p> </body></html> --fd8c6d25-H-- Apache-Error: [file "mod_authz_core.c"] [line 879] [level 3] AH01630: client denied by server configuration: /var/www/html/store.opensourceecology.org/htdocs/wp-includes/images/w-logo-blue-white-bg.png Stopwatch: 1727929177076046 856 (- - -) Stopwatch2: 1727929177076046 856; combined=26, p1=24, p2=0, p3=0, p4=0, p5=2, sr=0, sw=0, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.9.7 (http://www.modsecurity.org/). Server: Apache Engine-Mode: "ENABLED" --fd8c6d25-Z--