Maltfield Log/2025 Q2

From Open Source Ecology
Jump to navigation Jump to search

My work log from the second quarter of the year 2025. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

  1. Maltfield_Log
  2. User:Maltfield
  3. Special:Contributions/Maltfield

Sun Apr 11, 2025

  1. let's get Catarina that broken staging site for osemain on hetzner3
  2. Marcin still hasn't regained access to his ssh key (so he can update the ose keepass), but he did finally send me the password to our hetzner account
  3. so now I can order a second IPv4 address, as needed for obi & osemain to have two distinct sites on hetzner3
  4. I logged-into hetzner https://robot.hetzner.com/server
  5. I also typed a "name" into the blank "name" fields for our two servers. one is now called "hetzner2" and the new one "hetzner3"
  6. I clicked on the server for "hetzner3" and the tab "IPs".
    1. Then I clicked on "Order additional IPs / Nets"
    2. I selected "One additional IP with costs (€ 1.70 max. per month / € 0.0027 per hour + € 4.90 once-off setup)"
    3. it required me to enter a reason (IPv4 is scarce) to which I wrote:
we need to run two websites with the same domain name that are already running on our primary IPv4 address, and a client doesn't have IPv6 working at their office
    1. and I clicked "Apply for IP/subnet in obligation"
    2. I got a message; looks like it needs human approval
Your request for additional IPs/subnets was successfully sent. We will send you an email as soon as your IP/subnet is ready.
  1. I typed an email to Marcin and Catarina to notify them of this order
Hey Marcin,

As authorized on our last call, I ordered an additional IPv4 address for your hetzner account.

IPv4 addresses are scarce, and it appears that they need to approve it manually.

The cost is €1.70 per month + € 4.90 once-off setup.

This will allow us to run more than one website with the same domain off the same server. That will be needed for osemain and obi.

Once you finish rebuilding those websites on hetzner3 to use a new not-broken theme, we can cancel this second IP address.


Thank you,

Michael Altfield
https://www.michaelaltfield.net
PGP Fingerprint: 0465 E42F 7120 6785 E972  644C FE1B 8449 4E64 0D41

Note: If you cannot reach me via email, please check to see if I have changed my email address by visiting my website at https://email.michaelaltfield.net
  1. before I finished typing ^ that email, I got an email from hetzner indicating that we have a new IP
  2. I refreshed the hetzner wui, and now I see the new IP
  3. ...
  4. following-up on the bus factor, I added Catarina & Tom's ssh keys to their authorized_keys files on hetzner3
    1. I sent them both emails asking them to confirm access
  5. I also emailed Marcin asking if he installed zulucrypt yet to try to recover his old ssh key
  6. update: within a few hours, Marcin had successfully decrypted and mounted his old veracrypt volume using zuluCrypt
  7. he created this article on the wiki https://wiki.opensourceecology.org/wiki/Zulucrypt
  8. I found that he had previously documented scattered articles about backups, luks, veracrypt, pgp, cybersec general, etc in a ton of different articles. So I spent some time adding categories and "see also" sections to those articles, in hopes he will be more easily able to do this in the future
  9. I also asked him to please document what he needed for himself 5 years from now into a README file next to the 'ose-veracrypt' volume on his usb drive.
  10. Marcin confirmed that he was able to restore his ssh keys and ssh into hetzner3. awesome.
  11. ...
  12. I logged all my hours and sent an invoice to OSE for last month (Mar 2025)
  13. gah, I had obliterated half my 2025Q1 log. when I tried to restore it, I got a 413 error lgo
  14. I checked php and nginx; it's 10M. How did I write >10 MB of text in one quarter?
  15. there's too many layers on this server; I checked the logs
[Fri Apr 11 22:18:20.306872 2025] [:error] [pid 13182] [client 127.0.0.1:56606] [client 127.0.0.1] ModSecurity: Request body no files data length is larger than the configured limit (1000000).. Deny with code (413) [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "Z-mVLLwDarHC@6u2-5xhBgAAAAg"], referer: https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2025_Q1&action=edit
HTTP/1.1 413 Request Entity Too Large
Message: Request body no files data length is larger than the configured limit (1000000).. Deny with code (413)
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client 127.0.0.1] ModSecurity: Request body no files data length is larger than the configured limit (1000000).. Deny with code (413) [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "Z-mVLLwDarHC@6u2-5xhBgAAAAg"]
127.0.0.1 - - [11/Apr/2025:22:18:20 +0000] "POST /index.php?title=Maltfield_Log/2025_Q1&action=submit HTTP/1.0" 413 338 "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2025_Q1&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0"
146.70.199.124 - - [11/Apr/2025:22:18:20 +0000] "POST /index.php?title=Maltfield_Log/2025_Q1&action=submit HTTP/1.1" 413 338 "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2025_Q1&action=edit" "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0" "-"
  1. ok, so it's modsecurity?
  2. gah, that's a lot of files to review
[root@opensourceecology httpd]# find .  |grep -i security
./conf.d/mod_security.wordpress.include
./conf.d/mod_security.conf
./conf.modules.d/10-mod_security.conf
./modsecurity.d
./modsecurity.d/activated_rules
./modsecurity.d/activated_rules/modsecurity_crs_42_tight_security.conf
./modsecurity.d/activated_rules/modsecurity_crs_35_bad_robots.conf
./modsecurity.d/activated_rules/modsecurity_50_outbound.data
./modsecurity.d/activated_rules/modsecurity_crs_45_trojans.conf
./modsecurity.d/activated_rules/modsecurity_crs_48_local_exceptions.conf.example
./modsecurity.d/activated_rules/modsecurity_35_bad_robots.data
./modsecurity.d/activated_rules/modsecurity_crs_23_request_limits.conf
./modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf
./modsecurity.d/activated_rules/modsecurity_crs_49_inbound_blocking.conf
./modsecurity.d/activated_rules/modsecurity_crs_60_correlation.conf
./modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf
./modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf
./modsecurity.d/activated_rules/modsecurity_50_outbound_malware.data
./modsecurity.d/activated_rules/modsecurity_35_scanners.data
./modsecurity.d/activated_rules/modsecurity_40_generic_attacks.data
./modsecurity.d/activated_rules/modsecurity_crs_50_outbound.conf
./modsecurity.d/activated_rules/modsecurity_crs_47_common_exceptions.conf
./modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf
./modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf
./modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf
./modsecurity.d/activated_rules/modsecurity_crs_59_outbound_blocking.conf
./modsecurity.d/modsecurity_crs_10_config.conf.20181024.orig
./modsecurity.d/modsecurity_crs_10_config.conf
./modsecurity.d/do_not_log_passwords.conf
[root@opensourceecology httpd]#
  1. looks like it's SecRequestBodyLimit http://stackoverflow.com/questions/13887812/ddg#14690797
[root@opensourceecology httpd]# grep -irl 'BodyLimit' *
conf.d/mod_security.conf
modules/mod_security2.so
[root@opensourceecology httpd]#
  1. it's 13107200
[root@opensourceecology httpd]# grep -ir 'BodyLimit' *
conf.d/mod_security.conf:    SecRequestBodyLimit 13107200
conf.d/mod_security.conf:    SecRequestBodyLimitAction Reject
Binary file modules/mod_security2.so matches
[root@opensourceecology httpd]#
  1. docs say it's in bytes https://github.com/owasp-modsecurity/ModSecurity/wiki/Reference-Manual-(v2.x)#user-content-SecRequestBodyLimit
  2. so 13107200 / 1024 / 1024 = 12.5 MB.
  3. jesus that's a lot of data; I'm not gonna increase that in 4 places (nginx, apache, mod_security, php); let's just split it into two articles :(
  4. ...
  5. so Marcin is stressing urgancy to get Catarina a sandbox so she can rebuild osemain using some new theme that's not broken on the latest version of wordpress, php, etc on hetzner3
  6. I didn't want to do this site before the other less-priority ones, but it's just a sandbox
  7. I realized I never made a CHG file for osemain
  8. looks like I first did a snapshot Jan 31https://wiki.opensourceecology.org/wiki/Maltfield_Log/2025_Q1#Fri_Jan_31.2C_2025
  9. ugh, I just said I was "following the same guide as with the other sites"
    1. I was hoping to know which one to CHG to copy-from
    2. I guess it makes the most sense to copy from obi, which already has both a static and dynamic site setup (untested)
  10. ok, I made a first draft of our osemain CHG to migrate to hetnzer3 https://wiki.opensourceecology.org/wiki/CHG-2025-XX-XX_migrate_osemain_to_hetzner3
  11. oh, crap, I'm going to remove
Retrieved from "https://wiki.opensourceecology.org/index.php?title=Maltfield_Log/2025_Q2&oldid=306059"