Maltfield Log: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
No edit summary |
||
Line 24: | Line 24: | ||
# discovered that, despite the warnings [https://www.mediawiki.org/wiki/Manual:How_to_debug#Setting_up_a_debug_log_file on the documentation page], our debug logs are in the docroot x_x | # discovered that, despite the warnings [https://www.mediawiki.org/wiki/Manual:How_to_debug#Setting_up_a_debug_log_file on the documentation page], our debug logs are in the docroot x_x | ||
# wiki debug log file was stuck at ~2G & not growing. I moved it aside & touched the file, and new data came screaming in. | # wiki debug log file was stuck at ~2G & not growing. I moved it aside & touched the file, and new data came screaming in. | ||
=Fri Sep 15, 2017= | =Fri Sep 15, 2017= |
Revision as of 18:50, 18 September 2017
Links: OSE_Server
Sat Sep 16, 2017
- stripped-out ini_set requirements
perl -pi -e ' BEGIN{undef $/;} s%(.*)if(.*)function_exists(.*)ini_set[^}]*}%${1}\n/*****************************************************************************\n * disabling ini_set detction\n * it was intentionally disabled for security! -maltfield\n******************************************************************************\nif${2}function_exists${3}ini_set${4}\n*/%smg' /var/www/html/piwik.openbuildinginstitute.org/htdocs/core/testMinimumPhpVersion.php
- fixed permissions
vhostDir="/var/www/html/piwik.openbuildinginstitute.org/" piwikDocroot="${vhostDir}/htdocs" chown -R apache:apache "${vhostDir}" find "${vhostDir}" -type d -exec chmod 0750 {} \; find "${vhostDir}" -type f -exec chmod 0640 {} \; chown apache:apache-admins "${piwikDocroot}/config/config.ini.php" chmod 0440 "${piwikDocroot}/config/config.ini.php"
- began documenting the Piwik install process http://opensourceecology.org/wiki/Piwik
- hit issue when writing Piwik wiki page where the wiki would just timeout; this was not reproducable with my personal page
- discovered that, despite the warnings on the documentation page, our debug logs are in the docroot x_x
- wiki debug log file was stuck at ~2G & not growing. I moved it aside & touched the file, and new data came screaming in.
Fri Sep 15, 2017
- When Chris & I were digging, we found lots of evidence of sec incompetence from Piwik (requiring ini_set & marking requests to fix as "won't fix", getting sec audits & *not* publishing the results). This is usually a no-go, but since there's no better alternative && it's not necessary to make it public, we'll lock it behind a non-standard port && behind htpasswd (in addition to untrustwrothy application-level
- switched piwik.openbuildinginstitue.org back to port 4443 (from 443, which was just done for testing ssl hardning with qualy's ssllabs)
- created htpasswd file for obi's piwik
cd /var/www/html/piwik.openbuildinginstitute.org htpasswd -c -B .htpasswd admin chown apache:apache .htpasswd chmod 0400 .htpasswd
- removed the line "Require all granted" && changed AllowOverride to None && added basic auth config to /etc/httpd/conf.d/piwik.openbuildinginstitute.org
# we don't trust piwik due to obvious sec incompetence in the project, so we # restrict the whole site with basic auth *in addition* to the untrusted # applicaiton-level auth AuthType Basic AuthName "Authentication Required" AuthUserFile /var/www/html/piwik.openbuildinginstitute.org/.htpasswd Require valid-user
- unzipped latest pwik files to /var/www/html/piwik.openbuildinginstitute.org/htdocs/
- added '/var/www/html/piwik.openbuildinginstitute.org/' to php.ini basedir
Wed Sep 13, 2017
- Catarina had issues authing with 2FA on obi3
- disabled obi3 & sent her updated instructions to validate the accuracy of her phone's cock & to enable "relaxed mode" which will accept codes +/- 4 min from the current time
Tue Sep 12, 2017
- Catarina finished validation of content on obi3
- discovered that EDH isn't listed in our cipher suite, and therefore the quest to validate the dhparams is moot. We use ECDH.
- "We have three recommendations for correctly deploying Diffie-Hellman for TLS: ... 2. Deploy (Ephemeral) Elliptic-Curve Diffie-Hellman (ECDHE). Elliptic-Curve Diffie-Hellman (ECDH) key exchange avoids all known feasible cryptanalytic attacks, and modern web browsers now prefer ECDHE over the original, finite field, Diffie-Hellman. The discrete log algorithms we used to attack standard Diffie-Hellman groups do not gain as strong of an advantage from precomputation, and individual servers do not need to generate unique elliptic curves." -source: https://weakdh.org/sysadmin.html
- if we have client issues that force us to enable EDH, then I will further investigate this issue
Mon Sep 11, 2017
- emailed with marcin for d3d
- emailed with Catarina for obi3 vaidation update
- created DH key & added to apache config
mkdir -p /etc/pki/dhparam chown apache:apache /etc/pki/dhparam chmod 0500 /etc/pki/dhparam cd /etc/pki/dhparam openssl dhparam -out dhparam.pem 4096 chown apache:apache /etc/pki/dhparam/dhparam.pem chmod 0400 /etc/pki/dhparam/dhparam.pem
- attempted to add the dh key to apache config in /etc/httpd/conf.d/ssl.conf & /etc/httpd/conf.d/ssl.openbuildinginstitute.org
- discovered that the "SSLOpenSSLConfCmd" directive wasn't added until apache v2.4.7, but cent7 is pinned to 2.4.6. The solution is to append the dhparams file to the certificate, which should be done with the letsencrypt cron renewal
cat /etc/pki/dhparam/dhparam.pem >> /etc/letsencrypt/live/openbuildinginstitute.org/cert.pem cat /etc/pki/dhparam/dhparam.pem >> /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem
- could not validate the config; created StackOverflow question https://stackoverflow.com/questions/46164547/how-to-validate-dhparams-in-apache-2-4-6
Mon Sep 04, 2017
- generated obi lets encrypt certificate; these certbot args *might* need to change after we redirect all http -> https
yum install python-certbot-apache certbot certonly -v --webroot -w /var/www/html/obi/htdocs/ -d openbuildinginstitute.org -d www.openbuildinginstitute.org -d piwik.openbuildinginstitute.org chmod 0400 /etc/letsencrypt/archive/*/pri*
- a cron job should be added to run `certbot renew` before the current cert expires in 4 months on 2017-12-03
- apache https hardening
- first tested default configs with the qualys ssllabs.com test. got "C" due to bad ciphers, lack of PFS support, POODLE & RC4 attack vuln, & incomplete cert chain.
- used mozilla's ssl-config-generator with "Apache v2.4.6" + "Modern" + "Openssl v1.0.1e" https://mozilla.github.io/server-side-tls/ssl-config-generator/
<VirtualHost *:443> ... SSLEngine on SSLCertificateFile /path/to/signed_certificate SSLCertificateChainFile /path/to/intermediate_certificate SSLCertificateKeyFile /path/to/private/key # Uncomment the following directive when using client certificate authentication #SSLCACertificateFile /path/to/ca_certs_for_client_authentication # HSTS (mod_headers is required) (15768000 seconds = 6 months) Header always set Strict-Transport-Security "max-age=15768000" ... </VirtualHost> # modern configuration, tweak to your needs SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 SSLHonorCipherOrder on SSLCompression off # OCSP Stapling, only in httpd 2.3.3 and later SSLUseStapling on SSLStaplingResponderTimeout 5 SSLStaplingReturnResponderErrors off SSLStaplingCache shmcb:/var/run/ocsp(128000)
- determined SPKI public key hashes to use in HPKP per best practices guide https://community.letsencrypt.org/t/hpkp-best-practices-if-you-choose-to-implement/4625
- determined SPKI public key hashes for alternatie CA StartCom (which also issues free certs)
- decided *not* to pin SPKI public key hashes for alternate free CA CACert, which uses an md5 signature on their root ca, and has failed to fix it since this bug was issued in 2014 http://bugs.cacert.org/view.php?id=1305
- decided *not* to pin key hashes for StartCom, or wosign since it was detrusted by google, apple, & mozilla https://security.googleblog.com/2016/10/distrusting-wosign-and-startcom.html
- decided to pin key hashes for root certs provided by trusted CAs with free 90-day certs or free certs = cloudflare (free) and comodo & ssl.com (both 90 days free)
- cloudflare itself requires pinning digicert, addtrust, globalsign, and gtecybertrust (now digicert)
- comodo's root cert documentation is a clusterfuck that requires javascript & a gui, and there's a ton of them. decided to abandon this; we already picked-up a few CAs just from cloudflare
[root@hetzner2 ~]# cd /etc/letsencrypt/live/openbuildinginstitute.org/ [root@hetzner2 openbuildinginstitute.org]# openssl x509 -in cert.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c= [root@hetzner2 openbuildinginstitute.org]# openssl x509 -in chain.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg= [root@hetzner2 ~]# mkdir /var/tmp/letsencrypt [root@hetzner2 ~]# cd /var/tmp/letsencrypt/ [root@hetzner2 letsencrypt]# wget --quiet https://letsencrypt.org/certs/isrgrootx1.pem [root@hetzner2 letsencrypt]# cat isrgrootx1.pem -----BEGIN CERTIFICATE----- MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4 WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+ 0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ 3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5 ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq 4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc= -----END CERTIFICATE----- [root@hetzner2 letsencrypt]# openssl x509 -in isrgrootx1.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M= # copied & pasted IdenTrust's DST Root CA X3 Certificate from here https://www.identrust.com/certificates/trustid/root-download-x3.html # added '-----BEGIN CERTIFICATE-----' to top of file and # added '-----END CERTIFICATE-----' to end of file # per https://www.c-rieger.de/http-public-key-pinning-nginx-and-nextcloud/ [root@hetzner2 tmp]# vim /var/tmp/identrust.dst.root.x3.pem [root@hetzner2 tmp]# cat /var/tmp/identrust.dst.root.x3.pem -----BEGIN CERTIFICATE----- MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw 7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69 ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5 JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ -----END CERTIFICATE----- [root@hetzner2 tmp]# openssl x509 -in /var/tmp/identrust.dst.root.x3.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys= [root@hetzner2 tmp]# ############################################## # pin cloudflare root CA certs if Let's Encrypt disappears one day # ############################################## # https://support.cloudflare.com/hc/en-us/articles/115001186052-What-intermediates-and-roots-are-Cloudflare-issued-certs-signed-against- [root@hetzner2 openbuildinginstitute.org]# mkdir /var/tmp/cloudflare [root@hetzner2 openbuildinginstitute.org]# cd /var/tmp/cloudflare # CloudFlare Universal SSL Certs are signed by Comodo or GlobalSign # https://www.crt.sh/?d=1 # https://www.crt.sh/?d=88 [root@hetzner2 cloudflare]# wget --quiet -O addtrust.pem https://www.crt.sh/?d=1 [root@hetzner2 cloudflare]# cat addtrust.pem -----BEGIN CERTIFICATE----- MIIENjCCAx6gAwIBAgIBATANBgkqhkiG9w0BAQUFADBvMQswCQYDVQQGEwJTRTEU MBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFs IFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290 MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowbzELMAkGA1UEBhMCU0Ux FDASBgNVBAoTC0FkZFRydXN0IEFCMSYwJAYDVQQLEx1BZGRUcnVzdCBFeHRlcm5h bCBUVFAgTmV0d29yazEiMCAGA1UEAxMZQWRkVHJ1c3QgRXh0ZXJuYWwgQ0EgUm9v dDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALf3GjPm8gAELTngTlvt H7xsD821+iO2zt6bETOXpClMfZOfvUq8k+0DGuOPz+VtUFrWlymUWoCwSXrbLpX9 uMq/NzgtHj6RQa1wVsfwTz/oMp50ysiQVOnGXw94nZpAPA6sYapeFI+eh6FqUNzX mk6vBbOmcZSccbNQYArHE504B4YCqOmoaSYYkKtMsE8jqzpPhNjfzp/haW+710LX a0Tkx63ubUFfclpxCDezeWWkWaCUN/cALw3CknLa0Dhy2xSoRcRdKn23tNbE7qzN E0S3ySvdQwAl+mG5aWpYIxG3pzOPVnVZ9c0p10a3CitlttNCbxWyuHv77+ldU9U0 WicCAwEAAaOB3DCB2TAdBgNVHQ4EFgQUrb2YejS0Jvf6xCZU7wO94CTLVBowCwYD VR0PBAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wgZkGA1UdIwSBkTCBjoAUrb2YejS0 Jvf6xCZU7wO94CTLVBqhc6RxMG8xCzAJBgNVBAYTAlNFMRQwEgYDVQQKEwtBZGRU cnVzdCBBQjEmMCQGA1UECxMdQWRkVHJ1c3QgRXh0ZXJuYWwgVFRQIE5ldHdvcmsx IjAgBgNVBAMTGUFkZFRydXN0IEV4dGVybmFsIENBIFJvb3SCAQEwDQYJKoZIhvcN AQEFBQADggEBALCb4IUlwtYj4g+WBpKdQZic2YR5gdkeWxQHIzZlj7DYd7usQWxH YINRsPkyPef89iYTx4AWpb9a/IfPeHmJIZriTAcKhjW88t5RxNKWt9x+Tu5w/Rw5 6wwCURQtjr0W4MHfRnXnJK3s9EK0hZNwEGe6nQY1ShjTK3rMUUKhemPR5ruhxSvC Nr4TDea9Y355e6cJDUCrat2PisP29owaQgVR1EX1n6diIWgVIEM8med8vSTYqZEX c4g/VhsxOBi0cQ+azcgOno4uG+GMmIPLHzHxREzGBHNJdmAPx/i9F4BrLunMTA5a mnkPIAou1Z5jJh5VkpTYghdae9C8x49OhgQ= -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/addtrust.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU= [root@hetzner2 cloudflare]# wget --quiet -O globalsign.pem https://www.crt.sh/?d=88 [root@hetzner2 cloudflare]# cat globalsign.pem -----BEGIN CERTIFICATE----- MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkG A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw05ODA5MDExMjAw MDBaFw0yODAxMjgxMjAwMDBaMFcxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i YWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT aWduIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDaDuaZ jc6j40+Kfvvxi4Mla+pIH/EqsLmVEQS98GPR4mdmzxzdzxtIK+6NiY6arymAZavp xy0Sy6scTHAHoT0KMM0VjU/43dSMUBUc71DuxC73/OlS8pF94G3VNTCOXkNz8kHp 1Wrjsok6Vjk4bwY8iGlbKk3Fp1S4bInMm/k8yuX9ifUSPJJ4ltbcdG6TRGHRjcdG snUOhugZitVtbNV4FpWi6cgKOOvyJBNPc1STE4U6G7weNLWLBYy5d4ux2x8gkasJ U26Qzns3dLlwR5EiUWMWea6xrkEmCMgZK9FGqkjWZCrXgzT/LCrBbBlDSgeF59N8 9iFo7+ryUp9/k5DPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E BTADAQH/MB0GA1UdDgQWBBRge2YaRQ2XyolQL30EzTSo//z9SzANBgkqhkiG9w0B AQUFAAOCAQEA1nPnfE920I2/7LqivjTFKDK1fPxsnCwrvQmeU79rXqoRSLblCKOz yj1hTdNGCbM+w6DjY1Ub8rrvrTnhQ7k4o+YviiY776BQVvnGCv04zcQLcFGUl5gE 38NflNUVyRRBnMRddWQVDf9VMOyGj/8N7yy5Y0b2qvzfvGn9LhJIZJrglfCm7ymP AbEVtQwdpf5pLGkkeB6zpxxxYu7KyJesF12KwvhHhm4qxFYxldBniYUr+WymXUad DKqC5JlR3XC321Y9YeRq4VzW9v493kHMB65jUr9TU/Qr6cf9tveCX4XSQRjbgbME HMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A== -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/globalsign.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q= # CloudFlare Dedicated Certificates are signed by DigiCert # https://www.crt.sh/?d=76 [root@hetzner2 cloudflare]# wget --quiet -O digicert.pem https://www.crt.sh/?d=76 [root@hetzner2 cloudflare]# cat digicert.pem -----BEGIN CERTIFICATE----- MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1 BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3 DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92 9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0 Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/digicert.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o= # GTE CyberTrust was acquired by DigiCert [root@hetzner2 cloudflare]# wget --quiet -O gtecybertrust.pem https://www.crt.sh/?d=10 [root@hetzner2 cloudflare]# cat gtecybertrust.pem -----BEGIN CERTIFICATE----- MIICWjCCAcMCAgGlMA0GCSqGSIb3DQEBBAUAMHUxCzAJBgNVBAYTAlVTMRgwFgYD VQQKEw9HVEUgQ29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNv bHV0aW9ucywgSW5jLjEjMCEGA1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJv b3QwHhcNOTgwODEzMDAyOTAwWhcNMTgwODEzMjM1OTAwWjB1MQswCQYDVQQGEwJV UzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQLEx5HVEUgQ3liZXJU cnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0IEds b2JhbCBSb290MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCVD6C28FCc6HrH iM3dFw4usJTQGz0O9pTAipTHBsiQl8i4ZBp6fmw8U+E3KHNgf7KXUwefU/ltWJTS r41tiGeA5u2ylc9yMcqlHHK6XALnZELn+aks1joNrI1CqiQBOeacPwGFVw1Yh0X4 04Wqk2kmhXBIgD8SFcd5tB8FLztimQIDAQABMA0GCSqGSIb3DQEBBAUAA4GBAG3r GwnpXtlR22ciYaQqPEh346B8pt5zohQDhT37qw4wxYMWM4ETCJ57NE7fQMh017l9 3PR2VX2bY1QY6fDq81yx2YtCHrnAlU66+tXifPVoYb+O7AWXX1uw16OFNMQkpw0P lZPvy5TYnh+dXIVtx6quTx8itc2VrbqnzPmrC3p/ -----END CERTIFICATE----- [root@hetzner2 cloudflare]# openssl x509 -in /var/tmp/cloudflare/gtecybertrust.pem -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 EGn6R6CqT4z3ERscrqNl7q7RC//zJmDe9uBhS/rnCHU= ############################################# # pin ssl.com root CA certs if Let's Encrypt disappears one day # ############################################# # https://www.ssl.com/repository/ [root@hetzner2 cloudflare]# mkdir /var/tmp/ssl.com [root@hetzner2 cloudflare]# cd /var/tmp/ssl.com [root@hetzner2 ssl.com]# wget --quiet https://www.ssl.com/repository/SSLcom-RootCA.zip [root@hetzner2 ssl.com]# unzip SSLcom-RootCA.zip Archive: SSLcom-RootCA.zip inflating: SSLcom-RootCA-EV-RSA-4096-R2.pem inflating: SSLcomEVRootCertificationAuthorityECC.pem inflating: SSLcomRootCertificationAuthorityECC.pem inflating: SSLcomRootCertificationAuthorityRSA.pem [root@hetzner2 ssl.com]# cat SSLcom-RootCA-EV-RSA-4096-R2.pem Subject: CN=SSL.com EV Root Certification Authority RSA R2,O=SSL Corporation,L=Houston,ST=Texas,C=US Issuer: CN=SSL.com EV Root Certification Authority RSA R2,O=SSL Corporation,L=Houston,ST=Texas,C=US -----BEGIN CERTIFICATE----- MIIF6zCCA9OgAwIBAgIIVrYpzTS8ePYwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNV BAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEQMA4GA1UEBwwHSG91c3RvbjEYMBYGA1UE CgwPU1NMIENvcnBvcmF0aW9uMTcwNQYDVQQDDC5TU0wuY29tIEVWIFJvb3QgQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkgUlNBIFIyMB4XDTE3MDUzMTE4MTQzN1oXDTQy MDUzMDE4MTQzN1owgYIxCzAJBgNVBAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEQMA4G A1UEBwwHSG91c3RvbjEYMBYGA1UECgwPU1NMIENvcnBvcmF0aW9uMTcwNQYDVQQD DC5TU0wuY29tIEVWIFJvb3QgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkgUlNBIFIy MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAjzZlQOHWTcDXtOlG2mvq M0fNTPl9fb69LT3w23jhhqXZuglXaO1XPqDQCEGD5yhBJB/jchXQARr7XnAjssuf OePPxU7Gkm0mxnu7s9onnQqG6YE3Bf7wcXHswxzpY6IXFJ3vG2fThVUCAtZJycxa 4bH3bzKfydQ7iEGonL3Lq9ttewkfokxykNorCPzPPFTOZw+oz12WGQvE43LrrdF9 HSfvkusQv1vrO6/PgN3B0pYEW3p+pKk8OHakYo6gOV7qd89dAFmPZiw+B6KjBSYR aZfqhbcPlgtLyEDhULouisv3D5oi53+aNxPN8k0TayHRwMwi8qFG9kRpnMphNQcA b9ZhCBHqurj26bNg5U257J8UZslXWNvNh2n4ioYSA0e/ZhN2rHd9NCSFg83XqpyQ Gp8hLH94t2S42Oim9HizVcuE0jLEeK6jj2HdzghTreyI/BXkmg3mnxp3zkyPuBQV PWKchjgGAGYS5Fl2WlPAApiiECtoRHuOec4zSnaqW4EWG7WK2NAAe15itAnWhmMO pgWVSbooi4iTsjQc2KRVbrcc0N6ZVTsj9CLg+SlmJuwgUHfbSguPvuUCYHBBXtSu UDkiFCbLsjtzdFVHB3mBOagwE0TlBIqulhMlQg+5U8Sb/M3kHN48+qvWBkofZ6aY MBzdLNvcGJVXZsb/XItW9XcCAwEAAaNjMGEwDwYDVR0TAQH/BAUwAwEB/zAfBgNV HSMEGDAWgBT5YLvU49U09rj1BoAlp3PbRmmonjAdBgNVHQ4EFgQU+WC71OPVNPa4 9QaAJadz20ZpqJ4wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBW s47LCp1Jjr+kxJG7ZhcFUZh1++VQLHqe8RT6q9OKPv+RKY9ji9i0qVQBDb6Thi/5 Sm3HXvVX+cpVHBK+Rw82xd9qt9t1wkclf7nxY/hoLVUE0fKNsKTPvDxeH3jnpaAg cLAExbf3cqfeIg29MyVGjGSSJuM+LmOW2puMPfgYCdcDzH2GguDKBAdRUNf/ktUM 79qGn5nX67evaOI5JpS6aLe/g9Pqemc9YmeuJeVy6OLk7K4S9ksrPJ/psEDzOFSz /bdoyNrGj1E8svuR3Bznm53htw1yj+KkxKl4+esUrMZDBcJlOSgYAsOCsp0FvmXt ll9ldDz7CTUue5wT/RsPXcdtgTpWD8w74a8CLyKsRspGPKAcTNZEtF4uXBVmCeEm Kf7GUmG6sXP/wwyc5WxqlD8UykAWlYTzWamsX0xhk23RO8yilQwipmdnRC652dKK QbNmC1r7fSOl8hqw/96bg5Qu0T/fkreRrwU7ZcegbLHNYhLDkBvjJc40vG93drEQ w/cFGsDWr3RiSBd3kmmQYRzelYB0VI8YHMPzA9C/pEN1hlMYegouCRw2n5H9gooi S9EOUCXdywMMF8mDAAhONU2Ki+3wApRmLER/y5UnlhetCTCstnEXbosX9hwJ1C07 mKVx01QT2WDz9UtmT/rx7iASjbSsV7FFY6GsdqnC+w== -----END CERTIFICATE----- [root@hetzner2 ssl.com]# cat SSLcomEVRootCertificationAuthorityECC.pem -----BEGIN CERTIFICATE----- MIIClDCCAhqgAwIBAgIILCmcWxbtBZUwCgYIKoZIzj0EAwIwfzELMAkGA1UEBhMC VVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQKDA9T U0wgQ29ycG9yYXRpb24xNDAyBgNVBAMMK1NTTC5jb20gRVYgUm9vdCBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eSBFQ0MwHhcNMTYwMjEyMTgxNTIzWhcNNDEwMjEyMTgx NTIzWjB/MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjE0MDIGA1UEAwwrU1NMLmNv bSBFViBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IEVDQzB2MBAGByqGSM49 AgEGBSuBBAAiA2IABKoSR5CYG/vvw0AHgyBO8TCCogbR8pKGYfL2IWjKAMTH6kMA VIbc/R/fALhBYlzccBYy3h+Z1MzFB8gIH2EWB1E9fVwHU+M1OIzfzZ/ZLg1Kthku WnBaBu2+8KGwytAJKaNjMGEwHQYDVR0OBBYEFFvKXuXe0oGqzagtZFG22XKbl+ZP MA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUW8pe5d7SgarNqC1kUbbZcpuX 5k8wDgYDVR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA2gAMGUCMQCK5kCJN+vp1RPZ ytRrJPOwPYdGWBrssd9v+1a6cGvHOMzosYxPD/fxZ3YOg9AeUY8CMD32IygmTMZg h5Mmm7I1HrrW9zzRHM76JTymGoEVW/MSD2zuZYrJh6j5B+BimoxcSg== -----END CERTIFICATE----- [root@hetzner2 ssl.com]# cat SSLcomRootCertificationAuthorityECC.pem -----BEGIN CERTIFICATE----- MIICjTCCAhSgAwIBAgIIdebfy8FoW6gwCgYIKoZIzj0EAwIwfDELMAkGA1UEBhMC VVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQKDA9T U0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZpY2F0 aW9uIEF1dGhvcml0eSBFQ0MwHhcNMTYwMjEyMTgxNDAzWhcNNDEwMjEyMTgxNDAz WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hvdXN0 b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNvbSBS b290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IEVDQzB2MBAGByqGSM49AgEGBSuB BAAiA2IABEVuqVDEpiM2nl8ojRfLliJkP9x6jh3MCLOicSS6jkm5BBtHllirLZXI 7Z4INcgn64mMU1jrYor+8FsPazFSY0E7ic3s7LaNGdM0B9y7xgZ/wkWV7Mt/qCPg CemB+vNH06NjMGEwHQYDVR0OBBYEFILRhXMw5zUE044CkvvlpNHEIejNMA8GA1Ud EwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUgtGFczDnNQTTjgKS++Wk0cQh6M0wDgYD VR0PAQH/BAQDAgGGMAoGCCqGSM49BAMCA2cAMGQCMG/n61kRpGDPYbCWe+0F+S8T kdzt5fxQaxFGRrMcIQBiu77D5+jNB5n5DQtdcj7EqgIwH7y6C+IwJPt8bYBVCpk+ gA0z5Wajs6O7pdWLjwkspl1+4vAHCGht0nxpbl/f5Wpl -----END CERTIFICATE----- [root@hetzner2 ssl.com]# cat SSLcomRootCertificationAuthorityRSA.pem -----BEGIN CERTIFICATE----- MIIF3TCCA8WgAwIBAgIIeyyb0xaAMpkwDQYJKoZIhvcNAQELBQAwfDELMAkGA1UE BhMCVVMxDjAMBgNVBAgMBVRleGFzMRAwDgYDVQQHDAdIb3VzdG9uMRgwFgYDVQQK DA9TU0wgQ29ycG9yYXRpb24xMTAvBgNVBAMMKFNTTC5jb20gUm9vdCBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eSBSU0EwHhcNMTYwMjEyMTczOTM5WhcNNDEwMjEyMTcz OTM5WjB8MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxEDAOBgNVBAcMB0hv dXN0b24xGDAWBgNVBAoMD1NTTCBDb3Jwb3JhdGlvbjExMC8GA1UEAwwoU1NMLmNv bSBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IFJTQTCCAiIwDQYJKoZIhvcN AQEBBQADggIPADCCAgoCggIBAPkP3aMrfcvQKv7sZ4Wm5y4bunfh4/WvpOz6Sl2R xFdHaxh3a3by/ZPkPQ/CFp4LZsNWlJ4Xg4XOVu/yFv0AYvUiCVToZRdOQbngT0aX qhvIuG5iXmmxX9sqAn78bMrzQdjt0Oj8P2FI7bADFB0QDksZ4LtO7IZl/zbzXmcC C52GVWH9ejjt/uIZALdvoVBidXQ8oPrIJZK0bnoix/geoeOy3ZExqysdBP+lSgQ3 6YWkMyv94tZVNHwZpEpox7Ko07fKoZOI68GXvIz5HdkihCR0xwQ9aqkpk8zruFvh /l8lqjRYyMEjVJ0bmBHDOJx+PYZspQ9AhnwC9FwCTyjLrnGfDzrIM/4RJTXq/LrF YD3ZfBjVsqnTdXgDciLKOsMf7yzlLqn6niy2UUb9rwPW6mBo6oUWNmuF6R7As93E JNyAKoFBbZQ+yODJgUEAnl6/f8UImKIYLEJAs/lvOCdLToD0PYFH4Ih86hzOtXVc US4cK38acijnALXRdMbX5J+tB5O2UzU1/Dfkw/ZdFr4hc96SCvigY2q8lpJqPvi8 ZVWb3vUNiSYE/CUapiVpy8JtynziWV+XrOvvLsi81xtZPCvM8hnIk2snYxnP/Okm +Mpxm3+T/jRnhE6Z6/yzeAkzcLpmpnbtG3PrGqUNxCITIJRWCk4sbE6x/c+cCbqi M+2HAgMBAAGjYzBhMB0GA1UdDgQWBBTdBAkHovV6fVJTEpKV7jiAJQ2mWTAPBgNV HRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFN0ECQei9Xp9UlMSkpXuOIAlDaZZMA4G A1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAIBgRlCn7Jp0cHh5wYfGV cpNxJK1ok1iOMq8bs3AD/CUrdIWQPXhq9LmLpZc7tRiRux6n+UBbkflVma8eEdBc Hadm47GUBwwyOabqG7B52B2ccETjit3E+ZUfijhDPwGFpUenPUayvOUiaPd7nNgs PgohyC0zrL/FgZkxdMF1ccW+sfAjRfSda/wZY52jvATGGAslu1OJD7OAUN5F7kR/ q5R4ZJjT9ijdh9hwZXT7DrkT66cPYakylszeu+1jTBi7qUD3oFRuIIhxdRjqerQ0 cuAjJ3dctpDqhiVAq+8zD8ufgr6iIPv2tS0a5sKFsXQP+8hlAqRSAUfdSSLBv9jr a6x+3uxjMxW3IwiPxg+NQVrdjsW5j+VFP3jbutIbQLH+cU0/4IGiul607BXgk90I H37hVZkLId6Tngr75qNJvTYw/ud3sqB1l7UtgYgXZSD32pAAn8lSzDLKNXz1PQ/Y K9f1JmzJBjSWFupwWRoyeXkLtoh/D1JIPb9s2KJELtFOt3JY04kTlf5Eq/jXixtu nLwsoFvVagCvXzfh1foQC5ichucmj87w7G6KVwuA406ywKBjYZC6VWg3dGq2ktuf oYYitmUnDuy2n0Jg5GfCtdpBC8TTi2EbvPofkSvXRAdeuims2cXp71NIWuuA8ShY Ic2wBlX7Jz9TkHCpBB5XJ7k= -----END CERTIFICATE----- [root@hetzner2 ssl.com]# for file in $(ls *.pem); do echo /var/tmp/ssl.com/$file; openssl x509 -in /var/tmp/ssl.com/$file -pubkey | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64; done /var/tmp/ssl.com/SSLcomEVRootCertificationAuthorityECC.pem NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ= /var/tmp/ssl.com/SSLcom-RootCA-EV-RSA-4096-R2.pem fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A= /var/tmp/ssl.com/SSLcomRootCertificationAuthorityECC.pem oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo= /var/tmp/ssl.com/SSLcomRootCertificationAuthorityRSA.pem 0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=
- created backup CSRs per HPKP best practices
[root@hetzner2 ~]# mkdir /etc/pki/tls/hpkpBackupKeys [root@hetzner2 ~]# chown root:root /etc/pki/tls/hpkpBackupKeys [root@hetzner2 ~]# chmod 0700 /etc/pki/tls/hpkpBackupKeys/ [root@hetzner2 ~]# cd /etc/pki/tls/hpkpBackupKeys/ [root@hetzner2 hpkpBackupKeys]# openssl genrsa -out first.key 4096 Generating RSA private key, 4096 bit long modulus .................................................................++ ..............................................................................................................................................................................................................................++ e is 65537 (0x10001) [root@hetzner2 hpkpBackupKeys]# openssl req -new -key first.key -sha256 -out first.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []: Locality Name (eg, city) [Default City]: Organization Name (eg, company) [Default Company Ltd]: Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []: Email Address []: Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: [root@hetzner2 hpkpBackupKeys]# openssl genrsa -out second.key 4096 Generating RSA private key, 4096 bit long modulus ...............................................................................................................++ ...............................................................................................................................................++ e is 65537 (0x10001) [root@hetzner2 hpkpBackupKeys]# openssl req -new -key second.key -sha256 -out second.csr You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []:Missouri Locality Name (eg, city) [Default City]:Maysville Organization Name (eg, company) [Default Company Ltd]:Open Source Ecology Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []:opensourceecology.org Email Address []:marcin@opensourceecology.org Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: [root@hetzner2 hpkpBackupKeys]# chown -R root:root /etc/pki/tls/hpkpBackupKeys [root@hetzner2 hpkpBackupKeys]# chmod 0400 /etc/pki/tls/hpkpBackupKeys/* [root@hetzner2 hpkpBackupKeys]# openssl req -pubkey < first.csr | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA= [root@hetzner2 hpkpBackupKeys]# openssl req -pubkey < second.csr | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64 OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=
- now, taking all the SPKI public key hashes from above, we have the following line to add to our apache config:
Header set Public-Key-Pins "pin-sha256=\"changeme\"; pin-sha256=\"changeme=\"; pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=43200; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\""
- created /etc/httpd/conf.d/ssl.openbuildinginstitute.org to be included in all vhosts using the *.openbuildinginstitute.org certificate
<VirtualHost piwik.openbuildinginstitute.org:443> ServerName piwik.openbuildinginstitute.org ServerAlias piwik.openbuildinginstitute.org DocumentRoot "/var/www/html/piwik.openbuildinginstitute.org/htdocs" Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org </VirtualHost>
- updated <VirtualHost> block to include file above in /etc/httpd/conf.d/piwik.openbuildinginstitute.org
[root@hetzner2 conf.d]# cat ssl.openbuildinginstitute.org ################################################################################ # Purpose: To be included inside the <VirtualHost> block for all # *.openbuildinginstitute.org sites # # This has been hardened by Michael Altfield in 2017-09. # # For updating, I recommend: # * https://mozilla.github.io/server-side-tls/ssl-config-generator/ # # Followed by testing with ssllabs: # * https://www.ssllabs.com/ssltest ################################################################################ SSLEngine on SSLCertificateFile /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem SSLCertificateChainFile /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 SSLHonorCipherOrder on # HSTS so a browser will refuse a downgrade attack after their first visit # TODO: increase this back to 6-months #Header always set Strict-Transport-Security "max-age=15768000" Header always set Strict-Transport-Security "max-age=15552001" # TODO: increase this back to 1 month #Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=43200; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=3600; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" # enabling compression can be a security risk - see "CRIME" attack SSLCompression off
- updated /etc/httpd/conf.d/ssl.conf with general settings, though I'm not sure this ever actually gets usedu
[root@hetzner2 conf.d]# diff ssl.conf.orig ssl.conf 0a1,10 > ################################################################################ > # This has been hardened by Michael Altfield in 2017-09. > # > # For updating, I recommend: > # * https://mozilla.github.io/server-side-tls/ssl-config-generator/ > # > # Followed by testing with ssllabs: > # * https://www.ssllabs.com/ssltest > ################################################################################ > 75c85,88 < SSLProtocol all -SSLv2 --- > #SSLProtocol all -SSLv2 > # moved outside VirtualHost block (see below) > #SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 > SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 80c93,95 < SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA --- > #SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SEED:!IDEA > # moved outside VirtualHost block (see below) > #SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 93d107 < #SSLHonorCipherOrder on 100c114,117 < SSLCertificateFile /etc/pki/tls/certs/localhost.crt --- > #SSLCertificateFile /etc/pki/tls/certs/localhost.crt > > #SSLCertificateFile /etc/letsencrypt/live/opensourceecology.org/fullchain.pem > SSLCertificateFile /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem 107c124,127 < SSLCertificateKeyFile /etc/pki/tls/private/localhost.key --- > #SSLCertificateKeyFile /etc/pki/tls/private/localhost.key > > #SSLCertificateKeyFile /etc/letsencrypt/live/opensourceecology.org/privkey.pem > SSLCertificateKeyFile /etc/letsencrypt/live/openbuildinginstitute.org/privkey.pem 216,217c236,263 < </VirtualHost> < --- > ################################### > # other custom options -maltfield # > ################################### > > SSLProtocol -ALL -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2 > SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 > > SSLHonorCipherOrder on > > # HSTS so a browser will refuse a downgrade attack after their first visit > # TODO: increase this back to 6-months > #Header always set Strict-Transport-Security "max-age=15768000" > Header always set Strict-Transport-Security "max-age=15552001" > > # TODO: increase this back to 1 month > #Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=43200; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" > Header set Public-Key-Pins "pin-sha256=\"UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c=\"; pin-sha256=\"YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg=\"; pin-sha256=\"C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=\"; pin-sha256=\"Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys=\"; pin-sha256=\"lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU=\"; pin-sha256=\"K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q=\"; pin-sha256=\"Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o=\"; pin-sha256=\"EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU=\"; pin-sha256=\"NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ=\"; pin-sha256=\"fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A=\"; pin-sha256=\"oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo=\"; pin-sha256=\"0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo=\"; pin-sha256=\"MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA=\"; pin-sha256=\"OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU=\"; max-age=3600; includeSubDomains; report-uri=\"http:opensourceecology.org/hpkp-report\"" > > </VirtualHost> > > # enabling compression can be a security risk - see "CRIME" attack > SSLCompression off > > # OCSP Stapling > SSLUseStapling on > SSLStaplingResponderTimeout 5 > SSLStaplingReturnResponderErrors off > SSLStaplingCache shmcb:/var/run/ocsp(128000)
- Successfully got "A+" rating from ssllabs.com !!!
- added "https" documentation to OSE_Server
- further documented today's hpkp solution on my blog https://tech.michaelaltfield.net/2017/09/05/hpkp-best-practices-lets-encrypt/
Sat Sep 02, 2017
- created letsencrypt@opensourceecoloy.org & added to keepass
- follow-up with Chris & assigned jitsi videobridge task of documenting install commands on cent7 local vm with due date of 2017-10-01
Fri Sep 01, 2017
- created obi3 for updating obi without the theme updates..for now
- reviewed all changelogs for oshine theme & sent email to Catarina on issues of not updating the theme
- updated wordpress wiki article with wp-cli updates
- installed google-authenticator & google-authenticator-encourage-user-activation
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator --activate sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator-encourage-user-activation --activate
- activated google-authenticator plugin in wp wui
- changed default name to be obi-specific
defaultOtpAccountDescription="`basename $newVhostDir` wp" cd $newVhostDir/htdocs/wp-content/plugins/google-authenticator #sed -i "s/WordPressBlog/$defaultOtpAccountDescription/g" lang/google-authenticator.pot sed -i "s^\$GA_description\s=\s(\s[\"'].*[\"']^\$GA_description = ( '$defaultOtpAccountDescription'^" google-authenticator.php
- required 2FA for all users
- went to "Settings" -> "General" -> "Google Authenticator - Encourage User Activation"
- selected "Force" option
- Clicked "Save Changes"
- confirmed that if a user looses their 2FA config on their phone, an Admin can "reset" their account to let them in again
- The admin needs to login, go to "Users" -> "All Users"
- click "Edit" under the locked-out user
- uncheck the "Activate" checkbox under "Google Authenticator Settings"
- ...then the user will be able to login again, but will still be required to setup 2FA before they can do anything (locked to Subscriber) per the plugin = google-authenticator-encourage-user-activation
- installed & activated 'rename-wp-login' plugin
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install rename-wp-login --activate
- changed login slug by:
- "Settings" -> "Permalinks"
- changing the field under "Rename wp-login.php" -> "Login url" to 'ose-hidden-login'
- added section to "deny from all" for anyone attempting to hit '.../wp-login.php' to vhost config
- changed login slug by:
<LocationMatch .*wp-login.php> Deny From All </LocationMatch>
- confirmed that the default wp core providers a password strength meter, but does not enforce it
- installed 'force-strong-passwords' & validated that it blocked my new user from setting their password to 'password'
- also validated that sufficiently long all-lowercase-character passwords are considered "stong" — as they should be. Symbols are a negligible benefit to long passwords.
sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install force-strong-passwords --activate
- researched FOSS TOTP apps for android & found andOTP to be the best
- created 'obi4' && 'piwik' subdomains for obi for testing let's encrypt & piwik
- decided to have a distinct piwki site for obi & ose
- created vhost for "http://piwik.openbuildinginstitute.org:4443"
- opened port for 4443 in iptables
Sun Aug 28, 2017
- began investigating updating to v6 of oshine theme
- dug through the theme's code, but was unable to see an obvious update url & the necessary variables
- temporarily removed iptables rules blocking apache from making new connections outbound
- `tcpdump`ed all calls to brandexponents.com, and extracted the following GET request:
- brandexponents.com/oshin-plugins/oshine-purchase-verifier.php?installed_version=5.0&purchase_key=XYZ (which is also only http, so our purchase key is trivial to steal *facepalm*)
- this returned a json saying that the latest version is v5.0.5, which is what I downloaded yesterday though wp-cli. I assume that wp-cli uses this same method as the wp wui
- the theme itself is publically accessible for some reason here http://brandexponents.com/oshin-plugins/oshine.zip
- emailed Catarina stating that brandexponents hasn't made their new version available for update yet, even if it is published on themeforest
- began attempting to manually update & install plugins that oshin malicously-like expects to install itself with their own code & calls to the internet
cd /var/www/html/obi2/htdocs/wp-content manualPlugins='oshine-modules tatsu revslider masterslider' for plugin in $manualPlugins; do cp "../themes/oshin/lib/plugins/${plugin}.zip" . && unzip -o "${plugin}"; done
- went into the wp dashboard to enable the new Tatsu plugin that oshin was complaining it needed
- emailed Catarina for post-theme-and-plugin-updates validation
Sat Aug 26, 2017
- discovered wiki db was locked again, repeated mv of 'read-only-message' in docroot for wiki
- investigated konsoleh's config to see if there's any hetzner-level mysql backups triggering the lock, but found none
- probably found the culprit in /usr/home/osemain/cron, which contained many cron jobs that weren't listed in the crontab or in hetzner's knosleh wui. Moved this entire dir to /usr/home/osemain/cron/noBackup/deleteMeIn2018/usr/home/osemain/ & created an empty '/usr/home/osemain/cron' dir. These backups were supurfluious; we're doing a mysqldump of everything daily for months now.
- added documentation for the commands to produce the obi ephemeral vhost clone in the Wordpress article http://opensourceecology.org/wiki/Wordpress
- investigated missing '3Dmodels' render in obi2
- the file referenced is actually the original obi site, but it's in an iframe. there's a 200 OK, I determined that this is the browser is refusing to show it by respecting the "X-Frame-Options: SAMEORIGIN" header or similar. When I removed this option from the apache config & reloaded, it worked.
- to be consistent with the wp install upgrade documentation, '3Dmodels' should *really* be in the wp-content dir
- discovered unnecessary & empty 'upgrade' dir in wp-content. deleting.
- investigated missing images on '/how-it-works/'
- solved by repeating ip address find-and-replace done on 2017-08-15, but for 'openbuildginstitute.org' to 'openbuildinginstitute.org'
- this also solved all the other issues
- deleted obi2 & re-cloned
- began investigation of updating/cleanup of obi themes & plugins
- discovered 17 plugins
- 3 were begging to be updated in the admin panel
- deleted 3 unused plugins
- discovered 17 plugins
rm -rf htdocs/wp-content/plugins/hello.php rm -rf htdocs/wp-content/plugins/simple-301-redirects rm -rf htdocs/wp-content/plugins/password-protected
- we intentionally blocked apache from making outgoing connections with iptables, which makes cripples wordpress, so I use this to update plugins. the output can be opened en-mass in firefox using the "Copy All URLs" addon
cd wp-content/plugins for line in $(ls -1); do echo "https://wordpress.org/plugins/$line"; done
- looked for a better way to update plugins
- found wp-cli https://wp-cli.org/
- wp-cli is part of the official wordpress.org project https://developer.wordpress.org/cli/commands/
- wp-cli appears to be funded by automatic, bluehost, dreamhost, siteground, wp engine, etc (basically a ton of hosting companies that offer cheap wordpress hosting, which makes sense for them to want & fund)
- the wp-cli github repo was created in 2011-09. it was last updated 3 days ago https://github.com/wp-cli/wp-cli
- the most recent stable release is 1.3.0, which was released earlier this month. the relesae before that was a couple months prior. there's been 49 releases, usually a few months apart.
- this looks like a great solution. especially because large-scale wordpress stakeholders like bluehost are using it, it's been around a while, it's still being actively maintained, and that doesn't look like it would end anytime soon.
- breifly checked through the sourcecode. I don't see any ugly `curl -k | bash` bullshit. Actually, they have the certs included in the source, pinned. And it instaled from github over https. TOFU, but well above the sec of other options.
- there's no centos package wp-cli
- considered making wp-cli accessible to all users in the 'apache' group, but decided that wp-related updates should only be preformed by someone who can do a backup first, which is necessarily root. And we wouldn't want someone attempting an update unless they had the experience necessary to do a restore. All of these should just require a root user. Else they shouldn't be using the tool.
- installed wp-cli into /root/wp-cli
- found wp-cli https://wp-cli.org/
- looked for a better way to update plugins
sudo su - mkdir -p /root/.wp-cli cd /root/.wp-cli curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar mkdir -p /root/bin ln -s /root/.wp-cli/wp-cli.phar /root/bin/wp chown [[root:root]] -R /root/.wp-cli find /root/.wp-cli -type d -exec chmod 0700 {} \; find /root/.wp-cli -type f -exec chmod 0600 {} \; chmod 0700 /root/.wp-cli/wp-cli.phar
- added '/root/.wp-cli' to php.ini & restarted apache
- successfully updated akismet with wp-cli
[root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi/htdocs list ... +----------------------------+----------+-----------+---------+ | name | status | update | version | +----------------------------+----------+-----------+---------+ | akismet | inactive | available | 3.3 | | amr-shortcode-any-widget | active | available | 3.3 | | be-themes-one-click-import | active | none | 1.6 | | be-page-builder | active | none | 4.6.1 | | be-portfolio-post | active | none | 1.1 | | duplicate-page | active | available | 2.2 | | hello | inactive | none | 1.6 | | masterslider | active | none | 2.29.0 | | meta-box-conditional-logic | active | none | 1.0.8 | | meta-box-show-hide | active | none | 0.2.1 | | meta-box-tabs | active | none | 0.1.5 | | oa-open-graph-for-fb | active | none | 1.0.2 | | open-in-new-window-plugin | active | none | 2.4 | | password-protected | inactive | none | 2.0.3 | | simple-301-redirects | inactive | none | 1.07 | | revslider | active | none | 5.2.5 | | wordpress-importer | active | none | 0.6.3 | +----------------------------+----------+-----------+---------+ [root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi/htdocs update akismet ... Downloading update from https://downloads.wordpress.org/plugin/akismet.3.3.4.zip... Unpacking the update... Installing the latest version... Removing the old version of the plugin... Plugin updated successfully. Success: Updated 1 of 1 plugins. +---------+-------------+-------------+---------+ | name | old_version | new_version | status | +---------+-------------+-------------+---------+ | akismet | 3.3 | 3.3.4 | Updated | +---------+-------------+-------------+---------+ [root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi/htdocs list ... +----------------------------+----------+-----------+---------+ | name | status | update | version | +----------------------------+----------+-----------+---------+ | akismet | inactive | none | 3.3.4 | | amr-shortcode-any-widget | active | available | 3.3 | | be-themes-one-click-import | active | none | 1.6 | | be-page-builder | active | none | 4.6.1 | | be-portfolio-post | active | none | 1.1 | | duplicate-page | active | available | 2.2 | | hello | inactive | none | 1.6 | | masterslider | active | none | 2.29.0 | | meta-box-conditional-logic | active | none | 1.0.8 | | meta-box-show-hide | active | none | 0.2.1 | | meta-box-tabs | active | none | 0.1.5 | | oa-open-graph-for-fb | active | none | 1.0.2 | | open-in-new-window-plugin | active | none | 2.4 | | password-protected | inactive | none | 2.0.3 | | simple-301-redirects | inactive | none | 1.07 | | revslider | active | none | 5.2.5 | | wordpress-importer | active | none | 0.6.3 | +----------------------------+----------+-----------+---------+
- I skipped a ton of WARNINGs that wp-cli yells about. But it works regardless.
- updated the remaining plugins:
[root@hetzner2 .wp-cli]# wp plugin --path=/var/www/html/obi2/htdocs update --all ... Success: Updated 3 of 3 plugins. +--------------------------+-------------+-------------+---------+ | name | old_version | new_version | status | +--------------------------+-------------+-------------+---------+ | akismet | 3.3.3 | 3.3.4 | Updated | | amr-shortcode-any-widget | 3.3 | 3.6 | Updated | | duplicate-page.bak | 2.2 | 2.3 | Updated | +--------------------------+-------------+-------------+---------+
- updated themes with wp-cli
[root@hetzner2 .wp-cli]# wp --path=/var/www/html/obi2/htdocs theme update --all +-------+-------------+-------------+---------+ | name | old_version | new_version | status | +-------+-------------+-------------+---------+ | oshin | 4.3.1 | 5.0.5 | Updated | +-------+-------------+-------------+---------+
- then I read the documentation, and found that they tell you not to run it as root. https://make.wordpress.org/cli/handbook/common-issues/#error-yikes-it-looks-like-youre-running-this-as-root
- this is a good read. since our php is hardened (for now), bellwood's stance matches mine. in any case, it's a bad idea to run the wp-cli as root. So I'll set it up (and document it) using a non-root user https://github.com/wp-cli/wp-cli/pull/973
- added root to sudoers `gpasswd -a root wheel`
- created a new group 'apache-admins', which is only for users who need to be able to update files with passwords like 'wp-config.php'. Other users can be added to the group 'apache' and they will only be able to view files that don't contain passwords. Users who are added to 'apache-admins' should also be added to the group 'apache'
groupadd apache-admins chown [[apache:apache-admins]] /var/www/html/obi2/wp-config.php chmod 0440 /var/www/html/obi2/wp-config.php gpasswd -a cmota apache-admins gpasswd -a maltfield apache-admins gpasswd -a marcin apache-admins gpasswd -a crupp apache-admins gpasswd -a tgriffing apache-admins
- also giving write access to users in the apache group to the wp-content dir
find /$HOME/.wp-cli -type d -exec chmod 0770 {} \; find /$HOME/.wp-cli -type f -exec chmod 0660 {} \;
- also added documentation on how to set the correct permssions in a wordpress docroot to the Wordpress article http://opensourceecology.org/wiki/Wordpress#Proper_File.2FDirectory_Ownership_.26_Permissions
- installed wp-cli for a user = 'wp'
useradd wp gpasswd -a wp apache gpasswd -a wp apache-admins su - wp mkdir -p /$HOME/.wp-cli cd /$HOME/.wp-cli curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar mkdir -p /$HOME/bin ln -s /$HOME/.wp-cli/wp-cli.phar /$HOME/bin/wp chown wp:wp -R /$HOME/.wp-cli find /$HOME/.wp-cli -type d -exec chmod 0700 {} \; find /$HOME/.wp-cli -type f -exec chmod 0600 {} \; chmod 0700 /$HOME/.wp-cli/wp-cli.phar
- updated again using user wp = the right way
[root@hetzner2 ~]# sudo -u wp -i wp plugin --path=/var/www/html/obi2/htdocs list ... +----------------------------+----------+-----------+---------+ | name | status | update | version | +----------------------------+----------+-----------+---------+ | akismet | inactive | none | 3.3.4 | |----------------------------|----------|-----------|--------| | amr-shortcode-any-widget | active | available | 3.3 | | be-themes-one-click-import | active | none | 1.6 | | be-page-builder | active | none | 4.6.1 | | be-portfolio-post | active | none | 1.1 | | duplicate-page | active | available | 2.2 | | hello | inactive | none | 1.6 | | masterslider | active | none | 2.29.0 | | meta-box-conditional-logic | active | none | 1.0.8 | | meta-box-show-hide | active | none | 0.2.1 | | meta-box-tabs | active | none | 0.1.5 | | oa-open-graph-for-fb | active | none | 1.0.2 | | open-in-new-window-plugin | active | none | 2.4 | | password-protected | inactive | none | 2.0.3 | | simple-301-redirects | inactive | none | 1.07 | | revslider | active | none | 5.2.5 | | wordpress-importer | active | none | 0.6.3 | +----------------------------+----------+-----------+---------+
- successfully updated plugins with wp-cli running under the user 'wp'
sudo -u wp -i wp plugin --path=/var/www/html/obi2/htdocs update --all ... Success: Updated 2 of 2 plugins. +--------------------------+-------------+-------------+---------+ | name | old_version | new_version | status | +--------------------------+-------------+-------------+---------+ | amr-shortcode-any-widget | 3.3 | 3.6 | Updated | | duplicate-page | 2.2 | 2.3 | Updated | +--------------------------+-------------+-------------+---------+
- attempted to do an update of the wordpress core, but it appears that wp-cli necessarily needs access to '/tmp/'. This directly is necessarily insecure with 777 permissions, so it should never be included in the basedir of php.ini. Therefore, we'll just stick to using svn to update the core wp software, and just use wp-cli for updating plugins & themes
Fri Aug 25, 2017
- troubleshooting site-down alerts for oswh & obi
- both oswh & obi (which are located on distinct servers) went down. I confirmed the sites were inaccessible in the browser
- in fact, the dns wouldn't resolve to an ip address. I confirmed this on opendns & google nameservers from both my lan in NYC and from my personal server in the Neatherlands
- within a few hours, the ip address was being returned on dns lookup, and the sites came back online (not at the same time)
Tue Aug 15, 2017
- dialog with Chris on php, mod_security, & htaccess settings
- discovered that the WP-Piwik plugin had a major XSS vulnerability that allowed anyone to persistently insert arbitrary JS code into all WP pages without any authorization. Asked Chris to investigate if the issue was resolved, and--if not--what's the best way to integrate piwik into wordpress https://www.pluginvulnerabilities.com/tag/wp-piwik/
- discovered that piwik talks a lot about security, but I found many red flags that smell like bullshit
- their site requires ini_set to be enabled in php. Many people have complained about the security implications of this, and the piwik response was "won't fix" https://forum.piwik.org/t/workaround-possible-for-ini-set/6569/6
- piwik has mentioned that they've received many independent security audits from professionals in the past, but they don't include the reports from the audits! they claim things were fixed, but how can we know without transparency? I specifically want to see what the auditors said about ini_set, and other things. I've asked Chris to post in the forum for a link to the reports from the audit. If they can't or won't produce it to the public, that's a huge red flag. https://piwik.org/blog/2011/01/professional-security-audit-in-piwik/
- Chris mentioned that obi is throwing forbidden messages
- all the failed requests seem to occur to content referenced over ip address rather than openbuildgininstitute.org
- mod_security doesn't like it when people reference content over ip address
- regardless of mod_security, it's not very robust to have an ip address hard-coded into your content's links
- I decided the solution is to replace all occurrences of '138.201.84.223' with the domain 'openbuildinginstitute.org'
- I confirmed that the settings in the wp admin page defined wordpress address & site address by domain name, not ip address
- I dug deeper & searched all of the wp_options table, and confirmed only 1 reference to the ip address, which was in regard to the (now deprecated) ftp credentials, which is unrelated
- I decided that this must be done with a mysqldump backup -> cp -> sed -> db delete -> db import -> verify
- While this is annoying, it's our best option. And it is good practice, as it will probably be necessary to sed s/http/https/ in the near future
- emailed catarina for dns info & credentials
dbName=obi_db dbUser=obi_user dbPass=CHANGEME fromString=138.201.84.223 toString=openbuildinginstitute.org stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir service httpd stop # create backup of everything for good measure time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz # dump obi wordpress db contents time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql # make backup cp $dbName.$stamp.sql $dbName.$stamp.sql.orig # sed sed -i "s/$fromString/$toString/g" $dbName.$stamp.sql # verify grep "$fromString" $dbName.$stamp.sql | less grep "$toString" obi_db.$stamp.sql | less # delete db tables mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' | while read table; do mysql -u$dbUser -p$dbPass -sNe "DROP TABLE $dbName.$table;"; done # verify mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' # import mysql -u$dbUser -p$dbPass < obi_db.$stamp.sql # verify mysql -u$dbUser -p$dbPass $dbName -sNe 'show tables' service httpd start
- successfully finished the above change. verified site & absence of forbidden messages with firefox debugger.
- gained access to robot for managing hetzner2
- added creds to keepass
- confirmed that I have access to order an additional ip address, which will be needed for 2x seperate domains over https on 1 server over port 443
- added my email address to the "Main addresses" field
- saw that hetzner sells a package of 2x the servers we currently have as a "private cloud" option using openstack. This may be the best way to scale (while remaining powered by renewable energy) when needed, maybe. https://www.hetzner.com/cloud/private-cloud
- installed subversion
- began creating a clone of obi2 as a test for upgrading all its software
- new dir installs wordpress from subversion to make updates easier
- created obi2.conf vhost from obi.conf
- added "<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*> Deny From All </LocationMatch>" to obi2 vhost
oldVhostDir=/var/www/html/obi newVhostDir=/var/www/html/obi2 oldDbName=obi_db newDbName=obi2_db newDbUser=obi2_user newDbPass=CHANGEME mkdir -p $newVhostDir/htdocs pushd $newVhostDir/htdocs yum install subversion svn co https://core.svn.wordpress.org/tags/4.8.1 . find $newVhostDir -type d -exec chmod 750 {} \; find $newVhostDIr -type f -exec chmod 640 {} \; chown -R apache:apache $newVhostDir stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz time nice mysqldump -uroot -p --databases $oldDbName > $oldDbName.$stamp.sql cp $oldDbName.$stamp.sql $newDbName.$stamp.sql # replace the first 2 (non-comment) occurances of $OldDbName with $newDbName vim $newDbName.$stamp.sql time nice mysql -uroot -p -sNe "CREATE DATABASE $newDbName; USE $newDbName; SOURCE $oldDbName.$stamp.sql;" time nice mysql -uroot -p -sNe "GRANT ALL ON $newDbName.* TO '$newDbUser'@'localhost' IDENTIFIED BY '$newDbPass'; FLUSH PRIVILEGES;" popd rsync -av --progress $oldVhostDir/wp-config.php $newVhostDir/ chown apache:apache $newVhostDir/wp-config.php chmod 400 $newVhostDir/wp-config.php # change DB_NAME, DB_USER, DB_PASSWORD, WP_HOME, & WP_SITEURL vim $newVhostDir/wp-config.php # we want to copy files that don't exist yet in our new install dir. if a file exists in both, don't overwrite the new from the old rsync -av --progress --ignore-existing $oldVhostDir/htdocs/wp-content/ $newVhostDir/htdocs/wp-content/ rsync -av --progress $oldVhostDir/htdocs/.htaccess $newVhostDir/htdocs/
- attempted to log into new obi clone with updated wordpress
- after login, I was prompted to click a button to update the db, which I did. It completed in seconds.
- everything apeared to be working on the obi clone
- except just loading the domain directly 301 redirects--though index.php loads fine
- sent an email to Catarina for initial validation before I attempt to update plugins & themes
Mon Aug 14, 2017
- began investigating why the wiki became locked from edits
- "Warning: The database has been locked for maintenance, so you will not be able to save your edits right now. You may wish to copy and paste your text into a text file and save it for later...The administrator who locked it offered this explanation: Wiki backup in progress. Access should be restored in about 5 minutes"
- successfully connected to the DB locally
- got a permission error when I attempted to load Special:UnlockDB "You are not allowed to execute the action you have requested. "
- couldn't immediately find any backup-related extensions to mediawiki that may have triggered the lock
- found a file 'read-only-message' in the docroot
- after finding no safer way to restore the wiki to write-enabled mode, I renamed 'read-only-message' to 'read-only-message.20170814.bak'
- I confirmed an edit
- I saw the last edit was just under 3 days ago
- I emailed the OSE Devs
- I emailed Marcin for access to Special:UnlockDB
- I emailed Jozef for access to the OSE Dev contact list
- I emailed Tom to ask if he was aware of any mediaiwki-related backup processes or otherwise that would have triggered the db lock
Fri, Aug 11, 2017
- commented-out monitoring of kern.log (I got nearly 1,000 ossec alerts since enabling them almost a week ago
- added /var/log/kern.log, which was 42M since its first entry less than a month ago
- updated common log format for apache to track sessionid & time to generate
- confirmed that ossec is now reporting diffs in alerts
- installed mod_evasive
- confirmed that mod_evasive is running & actively blocking DOS attacks (or at least attempting to)
- disabled webdav by commenting out all lines in /etc/httpd/conf.modules.d/00-dav.conf
- commented-out mod_info in /etc/httpd/conf.modules.d/00-base.conf
- moved mod_security overrides to the vhost file (/etc/httpd/conf.d/obi.conf) by Location match (where needed)
- removed file /etc/httpd/modsecurity.d/modsecurity_crs_00_config.conf
- finished apache hardening (minus changes needed for https & related hardening)
- added documentation on mod_security to the OSE_Server page
- mysql (mariadb) hardening
- added 'skip-networking', 'skip-show-database', and 'local-infile=0' to /etc/my.cnf
- changed DB_HOST of oib/wp-config.php to 'localhost:/var/lib/mysql/mysql.sock'
- reset root password to a good, long, random passphrase (and updated keepass)
- updated /root/backups/backup.settings with the new mysql root password
- dropped unnecessary users: osemain, osewiki, & osewiki_w & unnecssary hosts '127.0.0.1' & '::1' & 'centos-72-64-minimal'. Only 2 entries remain: 'root'@'localhost' & 'wordpressuser'@'localhost'
- began hardening obi
- researched ~10 2FA/OTP wordpress plugins
- best option is probably 'google-authenticator' https://wordpress.org/plugins/google-authenticator
- this doesn't support forcing users to use 2FA, but this can be achieved with this additional plugin https://wordpress.org/plugins/google-authenticator-encourage-user-activation/
- best option is probably 'google-authenticator' https://wordpress.org/plugins/google-authenticator
- defined all salts in wp-config.php. Passwords in the db will only be updated when a user logs-in again. We'll force this to happen by resetting everyone's password once https is implemented.
- set DISALLOW_FILE_EDIT in wp-config.php
- renamed wordpress db & created new user
- researched ~10 2FA/OTP wordpress plugins
sudo su - oldDbName=wordpress newDbName=obi_db oldUsername=wordpressuser newUser=obi_user newPass=CHANGEME rootDbPass=CHANGEME stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir service httpd stop # create backup time nice mysqldump -uroot -p --all-databases | gzip -c > preBackup.all_databases.sql.gz # create new db echo "CREATE DATABASE $newDbName; GRANT ALL PRIVILEGES ON $newDbName.* TO '$newUser'@'localhost' IDENTIFIED BY '$newPass'; FLUSH PRIVILEGES;" | mysql -uroot -p$rootDbPass # move tables from old db to new db mysql -uroot -p$rootDbPass $oldDbName -sNe 'show tables' | while read table; do mysql -uroot -p$rootDbPass -sNe "RENAME TABLE $oldDbName.$table TO $newDbName.$table;"; done # update the wp-config.php file & start httpd
- verified that the site was working, then deleted the old user
DROP USER 'wordpressuser'@'localhost';
Tue, Aug 08, 2017
- keepass training meeting with Marcin & Christian
Sat, Aug 05, 2017
- added marcin to recieve monthly statuscake reports
- isolated the cookie obi issue to the "Secure" flag, so I enabled it to be just "HttpOnly" to provide some XSS protection, until we can enable the original line after implementing https
- installed git on hetzner2
- installed mod_security_crs
- added '/etc/httpd/modesecurity.d/modsecurity_crs_00_config.conf' to disable several rules, which were triggering false positives.
- added ossec local rule to calm down alerts of mod_security
- added /var/log/cron to ossec
- added /var/log/kern.log to ossec
- added ossec local rule to ignore alerts on kern.log for IN=eth0. We should get an alert if something tries to get out, but attempts to get in are unavoidably continuious
- also ignored OUT=lo; not sure why this is happening, but I don't think it's necessary
- added report_changes="yes" and realtime="yes" to the directories listed undersyscheck in /var/ossec/etc/ossec.conf
- updated log_format of mariadb logfile to 'mysql_log'
Fri, Aug 04, 2017
- confirmed that hetzner1's daily backups dropped from 39G to 14G starting on 2017-08-03 due to the cleanup from 08-02
- hardened keepas file
- now has a randomly generated 100-character password
- also requires the 4 KiB key file
- increased transformation rounds to 87654321, which takes about 5 seconds on my 2-year-old laptop
- moved to /etc/keepass, which is owned by root:keepass 770. The files inside are root:keepass 660.
- emailed Marcin & Christian for keepass hand-off/training session
Thr, Aug 03, 2017
- generated a 4 KiB key file for passwords keepass file
- discovered that dd will take an undeterministic size from /dev/random, regardless of bs & count. used head instead
- `head -c 4096 < /dev/random > ose.passwords.key`
Wed, Aug 02, 2017
- confirmed the hetzner1 2017-08-01 backup of 39G is on backup server
- created /usr/home/osemain/noBackup on hetzner1
- added "--exclude /usr/home/osemain/noBackup" to backup.sh on hetzner1 to reduce nightly backup size from 39G to 12G
- fixed wp-login.php cookie issue preventing logins on obi
- ERROR: Cookies are blocked or not supported by your browser. You must enable cookies to use WordPress
- fix was commenting-out "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" in /etc/httpd/conf/httpd.conf
Tue, Aug 01, 2017
- meeting with Lex
- sent Marcin my signed Liability Waiver for Aug 25 workshop
Mon Jul 31, 2017
- moved obi's wp-config.php up 1 dir to be outside the docroot & chmod to 400
- begun debugging oib wp-login issues (cookie error)
- confirmed that I cannot login
- did not see any error logs
- began reverting all session-related hardening changes to php.ini
Sat Jul 29, 2017
- multi-user keepass research
Fri Jul 28, 2017
- added christian to statuscake admin contact group
- emailed christian & marcin about gpg key creation for recieving ossec alerts
Thr Jul 27, 2017
- email to Devs about new github accounts for forking filament winder
- created account for christian on hetzner2
Thr Jul 20, 2017
- further docker research
- dialog with Lex about docker
- meeting with Christian about sysadmin assistance
Wed Jul 19, 2017
- keepass research
- preparing mediawiki file dump for Lex
- container research
Tue Jul 18, 2017
- emailed with Marcin about containers
- emailed Lex asking for his ssh public key
- created user on hetzner 2
- added user to sshaccess
- added Lex's new public key
- uploaded last night's dump of the wiki to /var/tmp/ with a symlink in his $HOME per Marcin's request
- confirmed that "fake" backup "from" 2017-07-01 of hetzner1 was not deleted, so it's safe to make changes to hetzner1 now
- moved 27G of assumed-unnecessary files from hetzner1 into '/usr/home/osemain/deleteMeIn2018/'
- if nothing is reported as broken by 2017-08-02, I'll validate that the files were included in the first-of-the-month August backup, then add an "exclude" argument to the tarball creation in 'backup.sh' for 'deleteMeIn2018'. And I will not be migrating these files to hetzner2.
- finishing php hardening
- changed upload_tmp_dir to '/var/lib/php/tmp_upload' per best practices
- added upload_tmp_dir to basedir
- set session.use_strict_mode = 1
- set session.cookie_httponly = 1
- changed PHPSESSID to custom value (session.name)
- changed session.save_path to "/var/lib/php/session"
- set the session.hash_function to 'sha512'
- disable_functions = ini_set,php_uname,getmyuid,getmypid,passthru,leak,listen,diskfreespace,tmpfile,link,ignore_user_abord,shell_exec,dl,set_time_limit,exec,system,highlight_file,source,show_source,fpaththru,virtual,posix_ctermid,posix_getcwd,posix_getegid,posix_geteuid,posix_getgid,posix_getgrgid,posix_getgrnam,posix_getgroups,posix_getlogin,posix_getpgid,posix_getpgrp,posix_getpid,posix,_getppid,posix_getpwnam,posix_getpwuid,posix_getrlimit,posix_getsid,posix_getuid,posix_isatty,posix_kill,posix_mkfifo,posix_setegid,posix_seteuid,posix_setgid,posix_setpgid,posix_setsid,posix_setuid,posix_times,posix_ttyname,posix_uname,proc_open,proc_close,proc_get_status,proc_nice,proc_terminate,phpinfo,popen,curl_exec,curl_multi_exec,parse_ini_file,allow_url_fopen,allow_url_include,pcntl_exec,chgrp,chmod,chown,lchgrp,lchown,putenv
- soap.wsdl_cache_dir = /var/lib/php/soap_cache
- began hardening apache
- reset permissions of files:
- chown -R apache:apache /var/www/html
- find /var/www/html/ -type d -exec chmod 750 {} \;
- find /var/www/html/ -type f -exec chmod 640 {} \;
- added ServerTokens Prod
- added ServerSignature Off
- removed DocumentRoot from the main /etc/httpd/conf/httpd.conf since we're using vhosts
- removed cooresponding Directory block
- added 'Options -Indexes -Includes' to all Directory blocks
- added "Order allow,deny/nAllow from all" to all Directory blocks
- set main "Directory /" block to only contain "Options -Indexes -Includes\nAllowOverride none"
- added 'FileETag None' option
- added "TraceEnable off" option
- recursively changed ownership of all files in /var/www/html/obi/htdocs to 'apache:apache'
- changed wp-config.php back to 'root:root'
- recursively changed ownership of '/etc/httpd/conf' & '/etc/httpd/conf.d' to apache:apache & set permissions to 750 (from root:root & 755)
- recursively changed permissions to 640 within the above dirs
- added LimitExcept block to deny all requests other than the 3 basic: GET, POST, & HEAD for all vhost Directory blocks
- added "Header edit Set-Cookie ^(.*)$ $1;HttpOnly;Secure" for XSS protection
- added "Header always append X-Frame-Options SAMEORIGIN" for clickjacking protection
- added mod_rewrite rules to disable HTTP 1.0 to avoid session hijacking risks
- decreased apache timeout to 60 (default is 300) to decrease DoS risk
- installed mod_security
- reset permissions of files:
Mon Jul 17, 2017
- validated with Marcin that obi's content saves are now functioning, following my php.ini changes from yesterday
Sun Jul 16, 2017
- Recieved an email from marcin describing issues adding content to OBI wordpress
- grepped through /var/log/error_log & found issues with php temp dir & max_input_vars due to recent php hardening that may be breaking wordpress changes
- relevant errors:
- [Sun Jul 16 04:34:18.532015 2017] [:error] [pid 21569] [client 184.157.59.85:45406] PHP Warning: Unknown: Input variables exceeded 100. To increase the limit change max_input_vars in php.ini. in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4509&action=edit
- [Sun Jul 16 07:11:02.922720 2017] [:error] [pid 24088] [client 184.157.59.85:49642] PHP Warning: File upload error - unable to create a temporary file in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4531&action=edit
- [Sun Jul 16 07:11:35.723209 2017] [:error] [pid 23510] [client 184.157.59.85:49646] PHP Warning: Unknown: open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/var/www/html/obi/:/var/www/html/osemain) in Unknown on line 0, referer: http://openbuildinginstitute.org/wp-admin/post.php?post=4531&action=edit
- changed upload_tmp_dir to '/var/lib/php/tmp_upload' per best practices (currently it's '/tmp')
- set the new tmp_upload dir to root:apache 770, unlike '/tmp' which is necessarily 777
- increased max_input_vars back to 1000, the default
- relevant errors:
- emailed with marcin about configuring his mail client with gpg for ossec reports
- statuscake started marking oswh as "recovered" at 04:08 ET, after a 9-hour false-negative streak. I confirmed that they have not responded to my support question about the issue.
Sat Jul 15, 2017
- email exchange with marcin about security limitations
- investigating statuscake's false-negative "outage" of opensourcewarehouse.org
- confirmed that 200 OK was returned with `curl -I "http://www.opensourcewarehouse.org/"`
- sent statuscake a support message about the false-negative
Fri Jul 14, 2017
- Got Christian's ssh pubkey & asked for availability for a meeting
- Got credentials for themeforest, but was unable to login due to a shitty 2FA over email
- Asked Catarina to login to themeforest & send me the osemain theme's "Item Purchase Code" so that I can register an account with the theme creator's forum https://support.livemeshthemes.com/wp-login.php?action=register
- gained access to themeforest & downloaded all necessary files
- fixed osemain wp theme issue
- I downloaded the theme archive from themeforest
- I scp'd the archive to /usr/home/osemain/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme.zip
- I extracted the archive & the archive within named enigmatic.zip to /usr/home/osemain/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme/enigmatic
- I changed directory to /usr/home/osemain/public_html/wp-content/themes
- I changed the current directory & the 'enigmatic' directory's permissions to '755'
- `mv enigmatic enigmatic.20170714.bak` && `rsync -av --progress ~/enigmaticTheme/20170714/themeforest-3919108-enigmatic-responsive-multipurpose-wp-theme/enigmatic .`
- I emailed Marcin for validation of the changes
- created a user on osemain for Tom
Thr Jul 13, 2017
- fixed migration oversite that broke all the obi pages but the main page
- new docroot was missing /var/log/http/obi/htdocs/.htaccess, which included mod_rewrite rules
- emailed Christian asking for portfolio & CV for risk analysis before user creation
- manually added an admin user for myself to the osemain wp db (ose_website) as uid = 55 using SQL commands directly
- login attempt produced a fatal php error:
- Fatal error: Call to undefined function ot_register_meta_box() in /usr/www/users/osemain/wp-content/themes/enigmatic/framework/presentation/metabox-manager.php on line 34
- created new user 'osemain' on mysql db of hetzner2 & database 'osemain' per the wordpress install guide
- copied last night's backup of the osemain db's mysqldump sql file to hetzner2 from backup server
- copied public_html from backup server to hetzner2 & extracted to /var/www/html/osemain.old/htdocs/
- created vhost at /etc/httpd/conf.d/osemain.conf
- manually reset theem through mysql to 'twentyeleven'
- successfully logged-in to osemain on the new server under new theme
- confirmed wp is currently 4.7.5 on osemain
Tue Jul 11, 2017
- verified that the cleanup of files on hetzner2 reduced the daily backup size from 20G to 1.3G
- The dir 'ose_wiki' (16G) & file w.tar.gz (9.4G) had been moved to /var/tmp/deleteMeIn2018/ a few days ago
- confirmed that ossec has been actively blocking abusive ip addresses
- <active-response> is configured to block these ip addresses via host-deny & iptables for 10 minutes
- logs are found in /var/ossec/logs/active-responses.log
- I was able to ban myself by failing ssh attempt 10 times in a row
- organized httpd vhost as prereq to harden php
- created /etc/httpd/conf.d/obi.conf
- moved all files from /var/www/html to /var/www/html/obi/htdocs/
- soon obi-related files that should be stored outside the docroot for security reasons can be stored in the 'obi' dir, but the docroot will be 'htdocs' ie: wp-config.php!
- hardened php
- disabled allow_url_fopen
- reduced max_input_time & max_execution_time to 30 seconds
- disabled expose_php
- decreased max_input_vars to 100
- decreased post_max_size & upload_max_filesize to 10M
- added open_basedir to whitelist the php dir "/var/www/html/obi/"
- researched our https options with multiple domain names running as vhosts on a single server
- sent an emal to Marcin & Catarina asking if they'd be interested in consolidating to a single domain with many subdomains to simplify the costs & config
- discovered that Hetzner charges € 0.84 / month for each additional ip address, which is our best non-consolidated option, as there's client support issues with SNI & running on separate ports is unpractical
- research shows that ~10% of internet users do *not* support SNI as of last year (2016)
- Catarina said she wants separate domains, and she said she'd cover the 10 eur/yr fee
Mon Jul 10, 2017
- installed make, gcc, gcc-c++, kernel-devs
- installed & started ossec service
- added config to monitor mariadb logs
- installed & started postfix service
- added TMOUT=14400 to '/root/.bash_profile' to force all root logins to timeout after 4 hours
- configured ossec to send pgp-encrypted emails
- installed procmail & mailx
- configured mailbox_command in /etc/postfix/main.cf
- imported my gpg public key to keyring @ /var/ossec/.gnupg/
- created /var/ossec/.procmailrc
- created script /var/ossec/sent_encrypted_alarm.sh
- emailed Catarina & gained admin access to OBI WP
- receieved many OSSEC alerts regarding brute-force attempts on wp-login.php from a distributed set of IPs
- researched wp-login.php hardning options, including 2FA, captcha, rate limiting plugins, and ossec active response iptables ip address blocking. I'm thinking we'll do all except the captcha, especially if 2FA can be enforced for all users.
Sun Jul 09, 2017
- discovered that hezner2 has 2x 250G disks: sda & sdb
- each disk has 3 partitions of exactly the same size, and is RAID1'd between their cooresponding partition numbers on each disk
- 1 = 34.4G
- 2 = 537M
- 215G
- RAID config is as follows:
- /dev/md0 = 34.3G = swap
- /dev/md1 = 536M = ext3. mounted at '/boot'. filesystem features includes has_journal.
- /dev/md2 = 215G = ext4. mounted at '/'. `df` total size is listed as 197G. flesystem features includes has_journal.
- currently, 83G are in-use out of 197G
- each disk has 3 partitions of exactly the same size, and is RAID1'd between their cooresponding partition numbers on each disk
- reset the root password on hetzner2. added new credential to keepass.
- updated all packages (210) with `yum update`
- hardened sshd_config
- set 'PermitRootLogin no'
- set 'PasswordAuthentication no'
- set 'PermitEmptyPasswords no'
- set 'IgnoreRhosts yes'
- set 'AllowGroups sshaccess'
- created group 'sshaccess' & added all 4 users
- added documentation to OSE_Server on how to add new users with ssh access to the server
- created new, hardened ssh host keys
- `ssh-keygen -f /etc/ssh/ssh_host_rsa_key -t rsa -b 4096 -o -a 100 -N `
- `ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -t ecdsa -b 521 -o -a 100 -N `
- `ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -t ed25519 -a 100 -N `
- moved some large, seemingly unnecessary files from /var/www/html/ to /var/tmp/deleteMeIn2018/ to reduce backup sizes
- updated logrotate config to decrease backup sizes
- added 'compress' & 'delaycompress' to /etc/logrotate.conf
- changed default rotation from 'weekly' to 'daily' & rotate from '4' to '32' to keep the same amount of data locally, but now compressed daily (per above)
- added /etc/logrotate.d/backups for our new backup log
- updated the hostname of hetzner2 to 'hetzner2.opensourceecology.org' in /etc/hosts, /etc/hostname, and using `hostnamectl`
- set timezone to UTC with `timedatectl set-timezone UTC`
- restarted rsyslog & confirmed hostname & date updates got pushed to /var/log/secure
- replaced firewalld with iptables
- configured iptables
- apache & mysql users cannot send traffic out unless it's already established or dns
- no traffic can come in over eth0 unless it's over ssh or http or established
Sat Jul 08, 2017
- researching GPL & BSD license info
- got an email from statuscake that the site was down, verified that hitting opensourceecology.org in my browser lead tme to the cloudflare error page. so confirmed that statuscake works for email alerts & outages even when using cloud flare's "always on" cache
- added status.opensourceecology.org CNAME to statuscake.com
- added open building institue to statuscake
- added documentation to OSE Server about statuscake
- emailed OSE Devs, providing them with the statuscake URL & asking if anything else should be monitored that I missed
- created fake backup '20170701' from June 5th's backup as a long-term backup that won't be deleted by the cleanup script (the 1st of every month is normally kept, but Hetzner brought our site down, so backups were missing from the 1st through 3rd) by setting the mtime of the files with 'find . -type f | xargs touch -d '20170701'
- Added 'keepass solution' to TODO on OSE Server
Fri Jul 07, 2017
- researching OSE wiki's contents for Licensing info
Thr Jul 06, 2017
- created email statuscake at opensourceecology.org & added credentials to keepass
- created statuscake & added crednetials to keepass
- meeting with marcin, who now has a password-protected ssh keypair that is verified to work with both ssh & filezilla (sftp)
- configured (untested) statuscake basic GET checks with a public reporting page https://uptime.statuscake.com/?TestID=itmHX7Pfj2
Tue Jul 04, 2017
- SSH is now working to Hetzner 1, and I validated that backups are running again
- The last backup before the outage that still existed on dreamhost was from 2017-06-30. I made a copy of this outside the 'backups/hetzner1/' dir in $HOME so it's preserved (not deleted by the cron) just in case
- The first backup we have after the outage is from today, 2017-07-04. This started automatically, indicating that the site was offline at 07:20 for the daily backup for 3 days in a row. That's pretty awful service from Hetzner!
- Verified that my email address was added to the admin/techincal emails list on the hetzner1 web console
- Submitted a support request asking for [1] an RCA of the outage they caused [2] a way to communicate with Hetzner that has confidentiality [3] access to "the Robot"
- Added Tom's ssh key to hetzner1's authorized_keys file
- Emailed Marcin how I can create a new Gapps Email account as prereq to signing up for statuscake
Mon Jul 03, 2017
- Site came back online with no word from hetzner, but we still cannot ssh in.
- Hetzner support is still unresponsive to my numerous emails
- Spent some time researching hetzner alternatives that use 100% renewable energy & offer dedicated servers for comparable prices
Sun Jul 02, 2017
- The site was still down this morning, and hetzner 1 was still inaccessible. No word from Marcin
- I sent an email to support@heztner.de asking if our server was offline, indicating that we lost it about the time of their maintenance window
- I logged into cloudflare, and saw that opensourceecology.org points to '78.46.3.178'. Note that this is distinct from `dig` results, as the cloudflare service is essentially a MITM
- could not ping or ssh into 78.46.3.178
- determined that our old 'dedi978.your-server.de` dns host resolves to the above-listed IP address. That whold endpoint is insaccessible. I hope that we can find a static IP address that's associated with our vServer, and reconfigure to use that directly.
Sat Jul 01, 2017
- 07/01 backups from hetzner #2 were successful
- 07/01 backups from hetzner #1 failed
- I checked if they were still rsync-ing, and was unable to ssh into the server
- I saw mail from Marcin indicating the server was out
- I could not ping our server on dedi978.your-server.de'
- I could not preform any admin actions from the hetzner web portal
- Google analtyics still showed traffic to our site, perhaps a false negative due to the cloudflare "always on" feature
- I found that hetzner recently did maintenance to their VHosts, indicating that our vServer's vHost may change, and that we should check "robot" to determine our new vHost https://www.hetzner-status.de/en.html
- I found the URL for the "robot" service, but was unable to login with the credentials given to me by Marcin https://robot.your-server.de/
- I messaged Marcin, asking if he had these credentials
Thr Jun 29, 2017
- confirmed that 2017-06-29 backups came in, though hetzner1's dropped to 33G--not sure why
Wed Jun 28, 2017
- watched Monday's meeting
- Added my profile to the OSE Developers wiki article, including avatar & badge upload
- Added email & website support to the Department template Template:Department
- Met with Marcin
- Documented the backup procedure on OSE Server
Tue Jun 27, 2017
- confirmed that 06/26 backups were successfully rsync'd to dreamhost from hetzner1 & hetzner2.
- the cleanup cron was still in NOP mode, as unlink was commented-out. manually ran cleanup as /home was 99% full. It's now 93% (257G avail) after manually running the cleanup script. I'll need to validate that this cron is fully working later this week.
- cleanup script on dreamhost leaves behind empty dirs. updated script to clean out empty dirs too.
- fixed LOCK TABLE permissions issue on wiki mysqldump. Solution was to add the '--single-transaction' argument to the mysqldump command. The file is now 169M after bz2 compression. That's bigger than all the other DBs combined, but still not unreasonably large.
- meeting with Catarina to generate an ssh keypair
- added ssh key to her authorized_keys file & set permissions. validated access successfully.
- set her password on the server & validated sudo permissions were already in-place.
- fixed permissions on 3DModels directory, set to apache:apache from root recursively
- added 'define('ALLOW_UNFILTERED_UPLOADS', true);' to obi's wp-config.php. Because of this, we should not allow untrusted people accounts on this wp site. Currently it's just marcin, me, & cmota. I also asked Catarina to limit files stored on wp to <10mb
Sat Jun 24, 2017
- Discovered that at least 9.6G out of the 51G on hetzner1 are log files. This should be manually cleaned for now. Hetzner2 should be configured with logrotate, where we delete the files after a few days (now that we'll have backups we don't need to retain logs since 2001.
- 7.5G /usr/home/osemain/www_logs
- 2.1G /usr/www/users/osemain/logs
- Discovered what appears to be a 9.4G (deprecated) backup at /usr/www/users/osemain/w.tar.gz
- Discovered another 9.6G (deprecated) backup at /usr/home/osemain/tmpd/upgrade/w.tar.gz
- Therefore, if I move these files off to /var/tmp/, it will cut the backup size of hetzner1 in less than half.
- backup cron failed to run on hetzener 2. /var/log/cron says '(time) ERROR (getpwnam() failed)'. I was mising the user between the times & the command (which is unnecessary in hetzner 1, as she uses crontab). Added 'root' as the 7th option before the command, and confirmed it works on non-07:20 test. Logging works as well.
- backup cron on hetzner 1 ran successfully, though it failed to log. Interesting to note, the filenames are timestamped to 05:20, even though the job is set to run at 07:20. This is because the system time is CEST (UTC+2), but the backup script explicitly generates the timestamp with the -u argument for UTC. This is ideal, as the time of the backup files are unambigious.
- finished configuring ssh-ident & secpanel for ssh key compartimentalization
- discovered that dreamhost does have a policy on "unlimited storage", which explicitly says you can't run a file-hosting site, and it seems that using it as a backup archive may violate their "unlimited" policy. We should not depend on dreamhost to not delete out data.. Ideally, we'd need a service that'd give us ~3TB.
- added cleanLocal.pl to marcin_ose@hancock.dreamhost.com:'/home/marcin_ose/bin/'
- created crontabs for daily deletion of hetzner1 & hetzner2 backup files from dreamhost that are 3 days old. Note that cleanLocal.pl will intentionally *not* delete any backups created on the 1st day of every month. These will have to be manually deleted every year or so if space becomes an issue. Hopefully I'll have all the backups in perfect shape by July 1st (1 week from today)
Fri Jun 23, 2017
- Enabling google analytics in cloudflare was wildly successful. We now have the entire day of 06-22 showing:
- 1,230 sessions. 1,086 users. 3,024 page views. 3,321 unique page views.
- 36% from US, 7% from India, 4% from Canada, then (in order, including all with >=1%): Germany, Australia, UK, Brazil, Neterlands, France, Phillippines, Spain, Italy, Malaysia, Poland, Chile, South Korea, South Africa.
- 76% desktop. 20% mobile. 4% tablet.
- 7% of page views are '/', 3% "/wiki/List_of_CAD_Programs", 3% '/gvcs', 3% '/gvcs/gvcs-machine-index/', 2% '/wiki/Cost_of_Living', 2% '/wiki/Main_Page', 2% '/wiki/Global_Village_Construction_Set'
- the biggest site 45% of traffic
- it's especially interesting that Cost_of_Living is more viewed than the wiki home. This page is the 14th result on DDG for 'average cost of living' (which gave us 2 hits yesterday from this search term), and it doesn't even show up on the first 10 pages of google.
- 61% of sessions come from a search engine. The top 2 search terms (making up 96%) were not provided.
- Referreal traffic is 13% of overall traffic, domains in-order are: youtube.com, pintrest.com, duckduckgo.com, com.google.android.googlequicksearchbox, facebook.com, mg.mail.yahoo.com, ecosia.org, reddit.com, waldenlabs.com
- yesterday, there were <50 users at: 12am, 4am, 5am, 6am, 7am, 2pm, 6pm, 7pm, 8pm, 9pm, 10pm, & 11pm. The lowest time was 33 users at 9pm. To avoid ambiguity, I changed the GA UI to UTC, but the hours didn't change--so who knows what this data means! I could also find no information on what "49 users at 1am" means. Is that 01:00-01:59? 00:01-01:00? 00:30-01:29? This wasn't hugely helpful, but it's reasonable to assume that the 00:00-04:00 US PT time is a low-traffic window (as most of our users geolocate to North America). Therefore, I'll start the backups with cron at 07:20 UTC.
- Hetzner 2 is set to use CEST, which is UTC+2. I'm not sure, but it may switch to CET = UTC+1 sometimes. FeF is UTC-6, but sometimes UTC-5. I'm currently in NYC, which is UTC-4, but sometimes UTC-5. But the OSE devs could be in any timezone. To avoid ambiguity, and ensure consistancy across logs, I'll be changing everything to UTC in the future.
- Added '/etc/cron.d/backup_to_dreamhost' to hetzner2 to kick-off a backup job at 07:20 UTC (at least it _will be_ UTC in the future; I'm not going to make that change [or any changes] until I've validated automated backups are working with no intervention for at least a few days in a row). Logs go to '/var/log/backups/backup.log'
- Added a line to hetzner 1's osemain crontab to initiate a backup at 07:20 (this will probably always be CEST). Logs go to /usr/home/osemain/backups/log/backup.log'
Thr Jun 22, 2017
- First successful execution of the backup.sh script on both servers without manual intervention with all nice & bandwith-caps in-place on both hetzner 1 &2
- hetzner 1's mysqldump of the wiki is still failing, requiring an unlock of a table & further research into the potential impact of the change
- hetzner1's full backup execution time (including cleaning old local backups, mysqldump, tarball creation, and rsync to dreamhost) is 11 hours. If this server weren't to be deprecated shortly, I'd switch to gz to reduce this, but as space is a concern & the server is temporary, this should suffice.
- hetzner2's full backup execution time is under 3 hours
Wed Jun 21, 2017
- confirmed access to dreamhost web ui
- confirmed that we have ssh key control from the dreamhost dashboard
- determined that 'opensourceecology.org' is purchased from dreamhost for $14/yr
- found subdomains blog, community, eerik, & forum
- found 10 databases on dreamhost: dp7civicrm (drupal 7 civicrm db), dp7crm (drupal 7 civicrm), oftblog (Blog), oftcivi (CiviCRM), oftdrupal, oftforum, oftjoomla, oftsurvey (For LimeSurvey), oftwiki (Wiki), openfarmtech_org (openfarmtech.org/osefriends). Are any in use & in need of backup? Will confirm with Marcin.
- confirmed that dreamhost does *not* offer us free https certs (other than letsencrypt.org); they're $15/yr through Comodo. We just go straight with letsencrypt.org
- found 5x users on dreamhost: marcin_ose (17G), ose_site (32G), ose_community (0.2G), osecolby (<0.1G), osebackup (<0.1G). The creds I've been given were for marcin_ose, and I'm putting the backups in marcin_ose@hancock.dreamhost.com:/home/marcin_ose/backups/{hetzner1,hetzner2}/$timestamp/
- The backup of the data on hetzner 1 finished after 7 hours with the following sizes (note it was all bz2 compressed):
- 22G public_html (uncompressed size is 31G)
- 17G $HOME (uncompressed size is 20G)
- 43M mysqldump-forum
- 2.4M mysqldump-osemain
- 1.2M mysqldump-openswh
- 125K mysqldump-fef
- 527 bytes mysqldump-wiki
- There was an issue encountered with the wiki db. I've found a solution command, but I need to research its side-effects in relation to mediawiki to ensure I don't cause any issues
- 23% space savings probably isn't worth the 7 hour slamming of the CPU to compress at bz2 levels, but I'll keep it this way on hetzner 1, as we've already exceeded our disk quota many times over. I'll switch to gz compression on hetzner 2.
- mysqldump: Got error: 1044: Access denied for user 'osewiki_w'@'%' to database 'osewiki' when using LOCK TABLES
- tx of the 38G from hetzner 1 to dreamhost took just under 1 hour with speeds between 2.82 MB/s - 15.31 MB/s
- I'll configure the automated rysnc to cap at 3 MB/s so it takes a reasonable 3-5 hours while reducing the risk of saturating the network bandwidth. TODO: determine the best 5-hour window when the box is most idle on a daily cycle.
- Added backup scripts to hetzner 2. It does a single root mysqldump + file backups of /etc/, /home/, /var/log/, /root/, and /var/www/. Unlike hetzner 1, we have ample disk space (113G available after the first backup), so I'm using gzip instead of bz2. This whole backup process took 21 min on hetzner2, producing 31G. Transferring this to dreamhost took another 31 min
- confirmed access to cloudflare acount
- their site's "analytics" app has an error & returns an empty data set in both firefox & chrome
- it appears that we do have caching enabled, and some other last-mile optimization for mobile
- hetzner 2 has 61G of unused RAM. We have plenty of RAM to run a fat cache. Reverse proxy or application-level proxy? Squid or nginx? First, we'll have to see what the bottlenecks are & what the page requests and static vs dynamic content looks like over a few weeks.
- WAF events show that 2-200-ish IPs are actively being blocked every day. The worst are attacks on 'wp-login'. This is something we can have ossec or a wp plugin for rate limiting handle with iptables for free
- Email Address Obfuscation is currently enabled. There is no good replacement for this afaik, besides author's awareness
- "Always on" is enabled. There is no reasonably cheap alternative to this, though it's not strictly a requirement.
- I added google analytics tracking through cloudflare temporarily so I can get an understanding of the hourly & weekly usage trends of the sites. Eventually I hope to deprecate google analytics for awstats
Tue Jun 20, 2017
- Determined only backups done on hetzner 1 is mediawiki using MediaWiki's built-in 'maintenance/dumpBackup.php' script to 'w/export/'
- Confirmed access to mysql databases for mediawiki, main ose wp site, open warehouse wp, fef wp, & oseforum vanilla on old server
- begun a keepass db as central location for safely storing OSE credentials
- sent email to Marcin, Tom, and Catarina to get their ssh public keys to populate their user's authorized_keys file prior to disabling password-based authentication
- added maltfield user to wheel for sudo access
- discovered that the only 2 dns entries on ghandi are: 'oswarehouse.org' & 'opensourcewarehouse.org'. oswarehouse.org is an unconfigured landing page, and opensourcewarehouse.org points to dreamhost a NS.
- confirmed ssh access to dreamhost. we're using 18G in our $HOME, and '/home' (which is likley shared with other customers) has 357G free. I'll be using this unlimited storage plan to store compressed tarballs of daily backups of the server's DBs, webroots, and important config file dirs (/etc/, /home/, /root/, etc)
- was unable to connect to dreamhost web console, messaged Marcin for proper credentials
- determined size of $HOME on hetzner 1 to be 20G, and size of its 'public_html/' dir to be 31G
- added backup scripts for mysqldumps of 5x DBs + all files in $HOME (except the backups themselves, of course) to '$HOME/backups/'
- I noticed that the maximum disk usage for hezner 1's opensourceecology.org domain (not sure how these arbitrary divisions are made between "domains" since everything is actually just thrown into the same '$HOME/public_html' dir--and where would '$HOME' fit anyway?) is 10G, but we're currently using 56G. Trying to create a backup has suspiciously caused my session to be terminated, and when I reconnected I found my screen session no longer existed, suggesting that my whole session was `kill`ed. Using `nice` produced less lethal repercussions.
Mon Jun 19, 2017
- Document as you go along
- Meeting with Marcin for knowledge transfer on credentials to servers & prioritizing steps to migrate off old server.
- Established secure channel for credentials exchange
- Still missing root db credentials & root access to old server Hetzner 1
- Created a 'maltfield' account on the server, added my ssh public key, and Enabled PubKeyAuthentication on sshd
- Gained access to opensourcecology.org Google Apps email account with Google Analytics access. I should use this for OSE-related user accounts going forward.
Sun Jun 11, 2017
- researching free/discounted cloud and/or hardware services for non-profits
- researching FOSS slack alternatives for real time chatting. decided wire is best.
- researched bug tracking vs issue tracking software. Found FOSS Request Tracker, OTRS, Liberum Help Desk, GLPI, and Faveo. Marcin mentioned investigations into Mantis
Fri Jun 9, 2017
- Backup Hetzner Old - Wiki, Wordpress, Opensourcewarehouse, Forum, Factor e Farm Blog
- Fix OSE Wordpress (Main Site)
- Move all to Hetzner New. New Hetzner has Openbuildinginstitute. OSE Server
- Install HTTPS
- Install Jitsi Videobridge
- Do dev work
- Discuss password management for IT Team
- Discuss OSE Website Wordpress Theme
Sun Jun 04, 2017
- Added my video to the FreeCAD_101#Self-Verifying_FreeCAD_Exam_Videos article
- Added my comment to disqus
Thr June 01, 2017
- Updated OSE Wiki with better instructions to install the Assembly 2 Workbench FreeCAD Assembly Workbench
- successfully finished sketching polylines w/ constraints to pocket my 2d initials into the 3d xyz cube
- recorded myself building another xyz cube, initials, pocketed. Sped-up video to 30-seconds, added soundtrack, & uploaded my finished freecad test to youtube]
- Created a new repo on my github or my OSE work, and committed/pushed my xyz cube work to here
- See video.fcstd for the freecad file that was created during the video's recording.
Tue May 30, 2017
- Finished MarthamEngineering's 3-part freecad youtube introduction series
- Began [Marcin's 2-part freecad tutorial]
- Updated OSE Wiki with better instructions to install Fastener's Workbench Fasteners Workbench in FreeCAD
- Successfully finished building my first xyz cube in FreeCAD
Mon May 29, 2017
- Began reading freecad documentation, watching videos on, & playing with freecad
- Began MarthamEngineering's FreeCAD Tutorials youtube video series, starting with [part 1]