Maltfield Log/2018 Q1: Difference between revisions
Jump to navigation
Jump to search
| Line 195: | Line 195: | ||
=Sun Mar 04, 2018= | =Sun Mar 04, 2018= | ||
# after enabling caching of 404s, the "Hit rates" graph of varnish4 in munin changed--the "cache misses" shrunk significantly. Now I just need to find & reduce the "Cache hits for pass" | # after enabling caching of 404s, the "Hit rates" graph of varnish4 in munin changed--the "cache misses" shrunk significantly. Now I just need to find & reduce the "Cache hits for pass" | ||
[[File:Munin varnishHitRate 20180315.png]] | |||
# began testing why a request for 'https://www.opensourceecology.org/robots.txt' produces a HIT-FOR-PASS | # began testing why a request for 'https://www.opensourceecology.org/robots.txt' produces a HIT-FOR-PASS | ||
<pre> | <pre> | ||
Revision as of 13:40, 15 March 2018
My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
See Also
Tue Mar 06, 2018
- got an ossec alert for brute-force attacks on wp-login.php
- this shouldn't happen, since we're just 403-forbidden blocking this page
- discovered that the httpd config for this block was commented-out in oswh; fixed
- this shouldn't happen, since we're just 403-forbidden blocking this page
- disk usage is higher than I'd like, hovering at 85%
- installed `ncdu` = an ncurses disk usage analyzer
- there's more usage in /var/tmp (61G) than in /var/www (37G)
- most of the space is in /var/tmp/backups_for_migration_from_hetzner1 (58G)
- removed the contents of /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/old/20180214 (30G)
[maltfield@hetzner2 20180214]$ pwd /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/old/20180214 [maltfield@hetzner2 20180214]$ du -sh * 296M core 1.4G db.sql 17G htdocs 8.0K imagesLargerThan20M.txt 191M mysqldump_wiki.20180120.sql.bz2 12G wiki_files.20180120.tar.gz [maltfield@hetzner2 20180214]$ du -sh 30G . [maltfield@hetzner2 20180214]$
- continuing wiki varnish config
- found that mediawiki is producing headers in its response that would prevent varnish from caching the main page!
[maltfield@hetzner2 ~]$ curl -I "http://127.0.0.1:8000/wiki/Main_Page" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Tue, 06 Mar 2018 16:41:16 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Tue, 06 Mar 2018 16:34:44 GMT X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520354476.8255; expires=Tue, 06-Mar-2018 16:42:16 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [maltfield@hetzner2 ~]$
- the recommended/documented varnish config on mediawiki's wiki shows the block respecting any of the "private|no-cache|no-store" Cache-Control headers sent by mediawiki commented-out in order to ignore mediawiki. https://www.mediawiki.org/wiki/Manual:Varnish_caching#Configuring_Varnish_4.x
In this example, the 'no-cache' flag is being ignored on pages served to anonymous-IP users. Such measures normally are only needed if a wiki is making extensive use of extensions which add this flag indiscriminately (such as a wiki packed with random <choose>/<option> Algorithm tags on the main page and various often-used templates).
# Called after a document has been successfully retrieved from the backend.
sub vcl_backend_response {
# set minimum timeouts to auto-discard stored objects
set beresp.grace = 120s;
if (beresp.ttl < 48h) {
set beresp.ttl = 48h;
}
if (!beresp.ttl > 0s) {
set beresp.uncacheable = true;
return (deliver);
}
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
return (deliver);
}
# if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
# set beresp.uncacheable = true;
# return (deliver);
# }
if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") {
set beresp.uncacheable = true;
return (deliver);
}
return (deliver);
}
- this issue was exactly what I just encountered on osmain, but in wordpress! There was a plugin named 'fundraising' that was preventing caching. Apparently there's an extension in mediawiki doing something similar.
- found a list of all extensions currently installed: https://wiki.opensourceecology.org/wiki/Special:Version
- disabled all extensions & confirmed that the issue was still present
- note that there's no easy way to do this; you have to just comment-out all the require & wfLoadExtension function calls in LocalSettings.php until Special:Version shows no extensions "installed"
[maltfield@hetzner2 ~]$ curl -I "http://127.0.0.1:8000/wiki/Main_Page?safemode=1" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Wed, 07 Mar 2018 00:19:58 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Wed, 07 Mar 2018 00:18:56 GMT X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520381998.2197; expires=Wed, 07-Mar-2018 00:20:58 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [maltfield@hetzner2 ~]$
- fuck, even when I comment-out all the skins (breaking the site), the issue is still present
[maltfield@hetzner2 ~]$ curl -I "http://127.0.0.1:8000/wiki/Main_Page?safemode=1" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Wed, 07 Mar 2018 00:22:42 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520382163.0842; expires=Wed, 07-Mar-2018 00:23:43 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [maltfield@hetzner2 ~]$
- found a list of all the files that have 'cache-control' in them in the wiki dir
[root@hetzner2 wiki.opensourceecology.org]# grep -irl 'cache-control' * htdocs/opensearch_desc.php htdocs/includes/actions/RawAction.php htdocs/includes/OutputPage.php htdocs/includes/AjaxResponse.php htdocs/includes/api/ApiMain.php htdocs/includes/api/i18n/ru.json htdocs/includes/api/i18n/de.jsone htdocs/includes/api/i18n/ru.jsone htdocs/includes/api/i18n/ba.json htdocs/includes/api/i18n/de.json htdocs/includes/api/i18n/ba.jsone htdocs/includes/OutputPage.phpee htdocs/includes/specials/SpecialRevisiondelete.php htdocs/includes/specials/SpecialUndelete.php htdocs/includes/specials/SpecialUploadStash.php htdocs/includes/specials/SpecialJavaScriptTest.php htdocs/includes/HeaderCallback.php htdocs/includes/resourceloader/ResourceLoader.php htdocs/includes/libs/filebackend/HTTPFileStreamer.php htdocs/includes/WebStart.php htdocs/includes/PHPVersionCheck.php htdocs/includes/OutputHandler.php htdocs/thumb.php htdocs/img_auth.php htdocs/extensions/ConfirmEdit/FancyCaptcha/FancyCaptcha.class.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/ConfirmAccount_body.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/UserCredentials_body.php wiki-error.log [root@hetzner2 wiki.opensourceecology.org]#
- found the possible sources
[root@hetzner2 wiki.opensourceecology.org]# for file in $(echo $files); do grep -il 'no-cache, no-store, max-age=0, must-revalidate' $file; done htdocs/includes/OutputPage.php htdocs/includes/OutputPage.phpee htdocs/includes/specials/SpecialRevisiondelete.php htdocs/includes/specials/SpecialUndelete.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/ConfirmAccount_body.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/UserCredentials_body.php [root@hetzner2 wiki.opensourceecology.org]#
- OutputPage appears to default to using the cache. The code that sets our undesired header is encapsulated in an if whoose condition is triggered when 'mEnableClientCache' is not true. It defaults to true
Mon Mar 05, 2018
- Updated log & hours
- checked munin, all looks good.
- removed the temp comments in front of the block to deny access to 'wp-login.php' from osemain.
- also noticed that (in addition to the blue "cache misses" graph nearly disappearing) the varnish4 "backend traffic" graph shows a great decrease in backend connections, especially the blue = "success", following my varnish change to cache backend responses with code = 404 not found
- began researching varnish integration with mediawiki
- this great article describes how Mediawiki's core was designed to work with Varnish specifically https://www.mediawiki.org/wiki/Manual:Varnish_caching#cite_note-2
- this is less of an issue for our wiki because we don't allow anonymous edits, but I added these lines to LocalSettings.php to record ip addresses using the "X-Forwarded-For" header instead of the Source IP, which would just be 127.0.0.1 with our varnish->apache architecture
$wgUseSquid = true;
$wgSquidServers = array( '127.0.0.1', 'wiki.opensourceecology.org' );
$wgUsePrivateIPs = true;
//Use $wgSquidServersNoPurge if you don't want MediaWiki to purge modified pages
//$wgSquidServersNoPurge = array('127.0.0.1');
- found a list of squid (relevant to varnish) related options https://www.mediawiki.org/wiki/Manual:Configuration_settings#Squid
Sun Mar 04, 2018
- after enabling caching of 404s, the "Hit rates" graph of varnish4 in munin changed--the "cache misses" shrunk significantly. Now I just need to find & reduce the "Cache hits for pass"
- began testing why a request for 'https://www.opensourceecology.org/robots.txt' produces a HIT-FOR-PASS
root@personal:~# curl -i https://www.opensourceecology.org/robots.txt HTTP/1.1 200 OK Server: nginx Date: Sun, 04 Mar 2018 19:10:42 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 182 Connection: keep-alive Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <https://www.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=ee8abcqulnd7f8e9jbv1gve33cht0fuohmje7bps0iokihfhnah4c5ib4r95vpv9muq2ccqms2vhjf469rfj75gv1bik423vkobovv3; path=/; HttpOnly;HttpOnly X-Varnish: 2109565 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" Sitemap: https://www.opensourceecology.org/sitemap.xml Sitemap: https://www.opensourceecology.org/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php root@personal:~#
- strangely, this file doesn't even exist!
[root@hetzner2 ~]# find /var/www/html/www.opensourceecology.org | grep -i robots.txt [root@hetzner2 ~]#
- it also works if I just hit apache directly, locally
[root@hetzner2 ~]# curl -i 'http://127.0.0.1:8000/robots.txt' -H 'Host: www.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 19:14:38 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <https://www.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=bear8i55veadv2osgrpsg4m6t45n4r1c1gdcveomhbftk26okrgvjmeafge1poj5a0kkrakqn5ghbb7s4r0bqmiic3o9l5djpu5nmc3; path=/; HttpOnly;HttpOnly Content-Length: 182 Content-Type: text/plain; charset=utf-8 Sitemap: https://www.opensourceecology.org/sitemap.xml Sitemap: https://www.opensourceecology.org/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php [root@hetzner2 ~]#
- indeed, this behaviour differs from fef, another one of our fef wordpress sites. note that the "Expires" & "Cache-Control" headers are absent in Apache's response
[root@hetzner2 ~]# curl -i 'http://127.0.0.1:8000/robots.txt' -H 'Host: fef.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 19:19:56 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <https://fef.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Content-Length: 67 Content-Type: text/plain; charset=utf-8 User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
- I'm beginning to think that it's one of the osemain plugins causing this issue
- it appears that there's some sort of apache-level (or os-level) caching going on too. note that the first request below takes 10 fucking seconds. the second takes less than 1 second.
[root@hetzner2 ~]# time curl -I 'http://127.0.0.1:8000/apple-touch-icon.png' -H 'Host: fef.opensourceecology.org' HTTP/1.1 404 Not Found Date: Sun, 04 Mar 2018 20:34:18 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://fef.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m10.150s user 0m0.002s sys 0m0.001s [root@hetzner2 ~]# time curl -I 'http://127.0.0.1:8000/apple-touch-icon.png' -H 'Host: fef.opensourceecology.org' HTTP/1.1 404 Not Found Date: Sun, 04 Mar 2018 20:34:35 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://fef.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m0.125s user 0m0.002s sys 0m0.001s [root@hetzner2 ~]#
- if I wait a bit (I've observed this after 3 minutes) & re-run the above, it takes another 10 seconds.
- it looks like the "expires" & "cache-control" headers are set by wp natively (or, at most, a plugin calling this native function) using wp_get_nocache_headers and related functions https://developer.wordpress.org/reference/functions/wp_get_nocache_headers/
- wp-includes/class-wp.php appears to applying the above function for 404s, that's annoying (ie: we want to cache 404s for requests that are slamming our site constantly, such as '/apple-touch-icon.png' https://developer.wordpress.org/reference/classes/wp/handle_404/
- removed google analytics code from osemain via Appearance -> Theme Options -> Footer
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-58526017-1', 'auto');
ga('send', 'pageview');
</script>
<pre>
# created a staging site for osemain so I could tear down plugins & themes to isolate the cause of the call to nocache_headers()
## renamed 'osemain' A record on CF DNS to 'staging' for staging.opensourceecology.org
## created /etc/httpd/conf.d/staging.opensourceecology.org.conf
## created staging vhost files & db. note that the max char limit for a db username is 16, so I used "osemain_s_user" & "osemain_s_db"
<pre>
source /root/backups/backup.settings
prodVhostDir=/var/www/html/www.opensourceecology.org
stagingVhostDir=/var/www/html/staging.opensourceecology.org
prodDbName=osemain_db
stagingDbName=osemain_s_db
stagingDbUser=osemain_s_user
stagingDbPass=CHANGEME
- updated the wp-config.php to be http, not https, so I can just query with curl (otherwise it 301's to https)
- successfully reproduced the issue on the staging site
[root@hetzner2 staging.opensourceecology.org]# time curl -I 'http://127.0.0.1:8000/' -H "Host: staging.opensourceecology.org" HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:45:23 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=b3kuv36bt8fi8j9j07heh3a8epvug5fdq4go0ajrgsv4gq4aevvpi2v5ms212o68pdq4tkrjfde79274bbus3nvs9fvam273i6vohg0; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m10.331s user 0m0.003s sys 0m0.000s [root@hetzner2 staging.opensourceecology.org]# time curl -I 'http://127.0.0.1:8000/' -H "Host: staging.opensourceecology.org" HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:45:35 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=p9gdj5vsrhh18t0ahlv60f68eq35slum7bjv3vb98v3rbplifj9a0e2lpqhh67lsdmi6h1d1krj2pah0ja0vaa8gvrgeokqkp0sha72; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m0.315s user 0m0.002s sys 0m0.001s [root@hetzner2 staging.opensourceecology.org]#
- I removed the plugins directory in the staging docroot & replaced it with an empty dir, and confirmed that the issue went away!
[root@hetzner2 wp-content]# mv plugins plugins.old [root@hetzner2 wp-content]# mkdir plugins [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:49:00 GMT Server: Apache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m12.423s user 0m0.002s sys 0m0.002s [root@hetzner2 wp-content]#
- I copied in my base plugin set from the old plugins dir, and confirmed that the headers look ok still
[root@hetzner2 wp-content]# ls plugins force-strong-passwords google-authenticator google-authenticator-encourage-user-activation rename-wp-login ssl-insecure-content-fixer vcaching [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:52:36 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m13.186s user 0m0.002s sys 0m0.002s [root@hetzner2 wp-content]#
- I iteratively copied-in the plugins one-by-one that were activated on the prod site, until I found the issue crop-up after I added the "fundraising" plugin
[root@hetzner2 wp-content]# rsync -a plugins.old/akismet/ plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:55:27 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m10.166s user 0m0.002s sys 0m0.001s [root@hetzner2 wp-content]# rsync -a plugins.old/brankic-photostream-widget plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:55:45 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m0.134s user 0m0.003s sys 0m0.000s [root@hetzner2 wp-content]# rsync -a plugins.old/duplicate-post plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:56:02 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m0.133s user 0m0.001s sys 0m0.002s [root@hetzner2 wp-content]# rsync -a plugins.old/fundraising plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:56:17 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=d679j00fdalqp4usqe0iuf71vs6jhl29bbdds5e2n56i0p15m1379vnnlj2c1hlunjn061l6rtkij0141mpsvagnei6g8anihk842i2; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m0.144s user 0m0.001s sys 0m0.002s
- I replaced the temp plugins dir with the original dir & confirmed the issue was present
root@hetzner2 wp-content]# rm -rf plugins [root@hetzner2 wp-content]# mv plugins.old plugins [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:58:51 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=uvjm8sqbdl1uqbmrr27g5qns54sg0enk177m1efqpuournqneir1qtruoqdu6ph3go9vqflrpg58gq1hrn09hoqrrs4a6ttfeukcgs3; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m11.097s user 0m0.001s sys 0m0.002s
- I moved out the 'fundraising' plugin, and confirmed that the issue was fixed by this one change
[root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:58:51 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=uvjm8sqbdl1uqbmrr27g5qns54sg0enk177m1efqpuournqneir1qtruoqdu6ph3go9vqflrpg58gq1hrn09hoqrrs4a6ttfeukcgs3; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m11.097s user 0m0.001s sys 0m0.002s
- I added staging.opensourceecology.org vhost configs to nginx & varnish
- updated cert to include the staging SAN
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/staging.opensourceecology.org/htdocs -d staging.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org -d munin.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- added '950120' [Possible Remote File Inclusion (RFI) Attack: Off-Domain Reference/Link] to the modsec ignore list in /var/ossec/rules/local_rules.xml as it was too spammy
- updated Wordpress documentation for file permissions & wp-cli troubleshooting per what was discovered during the osemain migration (for plugins, themes, & upgrade dirs)
- discovered that the $wgDebugComments variable puts the the mediawiki debug messages (which may contain sensitive user or server info!) directly in the html output! https://www.mediawiki.org/wiki/Manual:Configuration_settings#Debug/logging
- removed this from LocalSettings.php & added a strong warning message telling admins to *only* use $wgDebugLogFile, which exists *outside* the docroot
- added rules to ossec to detect & active response (block via iptables) brute force attempts over mediawiki
<!-- Mediawiki Special:UserLogin brute force --> <rule id="100060" level="3"> <if_sid>31530</if_sid> <url>Special:UserLogin</url> <regex>POST</regex> <description>Mediawiki login attempt.</description> </rule> <!-- If we see frequent mediawiki login POST's, it is likely a bot. --> <rule id="100061" level="8" frequency="6" timeframe="30"> <if_matched_sid>100060</if_matched_sid> <same_source_ip /> <descript
- tested the above rules & verified that I was banned via iptables when I kicked off a bunch of simultanious login attempts with curl
while true; do date; curl --data "wpName=superman&wpPassword1=letmein" "https://wiki.opensourceecology.org/index.php?title=Special:UserLogin"; echo; done
- note that I had to execute this in many terminals at once, as the request<-->response time was too long to actually trigger the ban
- I'm now finished with the general hardening of mediawiki. Next up: varnish integration (and hardening of *that* config)
Sat Mar 03, 2018
- all munin graphs look great. I'll let it populate for 2 weeks before delivering to Marcin.
- updated StatusCake for osemain from 'http://opensourceecology.org' to 'https://www.opensourceecology.org/'
- saw that, indeed, statuscake shows that the site sped up significantly post migration. Before, our load times averaged ~1.3 seconds. Now it's showing ~0.3 seconds
- munin is showing about a 45% hit rate. that's lower than I would expect.
- I found a command to get a list of the cache misses
[root@hetzner2 ~]# varnishtop -i BereqURL list length 38 hetzner2.opensourceecology.org 3.49 BereqURL /apple-touch-icon.png 3.45 BereqURL /apple-touch-icon-precomposed.png 2.31 BereqURL /favicon.png 2.28 BereqURL /apple-touch-icon-120x120.png 2.24 BereqURL /apple-touch-icon-120x120-precomposed.png 1.43 BereqURL /apple-touch-icon-152x152.png 1.40 BereqURL /apple-touch-icon-152x152-precomposed.png 0.57 BereqURL /category/proposals/feed/ 0.48 BereqURL /robots.txt 0.46 BereqURL /localhost/localhost/varnish_expunge-day.png 0.46 BereqURL /static/logo-h.png 0.46 BereqURL /localhost/localhost/varnish_bad-day.png 0.46 BereqURL /localhost/localhost/varnish_uptime-day.png 0.46 BereqURL /localhost/localhost/varnish_objects-day.png 0.46 BereqURL /localhost/localhost/varnish_threads-day.png 0.46 BereqURL /localhost/localhost/varnish_memory_usage-day.png 0.46 BereqURL /localhost/localhost/varnish_request_rate-day.png 0.46 BereqURL /localhost/localhost/varnish_transfer_rates-day.png 0.46 BereqURL /localhost/localhost/varnish_hit_rate-day.png 0.46 BereqURL /static/style-new.css 0.46 BereqURL /localhost/localhost/varnish_backend_traffic-day.png 0.46 BereqURL /varnish4-day.html 0.46 BereqURL /category/ted-fellows/ 0.38 BereqURL / ... [root@hetzner2 ~]#
- the above output shows "BereqURL" occurrences in `varnishlog` = backend requests = cache misses
- We see that the '/apple-touch-icon.png' file is requested about 3 times per minute. It's actually just a 404. This shows that it's not caching this 404, which would be a trivial saver
- We also see that the '/' is being requested ~0.38 times per second. So about every 3 seconds, varnish is troubling apache for the main page.
- why? I want this cached for 24 hours. Looks like we could be better optimized
- dug back in my logs to 2017-11-24, when I configured wordpress' apache no what to cache by response code
# Avoid caching error responses
#if (beresp.status == 404 || beresp.status >= 500) {
if ( beresp.status != 200 && beresp.status != 203 && beresp.status != 300 && beresp.status != 301 && beresp.status != 302 && beresp.status != 304 && beresp.status != 307 && beresp.status != 410 ) {
set beresp.ttl = 0s;
set beresp.grace = 15s;
}
- according to the (still outstanding) bug report I filed 3 months ago, I didn't cache 404 only because the developer of the worpdress plugin didn't cache 404. https://wordpress.org/support/topic/please-dont-cache-403-by-default/
- but varnish, by default, *does* cache 404 https://book.varnish-software.com/4.0/chapters/VCL_Basics.html#the-initial-value-of-beresp-ttl
- therefore, I think I *should* cache 404. Or at least experiment with it.
- I changed it on prod for www.opensourceecology.org to cache 404s too
# Avoid caching error responses
#if (beresp.status == 404 || beresp.status >= 500) {
if ( beresp.status != 200 && beresp.status != 203 && beresp.status != 300 && beresp.status != 301 && beresp.status != 302 && beresp.status != 304 && beresp.status != 307 && beresp.status != 410 && beresp.status != 404 ) {
set beresp.ttl = 0s;
set beresp.grace = 15s;
}
- I noticed that varnishlog shows some HITs with an already-expired header
- ReqURL / - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate, br - ReqHeader Accept-Encoding: gzip - VCL_call HASH - ReqHeader hash: #www.opensourceecology.org - VCL_return lookup - Hit 83893 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Sat, 03 Mar 2018 19:52:48 GMT - RespHeader Server: Apache - RespHeader X-VC-Enabled: true - RespHeader X-VC-TTL: 86400 - RespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT - RespHeader Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 - RespHeader Pragma: no-cache
- why was this "Expires: Thu, 19 Nov 1981 08:52:00 GMT"?
- why was this "Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0"?
- why was this "Pragma: no-cache"?
- a much simpler test is '/robots.txt'. Why is this not caching?
Fri Mar 02, 2018
- Marcin followed-up on osemain
- he said it's running much faster :)
- he found a modesec false-positive when attempting to modify https://www.opensourceecology.org/open-building-institute-nears-launch/
- I whitelisted 959070, sqli
- began doing some ossec tuning
- after osemain migrated to hetzner2, the alert volume has become unreasonably high, so I want to prevent alerts from emailing in a few cases
- rule 30411 = modsecurity where the following strings match
- 960020 = Pragma Header requires Cache-Control Header
- 960009 = Request Missing a User Agent Header
- rule 31123 = Web server 503 error code (Service unavailable)
- but we'll not disable 31163 = Multiple web server 503 error code (Service unavailable).
- rule 30411 = modsecurity where the following strings match
- after osemain migrated to hetzner2, the alert volume has become unreasonably high, so I want to prevent alerts from emailing in a few cases
- returned to configuring cacti
- created nginx, varnish, & http config files
- created log dirs for nginx & apache
- created cacti.opensourceecology.org DNS entry pointing to 138.201.84.243 on CloudFlare
- update /etc/php.ini to include the cacti dir
- created cacti db & user per https://skrinhitam.wordpress.com/2017/04/21/how-to-install-cacti-on-centos-7/
CREATE DATABASE cacti_db; CREATE USER 'cacti_user'@'localhost' IDENTIFIED BY 'CHANGEME'; GRANT ALL PRIVILEGES ON cacti_db.* TO 'cacti_user'@'localhost'; FLUSH PRIVILEGES;
- updated /usr/share/cacti/include/confgi.php with credentials
- confirmed that we're running cacti v1.1.36
[root@hetzner2 cacti]# cat /usr/share/cacti/include/cacti_version 1.1.36 [root@hetzner2 cacti]#
- according to the website, the latest version is 1.1.36 from 2018-02-25. So we're latest, and it was updated within a month ago. That's great. https://www.cacti.net/download_cacti.php
- finally got it connected; sockets was nontrivial; you actually have to define the port as 3306, though it's a misnomer (misnumber?)
$database_type = 'mysql'; $database_default = 'cacti_db'; $database_hostname = 'localhost'; $database_username = 'cacti_user'; $database_password = 'CHANGEME'; $database_port = '3306';
- this time I got an error that the db was not initialized (it's true; there's no tables!), but the sql file they're telling me to use doesn't exist!
The Cacti Database has not been initialized. Please initilize it before continuing. To initilize the Cacti database, issue the following commands either as root or using a valid account. mysqladmin -uroot -p create cacti mysql -uroot -p -e "grant all on cacti.* to 'someuser'@'localhost' identified by 'somepassword'" mysql -uroot -p -e "grant select on mysql.time_zone_name to 'someuser'@'localhost' identified by 'somepassword'" mysql -uroot -p cacti < /pathcacti/cacti.sql Where /pathcacti/ is the path to your Cacti install location. Change someuser and somepassword to match your site preferences. The defaults are cactiuser for both user and password. NOTE: When installing a remote poller, the config.php file must be writable by the Web Server account, and must include valid connection information to the main Cacti server. The file should be changed to read only after the install is completed.
- ah, the distro put it into /usr/share/doc/cacti-1.136/cacti.sql ..of course (?)
[root@hetzner2 cacti]# rpm -ql cacti | grep cacti.sql /usr/share/doc/cacti-1.1.36/cacti.sql [root@hetzner2 cacti]#
- I initialized the db with this file
mysql -u"cacti_user" -p"CHANGEME" cacti_db < /usr/share/doc/cacti-1.1.36/cacti.sql
- set `date.timezone = "UTC"` in /etc/php.ini
- I had to comment-out the default config's block that blocks all non-localhost traffic (my .htpasswd config should suffice)
<Directory /usr/share/cacti/> # <IfModule mod_authz_core.c> # # httpd 2.4 # Require host localhost # </IfModule> </Directory>
- granted the new cacti user SELECT permissions into the mysql.time_zone table
GRANT SELECT ON mysql.time_zone_name TO cacti_user@localhost; flush privileges;
- populated the mysql timezone database, following a backup
sudo su -
source /root/backups/backup.settings
stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
service httpd stop
# create backup of all DBs for good measure
time nice mysqldump -u"root" -p"${mysqlPass}" --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz
service httpd start
mysql_tzinfo_to_sql /usr/share/zoneinfo/ | mysql -u"root" -p"${mysqlPass}" mysql
- expanded the cert to include cacti
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org -d cacti.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- Cacti was very unhappy with our mariadb settings. I'm not concerned as this is just for 1 server, and I don't want to fuck with the db backend which is prod on so many sites. But I'll leave this here for records
MariaDB Tuning (/etc/my.cnf) - [ Documentation ] Note: Many changes below require a database restart Variable Current Value Recommended Value Comments version 5.5.56-MariaDB >= 5.6 MySQL 5.6+ and MariaDB 10.0+ are great releases, and are very good versions to choose. Make sure you run the very latest release though which fixes a long standing low level networking issue that was causing spine many issues with reliability. collation_server latin1_swedish_ci utf8_general_ci When using Cacti with languages other than English, it is important to use the utf8_general_ci collation type as some characters take more than a single byte. If you are first just now installing Cacti, stop, make the changes and start over again. If your Cacti has been running and is in production, see the internet for instructions on converting your databases and tables if you plan on supporting other languages. character_set_client latin1 utf8 When using Cacti with languages other than English, it is important to use the utf8 character set as some characters take more than a single byte. If you are first just now installing Cacti, stop, make the changes and start over again. If your Cacti has been running and is in production, see the internet for instructions on converting your databases and tables if you plan on supporting other languages. max_connections 151 >= 100 Depending on the number of logins and use of spine data collector, MariaDB will need many connections. The calculation for spine is: total_connections = total_processes * (total_threads + script_servers + 1), then you must leave headroom for user connections, which will change depending on the number of concurrent login accounts. max_allowed_packet 1048576 >= 16777216 With Remote polling capabilities, large amounts of data will be synced from the main server to the remote pollers. Therefore, keep this value at or above 16M. tmp_table_size 16M >= 64M When executing subqueries, having a larger temporary table size, keep those temporary tables in memory. join_buffer_size 0.125M >= 64M When performing joins, if they are below this size, they will be kept in memory and never written to a temporary file. innodb_file_per_table OFF ON When using InnoDB storage it is important to keep your table spaces separate. This makes managing the tables simpler for long time users of MariaDB. If you are running with this currently off, you can migrate to the per file storage by enabling the feature, and then running an alter statement on all InnoDB tables. innodb_doublewrite ON OFF With modern SSD type storage, this operation actually degrades the disk more rapidly and adds a 50% overhead on all write operations. innodb_additional_mem_pool_size 8M >= 80M This is where metadata is stored. If you had a lot of tables, it would be useful to increase this. innodb_lock_wait_timeout 50 >= 50 Rogue queries should not for the database to go offline to others. Kill these queries before they kill your system. innodb_flush_log_at_trx_commit 1 2 Setting this value to 2 means that you will flush all transactions every second rather than at commit. This allows MariaDB to perform writing less often.
- I setup /etc/snmpd/snmpd.conf with a basic config
- I got cacti to add the device & connect to it
- I couldn't get cacti to add a graph. debugging the apache logs show that it was trying to execute spine, which isn't installed
- I uncommented the poller.php defined in
- the poller was pretty unhappy about our hardened php.ini
[root@hetzner2 snmp]# sudo -u cacti /usr/share/cacti/poller.php PHP Warning: ini_set() has been disabled for security reasons in /usr/share/cacti/include/global.php on line 65 PHP Warning: file_exists(): open_basedir restriction in effect. File(localhost) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/lib/database.php on line 65 PHP Warning: file_exists(): open_basedir restriction in effect. File(/usr/local/spine/bin/spine) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/include/global_arrays.php on line 560 PHP Warning: php_uname() has been disabled for security reasons in /usr/share/cacti/poller.php on line 79 03/02/2018 21:21:41 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 1.168,2 seconds have passed since the last poll! PHP Warning: ini_set() has been disabled for security reasons in /usr/share/cacti/poller.php on line 256 PHP Warning: ini_set() has been disabled for security reasons in /usr/share/cacti/poller.php on line 257 PHP Warning: file_exists(): open_basedir restriction in effect. File(/bin/php) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/lib/poller.php on line 124 PHP Warning: system() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 143 PHP Warning: exec() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 131 PHP Warning: file_exists(): open_basedir restriction in effect. File(/bin/php) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/lib/poller.php on line 124 PHP Warning: system() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 143 PHP Warning: exec() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 131 PHP Warning: putenv() has been disabled for security reasons in /usr/share/cacti/lib/rrd.php on line 55 PHP Warning: popen() has been disabled for security reasons in /usr/share/cacti/lib/rrd.php on line 97 ^C03/02/2018 21:23:11 - POLLER: Poller[1] WARNING: Cacti Master Poller process terminated by user
- therefore, I think we need spine
- spine is not in the yum repos :(
- this isn't looking good https://forums.cacti.net/post-91042.html
- ok, I'm abandoning Cacti; let's try Munin. It emphasizes that it works well OOTB, and it appears to run perl on the back-end, only producing static html pages. That's the most ideal. Just give us a whole docroot that is totally static content!
- began attempting to get munin working
- I installed it with `yum install munin` and it automatically created a dir with static files in /var/www/html/munin
- I started the service `service munin-node start`
- It works! That. Was. Fucking. Awesome. Now I just need to wait 30 minutes per https://www.tecmint.com/install-munin-network-monitoring-in-rhel-centos-fedora/
- Holy hell, it includes varnish stats ootb!! What is this awesomeness?!?
- no going back; I'm renaming all the 'cacti' configs 'munin'
- changed the cert SAN from cacti to munin
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org -d munin.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- varnish graphs weren't populating; I think it's because we need the varnish4 plugin https://blog.raika.fr/index.php/2015/06/14/varnish-cache-4-et-munin-node/
pushd /etc/munin/plugins wget https://raw.githubusercontent.com/munin-monitoring/contrib/master/plugins/varnish/varnish4_ chmod +x varnish4_ mv varnish_hit_rate varnish_hit_rate.bak mv varnish_backend_traffic varnish_backend_traffic.bak mv varnish_bad varnish_bad.bak mv varnish_expunge varnish_expunge.bak mv varnish_memory_usage varnish_memory_usage.bak mv varnish_objects varnish_objects.bak mv varnish_request_rate varnish_request_rate.bak mv varnish_threads varnish_threads.bak mv varnish_transfer_rates varnish_transfer_rates.bak mv varnish_uptime varnish_uptime.bak ln -s varnish4_ varnish_hit_rate ln -s varnish4_ varnish_backend_traffic ln -s varnish4_ varnish_bad ln -s varnish4_ varnish_expunge ln -s varnish4_ varnish_memory_usage ln -s varnish4_ varnish_objects ln -s varnish4_ varnish_request_rate ln -s varnish4_ varnish_threads ln -s varnish4_ varnish_transfer_rates ln -s varnish4_ varnish_uptime ln -s varnish4_ varnish_hit_rate cat >> /etc/munin/plugin-conf.d/munin-node << EOF [varnish4_*] user root env.varnishstat /bin/varnishstat EOF munin-node-configure --shell service munin-node restart
- test shows it's working for hit_rate
[root@hetzner2 plugins]# munin-run varnish_hit_rate Line is not well formed (env.name) at /usr/share/perl5/vendor_perl/Munin/Node/Service.pm line 110. at /etc/munin/plugin-conf.d/munin-node line 10. Skipping the rest of the file at /usr/share/perl5/vendor_perl/Munin/Node/Service.pm line 110. client_req.value 726315 cache_miss.value 220489 cache_hitpass.value 17813 cache_hit.value 488013 [root@hetzner2 plugins]#
- confirmed that data is coming in, but under the category "webserver" instead of "varnish". I see no obvious way to change that *shrug*
Thr Mar 01, 2018
- began to prepare for my production migration of the osemain site (www.opensourceecology.org)
- because the domain name is shared by both osemain & the wiki (as opposed to having them on distinct domains or subdomains), the DNS cutover may cause issues with the wiki
- currently, the naked domain name 'opensourceecology.org' points to '78.46.3.178' = hetzner1
- currently, 'www' is a cloud-fronted CNAME to the above naked domain name 'opensourceecology.org'
- I will be deleting the above 'www' CNAME & replacing it with a non-CF'd A record pointing to '138.201.84.243' = hetzner2
- therefore, wiki requests made to 'opensourceecology.org/wiki/...' should be unimpacted. However, anything that links to the wiki at 'www.opensourceecology.org/wiki/...' will be impacted. This is a fairly minor concern that should be fixable with an (nginx?) redirect on hetzner2 for anything '/wiki/...', simply stripping-out the 'www'.
- there may be issues in the other direction, namely that someone wants to get our wordpress site by going to 'opensourceecology.org' but hits our wiki instead. I don't intend to fix this unless we see a ton of 404s or hear an outcry of issues.
- in any case, for good measure, I'll make the wiki read-only during the osemain migration by creating the file at hetzner1:/usr/home/osemain/public_html/w/read-only-message
- I did some research on how to put up a warning for the upcoming maintenance on the wiki as a banner on all pages, and I found this plugin--which is the same plugin that Wikipedia famously uses in their donation campaigns https://www.mediawiki.org/wiki/Extension:CentralNotice
- the above article actually recommended just using Sitenotice for a single wiki https://www.mediawiki.org/wiki/Manual:Interface/Sitenotice
- Sitenotice can be updated using the $wgSiteNotice variable in LocalSettings.php
- tested it on our stating site with success
$wgSiteNotice = '<div style="margin: 10px 0px; padding:12px; color: #9F6000; background-color: #FEEFB3;"><i class="fa fa-warning"></i>NOTICE: This wiki will be temporarily made READ-ONLY during a maintenance window today at 15:00 UTC. Please finish & save any pending changes to avoid data loss.</div>';
- applied the change to the production site at hetzner1. It worked!
- began the production migration of osemain CHG-2018-02-05
- first, I locked the prod wiki by creating this file
osemain@dedi978:~/public_html/w$ cat read-only-message The wiki is currently locked for maintenance. Please check back in a few hours. osemain@dedi978:~/public_html/w$
- I then began the full-system backups on both hetzner 1 & hetzner 2 simultaneously
# hetzner 1 # STEP 0: CREATE BACKUPS source /usr/home/osemain/backups/backup.settings /usr/home/osemain/backups/backup.sh # when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*' # hetzner 2 sudo su - # STEP 0: CREATE BACKUPS # for good measure, trigger a backup of the entire system's database & files: time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log # when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'
- 16:30: ugh, after 1.5 hours, the full backup was still running on hetzner1. I'll proceed anyway with the migration, but try to refrain from any removal of content from hetzner1 until after this backup completion is confirmed
- 16:31: began backup of osemain-specific db & files
- 15:33: osemain-specific backups complete
- confirmed that the system-level backups were done being made on hetzner2, but they were still pending scp to our storage server. eta on the web root file alone was over an hour, so I proceeded
- 15:36: scp'd the osemain backup files from hetzner1 to hetzner2. This completed in <1 min
- 15:38 finished polishing the db.sql dump & replaced the staging db with this fresh copy
- 15:40 finished creating new vhost files
- I got errors when attempting to update the plugins using wp-cli due to the hardened permissions. It's less-than-ideal, but I updated them to grant the apache group write permissions to 'wp-content/themes' & 'wp-content/plugins' as well as 'uploads'
- err, I actually needed to update the permissions in 'wp-content/upgrade'
- 16:04: finished wp-cli updates
- 16:05: destroyed the existing CNAME for 'www' on opensourceecology.org via CloudFlare & re-created it to point to '138.201.84.243' with a 2 min (the min) TTL.
- 16:11 confirmed the domain name change worked
- fixed /etc/nginx/conf.d/www.opensourceecology.org/conf to use 'www' instead of the temp subdomain 'osemain'
- fixed /etc/httpd/conf.d/000-www.opensourceecology.org.conf
- restarted both httpd & nginx
- fixed the wordpress domain in WP_HOME & WP_SITEURL in /var/www/html/www.opensourceecolgy.org/wp-config.php by replacing 'osemain' with 'www'
- got a cert warning from the browser; indeed, 'www' hasn't been added to the cert's SAN yet
- fixed the cert, replacing 'osemain' with 'www'
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- 16:18 successfully loaded the site without
- 16:19 confirmed that both system-wide backups haven't completed yet..
- confirmed that hitting 'opensourceecology.org' doesn't actually take you to the wiki--it still takes you to the old site. After the backups finish, I'll have to move the old wp site's content out of the docroot & install a redirect for queries to the naked root (opensourceecology.org/) to 301 redirect to 'https://www.opensourceecology.org'
- confirmed that the wiki was still accessible at 'http://opensourceecology.org/wiki/'
- confirmed that the wiki was *not* accessible at 'www.opensourceecology.org/wiki/' that now tries to access the wordpress site on hetzner2. I should add a redirect for this--at least for now
- 16:34 finished setting-up the new plugins in the wp dashboard
- when editing a post, I noticed a link pointing to 'blog.opensourceecology.org'. Checking CF, I see that was a CNAME to 'dedi978.your-server.de'. So I deleted this record & added one for 'blog' as a CNAME to 'www'
- 16:47 finished fixing content for knightlab iframes on: /contributors/, /community-true-fans/, & /history-timeline/
- 16:47 confirmed that the dns updated for blog.opensourceecology.org
- attempted to click the 'blog.' link on 'https://www.opensourceecology.org/community-true-fans/', but it resulted in a cert warning of course https://blog.opensourceecology.org/2009/01/towards-1000-global-villages-factor-e-live-distillations-part-1/
- added blog SAN to the cert
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -d blog.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- I got an error. I confirmed that let's encrypt supports CNAME, but perhaps I just need to wait longer for propagation https://security.stackexchange.com/questions/134354/why-does-acme-lets-encrypt-require-a-records-not-cname/134389
- 16:56 I confirmed that the system-wide backup on hetzner2 is complete (after 148 minutes), but it's still running on hetzner1
- 17:16 finished updating widgets from "text" to "custom html" on relevant workshops = /cnc-torch-table-workshop/, /eco-tractor_workshop/, /3d-printer-construction-set-workshop-3/
- re-tried the certbot renewal. failed again. actually, I think it's because nginx isn't directing it correctly
- updated the nginx & httpd configs with aliases for 'blog.opensourceecology.org'
- updated the varnish config to use a regex match
- restarted apache, varnish, and nginx
- that worked! the cert now has a SAN for 'blog.opensourceecology.org'
- attempted to click the 'blog.' link on 'https://www.opensourceecology.org/community-true-fans/'; this time it didn't result in a cert error, just a 404 *facepalm* https://blog.opensourceecology.org/2009/01/towards-1000-global-villages-factor-e-live-distillations-part-1/
- got the same on www, *shurg* https://www.opensourceecology.org/2009/01/towards-1000-global-villages-factor-e-live-distillations-part-1/
- 19:03 finished configuring nginx to point wiki requests sent to hetzner2 back to the prod wiki in /etc/nginx/conf.d/www.opensourceecology.org.conf
####################
# REDIRECT TO WIKI #
####################
# this is for backwards-compatibility; before ~ 2018-02, both the wiki and
# this site shared the same domain-name. So, just in case someone sends
# www.opensourceecology.org a query trying to find the wiki, let's send them
# to the right place..
location ~* '^/wiki/' {
# TODO: change this to 'https://wiki.opensourceecology.org' after the wiki
# has been migrated
return 301 http://opensourceecology.org$uri;
}
- 19:11 fixed the facebook iframe on the main page from 'http' to 'https'
- 19:13 the hetzner1 system backup is *still* not finished!
- 19:24 added a redirect from the opensour
# 2018-03-01: osemain was migrated to hetzner2 today; this rule is a catch-all
# to send non-wiki traffic to our hetzner2 server
RewriteCond %{REQUEST_URI} !(/wiki|/w) [NC]
RewriteRule ^(.*)$ https://www.opensourceecology.org/$1 [L,R=301,NC]
- confirmed that correctly requesting on the new www.opensourceecology.org site works https://www.opensourceecology.org/cnc-torch-table-workshop/
- confirmed that correctly requesting on the current wiki works http://opensourceecology.org/wiki/User:Maltfield
- confirmed that incorrectly requesting the wiki from the 'www' works https://www.opensourceecology.org/wiki/User:Maltfield
- confirmed that incorrectly requesting osemain from the naked domain (which is now just the wiki) works http://opensourceecology.org/cnc-torch-table-workshop/
- came back later in the evening & found the upload finished
- moved all of the old osemain wordpress files out of the docroot on on hetzner1:/usr/home/osemain/public_html into hetzner1:/usr/home/osemain/noBackup/deleteMeIn2019/osemain_olddocroot/
- fixed the image on /marcin-jakubowski/ to use 'https://www.opensourceecology.org/...' for Marcin's photo instead of 'http://opensourceecology.org/...'
- confirmed that clicking the 'update' button didn't clear varnish for the page
- confirmed that clicking the "Purge from Varnish" button successfully cleared the single varnish page
- replaced the "Site Logo Image" URL at Appearance -> Theme Options -> Site Logo from 'http://opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png' to 'https://www.opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png'
- first I had to fix a modsec false-positives
- 950001, sqli
- 973336, xss
- 958051, xss
- 973331, xss
- 973330, xss
- first I had to fix a modsec false-positives
- began researching rrdtool-like solution
- rrdtool vs mrtg vs cacti vs zabbix
- I'm leaning to cacti
- began installing cacti
- installed depends
yum install php56w-snmp net-snmp net-snmp-utils cacti
- updated the TODO section of OSE_Server#TODO
Tue Feb 27, 2018
- still no word back from Marcin on the resolution of all outstanding issues on osemain & the wiki
- returned to hardening mediawiki
- $vgVerifyMimeTypes was set to false in a block within LocalSettings.php:
# Workaround for bug in MediaWiki 1.16 # See http://www.mediawiki.org/wiki/Thread:Project:Support_desk/Error:_File_extension_does_not_match_MIME_type $wgVerifyMimeType = false; $wgStrictFileExtensions = false; $wgCheckFileExtensions = false; ##file extensions # Workaround for bug in MediaWiki 1.16 # See http://www.mediawiki.org/wiki/Thread:Project:Support_desk/Error:_File_extension_does_not_match_MIME_type $wgVerifyMimeType = false; $wgStrictFileExtensions = false; $wgCheckFileExtensions = false; $wgFileExtensions = array( 'png', 'gif', 'jpg', 'jpeg', 'ppt', 'pdf', 'psd', 'mp3','xls', 'swf', 'doc', 'odt', 'odc', 'odp', 'ods', 'odg', 'pod', 'mm', 'xls', 'mpp', 'svg', 'dxf', 'stp', 'blend', 'g', 'FCStd', 'dia', 'bz2', 'gz', 'tbz', 'tgz', '7z', 'xz'); # See also: http://www.mediawiki.org/wiki/Manual:$wgMimeTypeBlacklist
- I hardened this, replacing it with
################ # FILE UPLOADS # ################ # MIME $wgVerifyMimeType = true; # EXTENSIONS $wgStrictFileExtensions = true; $wgCheckFileExtensions = true; $wgFileExtensions = array( 'png', 'gif', 'jpg', 'jpeg', 'ppt', 'pdf', 'psd', 'mp3','xls', 'doc', 'odt', 'odc', 'odp', 'ods', 'odg', 'pod', 'mm', 'xls', 'mpp', 'svg', 'dxf', 'stp', 'blend', 'g', 'FCStd', 'dia', 'bz2', 'gz', 'tbz', 'tgz', '7z', 'xz');
- per the mediawiki guide on file uploads, we may still have issues with compressed archives (ie: zip), and require adding them to the wgTrustedMediaFormats array https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads#Configuring_file_types
- reading through all the upload-related variables https://www.mediawiki.org/wiki/Category:Upload_variables
- found the notion of a "Copy Upload" which means a subset of users can upload directly from a URL from a specified set of domains (ie: flickr) it uses the upload_by_url permission (AKA "Sideloading"_ https://www.mediawiki.org/wiki/Manual:$wgAllowCopyUploads
- found there's both an UploadWizard and a Upload dialog. The former is built-into mediawiki and the latter is a popular extension for uploading multiple files at once
- found a cool option for segregating images into dirs by hashing their name (for scaling past FS issues), but apparently we can't enable it after the first image has been uploaded; may need to be revisited if issues are encountered in the future https://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory
- found a good read describing the upload process https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads#Upload_directory
- found another list of upload-related variables https://www.mediawiki.org/wiki/Manual:Configuration_settings#Uploads
- did a deep clean-up of LocalSettings.php
- began researching 2FA for mediawiki
- this guide suggests that the main TOTP extension is OATHAuth https://www.mediawiki.org/wiki/Help:Two-factor_authentication
- added the OATHAuth extension to our meiawiki install
[root@hetzner2 extensions]# date Tue Feb 27 19:11:14 UTC 2018 [root@hetzner2 extensions]# pwd /var/www/html/wiki.opensourceecology.org/htdocs/extensions [root@hetzner2 extensions]# git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/OATHAuth.git Cloning into 'OATHAuth'... remote: Counting objects: 164, done remote: Finding sources: 100% (32/32) remote: Getting sizes: 100% (11/11) remote: Compressing objects: 100% (24018/24018) remote: Total 2210 (delta 13), reused 2200 (delta 12) Receiving objects: 100% (2210/2210), 405.31 KiB | 0 bytes/s, done. Resolving deltas: 100% (1561/1561), done. [root@hetzner2 extensions]#
- that failed as the extension said it required mediawiki >= v 1.31.0...but the latest stable version is 1.30!
==> wiki.opensourceecology.org/error_log <==
[Tue Feb 27 19:19:43.898430 2018] [:error] [pid 21700] [client 127.0.0.1:51702] PHP Fatal error: Uncaught exception 'Exception' with message 'OATHAuth is not compatible with the current MediaWiki core (version 1.30.0), it requires: >= 1.31.0.' in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php:261\nStack trace:\n#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php(148): ExtensionRegistry->readFromQueue(Array)\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(40): ExtensionRegistry->loadFromQueue()\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(114): require_once('/var/www/html/w...')\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(40): require('/var/www/html/w...')\n#4 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php on line 261
- sent an email to the author, Ryan Lane, asking where I can submit a bug report for the extension https://www.mediawiki.org/wiki/User:Ryan_lane
- err, I just noticed that his page lists that extension as "unmaintained" :(
- indeed, 2016-10 was the last update
- actually, that documentation is just out-of-date. The file was updated earlier this month on 2018-02-12 by Jayprakash12345 https://phabricator.wikimedia.org/rEOATd7bdefda31161ad680dd2a88993e3d17734c4bef
- the relevant bug is here https://phabricator.wikimedia.org/T187037
- I OAUTH approved the Phabricator app to access my mediawiki account and was successfully able to submit a reply to the dev that made the relevant breaking change on the relevant bug https://phabricator.wikimedia.org/T187037
- within the hour, Sam Reed (Reedy)--who works for the Wikimedia Foundation--responded stating that the documentation is incorrect, and that I should have instead checked-out the "REL1_30" branch during the clone https://en.wikipedia.org/wiki/User:Reedy
- attempted to download again
[root@hetzner2 extensions]# git clone -b REL1_30 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/OATHAuth.git Cloning into 'OATHAuth'... remote: Counting objects: 164, done remote: Finding sources: 100% (32/32) remote: Getting sizes: 100% (11/11) remote: Compressing objects: 100% (24018/24018) remote: Total 2210 (delta 13), reused 2200 (delta 12) Receiving objects: 100% (2210/2210), 405.31 KiB | 0 bytes/s, done. Resolving deltas: 100% (1561/1561), done. [root@hetzner2 extensions]#
- that fixed the error, but logging-in fails
Database error A database query error has occurred. This may indicate a bug in the software.[WpXJwYmXdCJvWua7xeiBhQAAAAA] 2018-02-27 21:12:33: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- attempted to run `php update.php` from the maintenance dir (as has fixed this previously), but I got a syntax error. Indeed, this was due to my faulty sed find-and-replace of ini_set calls
if (#ini_set( "memory_limit", $newLimit ) === false ) {
- fortunately, that appeared to be a one-time/first-pass issue due my earlier/failed regex arguments when testing the sed command. I removed the comment, then retried the final one that I actually documented, and it worked fine.
- I retried `php update.php, and it ran fine.
- I retried the login, and it also worked fine after the update.
- updated the documentation to include the logic to checkout the appropriate branch when checking out the extensions from git
- removed all git-installed extension dirs & downloaded them all again. verified that they all work. ran update.php, & successfully logged-in again
- went to my user settings & saw a new option to enable 2FA. did it successfully.
- logged-out, attempted to log back in. after entering my username & password, I was sent to another page asking for my TOTP token. I entered it, and I logged in successfully.
- I logged-out, and attempted to log back in again. This time I intentionally entered the wrong password, and it told me this without ever showing me the TOTP field.
- I attempted to login again. This time I used the correct password & an intentionally incorrect TOTP token. I was given an error "Verification failed" on the TOTP "Token" step. I entered the correct token, and I logged in successfully.
- note that I was not asked to type my password again after the totp token was entered incorrectly; that's kinda nice.
- begun researching how to require users (ie: all Administrators!) to use 2FA
- found a list of tasks for this extension https://phabricator.wikimedia.org/tag/mediawiki-extensions-oathauth/
- I set $wgSecureLogin to true in LocalSettings.php per https://phabricator.wikimedia.org/T55197
- found the specific task tracking this ask (forcing users to use 2FA). It was created over a year ago (2016-11) and last updated
Sun Feb 25, 2018
- updated the weekly meeting slides per Marcin's request
- Began investigating (9) Page Information = why the Page Information metadata article's "Display Title" field is blank
- Found relevant documentation on this field
- Discovered at least 1 page where it is *not* blank https://wiki.opensourceecology.org/wiki/CNC_Machine
- I deleted the entire contents of this page, and it still wasn't blank. So it wasn't the contents of the article that filled-in the field
- For comparison, Marcin's biography's article page's "Display title" field is empty https://wiki.opensourceecology.org/wiki/Marcin_Biography
- I tried adding 'DISPLAYTITLE:Marcin Biography' to the head of the article, but it remained blank
- I tried adding 'DISPLAYTITLE:Marcin Biography' to the tail of the article, and it filled-in!
- I removed the above, saved, and it went back to blank so something in the article appears to be blanking it out
- I removed the line 'OrigLang', and the field came back again. That appears to be the culprit.
- So this is the culprit https://wiki.opensourceecology.org/index.php?title=Template:OrigLang
- This was the work of MarkDilley, Venko, & Elifarley
- Confirmed that the spanish version of the article *does* have the filed properly set https://wiki.opensourceecology.org/wiki/Marcin_Biography/es
- I was not able to find an obvious solution to the OrigLang template, though I'm no expert in mediawiki templates
- Sent an email to Marcin describing the issue & asking if it has further implications and if it's a blocker for the migration
- Sent Marcin an email follow-up asking for validation that the (1b) Non-existent File redirects to front page issue has been fixed as well, following the varnish config update
- confirmed that awstats is working as expected for the fef & wiki sites
Sat Feb 24, 2018
- Marcin responded that he got past the 2FA login on osemain, but got a forbidden error when attempting to create new blog posts
- confirmed modsec issue
- whitelisted 973337, xss
- confirmed the fix; emailed Marcin for another validation pass
- Marcin responded to the wiki issues status. I'll list them all for a status update here in my log
- (1a) Can't upload files.
- (1b) Non-existent File redirects to front page = "If I use File:filename.jpg, it takes me to the front page of the wiki."
- (2) Content Missing (Youtube embeds?)
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- And Marcin confirmed that the only issue on this page was in-fact the missing youtube videos
- (3) Youtube Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (4) Issuu Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (5) Scrumy Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (6) Recent Changes issues
- Marcin says there's not many changes, and that's the issue. It is a fork of the other site, so I should confirm if this is really an issue
- (7) Ted Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (8) Vimeo Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (9) Page information
- Page information for all pages is slightly corrupted, and it never shows number of views - https://wiki.opensourceecology.org/index.php?title=Main_Page&action=info
- The Page Views missing is "won't fix" since this was removed by Mediawiki on 2014-10-20 in their release of mediawiki v1.25
- The "Display Title" field is still blank here
- (10) Popular Pages Missing
- Solved as "won't fix" since this was removed by Mediawiki on 2014-10-20 in their release of mediawiki v1.25
- (11) Page Statistics
- Solved as "won't fix" since this was removed by Mediawiki on 2014-10-20 in their release of mediawiki v1.25
- Therefore, the following wiki issues are outstanding:
- (1a) Can't upload files.
- (1b) Non-existent File redirects to front page = "If I use File:filename.jpg, it takes me to the front page of the wiki."
- (6) Recent Changes issues
- (9) Page information
- Emailed Marcin asking for confirmation of the fix for (1a), which I fixed already via permissions
- Began investigating (1b) again
- Discovered that it's a 301 redirect from our server, but only for the get variable URL. For example:
user@personal:~$ curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload" HTTP/1.1 200 OK Server: nginx Date: Sun, 25 Feb 2018 00:30:28 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image X-Frame-Options: DENY Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1519518628.2754; expires=Sun, 25-Feb-2018 00:31:28 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly X-Varnish: 1326749 Age: 0 Via: 1.1 varnish-v4 Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" user@personal:~$ curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload&wpDestFile=New.jpg" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sun, 25 Feb 2018 00:30:38 GMT Content-Type: text/html; charset=utf-8 Content-Length: 0 Connection: keep-alive X-Content-Type-Options: nosniff Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Sun, 25 Feb 2018 00:30:38 GMT Location: https://wiki.opensourceecology.org/wiki/Main_Page X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1519518638.9758; expires=Sun, 25-Feb-2018 00:31:38 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly X-Varnish: 1301183 Age: 0 Via: 1.1 varnish-v4 Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" user@personal:~$
- I tried the curl again, and found that the second query (with the wpDestFile GET variable appended) caused the *entire* GET variable sequence to be stripped by the time it made it to the mediawiki
# this is when I do `curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload&wpDestFile=New.jpg"` [root@hetzner2 wiki.opensourceecology.org]# tail -f wiki-error.log IP: 127.0.0.1 Start request GET /index.php HTTP HEADERS: X-REAL-IP: 173.234.159.236 X-FORWARDED-PROTO: https X-FORWARDED-PORT: 443 HOST: wiki.opensourceecology.org USER-AGENT: curl/7.38.0 ACCEPT: */* X-FORWARDED-FOR: 173.234.159.236, 127.0.0.1, 127.0.0.1 HASH: #wiki.opensourceecology.org ACCEPT-ENCODING: gzip X-VARNISH: 738663 [caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff ... # this is when I do `curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload" [root@hetzner2 wiki.opensourceecology.org]# tail -f wiki-error.log IP: 127.0.0.1 Start request GET /index.php?title=Special:Upload HTTP HEADERS: X-REAL-IP: 173.234.159.236 X-FORWARDED-PROTO: https X-FORWARDED-PORT: 443 HOST: wiki.opensourceecology.org USER-AGENT: curl/7.38.0 ACCEPT: */* X-FORWARDED-FOR: 173.234.159.236, 127.0.0.1, 127.0.0.1 HASH: #wiki.opensourceecology.org ACCEPT-ENCODING: gzip X-VARNISH: 738668 [caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff ...
- so the request mediawiki sees changes form "GET /index.php?title=Special:Upload" to "GET /index.php" when the second variable is added, which is why it just responds with a 301 redirect to the main page.
- confirmed that the correct request does make it to nginx
==> wiki.opensourceecology.org/access.log <== 173.234.159.236 - - [25/Feb/2018:00:39:24 +0000] "HEAD /index.php?title=Special:Upload HTTP/1.1" 200 0 "-" "curl/7.38.0" "-" 173.234.159.236 - - [25/Feb/2018:00:39:30 +0000] "HEAD /index.php?title=Special:Upload&wpDestFile=New.jpg HTTP/1.1" 301 0 "-" "curl/7.38.0" "-"
- confirmed that varnish does see the entire request, but then it appears to strip it!
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 173.234.159.236'" * << Request >> 1301342 - Begin req 1301341 rxreq - Timestamp Start: 1519519251.915667 0.000000 0.000000 - Timestamp Req: 1519519251.915667 0.000000 0.000000 - ReqStart 127.0.0.1 59862 - ReqMethod HEAD - ReqURL /index.php?title=Special:Upload&wpDestFile=New.jpg - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 173.234.159.236 - ReqHeader X-Forwarded-For: 173.234.159.236 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.38.0 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 173.234.159.236 - ReqHeader X-Forwarded-For: 173.234.159.236, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 173.234.159.236, 127.0.0.1 - ReqHeader X-Forwarded-For: 173.234.159.236, 127.0.0.1, 127.0.0.1 - ReqHeader X-VC-My-Purge-Key: JOaSAn72IJzrykJp1pfEWaECvUU8KvxZJnxSue3repId3qV8wJOHexjtuhi9r6Wv4FH9y9eFfiMjXX6hvxRrVOEWr2IaBVZMZ7ToEz8nLFdRyjyMkUGMANd6MHOzxiTJ - ReqUnset X-VC-My-Purge-Key: JOaSAn72IJzrykJp1pfEWaECvUU8KvxZJnxSue3repId3qV8wJOHexjtuhi9r6Wv4FH9y9eFfiMjXX6hvxRrVOEWr2IaBVZMZ7ToEz8nLFdRyjyMkUGMANd6MHOzxiTJ - ReqURL /index.php - VCL_return hash - VCL_call HASH - ReqHeader hash: #wiki.opensourceecology.org - VCL_return lookup - Debug "XXXX HIT-FOR-PASS" - HitPass 1301282 - VCL_call PASS - VCL_return fetch - Link bereq 1301343 pass - Timestamp Fetch: 1519519252.049599 0.133932 0.133932 - RespProtocol HTTP/1.0 - RespStatus 301 - RespReason Moved Permanently - RespHeader Date: Sun, 25 Feb 2018 00:40:51 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Vary: Accept-Encoding,Cookie - RespHeader Expires: Thu, 01 Jan 1970 00:00:00 GMT - RespHeader Cache-Control: private, must-revalidate, max-age=0 - RespHeader Last-Modified: Sun, 25 Feb 2018 00:40:52 GMT - RespHeader Location: https://wiki.opensourceecology.org/wiki/Main_Page - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Set-Cookie: cpPosTime=1519519252.0366; expires=Sun, 25-Feb-2018 00:41:52 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader X-VC-Req-Host: wiki.opensourceecology.org - RespHeader X-VC-Req-URL: /index.php - RespHeader X-VC-Req-URL-Base: /index.php - RespHeader X-VC-Cacheable: NO:Not cacheable, ttl: 0.000 - RespProtocol HTTP/1.1 - RespHeader X-Varnish: 1301342 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - RespUnset X-VC-Req-Host: wiki.opensourceecology.org - RespUnset X-VC-Req-URL: /index.php - RespUnset X-VC-Req-URL-Base: /index.php - RespHeader X-VC-Cache: MISS - RespUnset X-VC-Cache: MISS - RespUnset X-VC-Cacheable: NO:Not cacheable, ttl: 0.000 - VCL_return deliver - Timestamp Process: 1519519252.049641 0.133974 0.000043 - Debug "RES_MODE 0" - RespHeader Connection: close - Timestamp Resp: 1519519252.049679 0.134012 0.000037 - Debug "XXX REF 1" - ReqAcct 270 0 270 615 0 615 - End
- I checked the varnish config, and it looked like I had used the wordpress varnish config as a place-holder. That must have been stripping the variable in the request somehow. When I reduced the logic to a base config (with vhost support) by copying from the awstats configs, it fixed the issue.
- added php engine off declaration (as was used in mediawiki apache vhost config file) to the seedhome vhost config file
Tue Feb 20, 2018
- began to investigate (10) Popular Pages Missing
- found that the 'includes/specials/SpecialPopularpages.php' file exists on hetzner1 but not the fresh install on hetzner2
# hetzner1 osemain@dedi978:~/public_html/w$ date Tue Feb 20 20:23:00 CET 2018 osemain@dedi978:~/public_html/w$ pwd /usr/home/osemain/public_html/w osemain@dedi978:~/public_html/w$ ls includes/specials | grep -i popular SpecialPopularpages.php osemain@dedi978:~/public_html/w$ # hetzner2 [root@hetzner2 htdocs]# date Tue Feb 20 19:22:45 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/wiki.opensourceecology.org/htdocs [root@hetzner2 htdocs]# ls includes/specials | grep -i popular [root@hetzner2 htdocs]#
- I read that you can choose whether or not to enable counters with 'wgDisableCounter', so I grepped the entire new install dir for this string and found 1 result in includes/Setup.php:
[root@hetzner2 htdocs]# grep -irC 3 'wgDisableCounter' includes/
includes/Setup.php-
includes/Setup.php-// We don't use counters anymore. Left here for extensions still
includes/Setup.php-// expecting this to exist. Should be removed sometime 1.26 or later.
includes/Setup.php:if ( !isset( $wgDisableCounters ) ) {
includes/Setup.php: $wgDisableCounters = true;
includes/Setup.php-}
includes/Setup.php-
includes/Setup.php-if ( $wgMainWANCache === false ) {
[root@hetzner2 htdocs]#
- So it may have been removed. Next question: why?
- Found the merge that removed it from the mediawiki-commits mailing list https://www.mail-archive.com/mediawiki-commits@lists.wikimedia.org/msg193945.html
- Found the RFC that proposed removal of PopularPages https://www.mediawiki.org/wiki/Requests_for_comment/Removing_hit_counters_from_MediaWiki_core
- Some people in there say it's not accurate as it is simply an integer, and cannot provide unique hits. Their solution is to use Google Analytics or Piwik.
- Also found this in the changelog for the mediawiki v1.25 release https://www.mediawiki.org/wiki/MediaWiki_1.25#Hit_counters_removed
- In fact, this would be a useless figure, as mediawiki is only going to increment when varnish hits it. For our setup, this really does need to be on awstats monitoring nginx.
- I updated our awstats config to include our wiki site
[root@hetzner2 awstats]# cat /etc/awstats/awstats.wiki.opensourceecology.org.conf LogFile="/var/log/nginx/wiki.opensourceecology.org/access.log" SiteDomain="wiki.opensourceecology.org" HostAliases="wiki.opensourceecology.org 138.201.84.223" Include "common.conf" [root@hetzner2 awstats]# [root@hetzner2 awstats]# cat /etc/cron.d/awstats_generate_static_files 06 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now 16 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 17 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=seedhome.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 18 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=fef.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ 19 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ 20 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=wiki.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ [root@hetzner2 awstats]#
- emailed marcin saying that (9), (10), & (11) are "won't fix" and that we should use awstats again
- beginning to investigate (1a) Can't upload files.
- tailed the error logs when I attempted to do an upload, and I got this
==> wiki.opensourceecology.org/error_log <== [Tue Feb 20 20:38:29.746565 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.769788 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.770337 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.783048 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.783854 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.798665 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.799370 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [20/Feb/2018:20:38:29 +0000] "GET /api.php?action=query&format=json&formatversion=2&titles=File%3AGi%2Ejpg&prop=imageinfo&iiprop=uploadwarning HTTP/1.0" 200 135 "https://wiki.opensourceecology.org/wiki/Special:Upload" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" ==> wiki.opensourceecology.org/error_log <== [Tue Feb 20 20:38:38.802234 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.826196 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.826646 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.839300 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.840009 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.854396 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.855178 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.879773 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: set_time_limit() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 3121, referer: https://wiki.opensourceecology.org/wiki/Special:Upload ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [20/Feb/2018:20:38:38 +0000] "POST /wiki/Special:Upload HTTP/1.0" 429 18407 "https://wiki.opensourceecology.org/wiki/Special:Upload" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" ==> wiki.opensourceecology.org/error_log <== [Tue Feb 20 20:38:49.757596 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.781479 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.781864 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.807448 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.808226 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.853839 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.854676 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [20/Feb/2018:20:38:49 +0000] "GET /load.php?debug=false&lang=en&modules=mediawiki.helplink%2CsectionAnchor%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector HTTP/1.1" 200 43047 "https://wiki.opensourceecology.org/wiki/Special:Upload" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
- And the relevant line in the super-verbose 'wiki-error.log'
UploadBase::verifyFile: all clear; passing. [LocalFile] Failed to lock 'Gi.jpg' MediaWiki::preOutputCommit: primary transaction round committed
- that tells me the source of the error is a script named LocalFile
[root@hetzner2 htdocs]# find . | grep -i localfile ./includes/filerepo/file/LocalFile.php ./includes/filerepo/file/OldLocalFile.php ./includes/filerepo/file/UnregisteredLocalFile.php ./tests/phpunit/includes/filerepo/file/LocalFileTest.php [root@hetzner2 htdocs]#
- checked the obvious: permissions
[root@hetzner2 htdocs]# ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images | head -n 2 total 352K d---r-x--- 29 not-apache apache 4.0K Feb 14 03:19 .
- so it doesn't have write permissions to the directory
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
[ -d "${docrootDir_hetzner2}/images" ] || mkdir "${docrootDir_hetzner2}/images"
chown -R apache:apache "${docrootDir_hetzner2}/images"
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
- that's better
[root@hetzner2 htdocs]# ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images | head -n 2 total 352K drwxrwx--- 29 apache apache 4.0K Feb 14 03:19 .
- I tried the upload again, and it worked fine. That fixes (1a)
- these warnings that our hardened php.ini config is blocking calls to unsafe & disabled functions like ini_set() & putenv() clogging up the logs are annoying. Let's fix it & add it to the migration procedure
find includes/ -type f -exec sed -i 's%^\(\s*\)ini_set\(.*\)%\1#ini_set\2%' '{}' \;
find includes/ -type f -exec sed -i 's%^\(\s*\)putenv\(.*\)%\1#putenv\2%' '{}' \;
- that fixed it
- I documented this on Mediawiki#migrate_site_from_hetzner1_to_hetzner2
- began to investigate (1b)
- what's weird here is that this link totally works:
- https://wiki.opensourceecology.org/index.php?title=Special:Upload
- but as soon as you add the "&wpDestFile=New.jpg" to the URL, it sends me to the Main_Page
- when I tail the wiki-error.log file, I see the very beginning of the message shows it's just getting a "GET /index.php" request. So the problem must exist before mediawiki (apache? varnish? nginx?)
- what's weird here is that this link totally works:
IP: 127.0.0.1 Start request GET /index.php HTTP HEADERS: X-REAL-IP: 209.208.216.133 X-FORWARDED-PROTO: https X-FORWARDED-PORT: 443 HOST: wiki.opensourceecology.org USER-AGENT: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0 ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 ACCEPT-LANGUAGE: en-US,en;q=0.5 REFERER: https://wiki.opensourceecology.org/wiki/Hello UPGRADE-INSECURE-REQUESTS: 1 X-FORWARDED-FOR: 209.208.216.133, 127.0.0.1, 127.0.0.1 ACCEPT-ENCODING: gzip HASH: #wiki.opensourceecology.org X-VARNISH: 947159 [caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff [caches] LocalisationCache: using store LCStoreDB
- the apache logs show "GET /index.php" as well
127.0.0.1 - - [20/Feb/2018:21:39:50 +0000] "GET /index.php HTTP/1.1" 301 - "https://wiki.opensourceecology.org/wiki/Hello" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" 127.0.0.1 - - [20/Feb/2018:21:39:50 +0000] "GET /wiki/Main_Page HTTP/1.1" 200 71842 "https://wiki.opensourceecology.org/wiki/Hello" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
- Found that I can limit varnishlog output by domain:
varnishlog -q "ReqHeader ~ wiki.opensourceecology.org"
- Better, found that I can limit varnishlog output by the client's (my) ip address
varnishlog -q "X-Forwarded-For ~ 209.208.216.133"
- added the above to the useful commands under our varnish documentation page at Web_server_configuration#Useful_Commands
Mon Feb 19, 2018
- discovered that the bug report I submitted to the Open Office Online Suite to add the functionality of hyperlinks to their Impress Online (Google Docs Presentation replacement) got fixed! https://bugs.documentfoundation.org/show_bug.cgi?id=113361
- sent an email to Marcin, suggesting that we should revisit & submit bug reports, so that this could be a viable solution that meets our needs in a couple years.
- Marcin responded with a list of 11 issues with the Staging wiki
- (1a) Can't upload files.
- (1b) Non-existent File redirects to front page = "If I use File:filename.jpg, it takes me to the front page of the wiki."
- (2) Content Missing (Youtube embeds?)
- Conent missing on https://wiki.opensourceecology.org/wiki/Tractor_User_Manual
- (3) Youtube Embeds
- Videos not embedding - https://wiki.opensourceecology.org/wiki/CEB_Press_Fabrication_videos
- (4) Issuu Embeds
- Issuu not embedding on top - https://wiki.opensourceecology.org/wiki/Civilization_Starter_Kit_DVD_v0.01
- (5) Scrumy Embeds
- https://wiki.opensourceecology.org/wiki/Development_Team_Log . Scrumy doesn't appear to embed anywhere throughout wiki - here's another example - https://wiki.opensourceecology.org/wiki/Tractor_Construction_Set_2017
- (6) Recent Changes issues
- Recent Changes doesn't appear to be displaying properly - https://wiki.opensourceecology.org/index.php?title=Special:RecentChanges&limit=500
- (7) Ted Embeds
- Video embed not working on top of page. https://wiki.opensourceecology.org/wiki/Donate
- (8) Vimeo Embeds
- Video not embedded - https://wiki.opensourceecology.org/wiki/OSEmail
- (9) Page information
- Page information for all pages is slightly corrupted, and it never shows number of views - https://wiki.opensourceecology.org/index.php?title=Main_Page&action=info
- (10) Popular Pages Missing
- Popular Pages is missing from Data and tools section of https://wiki.opensourceecology.org/wiki/Special:SpecialPages
- (11) Page Statistics
- https://wiki.opensourceecology.org/wiki/Special:Statistics doesn't show page view statistics.
- I iterated through the list & attempted to reproduce the issues
- (1a) Can't upload files.
- I was able to reproduce this. Attempting to upload an issue resulted in the error message: 'Action Failed. Could not open lock file for "mwstore://local-backend/local-public/7/78/Image.jpg".'
- (1b) Non-existent File redirects to front page
- Marcin's test page has 2 links to non-existant files. Both redirect to the main page https://wiki.opensourceecology.org/wiki/Hello
- (2) Content Missing (Youtube Embeds?)
- I confirmed that the youtube video embed was missing. Replacing 'https://www.youtube.com/embed//' with 'https://www.youtube.com/embed/' fixed it.
- (3) Youtube Embeds
- Same as above. Replacing 'https://www.youtube.com/embed//' with 'https://www.youtube.com/embed/' fixed one of the videos.
- (4) Issuu Embeds
- confirmed. got a mosec issue when attempting to update the page, replacing the string 'https://static.issuu.com/webembed/' with 'https://static.issuu.com/webembed/'
- whitelisted 950018, generic attack = "universal pdf xss url detected"
- confirmed the fix of the string replacement above fixed this issue
- (5) Scrumy Embeds
- Confirmed that the embed from 'scrumy.com' is not working on the staging wiki
- attempted to make an edit, but got a modesec issue id = 950001
- whitelisted 950001, sqli attack
- Fixed the issue by replacing the string 'https://scrumy.com/' with 'https://scrumy.com/'
- (6) Recent Changes issues
- Marcin's description was vague, but I do see discrepancies between the 2x pages:
- It looks like the difference is that the diff & hist links moved from the left of the article name to the right of the article name. I should confirm with Marcin if this is the only issue && if it's actually OK
- (7) Ted Embeds
- confirmed that the ted talk embed is not loading
- attempted to fix it, but got a modesec forbidden on 958008
- whitelisted 958008, xss
- whitelisted 973329, xss
- fixed with replacing the string 'https://embed.ted.com/' with 'https://embed.ted.com/'
- also discovered that the flattr embed is broken on the Donate page. This is a consequence of us removing the Flattr Mediawiki plugin we were using, which was no longer maintained. I emailed Marcin and asked him to remove all content dependent on the Flattr plugin from the prod wiki.
- (8) Vimeo Embeds
- confirmed that the vimeo video embed is not working
- fixed by replacing the string 'https://player.vimeo.com/' with 'https://player.vimeo.com/'
- (9) Page information
- Confirmed that the Information page does not show the number of views, but it does on the prod site. I'm not sure if this was just removed in newer versions of Mediawiki or what. I'll have to investigate this further..
- Not sure what else is "corrupted" per Marcin's email. I'll have to ask him to clarify.
- (10) Popular Pages Missing
- Confirmed that the Popular Pages link is no longer present, but it is on the prod site. Not sure why. I'll have to investigate this further...
- (11) Page Statistics
- Confirmed that the "View statistics" that's present in the prod wiki is missing. May be related to (9). Further investigation is necessary...
- (1a) Can't upload files.
- Attempted to fix (2), (4), (5), and (8) using a simple string replacement using sed
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
pushd "${backupDir_hetzner2}/current"
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql
# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql
# fix youtube embeds
fromString='https://www.youtube.com/embed/'
toString='https://www.youtube.com/embed/'
sed -i "s^$fromString^$toString^g" db.sql
# fix issuu embeds
fromString='https://static.issuu.com/webembed/'
toString='https://static.issuu.com/webembed/'
sed -i "s^$fromString^$toString^g" db.sql
# fix scrumy embeds
fromString='https://scrumy.com/'
toString='https://scrumy.com/'
sed -i "s^$fromString^$toString^g" db.sql
# fix ted embeds
fromString='https://embed.ted.com/'
toString='https://embed.ted.com/'
sed -i "s^$fromString^$toString^g" db.sql
# fix vimeo embeds
fromString='https://player.vimeo.com/'
toString='https://player.vimeo.com/'
sed -i "s^$fromString^$toString^g" db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT, INSERT, UPDATE, DELETE ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
- unfortunately, that broke the wiki, resulting in an error
[WotzE6X9Jz1lrHTtbDhlLgAAAAI] 2018-02-20 01:00:03: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- running the update script fixed the error above!
pushd "${docrootDir_hetzner2}/maintenance"
php update.php
- bonus: this confirms that updates are not an issue using the hardened SELECT, INSERT, UPDATE, DELETE permissions for the db user
- emailed Marcin asking him to validate that I fixed (2), (4), (5), (7), & (8) above
Sun Feb 18, 2018
- documented a quick-start guide for 2FA http://opensourceecology.org/wiki/2FA#Quick_Start_Guide
- emailed Marcin the above guide/documentation
Thr Feb 15, 2018
- began following the mediawiki hardening guide https://www.mediawiki.org/wiki/Manual:Security
- confirmed that our permissions are ideal, and I updated the documentation with this Mediawiki#Proper_File.2FDirectory_Ownership_.26_Permissions
- confirmed that we're denying the only dir owned by the the web server's user (/images/, per necessity of requiring write access) cannot execute php scripts per the vhost config
[root@hetzner2 conf.d]# date Thu Feb 15 15:24:55 UTC 2018 [root@hetzner2 conf.d]# pwd /etc/httpd/conf.d [root@hetzner2 conf.d]# cat 00-wiki.opensourceecology.org.conf ... # don't execute any php files inside the uploads directory <LocationMatch "/images/"> php_flag engine off </LocationMatch> <LocationMatch "/images/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> Order Deny,Allow Deny from All </LocationMatch> ... [root@hetzner2 conf.d]#
- yes, we're using tls. yes, we use hsts (not to mention hpkp). ssllabs gives us an A+ on our httpsv setup https://www.ssllabs.com/ssltest/analyze.html?d=wiki.opensourceecology.org
- our php.ini config doesn't mention anything about register_globals. I confirmed that it must be off then, since we're running php v 5.6.31, and it's been the default to be off since v4.2.0 https://secure.php.net/manual/en/ini.core.php#ini.register-globals
- allow_url_fopen is already off
- session.use_trans_sid is already off
- the mysql config already has networking disabled (and iptables won't allow it too)
- changed the mediawiki article on our wiki to only grant "SELECT, INSERT, UPDATE and DELETE" permissions to the osewiki_user user per the hardening guide Mediawiki#migrate_site_from_hetzner1_to_hetzner2
- note this may break mediawiki updates from the maintenance scripts, but we can create a new user with all permissions, if necessary, and pass those directly into the update scripts using --dbuser & --dbpass as described here mediawikiwiki:Manual:Maintenance_scripts#Configuration
- added a variable '$wgSecretKey' to the LocalSettings.php file per mediawikwiki:Manual:$wgSecretKey
- I just used keepassx to give me a [a-zA-Z0-9]{128} string.
- the LocalSettings.php file is already located outside the docroot.
- updated the debugging options in LocalSettings.php to write to a log file outside the docroot
[root@hetzner2 ~]# touch /var/www/html/wiki.opensourceecology.org/wiki-error.log [root@hetzner2 ~]# chown apache:apache /var/log/httpd/wiki.opensourceecology.org/wiki-error.log [root@hetzner2 ~]# ls -lah /var/www/html/wiki.opensourceecology.org/wiki-error.log -rw-r--r-- 1 apache apache 0 Feb 15 16:46 /var/www/html/wiki.opensourceecology.org/wiki-error.log [root@hetzner2 ~]#
Wed Feb 14, 2018
- updated mediawiki migration process from findings yesterday Mediawiki#migrate_site_from_hetzner1_to_hetzner2
- Marcin found that clicking a link to a non-existing file on a wiki incorrectly redirects to the main page. it should link to the page to upload the missing file
- note that the following page *does* load
Tue Feb 13, 2018
- no response from Marcin yet about osemain validation
- confirmed that the awstats sites for opensourceecology.org was still working & got data for 2 consecutive days
- the icons were missing for ose awstats but not obi, so I copied them in-place
[maltfield@hetzner2 html]$ sudo cp -r awstats.openbuildinginstitute.org/htdocs/awstatsicons awstats.opensourceecology.org/htdocs/ [sudo] password for maltfield: [maltfield@hetzner2 html]$
- attempted to modify the wiki install to come straight from the tarball. Unfortunately, we can't do the git solution, since it doesn't include the vendor dir & we can't use Composer without destroying our php.ini security
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# cleanup current dir, making backups of old contents
pushd "${backupDir_hetzner2}"
mkdir "old/`date +%Y%m%d`"
mv current/* "old/`date +%Y%m%d`"
mv "${docrootDir_hetzner2}" "old/`date +%Y%m%d`"
# get backups from hetzner1
pushd current
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/
# download mediawiki core source code
wget https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz
tar -xzvf mediawiki-1.30.0.tar.gz
mkdir "${docrootDir_hetzner2}"
rsync -av --progress mediawiki-1.30.0/ "${docrootDir_hetzner2}/"
# copy-in our images from backups
rsync -av --progress "usr/www/users/osemain/w/images/" "${docrootDir_hetzner2}/images/"
# and move the lone image sticking in root into the images directory
rsync -av --progress "usr/www/users/osemain/w/ose-logo.png" "${docrootDir_hetzner2}/images/"
# create LocalSettings.php that just requires the file from outside the docroot
# write multi-line to file for documentation copy & paste
cat << EOF > "${docrootDir_hetzner2}/LocalSettings.php"
<?php
# including separate file that contains the database password so that it is not stored within the document root.
# For more info see:
# * https://www.mediawiki.org/wiki/Manual:Security
# * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening
\$docRoot = dirname( FILE );
require_once "\$docRoot/../LocalSettings.php";
?>
EOF
# extensions
pushd "${docrootDir_hetzner2}/extensions"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd
# skins
pushd "${docrootDir_hetzner2}/skins"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd
# set permissions
chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0040 "${vhostDir_hetzner2}/LocalSettings.php"
[ -d "${docrootDir_hetzner2}/images" ] || mkdir "${docrootDir_hetzner2}/images"
chown -R apache:apache "${docrootDir_hetzner2}/images"
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
# attempt to update
pushd ${docrootDir_hetzner2}/maintenance
php update.php
popd
- mediawiki dumbly appears to hard-code "/tmp" as the system's temporary directory in TempFSFile.php
[Wed Feb 14 01:47:56.364862 2018] [:error] [pid 3152] [client 127.0.0.1:54708] PHP Warning: is_dir(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fsfile/TempFSFile.php on line 90
- the above warning disappears when I tell it to just check the php config--which it really should be doing on its own. I do this by setting the following in LocalSettings.php
# tell mediaiwki to actually ask php what the tmp dir is, rather than assuming # that "/tmp" is the system's temporary directory $wgTmpDirectory = sys_get_temp_dir();
- update.php execution was successful
[root@hetzner2 maintenance]# php update.php ... Done in 0.5 s. [root@hetzner2 maintenance]#
- confirmed that I can login
- attempted to make an edit, but got a modsec forbidden
- 981318, sqli
- confirmed a successful edit!
- the site still looks like shit, probably because the CSS is being 403'd
- https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector
- https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint,shared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector
- https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=jquery.accessKeyLabel,byteLength,checkboxShiftClick,client,getAttrs,highlightText,mw-jump,suggestions,tabIndex,throttle-debounce%7Cmediawiki.RegExp,Title,api,cldr,jqueryMsg,language,notify,searchSuggest,storage,user,util%7Cmediawiki.api.user,watch%7Cmediawiki.language.data,init%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready,startup%7Cmediawiki.page.watch.ajax%7Csite%7Cskins.vector.js%7Cuser.defaults&skin=vector&version=19n53uv
- confirmed that the page *does* load without the GET vars https://wiki.opensourceecology.org/load.php
- reduced the issue to just the 'modules' GET var, as the rest load fine https://wiki.opensourceecology.org/load.php?debug=false&lang=en&only=styles&skin=vector
- the manual on this script says it could be htaccess https://www.mediawiki.org/wiki/Manual:Load.php
- indeed, the default install came with an htaccess file:
# Protect against bug 28235
<IfModule rewrite_module>
RewriteEngine On
RewriteCond %{QUERY_STRING} \.[^\\/:*?\x22<>|%]+(#|\?|$) [nocase]
RewriteRule . - [forbidden]
</IfModule>
- when I commented-out the above, it loaded fine.
- The bug it mentions (28235) is a XSS vuln in IE 6 https://phabricator.wikimedia.org/T30235
- so it appears that the .htaccess file is triggering a 403 because the query includes a GET variable with a dot (.) in it. Specifically, it's the "modules=site.styles" bit.
- Well, this issue only gives XSS risks to IE6 users. Honestly, I think the best thing to do is to ban IE6 and delete all the .htaccess files with this block
- added this block to the top of LocalSettings.php
# the .htaccess file that ships with mediawiki to prevent XSS attacks on IE6
# clients breaks our CSS loading. OSE's solution = ban IE6 and remove the
# offending .htaccess files
if(strpos($_SERVER['HTTP_USER_AGENT'], 'MSIE 6.') !== false) {
die( "Sorry, you must update your browser to use this site." );
}
- deleted the contents of the htdocs/.htaccess file (the entire file was just this bugfix for 28235
- confirmed that the site css looks good!
- the main page load still triggers a modsec alert
- 981260, sqli
- 950911, generic attack (response splitting?)
- 973302, xss
- 973324, xss
- 973317, xss
- 981255, sqli
- 958057, xss
- 958056, xss
- 973327, xss
- strangely, the sys_get_temp_dir() is wrong! Is this a bug in php?
[root@hetzner2 httpd]# grep -i 'upload_tmp_dir' /etc/php.ini upload_tmp_dir = "/var/lib/php/tmp_upload" [root@hetzner2 httpd]# echo "<?=sys_get_temp_dir();?>" | php /tmp [root@hetzner2 httpd]# echo "<?=php_ini_loaded_file();?>" | php /etc/php.ini [root@hetzner2 httpd]#
- anyway, I just manually set `$wgTmpDirectory = "/var/lib/php/tmp_upload"` in LocalSettings.php
- that fixed all the thumbnail creation issues.
- sent an email to Marcin & Catarina asking for validation
Mon Feb 12, 2018
- updated CHG-2018-02-05 with last week's fixes
- Continuing to fix issues on osemain
- /marcin-jakubowski/
- Marcin pointed out that his headshot is missing on his about page of the staging site
- The offensive html points to http://opensourceecology.org/wp-content/uploads/2014/02/MarcinHeadshot-300x300.png
<div class="center"><div class=" image-box aligncenter clearfix"><a href="#" title="Marcin Jakubowski"><img class="photo_frame rounded-img" src="http://opensourceecology.org/wp-content/uploads/2014/02/MarcinHeadshot-300x300.png" alt="Marcin Jakubowski"></a></div>
- I would expect this to be a mixed content error (http over https), but the firefox developer console doesn't mention this image specificially (though it does mention an image from creativecommons.org, licensebuttons.net, and our OSE_yellow-copy2.png logo.
- Anyway, when I replace the src attribute's 'http:' with 'https:osemain.', it works. Because of the name change, I'll just have to address this post-migration. I'll probably end-up doing a database-wide sed for 'http://opensourceecology.org/' to 'https://www.opensourceecology.org/
- was unable to reproduce Marcin's issue clicking the workshop link
- sent marcin an email following-up on his edit issues
- confirmed that the awstats issues are fixed for obi
- added awstats config files for awstats.opensourceecology.org
[root@hetzner2 awstats.opensourceecology.org]# ls -lah /etc/awstats | grep -i opensourceecology -rw-r--r-- 1 root root 175 Feb 12 21:37 awstats.fef.opensourceecology.org.conf -rw-r--r-- 1 root root 175 Feb 12 21:31 awstats.www.opensourceecology.org.conf [root@hetzner2 awstats.opensourceecology.org]#
- added cron jobs for ose sites:
[root@hetzner2 awstats.opensourceecology.org]# cat /etc/cron.d/awstats_generate_static_files 06 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now 16 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 17 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=seedhome.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 18 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=fef.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ 19 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ [root@hetzner2 awstats.opensourceecology.org]#
- fixed permissions on the .htpasswd file next to the docroot (one directory above the docroot)
[root@hetzner2 awstats.opensourceecology.org]# date Mon Feb 12 22:10:47 UTC 2018 [root@hetzner2 awstats.opensourceecology.org]# pwd /var/www/html/awstats.opensourceecology.org [root@hetzner2 awstats.opensourceecology.org]# ls -lah total 16K drwxr-xr-x 3 root root 4.0K Feb 9 21:27 . drwxr-xr-x 16 root root 4.0K Feb 9 20:33 .. drwxr-xr-x 3 root root 4.0K Feb 12 21:42 htdocs -r-------- 1 apache apache 138 Feb 9 21:27 .htpasswd [root@hetzner2 awstats.opensourceecology.org]#
- that really shouldn't be owned by apache (it should owned by a group with apache in it with read-only permissions), but I'll fix that later
- began to test fixes to the docroot permissions such that they're not owned by the apache user, but the apache user has read access
- first, I created the new docroot default file/dir owner: not-apache
[root@hetzner2 awstats.opensourceecology.org]# useradd --home-dir '/dev/null' --shell '/sbin/nologin' not-apache [root@hetzner2 awstats.opensourceecology.org]# tail -n1 /etc/passwd not-apache:x:1012:1013::/dev/null:/sbin/nologin [root@hetzner2 awstats.opensourceecology.org]#
- next, I added the apache user to the apache-admins group, so that it can have read-only permissions to the password-containing config files
- tested the permission files updates on seedhome
vhostDir="/var/www/html/seedhome.openbuildinginstitute.org"
wpDocroot="${vhostDir}/htdocs"
chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0060 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0070 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"
- after the above, I still had permission issues; that was fixed by restarting apache (which was necessary following the group membership, I guess)
- I tested logging-in & an edit. it worked.
- attempted to upload an image, but it failed
Unable to create directory wp-content/uploads/2017/12. Is its parent directory writable by the server?
- for one, the 'apache' user wasn't in the apache group. I fixed this:
[root@hetzner2 htdocs]# gpasswd -a apache apache Adding user apache to group apache [root@hetzner2 htdocs]# service ^C [root@hetzner2 htdocs]# httpd -t Syntax OK [root@hetzner2 htdocs]# service httpd restart Redirecting to /bin/systemctl restart httpd.service [root@hetzner2 htdocs]#
- still having issues. I guess to create new files/directories, the directory must be owned by the apache user, not just a group with write permission that the apache user is in (that works for read only)
- also, the above commands wouldn't work as a fresh install doesn't necessarily already have the ;uploads' directory, so I should add a mkdir. Updating the process:
vhostDir="/var/www/html/seedhome.openbuildinginstitute.org"
wpDocroot="${vhostDir}/htdocs"
chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"
[ -d "${wpDocroot}/wp-content/uploads" ] || mkdir "${wpDocroot}/wp-content/uploads"
chown -R apache:apache "${wpDocroot}/wp-content/uploads"
find "${wpDocroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
- that worked. updated the documentation Wordpress#Proper_File.2FDirectory_Ownership_.26_Permissions
- updated the apache vhost to not execute any php files from within the 'uploads' directory
[root@hetzner2 httpd]# grep -C 3 'uploads' conf.d/00-seedhome.openbuildinginstitute.org.conf </IfModule> </LocationMatch> # don't execute any php files inside the uploads directory <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> Order Deny,Allow Deny from All </LocationMatch> [root@hetzner2 httpd]#
Thr Feb 09, 2018
- did a theme update. The default themes updated, but the active theme (enigmatic) had no updates (or, at least, it's not available via the api)
[root@hetzner2 www.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme list
...
+------------------------+----------+-----------+---------+
| name | status | update | version |
+------------------------+----------+-----------+---------+
| enigmatic.20170714.bak | inactive | none | 3.1 |
| enigmatic | active | none | 3.5 |
| twentyeleven | inactive | none | 2.7 |
| twentyfifteen | inactive | none | 1.9 |
| twentyfourteen | inactive | available | 1.0 |
| twentyseventeen | inactive | none | 1.4 |
| twentysixteen | inactive | none | 1.4 |
| twentyten | inactive | none | 2.4 |
| twentythirteen | inactive | available | 1.1 |
| twentytwelve | inactive | available | 1.3 |
+------------------------+----------+-----------+---------+
[root@hetzner2 www.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme update --all
...
Downloading update from https://downloads.wordpress.org/theme/twentyfourteen.2.1.zip...
Using cached file '/home/wp/.wp-cli/cache/theme/twentyfourteen-2.1.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentythirteen.2.3.zip...
Using cached file '/home/wp/.wp-cli/cache/theme/twentythirteen-2.3.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentytwelve.2.4.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
+----------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------+-------------+-------------+---------+
| twentyfourteen | 1.0 | 2.1 | Updated |
| twentythirteen | 1.1 | 2.3 | Updated |
| twentytwelve | 1.3 | 2.4 | Updated |
+----------------+-------------+-------------+---------+
Success: Updated 3 of 3 themes.
[root@hetzner2 www.opensourceecology.org]#
- awstats for obi was going to '/var/log/nginx/access.log' instead of the vhost-specific '/var/log/nginx/www.openbuildinginstitute.org/access.log' file
- the permissions were fine; a reload fixed it (note the file size changed from 0 to 940 after the reload & I queried the site again
[root@hetzner2 ~]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/ total 19M drw-r--r-- 2 nginx nginx 4.0K Feb 9 18:20 . drwx------ 9 nginx nginx 4.0K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 0 Feb 9 04:47 access.log -rw-r--r-- 1 nginx nginx 377 Dec 24 01:30 access.log.1.gz -rw-r--r-- 1 nginx nginx 1.9M Dec 18 03:20 access.log-20171210.gz -rw-r--r-- 1 nginx nginx 1.3M Dec 24 01:20 access.log-20171219.gz -rw-r--r-- 1 nginx nginx 2.6M Jan 3 22:03 access.log-20171225.gz -rw-r--r-- 1 nginx nginx 239K Jan 4 21:36 access.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.2M Jan 18 14:43 access.log-20180105.gz -rw-r--r-- 1 nginx nginx 193K Jan 19 13:03 access.log-20180119.gz -rw-r--r-- 1 nginx nginx 288K Jan 20 20:20 access.log-20180120.gz -rw-r--r-- 1 nginx nginx 4.2M Feb 4 20:19 access.log-20180121.gz -rw-r--r-- 1 nginx nginx 1.7M Feb 9 02:08 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 3.3M Feb 9 18:19 access.log-20180209 -rw-r--r-- 1 nginx nginx 224 Dec 24 01:29 access.log.2.gz -rw-r--r-- 1 nginx nginx 510 Dec 24 01:21 access.log.3.gz -rw-r--r-- 1 nginx nginx 0 Feb 6 06:01 error.log -rw-r--r-- 1 nginx nginx 1.1K Nov 30 17:06 error.log-20171201.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 8 13:12 error.log-20171206.gz -rw-r--r-- 1 nginx nginx 1.8K Dec 12 23:31 error.log-20171210.gz -rw-r--r-- 1 nginx nginx 401 Dec 19 13:45 error.log-20171219.gz -rw-r--r-- 1 nginx nginx 388 Dec 24 05:48 error.log-20171224.gz -rw-r--r-- 1 nginx nginx 3.3K Jan 1 09:19 error.log-20171229.gz -rw-r--r-- 1 nginx nginx 359 Jan 3 23:05 error.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.8K Jan 18 13:55 error.log-20180105.gz -rw-r--r-- 1 nginx nginx 888 Feb 2 10:27 error.log-20180124.gz -rw-r--r-- 1 nginx nginx 319 Feb 5 17:04 error.log-20180206 [root@hetzner2 ~]# service nginx reload Redirecting to /bin/systemctl reload nginx.service [root@hetzner2 ~]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/ total 19M drw-r--r-- 2 nginx nginx 4.0K Feb 9 18:20 . drwx------ 9 nginx nginx 4.0K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 940 Feb 9 18:20 access.log -rw-r--r-- 1 nginx nginx 377 Dec 24 01:30 access.log.1.gz -rw-r--r-- 1 nginx nginx 1.9M Dec 18 03:20 access.log-20171210.gz -rw-r--r-- 1 nginx nginx 1.3M Dec 24 01:20 access.log-20171219.gz -rw-r--r-- 1 nginx nginx 2.6M Jan 3 22:03 access.log-20171225.gz -rw-r--r-- 1 nginx nginx 239K Jan 4 21:36 access.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.2M Jan 18 14:43 access.log-20180105.gz -rw-r--r-- 1 nginx nginx 193K Jan 19 13:03 access.log-20180119.gz -rw-r--r-- 1 nginx nginx 288K Jan 20 20:20 access.log-20180120.gz -rw-r--r-- 1 nginx nginx 4.2M Feb 4 20:19 access.log-20180121.gz -rw-r--r-- 1 nginx nginx 1.7M Feb 9 02:08 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 3.3M Feb 9 18:20 access.log-20180209 -rw-r--r-- 1 nginx nginx 224 Dec 24 01:29 access.log.2.gz -rw-r--r-- 1 nginx nginx 510 Dec 24 01:21 access.log.3.gz -rw-r--r-- 1 nginx nginx 0 Feb 6 06:01 error.log -rw-r--r-- 1 nginx nginx 1.1K Nov 30 17:06 error.log-20171201.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 8 13:12 error.log-20171206.gz -rw-r--r-- 1 nginx nginx 1.8K Dec 12 23:31 error.log-20171210.gz -rw-r--r-- 1 nginx nginx 401 Dec 19 13:45 error.log-20171219.gz -rw-r--r-- 1 nginx nginx 388 Dec 24 05:48 error.log-20171224.gz -rw-r--r-- 1 nginx nginx 3.3K Jan 1 09:19 error.log-20171229.gz -rw-r--r-- 1 nginx nginx 359 Jan 3 23:05 error.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.8K Jan 18 13:55 error.log-20180105.gz -rw-r--r-- 1 nginx nginx 888 Feb 2 10:27 error.log-20180124.gz -rw-r--r-- 1 nginx nginx 319 Feb 5 17:04 error.log-20180206 [root@hetzner2 ~]#
- the reload *should* be happening already by the logroate
[root@hetzner2 ~]# cat /etc/logrotate.d/nginx
/var/log/nginx/*log /var/log/nginx/*/*log {
create 0644 nginx nginx
daily
rotate 10
missingok
notifempty
compress
sharedscripts
prerotate
/bin/nice /usr/share/awstats/tools/awstats_updateall.pl now
endscript
postrotate
/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
endscript
}
[root@hetzner2 ~]#
- I simulated the rotate & kill, and saw the same undesired behaviour
[root@hetzner2 www.openbuildinginstitute.org]# mv access.log access.log.test [root@hetzner2 www.openbuildinginstitute.org]# touch access.log [root@hetzner2 www.openbuildinginstitute.org]# chown nginx:nginx access.log [root@hetzner2 www.openbuildinginstitute.org]# chmod 0644 access.log [root@hetzner2 www.openbuildinginstitute.org]# /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true [root@hetzner2 nginx]# tail -f /var/log/nginx/access.log /var/log/nginx/www.openbuildinginstitute.org/access.log /var/log/nginx/error.log /var/log/nginx/www.openbuildinginstitute.org/error.log ... ==> /var/log/nginx/access.log <== 98.242.98.106 - - [09/Feb/2018:18:30:05 +0000] "GET /wp-content/uploads/2016/02/kitchen_1.jpg HTTP/1.1" 301 178 "https://www.openbuildinginstitute.org/buildings/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-"
- interseting, when I did the kill again, this log popped-up
[root@hetzner2 nginx]# tail -f /var/log/nginx/access.log /var/log/nginx/www.openbuildinginstitute.org/access.log /var/log/nginx/error.log /var/log/nginx/www.openbuildinginstitute.org/error.log ... ==> /var/log/nginx/error.log <== 2018/02/09 18:31:08 [emerg] 26860#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26861#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26866#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26862#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26865#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26859#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26860#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26861#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26866#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26862#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26865#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26859#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26864#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26864#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26863#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26863#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
- but the permissions look fine
[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/access.log -rw-r--r-- 1 nginx nginx 0 Feb 9 18:26 /var/log/nginx/www.openbuildinginstitute.org/access.log
- ah, found the issue thanks to this https://stackoverflow.com/questions/37245926/nginx-write-logs-to-the-old-file-after-running-logrotate
[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/ total 9.0M drwx------ 9 nginx nginx 4.0K Feb 9 04:47 . drwxr-xr-x 13 root root 12K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 2.2M Feb 9 18:39 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 443K Feb 8 07:21 access.log-20180208.gz -rw-r--r-- 1 nginx nginx 4.0M Feb 9 04:46 access.log-20180209 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 249K Feb 9 18:37 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 13K Feb 8 07:21 error.log-20180208.gz -rw-r--r-- 1 nginx nginx 379K Feb 9 04:46 error.log-20180209 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 fef.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 forum.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 oswh.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 seedhome.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 wiki.opensourceecology.org drw-r--r-- 2 nginx nginx 4.0K Feb 9 18:26 www.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 www.opensourceecology.org [root@hetzner2 www.openbuildinginstitute.org]#
- if you look at the above output of the /var/log/nginx contents, it's obvious really. One dir is 0444 while the rest are 0755. We need that containing dir to have execute permissions!
[root@hetzner2 www.openbuildinginstitute.org]# chmod 0755 /var/log/nginx/www.openbuildinginstitute.org [root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/ total 9.0M drwx------ 9 nginx nginx 4.0K Feb 9 04:47 . drwxr-xr-x 13 root root 12K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 2.2M Feb 9 18:41 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 443K Feb 8 07:21 access.log-20180208.gz -rw-r--r-- 1 nginx nginx 4.0M Feb 9 04:46 access.log-20180209 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 250K Feb 9 18:40 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 13K Feb 8 07:21 error.log-20180208.gz -rw-r--r-- 1 nginx nginx 379K Feb 9 04:46 error.log-20180209 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 fef.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 forum.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 oswh.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 seedhome.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 wiki.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 18:26 www.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 www.opensourceecology.org [root@hetzner2 www.openbuildinginstitute.org]#
- that appears to have fixed it!
- I updated the documentation for creating new wordpress vhosts to use these permissions at the get-go Wordpress#Create_New_Wordpress_Vhost
- began investigating how the hell awstats is accessible on port 443 and, more importantly, how certbot is able to refresh its certificate
- I found the logs of attempting to load 'https://awstats.openbuildinginstitute.org/fuck' in '/var/log/nginx/seedhome.openbuildinginstitute.org/access.log'
[root@hetzner2 nginx]# date Fri Feb 9 19:08:10 UTC 2018 [root@hetzner2 nginx]# pwd /var/log/nginx [root@hetzner2 nginx]# tail -f *.log */*.log | grep -i fuck 98.242.98.106 - - [09/Feb/2018:19:04:10 +0000] "GET /fuck HTTP/1.1" 401 381 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-" [root@hetzner2 nginx]# grep -irl 'fuck' * seedhome.openbuildinginstitute.org/access.log [root@hetzner2 nginx]#
- I guess that site became our default/catch-all, picking up requests on obi's ip address (138.201.84.223) on port 443. And the nginx config file is generic enough to just pass the host header along to varnish using a dynamic variable ($host), so then apache actually serves the correct page.
- so I think the best thing to do is to explicitly define an nginx listen on port 443 for each awstats site, passing to varnish per usual (but logging in the correct place) && the extensible configs in the proper & expected place. Then I'll configure apache to have 2 distinct docroots for X-Forwarded-Port equal to 4443 vs 443. 4443 will give us the htpasswd-protected awstats files. 443 will go to publicly-accessible docroot where certbot will generate files to negotiate https cert updates.
- successfully tested changing the "X-Forwarded-Port 443" line in the nginx config to "X-Forwarded-Port $server_port"
[root@hetzner2 conf.d]# cat /etc/nginx/conf.d/awstats.openbuildinginstitute.org.conf
################################################################################
# File: awstats.openbuildinginstitute.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################
server {
include conf.d/secure.include;
include conf.d/ssl.openbuildinginstitute.org.include;
listen 138.201.84.223:4443;
listen 138.201.84.223:443;
#listen [::]:443;
server_name awstats.openbuildinginstitute.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
#root /var/www/html/SITE_DOWN/htdocs/;
#index index.html index.htm;
# force all requests to load exactly this page
#location / {
# try_files $uri /index.html;
#}
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
#proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Host $host;
}
}
[root@hetzner2 conf.d]#
- apache isn't very powerful at conditionals, so I decided to move the logic to nginx & varnish
- updated /etc/httpd/conf/httpd.conf to listen on 127.0.0.1:8010 for the certbot vhost
- successfully added 'awstats.opensourceecology.org' to cert SAN with certbot
[root@hetzner2 ~]# certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org ... IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/opensourceecology.org/privkey.pem Your cert will expire on 2018-05-10. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le [root@hetzner2 ~]# /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* [root@hetzner2 ~]# nginx -t && service nginx reload nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Redirecting to /bin/systemctl reload nginx.service [root@hetzner2 ~]#
- so first, nginx is listening on ports '443' && '4443' It sets the 'X-Forwarded-Port' Header to whichever it heard the request come in on (referenced by $server_port)
[root@hetzner2 conf.d]# date
Fri Feb 9 21:37:33 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/nginx/conf.d
[root@hetzner2 conf.d]# cat awstats.openbuildinginstitute.org.conf
################################################################################
# File: awstats.openbuildinginstitute.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################
server {
include conf.d/secure.include;
include conf.d/ssl.openbuildinginstitute.org.include;
listen 138.201.84.223:4443;
listen 138.201.84.223:443;
server_name awstats.openbuildinginstitute.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
#root /var/www/html/SITE_DOWN/htdocs/;
#index index.html index.htm;
# force all requests to load exactly this page
#location / {
# try_files $uri /index.html;
#}
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
#proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Host $host;
}
}
[root@hetzner2 conf.d]#
- nginx sends its requests to varnish. The varnish config for the awstats configs defines 2x distinct backends. Both are apache, but the ports are different. The standard 8000 name-based-vhost is used for the 4443 port && there's a new vhost running on 8010 specifically for this certbot vhost.
[root@hetzner2 varnish]# date
Fri Feb 9 21:39:47 UTC 2018
[root@hetzner2 varnish]# pwd
/etc/varnish
[root@hetzner2 varnish]# cat sites-enabled/awstats.openbuildinginstitute.org
################################################################################
# File: awstats.openbuildinginstitute.org.vcl
# Version: 0.3
# Purpose: Config file for awstats for obi
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################
vcl 4.0;
##########################
# VHOST-SPECIFIC BACKEND #
##########################
backend awstats_openbuildinginstitute_org {
.host = "127.0.0.1";
.port = "8000";
}
backend certbot_openbuildinginstitute_org {
.host = "127.0.0.1";
.port = "8010";
}
#####################################################################
# STUFF MOSTLY COPIED FROM vcaching WORDPRESS PLUGIN #
# https://plugins.svn.wordpress.org/vcaching/trunk/varnish-conf/v4/ #
#####################################################################
sub vcl_recv {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# is this an admin trying to access awstats or certbot trying to renew the
# https certificate?
if ( req.http.X-Forwarded-Port == "4443" ){
# this is an admin; send them to the auth-protected apache vhost
set req.backend_hint = awstats_openbuildinginstitute_org;
} else {
# this is not an admin accessing the hidden port; send them to the apache
# vhost with no content (that certbot uses)
set req.backend_hint = certbot_openbuildinginstitute_org;
}
}
}
sub vcl_hash {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
sub vcl_backend_response {
if ( beresp.backend.name == "awstats_openbuildinginstitute_org" ){
# TODO
}
}
sub vcl_synth {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
sub vcl_pipe {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
sub vcl_deliver {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
[root@hetzner2 varnish]# cat default.vcl
################################################################################
# File: default.vcl
# Version: 0.1
# Purpose: Main config file for varnish cache. Note that it's intentionally
# mostly bare to allow robust vhost-specific logic. Please see this
# for more info:
# * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2017-11-12
################################################################################
vcl 4.0;
############
# INCLUDES #
############
#
import std;
include "conf/acl.vcl";
include "lib/purge.vcl";
include "all-vhosts.vcl";
include "catch-all.vcl";
[root@hetzner2 varnish]# cat all-vhosts.vcl
################################################################################
# File: all-hosts.vcl
# Version: 0.8
# Purpose: meta config file that simply imports the site-specific vcl files
# stored in the 'sites-enabled' directory Please see this for more info
# * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2018-02-09
################################################################################
include "sites-enabled/staging.openbuildinginstitute.org";
include "sites-enabled/awstats.openbuildinginstitute.org";
include "sites-enabled/awstats.opensourceecology.org";
include "sites-enabled/www.openbuildinginstitute.org";
include "sites-enabled/www.opensourceecology.org";
include "sites-enabled/seedhome.openbuildinginstitute.org";
include "sites-enabled/fef.opensourceecology.org";
include "sites-enabled/oswh.opensourceecology.org";
include "sites-enabled/forum.opensourceecology.org";
include "sites-enabled/wiki.opensourceecology.org";
[root@hetzner2 varnish]#
- finally, here's the 2x distinct vhosts in apache
[root@hetzner2 conf.d]# date
Fri Feb 9 21:41:31 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/httpd/conf.d
[root@hetzner2 conf.d]# cat awstats.openbuildinginstitute.org.conf
<VirtualHost 127.0.0.1:8000>
ServerName awstats.openbuildinginstitute.org
DocumentRoot "/var/www/html/awstats.openbuildinginstitute.org/htdocs"
Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
</VirtualHost>
<Directory "/var/www/html/awstats.openbuildinginstitute.org/htdocs">
AuthType Basic
AuthName "Authentication Required"
AuthUserFile /var/www/html/awstats.openbuildinginstitute.org/.htpasswd
# it is unnecessary to allow certbot here, since it uses port 80 (which we
# send to port 443), but this server only listens on port 4443 (via nginx)
#Require expr %{REQUEST_URI} =~ m#^/.well-known/acme-challenge/#
Require valid-user
Options +Indexes +Includes
AllowOverride None
Order allow,deny
Allow from all
# Harden vhost docroot by blocking all request types except the 3 essentials
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
# disable mod_security with rules as needed (found by logs in:
# /var/log/httpd/modsec_audit.log
<Location "/">
<IfModule security2_module>
SecRuleEngine On
#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
</IfModule>
</Location>
[root@hetzner2 conf.d]# cat certbot.conf
################################################################################
# File: certbot.conf
# Version: 0.1
# Purpose: localhost-only-listening, http-only, name-based-vhost for serving
# an empty vhost that's to be used by certbot for refreshing our Let's
# Encrypt certificates. This is necessary for admin-only/private vhosts
# that operate on non-standard port with basic auth required. For
# example, see the configs for awstats.
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2018-02-09
# Updated: 2018-02-09
################################################################################
<VirtualHost 127.0.0.1:8010>
#ServerName awstats.openbuildinginstitute.org
DocumentRoot "/var/www/html/certbot/htdocs"
#Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
</VirtualHost>
<Directory "/var/www/html/certbot/htdocs">
Options +Indexes +Includes
AllowOverride None
Order allow,deny
Allow from all
# Harden vhost docroot by blocking all request types except the 3 essentials
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
# disable mod_security with rules as needed (found by logs in:
# /var/log/httpd/modsec_audit.log
<Location "/">
<IfModule security2_module>
SecRuleEngine On
#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
</IfModule>
</Location>
Thr Feb 08, 2018
- installed our minimal set of security wordpress plugins
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
...
Installing Google Authenticator (0.48)
Downloading installation package from https://downloads.wordpress.org/plugin/google-authenticator.0.48.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/google-authenticator-0.48.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'google-authenticator'...
Plugin 'google-authenticator' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
...
Installing Google Authenticator Encourage User Activation (0.2)
Downloading installation package from https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/google-authenticator-encourage-user-activation-0.2.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'google-authenticator-encourage-user-activation'...
Plugin 'google-authenticator-encourage-user-activation' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]# defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
[root@hetzner2 htdocs]# pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
/var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/google-authenticator /var/www/html/www.opensourceecology.org/htdocs
[root@hetzner2 google-authenticator]# sed -i "s^\$GA_description\s=\s(\s[\"'].*[\"']^\$GA_description = ( '$defaultOtpAccountDescription'^" google-authenticator.php
[root@hetzner2 google-authenticator]# popd
/var/www/html/www.opensourceecology.org/htdocs
[root@hetzner2 htdocs]# # install 'force-strong-passwords' plugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate
...
Installing Force Strong Passwords (1.8.0)
Downloading installation package from https://downloads.wordpress.org/plugin/force-strong-passwords.1.8.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/force-strong-passwords-1.8.0.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'force-strong-passwords'...
Plugin 'force-strong-passwords' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install rename-wp-login plugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate
...
Installing Rename wp-login.php (2.5.5)
Downloading installation package from https://downloads.wordpress.org/plugin/rename-wp-login.2.5.5.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/rename-wp-login-2.5.5.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'rename-wp-login'...
Plugin 'rename-wp-login' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install "SSL Insecure Content Fixer" pugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate
...
Installing SSL Insecure Content Fixer (2.5.0)
Downloading installation package from https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.5.0.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/ssl-insecure-content-fixer-2.5.0.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'ssl-insecure-content-fixer'...
Plugin 'ssl-insecure-content-fixer' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install "Varnish Caching" pugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate
...
Installing Varnish Caching (1.6.7)
Downloading installation package from https://downloads.wordpress.org/plugin/vcaching.1.6.7.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/vcaching-1.6.7.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'vcaching'...
Plugin 'vcaching' activated.
Success: Installed 1 of 1 plugins.
- updated existing wordpress plugins on osemain staging site on hetzner2
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
...
+----------------------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------------------+-------------+-------------+---------+
| akismet | 3.3 | 4.0.2 | Updated |
| cyclone-slider-2 | 2.12.4 | 3.2.0 | Updated |
| duplicate-post | 3.1.2 | 3.2.1 | Updated |
| insert-headers-and-footers | 1.4.1 | 1.4.2 | Updated |
| jetpack | 4.7.1 | 5.8 | Updated |
| ml-slider | 3.5 | 3.6.8 | Updated |
| post-types-order | 1.9.3 | 1.9.3.6 | Updated |
| recent-tweets-widget | 1.6.6 | 1.6.8 | Updated |
| shareaholic | 7.8.0.4 | 8.6.5 | Updated |
| share-on-diaspora | 0.7.1 | 0.7.2 | Updated |
| w3-total-cache | 0.9.5.2 | 0.9.6 | Updated |
| wp-smushit | 2.6.1 | 2.7.6 | Updated |
+----------------------------+-------------+-------------+---------+
Success: Updated 12 of 12 plugins.
[root@hetzner2 htdocs]#
- updated all the wp dashboard config settings for the new plugins
- discovered that the "Purge ALL Varnish Cache" button resulted in an error
Trying to purge URL : http://127.0.0.1:6081/.* => 403 Forbidden
- there's a cooresponding ossec alert & trigger by modsec:
--388c9d2b-A--
[09/Feb/2018:00:38:31 +0000] WnzthxK3iH4mmnFop@Wm5gAAAAc 127.0.0.1 37192 127.0.0.1 8000
--388c9d2b-B--
PURGE /.* HTTP/1.1
User-Agent: WordPress/4.9.3; https://osemain.opensourceecology.org
Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5
host: osemain.opensourceecology.org
X-VC-Purge-Method: regex
X-VC-Purge-Host: osemain.opensourceecology.org
Connection: Close, close
X-Forwarded-For: 127.0.0.1
X-Varnish: 45279
--388c9d2b-F--
HTTP/1.1 403 Forbidden
Content-Length: 204
Connection: close
Content-Type: text/html; charset=iso-8859-1
--388c9d2b-E--
--388c9d2b-H--
Message: Access denied with code 403 (phase 1). Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf"] [line "31"] [id "960032"] [rev "2"] [msg "Method is not allowed by policy"] [data "PURGE"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"]
Action: Intercepted (phase 1)
Stopwatch: 1518136711479621 232 (- - -)
Stopwatch2: 1518136711479621 232; combined=97, p1=83, p2=0, p3=0, p4=0, p5=14, sr=18, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"
--388c9d2b-Z--
- added id = 960032 to the httpd config whitelist
- varnish log alerts showed that the purge request was being sent to obi, so I added it to that apache config too
- confirmed this works in seedhome.openbuildinginstitue.org
- confirmed this works in fef.opensourceecology.org
- ah, I think this fails because the varnish config is defined to use 'www.opensourceecology.org' it's temporarily 'osemain.opensourceecology.org' due to a name collision with the prod site. I'm confident this will work once the name is correct after cuttover. ignoring this issue. in the meantime, I'm pretty sure nothing is being cached at all (which is actually nice for testing the staging site).
- the top-left logo shows up fine on my disposable vm, but not in my locked-down https-everywhere browser http://opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png
- should probably change the url to use https instead in Appearance -> Theme Options -> Site Logo to use https instead (note I can't test this until after the name change)
- discovered that the facebook iframe on the main page was missing
- solution was to change 'http' to 'https'
[segment class="first-segment"] <div class="center"><iframe style="border: none; overflow: hidden; width: 820px; height: 435px; background: white; float: center;" src="https://www.facebook.com/plugins/likebox.php?href=http%3A%2F%2Fwww.facebook.com%2FOpenSourceEcology&width=820&colorscheme=light&show_faces=true&border_color&stream=true&header=true&height=435" width="300" height="150" frameborder="0" scrolling="yes"></iframe></div> [/segment]
- confirmed that the old broken pages are still working
- /contributors/
- the top slider that Catarina said we've permenantly lost is still throwing an error; Marcin will remove this from the prod site, so this will be a non-issue at cutover
- the timeline shows up properly
- /community-true-fans/
- this page perfectly matches prod
- /history-timeline/
- this page perfectly matches prod
- /cnc-torch-table-workshop/
- Eventbright on left still working as desired
- videos embed issues are still present as expected, and a non-blocker to the migration
- /about-videos-3/
- missing video content is still (not) present as expected, and a non-blocker to the migration
- /contributors/
- all looks good to me. sent an email to Marcin asking for validation.
- awstats were not updating again
- found a permissions issue
[root@hetzner2 conf.d]# ls -lah /var/log/nginx/ total 11M drwx------ 9 nginx nginx 4.0K Feb 8 07:21 . drwxr-xr-x 13 root root 12K Feb 8 07:21 .. -rw-r--r-- 1 nginx nginx 3.5M Feb 9 02:03 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 143K Jan 30 05:02 access.log-20180130.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 4.8M Feb 8 07:21 access.log-20180208 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 323K Feb 9 02:02 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 7.8K Jan 30 05:00 error.log-20180130.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 433K Feb 8 07:21 error.log-20180208 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 root root 4.0K Feb 8 07:21 fef.opensourceecology.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 forum.opensourceecology.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 oswh.opensourceecology.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 seedhome.openbuildinginstitute.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 wiki.opensourceecology.org drw-r--r-- 2 nginx nginx 4.0K Feb 9 02:01 www.openbuildinginstitute.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 www.opensourceecology.org [root@hetzner2 conf.d]#
- those probably should all be owned by nginx
[root@hetzner2 nginx]# chown nginx:nginx fef.opensourceecology.org [root@hetzner2 nginx]# chown -R nginx:nginx fef.opensourceecology.org [root@hetzner2 nginx]# chown -R nginx:nginx forum.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx oswh.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx seedhome.openbuildinginstitute.org/ [root@hetzner2 nginx]# chown -R nginx:nginx wiki.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx www.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx www.openbuildinginstitute.org/ [root@hetzner2 nginx]# ls -lah total 11M drwx------ 9 nginx nginx 4.0K Feb 8 07:21 . drwxr-xr-x 13 root root 12K Feb 8 07:21 .. -rw-r--r-- 1 nginx nginx 3.5M Feb 9 02:06 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 143K Jan 30 05:02 access.log-20180130.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 4.8M Feb 8 07:21 access.log-20180208 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 324K Feb 9 02:06 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 7.8K Jan 30 05:00 error.log-20180130.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 433K Feb 8 07:21 error.log-20180208 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 fef.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 forum.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 oswh.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 seedhome.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 wiki.opensourceecology.org drw-r--r-- 2 nginx nginx 4.0K Feb 9 02:01 www.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 www.opensourceecology.org [root@hetzner2 nginx]#
- that appears to be working. Now I just have to wait another week or so..
- added an 'awstats' record to opensourceecology.org in cloudflare.com
Wed Feb 07, 2018
- Marcin responded to the osemain issues. Here's the status:
- /
- Marcin confirmed that front-page content (the video) this was fixed
- /contributors/
- Catarina will be looking into the "featured collaborators" slider issue on this page which is broken on production & staging
- I'll look into the "collaborators timeline" issue that is fine on prod, but missing on staging
- Marcin said the missing photos on the "collaborators timeline" should be dealt with later & not block this migration
- /community/
- Marcin updated the site, removing the broken Flattr embed http://opensourceecology.org/community/
- /community-true-fans/
- I need to investigate why these 2x sliders are broken in staging, but working in production
- /history-timeline/
- I need to investigate why these 2x sliders are broken in staging, but working in production
- /
/cnc-torch-table-workshop/
- Eventbright is truncated on the left
- The first youtube video shows up as a link, instead of as an embed = https://www.youtube.com/watch?v=JOH03vzDYQg
- The first vemo video shows up as a link, instead of as an embed = https://vimeo.com/23785186
/about-videos-3/
- Marcin said the videos are missing. I asked him to attempt to add them & if he encountered any issues, send me the errors so I can try to reproduce & fix it.
- began debugging osemain staging slider issues on /contributors/
- first, none of the plugins were enabled. This was a permission issue, solved by following the guide at Wordpress
vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"
- next "revolution slider" wasn't enabled not sure why. I enabled it and the one named "Revolution Slider Patch"
- finally, I saw that the last slider wasn't actually a slider; it was just an iframe to an http site = http://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0ArpE5Y9PpJCXdFhleTktTkoyeHNFUXktUXJCMGVkbVE&font=Bevan-PotanoSans&maptype=toner&lang=en&start_at_end=true&hash_bookmark=true&height=650#16
- replacing the above iframe's http protocol with https fixed the issue, but first I had to whitelist some modsec triggers to be able to update the page
- 958057, XSS
- 958056, XSS
- 973327, XSS
- began debugging issues on /community-true-fans/
- same issue, there was an iframe linking to an http page on cdn.knightlab.com; changing to https fixed it https://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0ArpE5Y9PpJCXdEhYb1NkR1dGRnhPV2FGYndseUg2WkE&font=Merriweather-NewsCycle&maptype=toner&lang=en&start_at_end=true&start_at_slide=34&height=650
- began debugging issues on /history-timeline/
- same issue, there was an iframe linking to an http page on cdn.knightlab.com; changing to https fixed it https://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0AkNG-lv1ELQvdEMxdzRnU2VFbUllZ2Y0cnZPRld3SXc&font=Bevan-PotanoSans&maptype=toner&lang=en&hash_bookmark=true&start_at_slide=23&height=650
- began troubleshooting /cnc-torch-table-workshop/
- found a section on the page edit called "Choose Custom Sidebar" and a drop-down for "Primary Sidebar Choice" said "CNC Torch Table Workshop". So I opened Appearence -> Widgets in a new tab, and found the "CNC Torcn Table Workshop" widget with this contents:
<a href="http://opensourceecology.org/CNC-torch-table-workshop/#overview">Overview</a> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#learning">Learning Outcomes</a> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#schedule">Schedule</a> <a href="http://opensourceecology.org/CNC-torch-table-workshop#registration">Registration</a> <div style="width:195px; text-align:center;" ><iframe src="https://www.eventbrite.com/countdown-widget?eid=38201763503" frameborder="0" height="322" width="195" marginheight="0" marginwidth="0" scrolling="no" allowtransparency="true"></iframe><div style="font-family:Helvetica, Arial; font-size:12px; padding:10px 0 5px; margin:2px; width:195px; text-align:center;" ><a class="powered-by-eb" style="color: #ADB0B6; text-decoration: none;" target="_blank" href="http://www.eventbrite.com/">Powered by Eventbrite</a></div></div>
- I changed the height of "322" of the iframe to "999", pressed save, and cleared the varnish cache. That didn't help
- I added ";height:999px" to the containing div, pressed save, and cleared the varnish cache.
- I replaced the "CNC Torch Table Workshop" widget of type "Text" with one of the same contents of type "Custom HTML", and it worked.
- the "Custom HTML" widget was added in wordpress v4.8.1 https://wptavern.com/wordpress-4-8-1-adds-a-dedicated-custom-html-widget
- this makes sense, as this migration includes the update of core wp from v4.7.9 to v4.9.2
- I had to add some breaks to make the links on separate lines; I guess that's what "Text" does for you--and also breaks dynamic height? Here's the new contents:
- ugh, fucking hell. I can't copy from the new "Custom HTML widget?" It adds line numbers to my clipboard, breaking copy-and-paste to/from the window. This is horrible!
- I tried switching back to "text" so I could just try to uncheck the "automatically add paragraphs" checkbox, but that's only a "legacy" thing (as the link above points-out. Once you delete a text box & re-add it, that checkbox disappears :(
- it looks like this new "custom html" widget is powered by codemirror
- ok, I found the solution to the annoying codemirror issues, but it can't be fixed site-wide. It must be fixed on a per-user basis by going to the user's profile & checking the "Disable syntax highlighting when editing code" checkbox https://make.wordpress.org/core/tag/codemirror/
- ok, here's the new contents with line breaks:
- the "Custom HTML" widget was added in wordpress v4.8.1 https://wptavern.com/wordpress-4-8-1-adds-a-dedicated-custom-html-widget
<a href="http://opensourceecology.org/CNC-torch-table-workshop/#overview">Overview</a> <br/> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#learning">Learning Outcomes</a> <br/> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#schedule">Schedule</a> </> <a href="http://opensourceecology.org/CNC-torch-table-workshop#registration">Registration</a><div style="width:195px; text-align:center;" > <br/> <div style="width:195px; text-align:center;" ><iframe src="https://www.eventbrite.com/countdown-widget?eid=38201763503" frameborder="0" height="322" width="195" marginheight="0" marginwidth="0" scrolling="no" allowtransparency="true"></iframe><div style="font-family:Helvetica, Arial; font-size:12px; padding:10px 0 5px; margin:2px; width:195px; text-align:center;" ><a class="powered-by-eb" style="color: #ADB0B6; text-decoration: none;" target="_blank" href="http://www.eventbrite.com/">Powered by Eventbrite</a></div></div>
- I emailed marcin; he said that the issue embedding the 2x videos on this page shouldn't be a blocker & confirmed that I should proceed with the migration with the content missing
- began debugging the missing videos on /about-video-3/
- there's literally nothing in the textarea for this page in wordpress. It's empty for both staging & production *shrug*
- I emailed marcin; he said this shouldn't be a blocker & confirmed that I should proceed with the migration with the content missing
- So the osemain issues that remain to be fixed are:
- /contributors/
- Catarina will be looking into the "featured collaborators" slider issue on this page which is broken on production & staging
- /contributors/
/cnc-torch-table-workshop/
Tue Feb 06, 2018
- the osemain staging site was created on 2018-01-03
[maltfield@hetzner2 backups_for_migration_from_hetzner1]$ date Tue Feb 6 15:53:00 UTC 2018 [maltfield@hetzner2 backups_for_migration_from_hetzner1]$ pwd /var/tmp/backups_for_migration_from_hetzner1 [maltfield@hetzner2 backups_for_migration_from_hetzner1]$ ls -lah | grep -i osemain drwxr-xr-x 4 root root 4.0K Jan 3 21:32 osemain_20180103 [maltfield@hetzner2 backups_for_migration_from_hetzner1]$
- I refreshed it for today = 2018-02-06
# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180206"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
dbUser_hetzner2="osemain_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
- I updated the osemain site with a new wp core, but stopped short of installing our new security wp plugins or attempting to update the existing wp plugins. Here's a breakdown of what I found with the site:
- /
- vid fixed w/ update
- /contributors/
- revolution slider is broken on the production site with error message = "Revolution Slider Error: Slider with alias collabslider not found."
- collaborators timeline is missing entirely on the staging site
- many photos 404ing for the collaborators timeline on the production site
- /community/
- /
/community-true-fans/
- 2x sliders missing on staging; working fine on production
/history-timeline/
- 2x sliders missing on staging; working fine on production
/cnc-torch-table-workshop/
- Eventbright is truncated on the left
- The first youtube video shows up as a link, instead of as an embed = https://www.youtube.com/watch?v=JOH03vzDYQg
- The first vemo video shows up as a link, instead of as an embed = https://vimeo.com/23785186
/about-videos-3/
- Marcin said the videos are missing. I asked him to attempt to add them & if he encountered any issues, send me the errors so I can try to reproduce & fix it.
Mon Feb 05, 2018
- DIdn't hear back from Marcin wrt my email about the osemigration, so I decided to delay the migration CHG-2018-02-05
Sun Feb 04, 2018
- Marcin q confirmed that we can migrate the osemain wp site on Monday, Feb 5th.
- I began documenting this CHG process at CHG-2018-02-05
- I found that many of the plugins 3 of the active plugins had pending updates: fundraising, wpmdev-updates, and wp-smush-pro
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------------------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------------------------------+----------+-----------+---------+
| akismet | active | none | 3.3 |
| brankic-photostream-widget | active | none | 1.3.1 |
| cyclone-slider-2 | inactive | none | 2.12.4 |
| duplicate-post | active | none | 3.1.2 |
| flattr | inactive | none | 1.2.2 |
| force-strong-passwords | active | none | 1.8.0 |
| fundraising | active | available | 2.6.4.5 |
| google-authenticator | active | none | 0.48 |
| google-authenticator-encourage-user-activat | active | none | 0.2 |
| ion | | | |
| hello | inactive | none | 1.6 |
| insert-headers-and-footers | inactive | none | 1.4.1 |
| jetpack | active | none | 4.7.1 |
| ml-slider | active | none | 3.5 |
| ml-slider-pro | active | none | 2.6.6 |
| open-in-new-window-plugin | inactive | none | 2.4 |
| patch-for-revolution-slider | active | none | 2.4.1 |
| post-types-order | inactive | none | 1.9.3 |
| really-simple-facebook-twitter-share-button | inactive | none | 4.5 |
| s | | | |
| recent-tweets-widget | active | none | 1.6.6 |
| rename-wp-login | active | none | 2.5.5 |
| revision-control | inactive | none | 2.3.2 |
| revslider | active | none | 4.3.8 |
| shareaholic | inactive | none | 7.8.0.4 |
| share-on-diaspora | inactive | none | 0.7.1 |
| ssl-insecure-content-fixer | active | none | 2.5.0 |
| vcaching | active | none | 1.6.7 |
| w3-total-cache | inactive | none | 0.9.5.2 |
| wp-memory-usage | inactive | none | 1.2.2 |
| wp-optimize | inactive | none | 2.1.1 |
| wp-facebook-open-graph-protocol | inactive | none | 2.0.13 |
| wpmudev-updates | active | available | 4.2 |
| wp-smushit | inactive | none | 2.6.1 |
| wp-smush-pro | active | available | 2.4.5 |
+---------------------------------------------+----------+-----------+---------+
[WPMUDEV API Error] 4.2 | User has blocked requests through HTTP. ((unknown URL) [500])
[root@hetzner2 ~]#
- that update attempt failed
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Warning: Update package not available.
Warning: Update package not available.
Warning: Update package not available.
+-----------------+-------------+-------------+--------+
| name | old_version | new_version | status |
+-----------------+-------------+-------------+--------+
| fundraising | 2.6.4.5 | 2.6.4.9 | Error |
| wpmudev-updates | 4.2 | 4.4 | Error |
| wp-smush-pro | 2.4.5 | 2.7.6 | Error |
+-----------------+-------------+-------------+--------+
Success: Plugins already updated.
[root@hetzner2 ~]#
- after commenting-out the WP_HTTP_BLOCK_EXTERNAL set to 'true' in wp-config.php, however, I see there were many more updates available
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------------------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------------------------------+----------+-----------+---------+
| akismet | active | available | 3.3 |
| brankic-photostream-widget | active | none | 1.3.1 |
| cyclone-slider-2 | inactive | available | 2.12.4 |
| duplicate-post | active | available | 3.1.2 |
| flattr | inactive | none | 1.2.2 |
| force-strong-passwords | active | none | 1.8.0 |
| fundraising | active | available | 2.6.4.5 |
| google-authenticator | active | none | 0.48 |
| google-authenticator-encourage-user-activat | active | none | 0.2 |
| ion | | | |
| hello | inactive | none | 1.6 |
| insert-headers-and-footers | inactive | available | 1.4.1 |
| jetpack | active | available | 4.7.1 |
| ml-slider | active | available | 3.5 |
| ml-slider-pro | active | none | 2.6.6 |
| open-in-new-window-plugin | inactive | none | 2.4 |
| patch-for-revolution-slider | active | none | 2.4.1 |
| post-types-order | inactive | available | 1.9.3 |
| really-simple-facebook-twitter-share-button | inactive | none | 4.5 |
| s | | | |
| recent-tweets-widget | active | available | 1.6.6 |
| rename-wp-login | active | none | 2.5.5 |
| revision-control | inactive | none | 2.3.2 |
| revslider | active | none | 4.3.8 |
| shareaholic | inactive | available | 7.8.0.4 |
| share-on-diaspora | inactive | available | 0.7.1 |
| ssl-insecure-content-fixer | active | none | 2.5.0 |
| vcaching | active | none | 1.6.7 |
| w3-total-cache | inactive | available | 0.9.5.2 |
| wp-memory-usage | inactive | none | 1.2.2 |
| wp-optimize | inactive | none | 2.1.1 |
| wp-facebook-open-graph-protocol | inactive | none | 2.0.13 |
| wpmudev-updates | active | available | 4.2 |
| wp-smushit | inactive | available | 2.6.1 |
| wp-smush-pro | active | available | 2.4.5 |
+---------------------------------------------+----------+-----------+---------+
[root@hetzner2 ~]#
- then attempting updates again was partially successful
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pea
r:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.
opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/va
r/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Downloading update from https://downloads.wordpress.org/plugin/akismet.4.0.2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/akismet-4.0.2.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/cyclone-slider-2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/cyclone-slider-2-3.2.0.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/duplicate-post.3.2.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
Downloading update from https://downloads.wordpress.org/plugin/insert-headers-and-footers.1.4.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/jetpack.5.7.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/ml-slider.3.6.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from http://wp-updates.com/api/1/download/plugin/sXpq1rDdVNBB4r8Tm-EPBg/ml-slider-pro...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/post-types-order.1.9.3.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/recent-tweets-widget.1.6.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/shareaholic.8.6.1.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/shareaholic-8.6.1.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/share-on-diaspora.0.7.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/w3-total-cache.0.9.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
Downloading update from https://downloads.wordpress.org/plugin/wp-smushit.2.7.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
+----------------------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------------------+-------------+-------------+---------+
| akismet | 3.3 | 4.0.2 | Updated |
| cyclone-slider-2 | 2.12.4 | 3.2.0 | Updated |
| duplicate-post | 3.1.2 | 3.2.1 | Updated |
| fundraising | 2.6.4.5 | 2.6.4.9 | Updated |
| insert-headers-and-footers | 1.4.1 | 1.4.2 | Updated |
| jetpack | 4.7.1 | 5.7.1 | Updated |
| ml-slider | 3.5 | 3.6.8 | Updated |
| ml-slider-pro | 2.6.6 | 2.7.1 | Updated |
| post-types-order | 1.9.3 | 1.9.3.6 | Updated |
| recent-tweets-widget | 1.6.6 | 1.6.8 | Updated |
| shareaholic | 7.8.0.4 | 8.6.1 | Updated |
| share-on-diaspora | 0.7.1 | 0.7.2 | Updated |
| w3-total-cache | 0.9.5.2 | 0.9.6 | Updated |
| wpmudev-updates | 4.2 | 4.4 | Updated |
| wp-smushit | 2.6.1 | 2.7.6 | Updated |
| wp-smush-pro | 2.4.5 | 2.7.7 | Updated |
+----------------------------+-------------+-------------+---------+
Success: Updated 16 of 16 plugins.
[root@hetzner2 ~]#
- what follows showed everything was updated, except the same 3 plugins that fail. So it's probably those specific plugins' fault. Let's move on..
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
...
+---------------------------------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------------------------------+----------+-----------+---------+
| akismet | active | none | 4.0.2 |
| brankic-photostream-widget | active | none | 1.3.1 |
| cyclone-slider-2 | inactive | none | 3.2.0 |
| duplicate-post | active | none | 3.2.1 |
| flattr | inactive | none | 1.2.2 |
| force-strong-passwords | active | none | 1.8.0 |
| fundraising | active | available | 2.6.4.5 |
| google-authenticator | active | none | 0.48 |
| google-authenticator-encourage-user-activat | active | none | 0.2 |
| ion | | | |
| hello | inactive | none | 1.6 |
| insert-headers-and-footers | inactive | none | 1.4.2 |
| jetpack | active | none | 5.7.1 |
| ml-slider | active | none | 3.6.8 |
| ml-slider-pro | active | none | 2.7.1 |
| open-in-new-window-plugin | inactive | none | 2.4 |
| patch-for-revolution-slider | active | none | 2.4.1 |
| post-types-order | inactive | none | 1.9.3.6 |
| really-simple-facebook-twitter-share-button | inactive | none | 4.5 |
| s | | | |
| recent-tweets-widget | active | none | 1.6.8 |
| rename-wp-login | active | none | 2.5.5 |
| revision-control | inactive | none | 2.3.2 |
| revslider | active | none | 4.3.8 |
| shareaholic | inactive | none | 8.6.1 |
| share-on-diaspora | inactive | none | 0.7.2 |
| ssl-insecure-content-fixer | active | none | 2.5.0 |
| vcaching | active | none | 1.6.7 |
| w3-total-cache | inactive | none | 0.9.6 |
| wp-memory-usage | inactive | none | 1.2.2 |
| wp-optimize | inactive | none | 2.1.1 |
| wp-facebook-open-graph-protocol | inactive | none | 2.0.13 |
| wpmudev-updates | active | available | 4.2 |
| wp-smushit | inactive | none | 2.7.6 |
| wp-smush-pro | active | available | 2.4.5 |
+---------------------------------------------+----------+-----------+---------+
[root@hetzner2 ~]#
- Found a bunch of content issues on the site when doing a quick spot-check. Emailed Marcin for revalidation & a go/no-go decision for the cutover tomorrow
- error "Revolution Slider Error: Slider with alias collabslider not found."
* http://opensourceecology.org/contributors/ * https://osemain.opensourceecology.org/contributors/
- flattr embed is broken on donate section
* http://opensourceecology.org/community/ * https://osemain.opensourceecology.org/community/
- slider for True Fans isn't appearing
* http://opensourceecology.org/community-true-fans/ * https://osemain.opensourceecology.org/community-true-fans/
- added a CHG for the forum cutover CHG-2018-02-04
- followed the above guide. We're now storing the dynamic (php) vanilla files outside the docroot, next to the static content. Of course, the static content takes up more space, but it's not too bad.
[root@hetzner2 forum.opensourceecology.org]# date Sun Feb 4 18:13:00 UTC 2018 [root@hetzner2 forum.opensourceecology.org]# pwd /var/www/html/forum.opensourceecology.org [root@hetzner2 forum.opensourceecology.org]# du -sh * 2.7G htdocs 173M vanilla_docroot_backup.20180113 [root@hetzner2 forum.opensourceecology.org]#
- updated dns of forum.opensourceecology.org to point to hetzner2 instead of hetzner1
- deleted the CNAME entry for 'forum' pointing to dedi978.your-server.de
- added an A entry for 'forum' pointing to 138.201.84.243
- updated the name 'stagingforum' to 'forum' in /etc/httpd/conf.d/00-forum.opensourceecology.org.conf
- updated the name 'stagingforum' to 'forum' in /etc/nginx/conf.d/forum.opensourceecology.org.conf
- updated the opensourceecology.org certificate
[root@hetzner2 forum.opensourceecology.org]# certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org ... IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/opensourceecology.org/privkey.pem Your cert will expire on 2018-05-05. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le [root@hetzner2 forum.opensourceecology.org]# /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* [root@hetzner2 forum.opensourceecology.org]# nginx -t && service nginx reload nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Redirecting to /bin/systemctl reload nginx.service [root@hetzner2 forum.opensourceecology.org]#
- Did an unsafe varnish cache clear
service varnish restart
- confirmed that the static-content site was loading on the new url https://forum.opensourceecology.org
- updated our wiki (where the forum links to for info on how to create an account) with a notice that the site was deprecated in 2018 Vanilla_Forums
- began to test sending our encrypted ossec alerts to multiple recipients
- discovered that our spf record is invalid. I looked up the proper way to do this on cloudflare https://support.cloudflare.com/hc/en-us/articles/200168626-How-do-I-add-a-SPF-record-
- I had "spf" in the "Name" column, but it should have been "@". I updated this, and then Cloudflare automatically replaced the "@" that I typed with "opensourceecology.org"
- after the change above, I got a change in the lookup tool; success! https://mxtoolbox.com/SuperTool.aspx?action=spf%3aopensourceecology.org&run=toolpage
- updated the sent_encrypted_alarm.sh script to support multiple recipients
[root@hetzner2 ossec]# date
Mon Feb 5 01:39:24 UTC 2018
[root@hetzner2 ossec]# pwd
/var/ossec
[root@hetzner2 ossec]# cat sent_encrypted_alarm.sh
#!/bin/bash
# store the would-be plaintext email body
plaintext=`/usr/bin/formail -I ""`
# loop through all recipients & send them individually-encrypted mail
recipients="michael@opensourceecology.org marcin@opensourceecology.org"
for recipient in $(echo $recipients); do
# leave nothing unencrypted, including the subject!
echo "${plaintext}" | /usr/bin/gpg --homedir /var/ossec/.gnupg --trust-model always -ear "${recipient}" | /usr/bin/mail -r noreply@opensourceecology.org -s "" "${recipient}"
done
exit 0
- found & documented command to clear varnish cache without actually restarting the process Web_server_configuration
varnishadm 'ban req.url ~ "."'
Thr Feb 01, 2018
- confirmed that the forum wget retry worked
- rsync'd the wget into the document root, and a spot-checked showed that most pages worked
- saw an issue (Forbidden) with all the tagged pages, ie https://stagingforum.opensourceecology.org/discussions/tagged/water.html
[Thu Feb 01 17:29:02.262787 2018] [core:crit] [pid 30854] (9)Bad file descriptor: [client 127.0.0.1:45488] AH00529: /var/www/html/forum.opensourceecology.org/htdocs/discussions/tagged/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/html/forum.opensourceecology.org/htdocs/discussions/tagged/' is executable
- the issue appears to be with the directory named '.htaccess' for the tag of the same name
[root@hetzner2 htdocs]# ls -lah discussions/tagged/.htaccess total 64K drwxr-xr-x 2 root root 4.0K Feb 1 11:36 . drwxr-xr-x 573 root root 36K Feb 1 17:27 .. -rw-r--r-- 1 root root 1.7K Feb 1 10:33 feed.rss -rw-r--r-- 1 root root 17K Feb 1 11:36 p1.html [root@hetzner2 htdocs]#
- removing the above dir worked. The consequence is we will get a 404 for any links to this tag, but that's an acceptable loss IMO; te discussions themselves will still be accessible
- removed the forums dir from /etc/php.ini since we're now just static content
- restarted httpd
- confirmed that awstats has been updating, but--since today happens to be the first of the month--there's no data yet. I'll check again in a week or so.
Wed Jan 31, 2018
- moved the wget'd forums into the docroot & found a lot of Forbidden responses to my requests
- for some reason, the pages are named 'p1' instead of 'index.html' The "Forbidden" is caused by the server's refusal to do a directory listing.
- solution: add a symlink to "p1" from "index.html" in every dir with a "p1" but no "index.html"
[root@hetzner2 htdocs]# for p1 in $(find . -name p1); do dir=`dirname $p1`; pushd $dir; ln -s "p1" "index.html"; popd; done
- that helped, but there's still actually missing content (ie: https://stagingforum.opensourceecology.org/categories/other-languages/)
[root@hetzner2 htdocs]# ls -lah categories/other-languages/ total 12K drwxrwxrwx 2 root root 4.0K Jan 25 16:23 . drwxrwxrwx 55 root root 4.0K Jan 25 20:52 .. -rwxrwxrwx 1 root root 3.5K Jan 25 16:23 feed.rss [root@hetzner2 htdocs]#
- attempting again with a different command from https://www.linuxjournal.com/content/downloading-entire-web-site-wget
[root@hetzner2 oseforum.20180201]# date Thu Feb 1 03:39:34 UTC 2018 [root@hetzner2 oseforum.20180201]# pwd /var/tmp/deleteMeIn2019/oseforum.20180201 [root@hetzner2 oseforum.20180201]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "forum.opensourceecology.org" "http://forum.opensourceecology.org" <pre> ## if the above doesn't work, I'll try some of the options in the comments (ie: -m, --wait, --limit-rate, etc) =Tue Jan 30, 2018= # Trained Marcin on PGP =Fri Jan 26, 2018= # Emailed Marcin about setting up PGP # Confirmed that the wget of the forums finished <pre> [root@hetzner2 wget]# pwd /var/tmp/deleteMeIn2019/oseoforum.20180125/wget [root@hetzner2 ~]# wget -r -p -e robots=off http://forum.opensourceecology.org ... FINISHED --2018-01-26 00:53:29-- Total wall clock time: 8h 39m 3s Downloaded: 96387 files, 2.2G in 1m 33s (24.5 MB/s) [root@hetzner2 wget]# du -sh forum.opensourceecology.org/* 48K forum.opensourceecology.org/activity 264K forum.opensourceecology.org/applications 300K forum.opensourceecology.org/cache 3.0M forum.opensourceecology.org/categories 27M forum.opensourceecology.org/dashboard 1.4G forum.opensourceecology.org/discussion 15M forum.opensourceecology.org/discussions 873M forum.opensourceecology.org/entry 148K forum.opensourceecology.org/index.html 744K forum.opensourceecology.org/plugins 72M forum.opensourceecology.org/profile 12K forum.opensourceecology.org/search 28K forum.opensourceecology.org/search?Search=#000000&Mode=like 28K forum.opensourceecology.org/search?Search=#0&Mode=like 16K forum.opensourceecology.org/search?Search=#13Lifetime&Mode=like 16K forum.opensourceecology.org/search?Search=#197187&Mode=like 32K forum.opensourceecology.org/search?Search=#1&Mode=like 28K forum.opensourceecology.org/search?Search=#2&Mode=like 16K forum.opensourceecology.org/search?Search=#363636&Mode=like 28K forum.opensourceecology.org/search?Search=#3&Mode=like 16K forum.opensourceecology.org/search?Search=#458&Mode=like 20K forum.opensourceecology.org/search?Search=#4&Mode=like 28K forum.opensourceecology.org/search?Search=#5&Mode=like 20K forum.opensourceecology.org/search?Search=#6&Mode=like 16K forum.opensourceecology.org/search?Search=#7&Mode=like 16K forum.opensourceecology.org/search?Search=#8-High&Mode=like 28K forum.opensourceecology.org/search?Search=#8&Mode=like 16K forum.opensourceecology.org/search?Search=#9-Industrial&Mode=like 16K forum.opensourceecology.org/search?Search=#9&Mode=like 16K forum.opensourceecology.org/search?Search=#Another&Mode=like 16K forum.opensourceecology.org/search?Search=#apollo&Mode=like 16K forum.opensourceecology.org/search?Search=#Cloud&Mode=like 16K forum.opensourceecology.org/search?Search=#Edit&Mode=like 16K forum.opensourceecology.org/search?Search=#Fabiofranca&Mode=like 16K forum.opensourceecology.org/search?Search=#freecad&Mode=like 16K forum.opensourceecology.org/search?Search=#HUBCAMP&Mode=like 16K forum.opensourceecology.org/search?Search=#openhardware&Mode=like 16K forum.opensourceecology.org/search?Search=#opensourceecology&Mode=like 16K forum.opensourceecology.org/search?Search=#qi-hardware&Mode=like 16K forum.opensourceecology.org/search?Search=#qihardware&Mode=like 16K forum.opensourceecology.org/search?Search=#REDIRECT&Mode=like 16K forum.opensourceecology.org/search?Search=#toc&Mode=like 16K forum.opensourceecology.org/search?Search=#toctitle&Mode=like 24K forum.opensourceecology.org/themes 25M forum.opensourceecology.org/uploads 764K forum.opensourceecology.org/vanilla [root@hetzner2 wget]#
Thr Jan 25, 2018
- Marcin said that the forums initial validation is complete
- discovered that there was an 'oseforum' directory inside the forums docroot that had a duplicate copy of all the files in the docroot.
- I moved this director to
- I began a quick investigation on the low-hanging-fruit to harden Vanilla
- I didn't find any good guides. Lots of discussion about permissions, though. https://open.vanillaforums.com/discussion/748/security
- the fucking installation guide tells us to make the conf directory inside the docroot *and* be writeable by the webserver. That's a foundational red-flag! https://github.com/vanilla/vanilla/blob/master/README.md
- probably the most important step to securing the forums would be updating the core Vanilla version
- The version of Vanilla Forums that we're running is Version 2.0.18.1
- The current version of Vanilla Forums is 2.5
- The number of plugins that we're using tells me that an update of the Vanilla Forums core code probably going to be extremely non-trivial
[root@hetzner2 htdocs]# date Thu Jan 25 13:54:16 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/forum.opensourceecology.org/htdocs [root@hetzner2 htdocs]# grep 'EnabledPlugins' conf/config.php // EnabledPlugins $Configuration['EnabledPlugins']['HtmLawed'] = 'HtmLawed'; $Configuration['EnabledPlugins']['embedvanilla'] = 'embedvanilla'; $Configuration['EnabledPlugins']['Tagging'] = 'Tagging'; $Configuration['EnabledPlugins']['Flagging'] = 'Flagging'; $Configuration['EnabledPlugins']['Liked'] = 'Liked'; $Configuration['EnabledPlugins']['OpenID'] = 'OpenID'; $Configuration['EnabledPlugins']['Twitter'] = 'Twitter'; $Configuration['EnabledPlugins']['Sitemaps'] = 'Sitemaps'; $Configuration['EnabledPlugins']['GoogleSignIn'] = 'GoogleSignIn'; $Configuration['EnabledPlugins']['Facebook'] = 'Facebook'; $Configuration['EnabledPlugins']['FileUpload'] = 'FileUpload'; $Configuration['EnabledPlugins']['VanillaStats'] = 'VanillaStats'; $Configuration['EnabledPlugins']['EMailSubscribe'] = 'EMailSubscribe'; $Configuration['EnabledPlugins']['Emotify'] = 'Emotify'; $Configuration['EnabledPlugins']['VanillaInThisDiscussion'] = 'VanillaInThisDiscussion'; $Configuration['EnabledPlugins']['ProxyConnect'] = 'ProxyConnect'; $Configuration['EnabledPlugins']['ProxyConnectManual'] = 'ProxyConnectManualPlugin'; $Configuration['EnabledPlugins']['cleditor'] = 'cleditor'; $Configuration['EnabledPlugins']['TinyMCE'] = 'TinyMCE'; $Configuration['EnabledPlugins']['Categories2Menu'] = TRUE; $Configuration['EnabledPlugins']['SplitMerge'] = TRUE; $Configuration['EnabledPlugins']['Voting'] = TRUE; $Configuration['EnabledPlugins']['FeedDiscussions'] = TRUE; $Configuration['EnabledPlugins']['StopForumSpam'] = TRUE; $Configuration['EnabledPlugins']['Minify'] = TRUE; $Configuration['EnabledPlugins']['Cleanser'] = TRUE; $Configuration['EnabledPlugins']['RegistrationRestrictLogger'] = TRUE; [root@hetzner2 htdocs]#
- I found no obvious way to make the forums read-only from the application-side
- except an old extension that hasn't been updated since 2005 https://open.vanillaforums.com/addon/readonly-plugin
- of course, we could achieve read-only by:
- revoking all access privlidges to the db user = ${dbUser_hetzner2} = 'oseforums_user' && granting them SELECT only access
- And making all of the files/directories read-only
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
dbUser_hetzner2="oseforum_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
time nice mysql -uroot -p${mysqlPass} -sNe "REVOKE ALL ON ${dbName_hetzner2}.* FROM '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"
# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0550 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0440 {} \;
chown apache:apache-admins "${docrootDir_hetzner2}/conf/config.php"
chmod 0440 "${docrootDir_hetzner2}/conf/config.php"
- ran the above commands & confirmed that the permissions changed to SELECT-only in the db
[root@hetzner2 htdocs]# nice mysql -uroot -p${mysqlPass} mysql -sNe "select * from db where Db = 'oseforum_db';"
localhost oseforum_db oseforum_user Y N N N N N N N N N N N N N N N N N N
[root@hetzner2 htdocs]#
- attempted to login, but then I found that the fucking forums are leaking back to the user our server-side error message (in json), that includes the username & hostname of the db server in the response to our login query! https://stagingforum.opensourceecology.org/entry/signin
{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}
- And then it created a popup window leaking all this info!
FATAL ERROR IN: Gdn_Database.Query();
"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'" update GDN_User User set DateLastActive = :DateLastActive, DateUpdated = :DateUpdated, UpdateIPAddress = :UpdateIPAddress where UserID = :UserID LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php > 276: > 277: if (!is_object($PDOStatement)) { > 278: trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR); > 279: } else if ($PDOStatement->execute($InputParameters) === FALSE) { >>> 280: trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR); > 281: } > 282: } else { > 283: $PDOStatement = $this->Connection()->query($Sql); > 284: } BACKTRACE: [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start(); [/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator(); [/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();
- found the source of the logs being sent to the user as the line:
$Configuration['Garden']['Errors']['MasterView'] = 'deverror.master.php';
- fixed the log leaking by commenting-out the above line & replacing it with:
$Configuration['Garden']['Errors']['LogEnabled'] = TRUE; $Configuration['Garden']['Errors']['LogFile'] = '';
- the above setting forced writes to the apache-defined error logfile at /var/log/httpd/forum.opensourceecology.org/error_log
[root@hetzner2 httpd]# date Thu Jan 25 15:41:25 UTC 2018 [root@hetzner2 httpd]# pwd /var/log/httpd [root@hetzner2 httpd]# tail -f forum.opensourceecology.org/access_log forum.opensourceecology.org/error_log ... ==> forum.opensourceecology.org/error_log <== [Thu Jan 25 15:39:48.700738 2018] [:error] [pid 24269] [client 127.0.0.1:56072] [Garden] /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php, 280, Gdn_Database.Query(), UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User', update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID
- ugh, but the response is actually still leaking info
<h1>FATAL ERROR IN: Gdn_Database.Query();</h1>
<div class="AjaxError">"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'"
update GDN_User User set
DateLastActive = :DateLastActive,
DateUpdated = :DateUpdated,
UpdateIPAddress = :UpdateIPAddress
where UserID = :UserID
LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php
> 276:
> 277: if (!is_object($PDOStatement)) {
> 278: trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR);
> 279: } else if ($PDOStatement->execute($InputParameters) === FALSE) {
>>> 280: trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR);
> 281: }
> 282: } else {
> 283: $PDOStatement = $this->Connection()->query($Sql);
> 284: }
BACKTRACE:
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start();
[/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator();
[/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();
</div>
- unfortunately, after a single login attempt, all future queries of any kind produce the same "Bonk" generic error message now. It appears that the fix is to clear the cookie named "Vanilla" for the domain on the client-side. By default, this cookie sticks-around for 1 month.
# I changed the LogEnabled flag to FALSE. That helped, but then the logs disappeared from the apache logfile & the response still contained the error (though a shorter one)!
<pre>
{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}
- I'm pretty fucking over Vanilla. They seem to not give a shit about security.
- Note: the leaks above are nothing more than we've openly disclosed in our specs & docs. However, error messages often contain very sensitive info. Sometimes they contain passwords. Hence, the fact that this is being leaked in general is a huge red flag. The fact that it's considered "normal" behaviour is terrifying. The fact that I googled around for a fix of this issue, and the only thing I found was a developer suggesting to use a browser-side debugger to find the server-side error messages is mortifying. Sending server-side errors to the client should be a opt-in feature. It should not be the default. But Vanilla seems to have it as the baked-in default with not only no way to disable it, but also no obvious way (if at all?) to disable it. This tells me that they're either incompetent or just don't care. That's not what we want from the developers of our tools.## phpbb looks a bit better, they have at least a few blog posts tagged "security" https://blog.phpbb.com/tag/security/
- phpbb has a wiki with a guide recommending to move config.php out of the docroot. Not one of the guides I found for Vanilla mentioned that obvious requirement. They were occupied with permissions, and arguing if taking away the execute bit mattered to php files (read: it doesn't). https://wiki.phpbb.com/Hardening_Tips
- began downloading a static-content snapshot of our forums with wget
[root@hetzner2 wget]# date Thu Jan 25 16:49:39 UTC 2018 [root@hetzner2 wget]# pwd /var/tmp/deleteMeIn2019/oseoforum.20180125/wget [root@hetzner2 wget]# wget -r -p -e robots=off http://forum.opensourceecology.org
- I'm concerned that the output above may be insanely long, since people can permalink to comments within discussions. So effectively we'll store a single thread N times where N is the number of comments in the thread :(
- sent an email to Marcin asking if he was ready to fully archive our fourms to static content
- I discovered that my defined hardened permissions for Wordpress should be improved such that
- All of the files & directories that don't need rw permissions should not have write permissions. That's every file in a wordpress docroot except the folder "wp-content/uploads" and its subfiles/dirs.
- World permissions (not-user && not-group) for all files & directories inside the docroot (and including the docroot dir itself!) should be set to 0 for all files & all directories.
- Excluding 'wp-content/uploads/', these files should also not be owned by the user that runs a webserver (in cent, that's the 'apache' user). For even if the file is set to '0400', but it's owned by the 'apache' user, the 'apache' user can ignore the permissions & write to it anyway.
- Excluding 'wp-content/uploads/', all directories in the docroot (including the docroot dir itself!) should be owned by a group that contains the user that runs our webserver (in cent, that's the apache user). The permissions for this group must be not include write access for files or directories. For even if a file is set to '0040', but the containing directory is '0060', any user in the group that owns the directory can delete the existing file and replace it with a new file, effectively ignoring the read-only permission set for the file.
- currently, our documentation reads:
vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"
- but I believe the (untested) update should be:
vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"
chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0060 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0070 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"
- ...such that:
- the 'not-apache' user is a new user that doesn't run any software (ie: a daemon such as a web server) and whose shell is "/sbin/nologin" and home is "/dev/null".
- the apache user is now in the apache-admins group
- the result will now be that:
- ...such that:
- a compromised web server can no longer write data to a docroot (ie: adding malicious code to a php script) outside the 'wp-content/uploads/' directory
- for anyone to make changes to any files in the docroot (other than 'wp-content/uploads/'), they must be the root user. I think this is fair if they don't have the skills necessary to become root, they probably shouldn't fuck with the wp core files anyway.
- however, as with before, any user in the 'apache' group can read most files in the docroot. So, if we want an OSE Developer with ssh access to be able to access our server's files read-only, we should add them to the 'apache' group. If we trust them with passwords as well, we should additionally add them to the 'apache-admins' group.
Tue Jan 23, 2018
- found the best way to browse the git sourcecode of the mediawiki core at this URL https://phabricator.wikimedia.org/source/mediawiki/tags/master/
- for example, for v1.30.0 https://phabricator.wikimedia.org/source/mediawiki/browse/REL1_30/
- or for the 'vendor' directory https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/
- using composer to populate the 'vendor' directory does not seem like a practical options with our hardened php config.
- I attempted to download the actual tarball, copy-in the 'vendor' directory, update permissions, and then run the update
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
mkdir /var/tmp/mediawiki-1.30.0
pushd /var/tmp/mediawiki-1.30.0
wget https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz
tar -xzvf mediawiki-1.30.0.tar.gz
cd mediawiki-1.30.0
rm -rf /var/www/html/wiki.opensourceecology.org/htdocs/vendor
cp -r vendor /var/www/html/wiki.opensourceecology.org/htdocs/
# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"
/var/www/html/wiki.opensourceecology.org/htdocs/maintenance
php update.php
popd
- now I got a fatal php error
[Tue Jan 23 22:02:07.182288 2018] [:error] [pid 31837] [client 127.0.0.1:37506] PHP Fatal error: Class 'Wikimedia\\WrappedString' not found in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Gadgets/GadgetHooks.php on line 217
- I changed the require_once() to wfLoadExtension() in LocalSettings.php
- I found that the directory path for the "WrappedString.php" file is distinct for the vendors dirs in [a] the git repo vs [b] the extracted tarball release.
# the tarball has an src dir with the 2 files (WrappedString.php & WrappedStringList.php) [root@hetzner2 htdocs]# ls -lah /var/tmp/mediawiki-1.30.0/mediawiki-1.30.0/vendor/wikimedia/wrappedstring/src/ total 16K drwxr-xr-x 2 501 games 4.0K Dec 8 23:20 . drwxr-xr-x 3 501 games 4.0K Dec 8 23:20 .. -rw-r--r-- 1 501 games 3.5K Dec 8 23:20 WrappedStringList.php -rw-r--r-- 1 501 games 3.3K Dec 8 23:20 WrappedString.php # but the git repo has an src dir with 2 dirs = WrappedString & Wikimedia. Both of *these* dirs have the 2 files https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/wikimedia/wrappedstring/src/ [root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/ total 16K drwxr-xr-x 4 root root 4.0K Jan 22 21:22 . drwxr-xr-x 3 root root 4.0K Jan 22 21:22 .. drwxr-xr-x 2 root root 4.0K Jan 22 21:22 Wikimedia drwxr-xr-x 2 root root 4.0K Jan 22 21:22 WrappedString [root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/WrappedString/ total 16K drwxr-xr-x 2 root root 4.0K Jan 22 21:22 . drwxr-xr-x 4 root root 4.0K Jan 22 21:22 .. -rw-r--r-- 1 root root 98 Jan 22 21:22 WrappedStringList.php -rw-r--r-- 1 root root 90 Jan 22 21:22 WrappedString.php [root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/Wikimedia/ total 16K drwxr-xr-x 2 root root 4.0K Jan 22 21:22 . drwxr-xr-x 4 root root 4.0K Jan 22 21:22 .. -rw-r--r-- 1 root root 3.6K Jan 22 21:22 WrappedStringList.php -rw-r--r-- 1 root root 3.4K Jan 22 21:22 WrappedString.php [root@hetzner2 htdocs]#
Mon Jan 22, 2018
- Marcin found that editing posts on the stagingforum failed with "You don't have permission to access /vanilla/post/editdiscussion/X on this server."
- I coorelated this to a 403 error caused by modsecurity
--e381b03e-A--
[22/Jan/2018:14:02:11 +0000] WmXu49lQF2aUKXkcaav20AAAAAU 127.0.0.1 52050 127.0.0.1 8000
--e381b03e-B--
POST /vanilla/post/editdiscussion/237 HTTP/1.0
X-Real-IP: 173.234.159.236
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: stagingforum.opensourceecology.org
Content-Length: 3125
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: https://stagingforum.opensourceecology.org/vanilla/post/editdiscussion/237
Cookie: Vanilla=20082-1519221520%7C48d7a44f72d9d86b5e70a0da0e4b83c8%7C1516629520%7C20082%7C1519221520; Vanilla-Volatile=20082-1516802320%7C2271357054a693999ba33b7013e7c80e%7C1516629520%7C20082%7C1516802320
DNT: 1
X-Forwarded-For: 173.234.159.236, 127.0.0.1
X-Varnish: 34288
--e381b03e-C--
Discussion%2FTransientKey=82P0S34XB3Q0&Discussion%2Fhpt=&Discussion%2FDiscussionID=237&Discussion%2FDraftID=0&Discussion%2FName=Roof&Discussion%2FCategoryID=1&Discussion%2FBody=I+think+we+need+to+address+what+seems+to+be+a+hole+in+the+GVCS+-+constructing+a+good+roof.+Shelter+is+one+of+the+big+requirements+we+have+to+cover%2C+and+living+in+the+pacific+northwest%2C+I+am+painfully+aware+of+the+importance+and+costs+associated+with+a+roof.+Compressed+earth+blocks+cover%2C+to+a+decent+extent%2C+building+walls.+Floors+are+somewhat+covered+by+the+dimensional+saw+mill%2C+compressed+earth+blocks%2C+and+concrete.+However%2C+what+isn't+taken+into+account+yet+is+the+most+expensive+and+difficult+part+of+a+house+-+the+roof.%3Cbr%3E%3Cbr%3EI+think+figuring+out+a+good+roofing+solution+should+be+one+of+the+primary+focal+points+for+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2Fcomment%2F861%23Comment_861%22%3Efirst+wave%3C%2Fa%3E%3C%2Fu%3E+-+getting+the+basics+of+housing%2Ffood%2Fwater%2Fpower+up+and+running%2C+as+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F227%2Fbootstrapping-the-gvcs-priorities%22%3EGVCS+bootstrapping%3C%2Fa%3E%3C%2Fu%3E+thread.%3Cbr%3E%3Cbr%3EA+good+example+of+this+need+is+the+Factor+E+Farm+workshop+itself.+To+make+a+workshop+to+stage+the+development+of+the+GVCS%2C+they+were+able+to+make+part+of+the+structure+out+of+compressed+earth+block+columns.+However%2C+the+vast+majority+of+the+cost+of+the+structure+was+in+building+the+roof.+They+had+to+use+off-the-shelf+metal+roofing%2C+which+was+paid+for+through+donations%3A+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconstruction-time%2F%22%3Edonations+for+roofing+materials%3C%2Fa%3E%3C%2Fu%3E%2C+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%22%3Ebuilding+the+roof%3C%2Fa%3E%3C%2Fu%3E.%3Cbr%3E%3Cbr%3EAs+you+can+see%2C+building+a+roof+was+one+of+the+first+roadblocks%2C+and+was+solved+through+just+buying+one.%3Cbr%3E%26nbsp%3B%3Cbr%3EPerhaps+we+can+brainstorm+and+come+up+with+some+good+solutions.+I've+already+started+some+sections+and+given+some+background+requirements+on+the+wiki.+One+option+I+find+very+intriguing+myself+is+ferrocement.+You+can+see+it+being+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%23comments%22%3Ecomments%3C%2Fa%3E%3C%2Fu%3E+of+the+building+the+roof+blog+entry+for+the+farm.+I+made+a+thread+about+the+need+for+an+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F214%22%3Eopen+source+metal+lath+making+machine%3C%2Fa%3E%3C%2Fu%3E+with+roofing+as+one+of+its+applications.%3Cbr%3E%3Cbr%3EI've+started+a+wiki+entry+for+the+roof+here%3A+%3Cu%3Ehttp%3A%2F%2Fopenfarmtech.org%2Fwiki%2FRoof%3C%2Fu%3E+As+we+come+up+with+ideas+we+can+add+them.%3Cbr%3E%3Cbr%3EEdit%3A+This+is+Michael+attempting+to+edit+a+roof+post.%3Cbr%3E&Checkboxes%5B%5D=Announce&Checkboxes%5B%5D=Closed&Discussion%2FTags=&DeliveryType=VIEW&DeliveryMethod=JSON&Discussion/Save=Save
--e381b03e-F--
HTTP/1.1 403 Forbidden
Content-Length: 233
Connection: close
Content-Type: text/html; charset=iso-8859-1
--e381b03e-E--
--e381b03e-H--
Message: Access denied with code 403 (phase 2). Pattern match "\\W{4,}" at ARGS:Discussion/Body. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "37"] [id "960024"] [rev "2"] [msg "Meta-Character Anomaly Detection Alert - Repetative Non-Word Characters"] [data "Matched Data: > - found within ARGS:Discussion/Body: I think we need to address what seems to be a hole in the GVCS - constructing a good roof. Shelter is one of the big requirements we have to cover, and living in the pacific northwest, I am painfully aware of the importance and costs associated with a roof. Compressed earth blocks cover, to a decent extent, building walls. Floors are somewhat covered by the dimensional saw mill, compressed earth blocks, and concrete. However, what isn't taken into acc..."] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"]
Action: Intercepted (phase 2)
Stopwatch: 1516629731152710 755 (- - -)
Stopwatch2: 1516629731152710 755; combined=355, p1=104, p2=236, p3=0, p4=0, p5=14, sr=20, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"
--e381b03e-Z--
- realized that most of the whitelisting was done to just the 'wp-admin' dir, so I made it site-wide on the forums
- Catarina reported successfully that she was able to upload 3M images to fef && also connect via sftp using filezilla.
- marked fef migration as complete CHG-2018-01-03
- moved LocalSettings.php out of the document root per these hardening guides
[root@hetzner2 htdocs]# date Mon Jan 22 20:26:16 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/wiki.opensourceecology.org/htdocs [root@hetzner2 htdocs]# cat LocalSettings.php <?php # including separate file that contains the database password so that it is not stored within the document root. # For more info see: # * https://www.mediawiki.org/wiki/Manual:Security # * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening $docRoot = dirname( FILE ); require_once "$docRoot/../LocalSettings.php"; ?> [root@hetzner2 htdocs]#
- this also required me to change the original LocalSettings.php file (now outside the docroot in /var/www/html/wiki.opensourceecology.org) so that IP is inside of 'htdocs'
# If you customize your file layout, set $IP to the directory that contains
# the other MediaWiki files. It will be used as a base to locate files.
if( defined( 'MW_INSTALL_PATH' ) ) {
$IP = MW_INSTALL_PATH;
} else {
$IP = dirname( FILE ) . "/htdocs" ;
}
- decided to move the logo to 'images/' instead of just the base docroot to simplify future updates
- installed composer per the upgrade guide https://www.mediawiki.org/wiki/Manual:Upgrading
- which linked to https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries
- else, I got this error
MediaWiki 1.30 internal error Installing some external dependencies (e.g. via composer) is required. External dependencies MediaWiki now also has some external dependencies that need to be installed via composer or from a separate git repo. Please see mediawiki.org for help on installing the required components.
- and the actual install is
yum install composer
- and I had to add /usr/share/php/ to open_basedir in /etc/php.ini
- the composer steps themselves are included in the code block below
- found that several of our MediaWiki extensions are no longer maintained
- TreeAndMenu https://www.mediawiki.org/wiki/Extension:TreeAndMenu
- RSSReader https://www.mediawiki.org/wiki/Extension:RSS_Reader
- google-coop https://www.mediawiki.org/wiki/Extension_talk:Google_Custom_Search_Engine
- Mtag https://www.mediawiki.org/wiki/MediaWiki_and_LaTeX_on_a_host_with_shell_access
- PayPal https://www.mediawiki.org/wiki/Extension:Paypal
- Flattr https://www.mediawiki.org/wiki/Extension:Flattr
- JSWikiGnatt https://www.mediawiki.org/wiki/Extension:JSWikiGantt
- ProxyConnect https://www.mediawiki.org/wiki/Extension:ProxyConnect
- FreeMind https://www.mediawiki.org/wiki/Extension:FreeMind
- commented-out the entire function fnAddPersonalUrls() a section from LocalSettings.php that caused the following error
[Mon Jan 22 22:25:45.130218 2018] [:error] [pid 26913] [client 127.0.0.1:50368] PHP Fatal error: Call to undefined method User::getSkin() in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 421
- upgraded mediawiki install, including a switch to a git-backed core download to make future updates easier
- removed googleAnalytics require from LocalSettings.php, because privacy matters && awstats works.
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# clone the latest stable version of the Mediawiki core code from git
pushd ${backupDir_hetzner2}/current
time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git
pushd core
latestStableMediawikiVersion=`git tag -l | sort -V | grep -E '^[0-9\.]*$' | tail -n1`
git checkout "${latestStableMediawikiVersion}"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git
# extensions
pushd extensions
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd
#cp -r ../../../old/htdocs/extensions/TreeAndMenu .
#cp -r ../../../old/htdocs/extensions/RSSReader .
#cp -r ../../../old/htdocs/extensions/google-coop.php
#cp -r ../../../old/htdocs/extensions/Mtag.php
#cp -r ../../../old/htdocs/extensions/PayPal.php
#cp -r ../../../old/htdocs/extensions/Flattr
#cp -r ../../../old/htdocs/extensions/JSWikiGantt
#cp -r ../../../old/htdocs/extensions/ProxyConnect
popd
# skins
pushd skins
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd
popd
# copy core into the docroot
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
mkdir "${docrootDir_hetzner2}"
time nice rsync -av --progress core/ "${docrootDir_hetzner2}/"
# copy back essential content
rsync -av --progress "../old/LocalSettings.php" "${vhostDir_hetzner2}/"
rsync -av --progress "../old/htdocs/LocalSettings.php" "${docrootDir_hetzner2}/"
rsync -av --progress "../old/htdocs/images" "${docrootDir_hetzner2}/"
# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"
- having some 404 issues with images
<td><a href="http://www.shuttleworthfoundation.org/"><img alt="Shuttleworth funded-02---web.jpg" src="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg" width="200" height="55" srcset="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/300px-Shuttleworth_funded-02---web.jpg 1.5x, /wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/400px-Shuttleworth_funded-02---web.jpg 2x" /></a>
- the above links to (400) https://wiki.opensourceecology.org/wiki/Images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
- but it should link to (200) https://wiki.opensourceecology.org/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
- I logged into the hetzner1 konsleH config & tried to pull up the apache config there (since we can't see it in ssh on this annoying shared server). In the wui, click 'opensourceecology.org' -> Services -> Server Configuration -> Click the yellow wrench next to public_html -> Enhanced View:
Options All -Indexes
# Site-wide redirects
#Redirect 301 /community http://community.opensourceecology.org
#Redirect 301 /forum http://forum.opensourceecology.org
#Redirect 301 /weblog http://blog.opensourceecology.org
#RedirectMatch 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us
# Redirect 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us
# http://openfarmtech.org/w/index.php?title=CiviCRM_tech_notes&action=edit&redlink=1
# RedirectMatch 301 /CiviCRM http://openfarmtech.org/civicrm
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www\.
RewriteRule ^(.*)$ http://opensourceecology.org/$1 [R=301,L,QSA]
#RewriteRule ^civicrm/(.*)$ /community/civicrm/$1 [PT,L,QSA]
RewriteRule ^wiki/(.*)$ /w/index.php?title=$1 [PT,L,QSA]
#RewriteRule ^wiki/(.*:.*)$ wiki/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/(.*:.*)$ /w/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/*$ /w/index.php [L,QSA,T=text/html]
RewriteRule ^index.php/(.*)$ /wiki/$1?old-url=slash [R=permanent,L,QSA]
# RewriteRule ^/*$ /w/index.php [L,QSA]
# RewriteLog "/home/marcin_ose/openfarmtech.org/rewrite.log"
# RewriteLogLevel 3
# See http://docs.jboss.org/jbossweb/3.0.x/proxy-howto.html
# ProxyPass /OpenKM http://localhost:8080/OpenKM
# ProxyPassReverse /OpenKM http://localhost:8080/OpenKM
# RewriteRule ^OpenKM/(.*)$ http://localhost:8080/OpenKM/$1 [P]
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
- actually, just setting `$wgScriptPath = ""` in LocalSettings.php fixed everything!!
- attempting to login produces an error message
Database error A database query error has occurred. This may indicate a bug in the software.[WmZ1UeWiTZqK5j1xUV@YaQAAAAo] 2018-01-22 23:35:46: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- err, I found my password leaked in cleartext to /var/log/httpd/modsec_audit.log
- added "wpPassword" to sanitiseArg list in /etc/httpd/modexecurity.d/do_not_log_passwords.conf
- err, I found my password leaked in cleartext to /var/log/httpd/modsec_audit.log
[root@hetzner2 modsecurity.d]# date Mon Jan 22 23:46:06 UTC 2018 [root@hetzner2 modsecurity.d]# cd /etc/httpd [root@hetzner2 httpd]# cat modsecurity.d/do_not_log_passwords.conf ################################################################################ # File: modsecurity.d/do_not_log_passwords.conf # Version: 0.2 # Purpose: Defines a list of POST arguments that contain passwords and instructs # modsecurity to sanitise-out the values of these variables (with **s) # when logging to files (ie: /var/log/httpd/modsec_audit.log) # Author: Michael Altfield <michael@opensourceecology.org> # Created: 2017-12-16 # Updated: 2018-01-22 ################################################################################ SecAction "nolog,phase:2,id:131,sanitiseArg:password,sanitiseArg:Password,sanitiseArg:wpPassword,sanitiseArg:newPassword,sanitiseArg:oldPassword,sanitiseArg:pwd" [root@hetzner2 httpd]#
- it's possible that the login issue is due to a pending update.php execution required, but I'm getting an error when I attempt to run it
[root@hetzner2 maintenance]# php update.php PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53 PHP Warning: is_dir(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/forum.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fsfile/TempFSFile.php on line 90 PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127 PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 MediaWiki 1.30.0 Updater oojs/oojs-ui: 0.25.1 installed, 0.23.0 required. wikimedia/ip-set: 1.2.0 installed, 1.1.0 required. wikimedia/relpath: 2.1.1 installed, 2.0.0 required. wikimedia/remex-html: 1.0.2 installed, 1.0.1 required. wikimedia/running-stat: 1.2.1 installed, 1.1.0 required. wikimedia/wrappedstring: 2.3.0 installed, 2.2.0 required. Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies [root@hetzner2 maintenance]#
- I can't run composer, since it requires some unsafe, disabled php functions. That's why I chose to grab them via git (`git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git`). But apparently it gave me old versions?
- re-reading the error message, it says I have too new of versions!
- looks like the branches aren't maintained; we want REL1_30 but it isn't there
[root@hetzner2 vendor]# date Tue Jan 23 00:01:57 UTC 2018 [root@hetzner2 vendor]# pwd /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor [root@hetzner2 vendor]# git branch -r origin/HEAD -> origin/master origin/fundraising/REL1_27 origin/master origin/wmf/1.31.0-wmf.15 origin/wmf/1.31.0-wmf.16 origin/wmf/1.31.0-wmf.17 [root@hetzner2 vendor]#
Sat Jan 20, 2018
- the db backup & file backup of the wiki finished in a little over an hour
...
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$ time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
...
real 71m2.031s
user 17m36.700s
sys 1m38.868s
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$
- I initiated an scp of the data to hetnzer2
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# STEP 1: COPY FROM HETZNER1
mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/
- transfer of hetzner1 backups to hetzner2 finished after 8 minutes
- added wiki.opensourceecology.org A record in cloudflare DNS to 138.201.84.243
- disabled CDN/DNS proxy in cloudflare for the 'network' subdomain
- decreased TTL of all DNS entries to lowest time possible = 2 minutes for all entries except 'www', 'opensourceecology.org', and 'blog' (all 3 use the CDN for now, and are thus required to be set to TTL = 'automatic')
- created/updated necessary config files for forum.opensourceecology.org
- /etc/httpd/conf.d/00-wiki.opensourceecology.org.conf
- /etc/varnish/sites-enabled/wiki.opensourceecology.org
- /etc/varnish/all-vhosts.vcl
- /etc/nginx/conf.d/wiki.opensourceecology.org.conf
- created necessary dirs
- /var /www/html/wiki.opensourceecology.org/htdocs
- /var/log/httpd/wiki.opensourceecology.org
- /var/log/nginx/wiki.opensourceecology.org
- updated /etc/php.ini to include "/var/www/html/forum.opensourceecology.org/" in /etc/php.in's "open_basedir"
- added wiki SAN to opensourceecology.org cert
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d stagingforum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- created osewiki_db mysql database on hetzner2 from dump from hetzner1
- copied htdocs files on hetzner2 from backup on hetzner1
- updated LocalSettings.php with new db name & credentials
- updated permissions of docroot files
- commented-out $wgScriptPath & $wgStylePath to remove the "/w/" subdir in LocalSettings.php
- removed the "/w/" subdir prefix from $wgLogo in LocalSettings.php
- commented-out $wgServer & $wgFavicon to prevent a 301 redirect back to the naked domain opensourceecology.org
- commented-out $wgCacheDirectory & $wgUseFileCache. And `rm -rf cache`. This directory is *supposed* to be located outside the docroot for security reasons. But we won't need this feature as we use varnish https://www.mediawiki.org/wiki/Manual:Security#File_permissions
- added "Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php" to /etc/httpd/conf.d/00-wiki.opensourceecology.org
- commented-out debugging lines in LocalSettings.php
################################################################
# Debugging
# error_reporting(E_ALL | E_STRICT);
# error_reporting(E_ALL);
# ini_set("display_errors", 1);
# $wgShowExceptionDetails = true; ## Verbose output to user
#$wgShowSQLErrors = true;
#$wgDebugDumpSql = true;
#$wgDebugLogFile = "/usr/home/osemain/public_html/logs/wiki-error.log";
#$wgDebugLogFile = "/home/oft_site/logs/wiki-error.log";
# $wgDebugRawPage = true;
# $wgDebugComments = true;
################################################################
- saw some errors for generating temporary thumbnails
Error creating thumbnail: Unable to save thumbnail to destination
- attempted setting the image directory to be writeable, to no avail
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
- began investigating the guide to install Mediawiki via git https://www.mediawiki.org/wiki/Download_from_Git
time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git
pushd ${backupDir_hetzner2}/current
- determined that the latest version of Mediawiki is v1.30.0, and that we're currently running v1.24.2 https://wiki.opensourceecology.org/wiki/Special:Version
Thr Jan 18, 2018
- fixed awstats cron job & config files
- confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
- did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site
* https://core.trac.wordpress.org/ticket/17048
- there's no good, robust solution.
* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress * https://wordpress.org/plugins/relative-url/ * https://wordpress.org/plugins/root-relative-urls/ * https://deluxeblogtips.com/relative-urls/ * http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/
- take away is:
- Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
- However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.
- attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
- using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf
[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf # allow large posts for image uploads #client_max_body_size 1k; #client_max_body_size 900k; client_max_body_size 10M;
- next, I got a 403 error from /wp-admin/async-upload.php
- /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:
- next, I got a 403 error from /wp-admin/async-upload.php
==> /var/log/httpd/fef.opensourceecology.org/error_log <== [Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"]
- as above, whitelisted rule IDs:
- 960915, multipart_unmatched_boundary
- 200003, multipart_unmatched_boundary
- as above, whitelisted rule IDs:
- moved '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org' to '/usr/home/osemain/noBackup/deleteMeIn2019/oswh_olddocroot'
- added a 301 redirect from 'http://opensourcewarehouse.org' to 'https://oswh.opensourceecology.org' in new file = '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org/index.php'
Sat Jan 13, 2018
- Meeting with Marcin
Fri Jan 12, 2018
- finished configuring oswh wp plugins
- determined that the oswh hack was responsible for injecting pop-ups for a "windows defender" phishing site on some subset of page loads
- gained access to oseforums, oseblog, osecivi users via ssh on our non-dedicated server (hetzner1)
- checked /etc/passwd & found another 8x org-specific users with home directories that I couldn't access still
- emailed hetzner for advice on how to gain ssh access to these users' home directories
addon:x:1011:1011:addontest.opensourceecology.org:/usr/home/addon:bin/false oseirc:x:1018:1018:irc.opensourceecology.org:/usr/home/oseirc:bin/false oseholla:x:1019:1019:holla.opensourceecology.org:/usr/home/oseholla:bin/false osesurv:x:1020:1020:survey.opensourceecology.org:/usr/home/osesurv:/bin/bash sandbox:x:1021:1021:sandbox.opensourceecology.org:/usr/home/sandbox:/bin/false microft:x:1022:1022:test.opensourceecology.org:/usr/home/microft:/bin/bash zabbix:x:118:121::/var/run/zabbix:/bin/false openswh:x:1023:1023:opensourcewarehouse.org:/usr/home/openswh:/bin/false
- created backup of oseforum's docroot (58M) & db (44M). Both sizes are bzip2'd.
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
dbUser_hetzner2="oseforum_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
Mon Jan 08, 2018
- tried installing the fresh version of the Eventor theme to the website running the old wp core, but it was still broken
- reverted to the docroot I backed-up before attempting to try the outdated wp core && installed _that_ Eventor theme to the fresh version 1.7 that was just downloaded, and the site actually worked!
- finally, attempted the wp-cli commands to update themes & plugins && install our minimal set of sec plugins. The site was still functional after
Fri Jan 05, 2018
- investigation of minor fef issues
Thr Jan 04, 2018
- downloaded the Eventor theme v 1.7, thanks to Simone's contact with Themes Kingdom
- Hetzner responded saying we can use WebFTP to uplaod to $HOME by clicking "the server at the top"
- Marcin responded with some issues with osemain's ephemeral clone
- Catarina found some linking issues in fef
- I brought the site down & did a string replacement for all occurrences of 'http://opensourceecology.org/fef' to '/', brought the site back up, and asked Catarina to check again
- updated documentation at Wordpress#replace_strings_everywhere_in_wp_database_backend
Wed Jan 03, 2018
- migrated fef to hetzner2 CHG-2018-01-03
- updated statuscake for obi to hit 'https://www.openbuildinginstitute.org'
- updated statuscake for fef to hit 'https://fef.opensourceecology.org'
- ensured that ssh was activated for all domains/users on our (apparently dedicated, per hetzner support) hetzner1 server (but without root access) via the konsoleh site -> click on the server -> Account Management -> SSH access -> Select domain (for each) -> Next
- the kosoleh wui only allowed editing files in the docroot, not the user's home-dir, which prevented me from actually adding my ssh pubic key to $HOME/.ssh/authorized_keys file
- I emailed hetzner support back asking if [a] they could just add my pub key to all our user account's authorized_keys files or [b] tell me how I could reset all the user's passwords
- oswh was cannibalized by a virus & is awaiting a fresh version of the theme. the forums is awaiting access to the user account. I'm now going to work on beginning the migration of osemain
- it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
- archive
- w
- logs
- mediawiki-1.24.2.extra
- the entire dir is 23G. Excluding the above, it's ~ 0.7G
- it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
####################
# run on hetzner1 #
####################
# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh
# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'
# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
dbUser_hetzner1="${mysqlUser_osemain}"
dbPass_hetzner1="${mysqlPass_osemain}"
# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}
# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}
- the gz-compressed tarball generated from above was 353M.
# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180103"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
dbUser_hetzner2="osemain_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
- created domain name 'osemain.opensourceecology.org' for testing the osemain site on hetzner2
- using above vars, I followed the guide to migrate the files & db data from hetzner1 to hetzner2 Wordpress#migrate_site_from_hetzner1_to_hetzner2
- created necessary files & dirs:
- /etc/httpd/conf.d/00-www.opensourceecology.org.conf
- /etc/varnish/sites-enabled/www.opensourceecology.org
- /etc/nginx/conf.d/www.opensourceecology.org.conf
- this file has a temporary override for the 'Host' header passed to varnish, since the staging url is going to be 'osemain.opensourceecology.org' but the prod site will be 'opensourceecology.org'
- /var/log/httpd/www.opensourceecology.org
- /var/log/nginx/www.opensourceecology.org
- updated necessary files
- /etc/varnish/all-vhosts.vcl
- /etc/php.ini
- finished setting up ephemeral clone of osemain at https://osemain.opensourceecology.org
- sent email to Marcin & Catarina for validation
Tue Jan 02, 2018
- got an email from Simone Cicero stating that she emailed Themes Kingdom for a clean copy of Eventor 1.7
- emailed back-and-forth with hetzner
- learned that the forums are in /usr/www/users/oseforum/
- learned that we have a bunch of users on this box, and it might even be dedicated just for us (though without root access)
osemain@dedi978:~$ grep 'ose' /etc/group users:x:100:osemain,addon,osecivi,oseblog,oseforum,oseirc,oseholla,osesurv,sandbox,microft,openswh osemain:x:1010: osecivi:x:1014: oseblog:x:1015: oseforum:x:1016: oseirc:x:1018: oseholla:x:1019: osesurv:x:1020:
- but I couldn't actually access the home dirs of the other users through 'osemain'
osemain@dedi978:~$ date Tue Jan 2 16:21:13 CET 2018 osemain@dedi978:~$ ls -lah /usr/home/ ls: cannot open directory /usr/home/: Permission denied osemain@dedi978:~$ ls -lah /usr/home/addon ls: cannot open directory /usr/home/addon: Permission denied osemain@dedi978:~$ ls -lah /usr/home/osecivi ls: cannot open directory /usr/home/osecivi: Permission denied osemain@dedi978:~$ ls -lah /usr/home/oseblog ls: cannot open directory /usr/home/oseblog: Permission denied osemain@dedi978:~$ ls -lah /usr/home/oseirc ls: cannot open directory /usr/home/oseirc: Permission denied osemain@dedi978:~$ ls -lah /usr/home/oseforum ls: cannot open directory /usr/home/oseforum: Permission denied osemain@dedi978:~$ ls -lah /usr/home/osesurv ls: cannot open directory /usr/home/osesurv: Permission denied osemain@dedi978:~$ ls -lah /usr/home/openswh ls: cannot open directory /usr/home/openswh: Permission denied
- so I asked hetzner support to add the 'osemain' user to all the other users groups listed above, and I asked them to find any other accounts that we own that I may have missed