Revision as of 20:08, 12 February 2018

My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

Thr Feb 09, 2018

did a theme update. The default themes updated, but the active theme (enigmatic) had no updates (or, at least, it's not available via the api)

[root@hetzner2 www.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme list
...
+------------------------+----------+-----------+---------+
| name                   | status   | update    | version |
+------------------------+----------+-----------+---------+
| enigmatic.20170714.bak | inactive | none      | 3.1     |
| enigmatic              | active   | none      | 3.5     |
| twentyeleven           | inactive | none      | 2.7     |
| twentyfifteen          | inactive | none      | 1.9     |
| twentyfourteen         | inactive | available | 1.0     |
| twentyseventeen        | inactive | none      | 1.4     |
| twentysixteen          | inactive | none      | 1.4     |
| twentyten              | inactive | none      | 2.4     |
| twentythirteen         | inactive | available | 1.1     |
| twentytwelve           | inactive | available | 1.3     |
+------------------------+----------+-----------+---------+
[root@hetzner2 www.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme update --all
...
Downloading update from https://downloads.wordpress.org/theme/twentyfourteen.2.1.zip...
Using cached file '/home/wp/.wp-cli/cache/theme/twentyfourteen-2.1.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentythirteen.2.3.zip...
Using cached file '/home/wp/.wp-cli/cache/theme/twentythirteen-2.3.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentytwelve.2.4.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme... 
Theme updated successfully.
+----------------+-------------+-------------+---------+
| name           | old_version | new_version | status  |
+----------------+-------------+-------------+---------+
| twentyfourteen | 1.0         | 2.1         | Updated |
| twentythirteen | 1.1         | 2.3         | Updated |
| twentytwelve   | 1.3         | 2.4         | Updated |
+----------------+-------------+-------------+---------+
Success: Updated 3 of 3 themes.
[root@hetzner2 www.opensourceecology.org]#

awstats for obi was going to '/var/log/nginx/access.log' instead of the vhost-specific '/var/log/nginx/www.openbuildinginstitute.org/access.log' file
the permissions were fine; a reload fixed it (note the file size changed from 0 to 940 after the reload & I queried the site again

[root@hetzner2 ~]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/
total 19M
drw-r--r-- 2 nginx nginx 4.0K Feb  9 18:20 .
drwx------ 9 nginx nginx 4.0K Feb  9 04:47 ..
-rw-r--r-- 1 nginx nginx    0 Feb  9 04:47 access.log
-rw-r--r-- 1 nginx nginx  377 Dec 24 01:30 access.log.1.gz
-rw-r--r-- 1 nginx nginx 1.9M Dec 18 03:20 access.log-20171210.gz
-rw-r--r-- 1 nginx nginx 1.3M Dec 24 01:20 access.log-20171219.gz
-rw-r--r-- 1 nginx nginx 2.6M Jan  3 22:03 access.log-20171225.gz
-rw-r--r-- 1 nginx nginx 239K Jan  4 21:36 access.log-20180104.gz
-rw-r--r-- 1 nginx nginx 3.2M Jan 18 14:43 access.log-20180105.gz
-rw-r--r-- 1 nginx nginx 193K Jan 19 13:03 access.log-20180119.gz
-rw-r--r-- 1 nginx nginx 288K Jan 20 20:20 access.log-20180120.gz
-rw-r--r-- 1 nginx nginx 4.2M Feb  4 20:19 access.log-20180121.gz
-rw-r--r-- 1 nginx nginx 1.7M Feb  9 02:08 access.log-20180205.gz
-rw-r--r-- 1 nginx nginx 3.3M Feb  9 18:19 access.log-20180209
-rw-r--r-- 1 nginx nginx  224 Dec 24 01:29 access.log.2.gz
-rw-r--r-- 1 nginx nginx  510 Dec 24 01:21 access.log.3.gz
-rw-r--r-- 1 nginx nginx    0 Feb  6 06:01 error.log
-rw-r--r-- 1 nginx nginx 1.1K Nov 30 17:06 error.log-20171201.gz
-rw-r--r-- 1 nginx nginx 1.2K Dec  8 13:12 error.log-20171206.gz
-rw-r--r-- 1 nginx nginx 1.8K Dec 12 23:31 error.log-20171210.gz
-rw-r--r-- 1 nginx nginx  401 Dec 19 13:45 error.log-20171219.gz
-rw-r--r-- 1 nginx nginx  388 Dec 24 05:48 error.log-20171224.gz
-rw-r--r-- 1 nginx nginx 3.3K Jan  1 09:19 error.log-20171229.gz
-rw-r--r-- 1 nginx nginx  359 Jan  3 23:05 error.log-20180104.gz
-rw-r--r-- 1 nginx nginx 3.8K Jan 18 13:55 error.log-20180105.gz
-rw-r--r-- 1 nginx nginx  888 Feb  2 10:27 error.log-20180124.gz
-rw-r--r-- 1 nginx nginx  319 Feb  5 17:04 error.log-20180206
[root@hetzner2 ~]# service nginx reload
Redirecting to /bin/systemctl reload nginx.service
[root@hetzner2 ~]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/
total 19M
drw-r--r-- 2 nginx nginx 4.0K Feb  9 18:20 .
drwx------ 9 nginx nginx 4.0K Feb  9 04:47 ..
-rw-r--r-- 1 nginx nginx  940 Feb  9 18:20 access.log
-rw-r--r-- 1 nginx nginx  377 Dec 24 01:30 access.log.1.gz
-rw-r--r-- 1 nginx nginx 1.9M Dec 18 03:20 access.log-20171210.gz
-rw-r--r-- 1 nginx nginx 1.3M Dec 24 01:20 access.log-20171219.gz
-rw-r--r-- 1 nginx nginx 2.6M Jan  3 22:03 access.log-20171225.gz
-rw-r--r-- 1 nginx nginx 239K Jan  4 21:36 access.log-20180104.gz
-rw-r--r-- 1 nginx nginx 3.2M Jan 18 14:43 access.log-20180105.gz
-rw-r--r-- 1 nginx nginx 193K Jan 19 13:03 access.log-20180119.gz
-rw-r--r-- 1 nginx nginx 288K Jan 20 20:20 access.log-20180120.gz
-rw-r--r-- 1 nginx nginx 4.2M Feb  4 20:19 access.log-20180121.gz
-rw-r--r-- 1 nginx nginx 1.7M Feb  9 02:08 access.log-20180205.gz
-rw-r--r-- 1 nginx nginx 3.3M Feb  9 18:20 access.log-20180209
-rw-r--r-- 1 nginx nginx  224 Dec 24 01:29 access.log.2.gz
-rw-r--r-- 1 nginx nginx  510 Dec 24 01:21 access.log.3.gz
-rw-r--r-- 1 nginx nginx    0 Feb  6 06:01 error.log
-rw-r--r-- 1 nginx nginx 1.1K Nov 30 17:06 error.log-20171201.gz
-rw-r--r-- 1 nginx nginx 1.2K Dec  8 13:12 error.log-20171206.gz
-rw-r--r-- 1 nginx nginx 1.8K Dec 12 23:31 error.log-20171210.gz
-rw-r--r-- 1 nginx nginx  401 Dec 19 13:45 error.log-20171219.gz
-rw-r--r-- 1 nginx nginx  388 Dec 24 05:48 error.log-20171224.gz
-rw-r--r-- 1 nginx nginx 3.3K Jan  1 09:19 error.log-20171229.gz
-rw-r--r-- 1 nginx nginx  359 Jan  3 23:05 error.log-20180104.gz
-rw-r--r-- 1 nginx nginx 3.8K Jan 18 13:55 error.log-20180105.gz
-rw-r--r-- 1 nginx nginx  888 Feb  2 10:27 error.log-20180124.gz
-rw-r--r-- 1 nginx nginx  319 Feb  5 17:04 error.log-20180206
[root@hetzner2 ~]#

1. the reload *should* be happening already by the logroate

[root@hetzner2 ~]# cat /etc/logrotate.d/nginx 
/var/log/nginx/*log /var/log/nginx/*/*log {
	create 0644 nginx nginx
	daily
	rotate 10
	missingok
	notifempty
	compress
	sharedscripts
	prerotate
		/bin/nice /usr/share/awstats/tools/awstats_updateall.pl now
	endscript
	postrotate
		/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
	endscript
}

[root@hetzner2 ~]#

1. I simulated the rotate & kill, and saw the same undesired behaviour

[root@hetzner2 www.openbuildinginstitute.org]# mv access.log access.log.test
[root@hetzner2 www.openbuildinginstitute.org]# touch access.log
[root@hetzner2 www.openbuildinginstitute.org]# chown nginx:nginx access.log
[root@hetzner2 www.openbuildinginstitute.org]# chmod 0644 access.log
[root@hetzner2 www.openbuildinginstitute.org]# /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
[root@hetzner2 nginx]# tail -f /var/log/nginx/access.log /var/log/nginx/www.openbuildinginstitute.org/access.log /var/log/nginx/error.log /var/log/nginx/www.openbuildinginstitute.org/error.log
...
==> /var/log/nginx/access.log <==
98.242.98.106 - - [09/Feb/2018:18:30:05 +0000] "GET /wp-content/uploads/2016/02/kitchen_1.jpg HTTP/1.1" 301 178 "https://www.openbuildinginstitute.org/buildings/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-"

1. interseting, when I did the kill again, this log popped-up

[root@hetzner2 nginx]# tail -f /var/log/nginx/access.log /var/log/nginx/www.openbuildinginstitute.org/access.log /var/log/nginx/error.log /var/log/nginx/www.openbuildinginstitute.org/error.log
...
==> /var/log/nginx/error.log <==
2018/02/09 18:31:08 [emerg] 26860#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26861#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26866#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26862#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26865#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26859#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26860#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26861#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26866#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26862#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26865#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26859#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26864#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26864#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26863#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied)
2018/02/09 18:31:08 [emerg] 26863#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)

1. but the permissions look fine

[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/access.log
-rw-r--r-- 1 nginx nginx 0 Feb  9 18:26 /var/log/nginx/www.openbuildinginstitute.org/access.log

1. ah, found the issue thanks to this https://stackoverflow.com/questions/37245926/nginx-write-logs-to-the-old-file-after-running-logrotate

[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/
total 9.0M
drwx------  9 nginx nginx 4.0K Feb  9 04:47 .
drwxr-xr-x 13 root  root   12K Feb  9 04:47 ..
-rw-r--r--  1 nginx nginx 2.2M Feb  9 18:39 access.log
-rw-r--r--  1 nginx nginx  222 Dec 24 01:33 access.log.1.gz
-rw-r--r--  1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz
-rw-r--r--  1 nginx nginx 186K Feb  1 06:41 access.log-20180201.gz
-rw-r--r--  1 nginx nginx 107K Feb  2 08:54 access.log-20180202.gz
-rw-r--r--  1 nginx nginx  83K Feb  3 05:49 access.log-20180203.gz
-rw-r--r--  1 nginx nginx  81K Feb  4 04:48 access.log-20180204.gz
-rw-r--r--  1 nginx nginx 265K Feb  5 09:12 access.log-20180205.gz
-rw-r--r--  1 nginx nginx 351K Feb  6 06:00 access.log-20180206.gz
-rw-r--r--  1 nginx nginx 458K Feb  7 07:35 access.log-20180207.gz
-rw-r--r--  1 nginx nginx 443K Feb  8 07:21 access.log-20180208.gz
-rw-r--r--  1 nginx nginx 4.0M Feb  9 04:46 access.log-20180209
-rw-r--r--  1 nginx nginx  420 Dec 24 01:32 access.log.2.gz
-rw-r--r--  1 nginx nginx  207 Dec 24 01:29 access.log.3.gz
-rw-r--r--  1 nginx nginx  187 Dec 24 01:28 access.log.4.gz
-rw-r--r--  1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz
-rw-r--r--  1 nginx nginx  271 Dec 24 01:21 access.log.6.gz
-rw-r--r--  1 nginx nginx 249K Feb  9 18:37 error.log
-rw-r--r--  1 nginx nginx  211 Dec 24 01:33 error.log.1.gz
-rw-r--r--  1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz
-rw-r--r--  1 nginx nginx 9.0K Feb  1 06:41 error.log-20180201.gz
-rw-r--r--  1 nginx nginx 7.8K Feb  2 08:52 error.log-20180202.gz
-rw-r--r--  1 nginx nginx 6.0K Feb  3 05:49 error.log-20180203.gz
-rw-r--r--  1 nginx nginx 6.2K Feb  4 04:48 error.log-20180204.gz
-rw-r--r--  1 nginx nginx 8.5K Feb  5 09:03 error.log-20180205.gz
-rw-r--r--  1 nginx nginx 9.6K Feb  6 06:00 error.log-20180206.gz
-rw-r--r--  1 nginx nginx  13K Feb  7 07:35 error.log-20180207.gz
-rw-r--r--  1 nginx nginx  13K Feb  8 07:21 error.log-20180208.gz
-rw-r--r--  1 nginx nginx 379K Feb  9 04:46 error.log-20180209
-rw-r--r--  1 nginx nginx  592 Dec 24 01:33 error.log.2.gz
-rw-r--r--  1 nginx nginx  413 Dec 24 01:29 error.log.3.gz
-rw-r--r--  1 nginx nginx  210 Dec 24 01:28 error.log.4.gz
-rw-r--r--  1 nginx nginx  211 Dec 24 01:27 error.log.5.gz
-rw-r--r--  1 nginx nginx  506 Dec 24 01:24 error.log.6.gz
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 fef.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 forum.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 oswh.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 seedhome.openbuildinginstitute.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 wiki.opensourceecology.org
drw-r--r--  2 nginx nginx 4.0K Feb  9 18:26 www.openbuildinginstitute.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 www.opensourceecology.org
[root@hetzner2 www.openbuildinginstitute.org]#

1. if you look at the above output of the /var/log/nginx contents, it's obvious really. One dir is 0444 while the rest are 0755. We need that containing dir to have execute permissions!

[root@hetzner2 www.openbuildinginstitute.org]# chmod 0755 /var/log/nginx/www.openbuildinginstitute.org
[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/
total 9.0M
drwx------  9 nginx nginx 4.0K Feb  9 04:47 .
drwxr-xr-x 13 root  root   12K Feb  9 04:47 ..
-rw-r--r--  1 nginx nginx 2.2M Feb  9 18:41 access.log
-rw-r--r--  1 nginx nginx  222 Dec 24 01:33 access.log.1.gz
-rw-r--r--  1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz
-rw-r--r--  1 nginx nginx 186K Feb  1 06:41 access.log-20180201.gz
-rw-r--r--  1 nginx nginx 107K Feb  2 08:54 access.log-20180202.gz
-rw-r--r--  1 nginx nginx  83K Feb  3 05:49 access.log-20180203.gz
-rw-r--r--  1 nginx nginx  81K Feb  4 04:48 access.log-20180204.gz
-rw-r--r--  1 nginx nginx 265K Feb  5 09:12 access.log-20180205.gz
-rw-r--r--  1 nginx nginx 351K Feb  6 06:00 access.log-20180206.gz
-rw-r--r--  1 nginx nginx 458K Feb  7 07:35 access.log-20180207.gz
-rw-r--r--  1 nginx nginx 443K Feb  8 07:21 access.log-20180208.gz
-rw-r--r--  1 nginx nginx 4.0M Feb  9 04:46 access.log-20180209
-rw-r--r--  1 nginx nginx  420 Dec 24 01:32 access.log.2.gz
-rw-r--r--  1 nginx nginx  207 Dec 24 01:29 access.log.3.gz
-rw-r--r--  1 nginx nginx  187 Dec 24 01:28 access.log.4.gz
-rw-r--r--  1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz
-rw-r--r--  1 nginx nginx  271 Dec 24 01:21 access.log.6.gz
-rw-r--r--  1 nginx nginx 250K Feb  9 18:40 error.log
-rw-r--r--  1 nginx nginx  211 Dec 24 01:33 error.log.1.gz
-rw-r--r--  1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz
-rw-r--r--  1 nginx nginx 9.0K Feb  1 06:41 error.log-20180201.gz
-rw-r--r--  1 nginx nginx 7.8K Feb  2 08:52 error.log-20180202.gz
-rw-r--r--  1 nginx nginx 6.0K Feb  3 05:49 error.log-20180203.gz
-rw-r--r--  1 nginx nginx 6.2K Feb  4 04:48 error.log-20180204.gz
-rw-r--r--  1 nginx nginx 8.5K Feb  5 09:03 error.log-20180205.gz
-rw-r--r--  1 nginx nginx 9.6K Feb  6 06:00 error.log-20180206.gz
-rw-r--r--  1 nginx nginx  13K Feb  7 07:35 error.log-20180207.gz
-rw-r--r--  1 nginx nginx  13K Feb  8 07:21 error.log-20180208.gz
-rw-r--r--  1 nginx nginx 379K Feb  9 04:46 error.log-20180209
-rw-r--r--  1 nginx nginx  592 Dec 24 01:33 error.log.2.gz
-rw-r--r--  1 nginx nginx  413 Dec 24 01:29 error.log.3.gz
-rw-r--r--  1 nginx nginx  210 Dec 24 01:28 error.log.4.gz
-rw-r--r--  1 nginx nginx  211 Dec 24 01:27 error.log.5.gz
-rw-r--r--  1 nginx nginx  506 Dec 24 01:24 error.log.6.gz
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 fef.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 forum.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 oswh.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 seedhome.openbuildinginstitute.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 wiki.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 18:26 www.openbuildinginstitute.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  9 04:47 www.opensourceecology.org
[root@hetzner2 www.openbuildinginstitute.org]#

1. that appears to have fixed it!
2. I updated the documentation for creating new wordpress vhosts to use these permissions at the get-go Wordpress#Create_New_Wordpress_Vhost
began investigating how the hell awstats is accessible on port 443 and, more importantly, how certbot is able to refresh its certificate
1. I found the logs of attempting to load 'https://awstats.openbuildinginstitute.org/fuck' in '/var/log/nginx/seedhome.openbuildinginstitute.org/access.log'

[root@hetzner2 nginx]# date
Fri Feb  9 19:08:10 UTC 2018
[root@hetzner2 nginx]# pwd
/var/log/nginx
[root@hetzner2 nginx]# tail -f *.log */*.log | grep -i fuck
98.242.98.106 - - [09/Feb/2018:19:04:10 +0000] "GET /fuck HTTP/1.1" 401 381 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-"
[root@hetzner2 nginx]# grep -irl 'fuck' *
seedhome.openbuildinginstitute.org/access.log
[root@hetzner2 nginx]#

1. I guess that site became our default/catch-all, picking up requests on obi's ip address (138.201.84.223) on port 443. And the nginx config file is generic enough to just pass the host header along to varnish using a dynamic variable ($host), so then apache actually serves the correct page.
2. so I think the best thing to do is to explicitly define an nginx listen on port 443 for each awstats site, passing to varnish per usual (but logging in the correct place) && the extensible configs in the proper & expected place. Then I'll configure apache to have 2 distinct docroots for X-Forwarded-Port equal to 4443 vs 443. 4443 will give us the htpasswd-protected awstats files. 443 will go to publicly-accessible docroot where certbot will generate files to negotiate https cert updates.
3. successfully tested changing the "X-Forwarded-Port 443" line in the nginx config to "X-Forwarded-Port $server_port"

[root@hetzner2 conf.d]# cat /etc/nginx/conf.d/awstats.openbuildinginstitute.org.conf 
################################################################################
# File:    awstats.openbuildinginstitute.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
#          protection, and passing to varnish cache (varnish then passes to
#          apache)
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################

server {

   include conf.d/secure.include;
   include conf.d/ssl.openbuildinginstitute.org.include;

   listen 138.201.84.223:4443;
   listen 138.201.84.223:443;
   #listen [::]:443;

   server_name awstats.openbuildinginstitute.org;

		#############
		# SITE_DOWN #
		#############
		# uncomment this block && restart nginx prior to apache work to display the
		# "SITE DOWN" webpage for our clients

		#root /var/www/html/SITE_DOWN/htdocs/;
   #index index.html index.htm;

		# force all requests to load exactly this page
		#location / {
		#       try_files $uri /index.html;
		#}

		###################
		# SEND TO VARNISH #
		###################

   location / {
	  proxy_pass http://127.0.0.1:6081;
	  proxy_set_header X-Real-IP $remote_addr;
	  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	  proxy_set_header X-Forwarded-Proto https;
	  #proxy_set_header X-Forwarded-Port 443;
	  proxy_set_header X-Forwarded-Port $server_port;
	  proxy_set_header Host $host;
   }

}

[root@hetzner2 conf.d]#

1. apache isn't very powerful at conditionals, so I decided to move the logic to nginx & varnish
2. updated /etc/httpd/conf/httpd.conf to listen on 127.0.0.1:8010 for the certbot vhost
successfully added 'awstats.opensourceecology.org' to cert SAN with certbot

[root@hetzner2 ~]# certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org
...
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/opensourceecology.org/privkey.pem
   Your cert will expire on 2018-05-10. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

[root@hetzner2 ~]#  /bin/chmod 0400 /etc/letsencrypt/archive/*/pri*
[root@hetzner2 ~]#  nginx -t && service nginx reload
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Redirecting to /bin/systemctl reload nginx.service
[root@hetzner2 ~]#

1. so first, nginx is listening on ports '443' && '4443' It sets the 'X-Forwarded-Port' Header to whichever it heard the request come in on (referenced by $server_port)

[root@hetzner2 conf.d]# date
Fri Feb  9 21:37:33 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/nginx/conf.d
[root@hetzner2 conf.d]# cat awstats.openbuildinginstitute.org.conf 
################################################################################
# File:    awstats.openbuildinginstitute.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
#          protection, and passing to varnish cache (varnish then passes to
#          apache)
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################

server {

   include conf.d/secure.include;
   include conf.d/ssl.openbuildinginstitute.org.include;

   listen 138.201.84.223:4443;
   listen 138.201.84.223:443;

   server_name awstats.openbuildinginstitute.org;

		#############
		# SITE_DOWN #
		#############
		# uncomment this block && restart nginx prior to apache work to display the
		# "SITE DOWN" webpage for our clients

		#root /var/www/html/SITE_DOWN/htdocs/;
   #index index.html index.htm;

		# force all requests to load exactly this page
		#location / {
		#       try_files $uri /index.html;
		#}

		###################
		# SEND TO VARNISH #
		###################

   location / {
	  proxy_pass http://127.0.0.1:6081;
	  proxy_set_header X-Real-IP $remote_addr;
	  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	  proxy_set_header X-Forwarded-Proto https;
	  #proxy_set_header X-Forwarded-Port 443;
	  proxy_set_header X-Forwarded-Port $server_port;
	  proxy_set_header Host $host;
   }

}
[root@hetzner2 conf.d]#

1. nginx sends its requests to varnish. The varnish config for the awstats configs defines 2x distinct backends. Both are apache, but the ports are different. The standard 8000 name-based-vhost is used for the 4443 port && there's a new vhost running on 8010 specifically for this certbot vhost.

[root@hetzner2 varnish]# date
Fri Feb  9 21:39:47 UTC 2018
[root@hetzner2 varnish]# pwd
/etc/varnish
[root@hetzner2 varnish]# cat sites-enabled/awstats.openbuildinginstitute.org
################################################################################
# File:    awstats.openbuildinginstitute.org.vcl
# Version: 0.3
# Purpose: Config file for awstats for obi
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################

vcl 4.0;

##########################
# VHOST-SPECIFIC BACKEND #
##########################

backend awstats_openbuildinginstitute_org {
		.host = "127.0.0.1";
		.port = "8000";
}

backend certbot_openbuildinginstitute_org {
		.host = "127.0.0.1";
		.port = "8010";
}

#####################################################################
# STUFF MOSTLY COPIED FROM vcaching WORDPRESS PLUGIN                #
# https://plugins.svn.wordpress.org/vcaching/trunk/varnish-conf/v4/ #
#####################################################################

sub vcl_recv {

		if ( req.http.host == "awstats.openbuildinginstitute.org" ){

				# is this an admin trying to access awstats or certbot trying to renew the
				# https certificate?
				if ( req.http.X-Forwarded-Port == "4443" ){
						# this is an admin; send them to the auth-protected apache vhost

						set req.backend_hint = awstats_openbuildinginstitute_org;

				} else { 
						# this is not an admin accessing the hidden port; send them to the apache
						# vhost with no content (that certbot uses)

						set req.backend_hint = certbot_openbuildinginstitute_org;

				}

		}
}

sub vcl_hash {

		if ( req.http.host == "awstats.openbuildinginstitute.org" ){

				# TODO   

		}
}

sub vcl_backend_response {

		if ( beresp.backend.name == "awstats_openbuildinginstitute_org" ){

				# TODO   

		}

}

sub vcl_synth {

		if ( req.http.host == "awstats.openbuildinginstitute.org" ){

				# TODO   

		}

}

sub vcl_pipe {

		if ( req.http.host == "awstats.openbuildinginstitute.org" ){

				# TODO   

		}
}

sub vcl_deliver {

		if ( req.http.host == "awstats.openbuildinginstitute.org" ){

				# TODO   

		}
}
[root@hetzner2 varnish]# cat default.vcl 
################################################################################
# File:    default.vcl
# Version: 0.1
# Purpose: Main config file for varnish cache. Note that it's intentionally
#          mostly bare to allow robust vhost-specific logic. Please see this
#          for more info:
#            * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2017-11-12
################################################################################

vcl 4.0;

############
# INCLUDES #
############
#
import std;

include "conf/acl.vcl";
include "lib/purge.vcl";


include "all-vhosts.vcl";
include "catch-all.vcl";
[root@hetzner2 varnish]# cat all-vhosts.vcl 
################################################################################
# File:    all-hosts.vcl
# Version: 0.8
# Purpose: meta config file that simply imports the site-specific vcl files
#          stored in the 'sites-enabled' directory Please see this for more info
#            * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2018-02-09
################################################################################

include "sites-enabled/staging.openbuildinginstitute.org";
include "sites-enabled/awstats.openbuildinginstitute.org";
include "sites-enabled/awstats.opensourceecology.org";
include "sites-enabled/www.openbuildinginstitute.org";
include "sites-enabled/www.opensourceecology.org";
include "sites-enabled/seedhome.openbuildinginstitute.org";
include "sites-enabled/fef.opensourceecology.org";
include "sites-enabled/oswh.opensourceecology.org";
include "sites-enabled/forum.opensourceecology.org";
include "sites-enabled/wiki.opensourceecology.org";
[root@hetzner2 varnish]#

1. finally, here's the 2x distinct vhosts in apache

[root@hetzner2 conf.d]# date
Fri Feb  9 21:41:31 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/httpd/conf.d
[root@hetzner2 conf.d]# cat awstats.openbuildinginstitute.org.conf 
<VirtualHost 127.0.0.1:8000>
		ServerName awstats.openbuildinginstitute.org
		DocumentRoot "/var/www/html/awstats.openbuildinginstitute.org/htdocs"

		Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org

		<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
				Deny From All
		</LocationMatch>

</VirtualHost>

<Directory "/var/www/html/awstats.openbuildinginstitute.org/htdocs">

		AuthType Basic
		AuthName "Authentication Required"
		AuthUserFile /var/www/html/awstats.openbuildinginstitute.org/.htpasswd

		# it is unnecessary to allow certbot here, since it uses port 80 (which we
		# send to port 443), but this server only listens on port 4443 (via nginx)
		#Require expr %{REQUEST_URI} =~ m#^/.well-known/acme-challenge/#
		Require valid-user

		Options +Indexes +Includes
		AllowOverride None

		Order allow,deny
		Allow from all

		# Harden vhost docroot by blocking all request types except the 3 essentials
		<LimitExcept GET POST HEAD>
				deny from all
		</LimitExcept>

</Directory>

# disable mod_security with rules as needed (found by logs in:
#  /var/log/httpd/modsec_audit.log
<Location "/">
		<IfModule security2_module>
				SecRuleEngine On
				#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
		</IfModule>
</Location>
[root@hetzner2 conf.d]# cat certbot.conf 
################################################################################
# File:    certbot.conf
# Version: 0.1
# Purpose: localhost-only-listening, http-only, name-based-vhost for serving
#          an empty vhost that's to be used by certbot for refreshing our Let's
#          Encrypt certificates. This is necessary for admin-only/private vhosts
#          that operate on non-standard port with basic auth required. For
#           example, see the configs for awstats.
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2018-02-09
# Updated: 2018-02-09
################################################################################
<VirtualHost 127.0.0.1:8010>
		#ServerName awstats.openbuildinginstitute.org
		DocumentRoot "/var/www/html/certbot/htdocs"

		#Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org

		<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
				Deny From All
		</LocationMatch>

</VirtualHost>

<Directory "/var/www/html/certbot/htdocs">

		Options +Indexes +Includes
		AllowOverride None

		Order allow,deny
		Allow from all

		# Harden vhost docroot by blocking all request types except the 3 essentials
		<LimitExcept GET POST HEAD>
				deny from all
		</LimitExcept>

</Directory>

# disable mod_security with rules as needed (found by logs in:
#  /var/log/httpd/modsec_audit.log
<Location "/">
		<IfModule security2_module>
				SecRuleEngine On
				#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
		</IfModule>
</Location>

Thr Feb 08, 2018

installed our minimal set of security wordpress plugins

[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
...
Installing Google Authenticator (0.48)
Downloading installation package from https://downloads.wordpress.org/plugin/google-authenticator.0.48.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/google-authenticator-0.48.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'google-authenticator'...
Plugin 'google-authenticator' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
...
Installing Google Authenticator  Encourage User Activation (0.2)
Downloading installation package from https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/google-authenticator-encourage-user-activation-0.2.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'google-authenticator-encourage-user-activation'...
Plugin 'google-authenticator-encourage-user-activation' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]# defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp" 
[root@hetzner2 htdocs]# pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
/var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/google-authenticator /var/www/html/www.opensourceecology.org/htdocs
[root@hetzner2 google-authenticator]# sed -i "s^\$GA_description\s=\s(\s[\"'].*[\"']^\$GA_description = ( '$defaultOtpAccountDescription'^" google-authenticator.php
[root@hetzner2 google-authenticator]# popd
/var/www/html/www.opensourceecology.org/htdocs
[root@hetzner2 htdocs]# # install 'force-strong-passwords' plugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate
...
Installing Force Strong Passwords (1.8.0)
Downloading installation package from https://downloads.wordpress.org/plugin/force-strong-passwords.1.8.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/force-strong-passwords-1.8.0.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'force-strong-passwords'...
Plugin 'force-strong-passwords' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install rename-wp-login plugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate
...
Installing Rename wp-login.php (2.5.5)
Downloading installation package from https://downloads.wordpress.org/plugin/rename-wp-login.2.5.5.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/rename-wp-login-2.5.5.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'rename-wp-login'...
Plugin 'rename-wp-login' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install "SSL Insecure Content Fixer" pugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate
...
Installing SSL Insecure Content Fixer (2.5.0)
Downloading installation package from https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.5.0.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/ssl-insecure-content-fixer-2.5.0.zip'... 
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'ssl-insecure-content-fixer'...
Plugin 'ssl-insecure-content-fixer' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install "Varnish Caching" pugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate
...
Installing Varnish Caching (1.6.7)
Downloading installation package from https://downloads.wordpress.org/plugin/vcaching.1.6.7.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/vcaching-1.6.7.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'vcaching'...
Plugin 'vcaching' activated.
Success: Installed 1 of 1 plugins.

updated existing wordpress plugins on osemain staging site on hetzner2

[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
...
+----------------------------+-------------+-------------+---------+
| name                       | old_version | new_version | status  |
+----------------------------+-------------+-------------+---------+
| akismet                    | 3.3         | 4.0.2       | Updated |
| cyclone-slider-2           | 2.12.4      | 3.2.0       | Updated |
| duplicate-post             | 3.1.2       | 3.2.1       | Updated |
| insert-headers-and-footers | 1.4.1       | 1.4.2       | Updated |
| jetpack                    | 4.7.1       | 5.8         | Updated |
| ml-slider                  | 3.5         | 3.6.8       | Updated |
| post-types-order           | 1.9.3       | 1.9.3.6     | Updated |
| recent-tweets-widget       | 1.6.6       | 1.6.8       | Updated |
| shareaholic                | 7.8.0.4     | 8.6.5       | Updated |
| share-on-diaspora          | 0.7.1       | 0.7.2       | Updated |
| w3-total-cache             | 0.9.5.2     | 0.9.6       | Updated |
| wp-smushit                 | 2.6.1       | 2.7.6       | Updated |
+----------------------------+-------------+-------------+---------+
Success: Updated 12 of 12 plugins.
[root@hetzner2 htdocs]#

updated all the wp dashboard config settings for the new plugins
discovered that the "Purge ALL Varnish Cache" button resulted in an error

Trying to purge URL : http://127.0.0.1:6081/.* => 403 Forbidden

1. there's a cooresponding ossec alert & trigger by modsec:

--388c9d2b-A--
[09/Feb/2018:00:38:31 +0000] WnzthxK3iH4mmnFop@Wm5gAAAAc 127.0.0.1 37192 127.0.0.1 8000
--388c9d2b-B--
PURGE /.* HTTP/1.1
User-Agent: WordPress/4.9.3; https://osemain.opensourceecology.org
Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5
host: osemain.opensourceecology.org
X-VC-Purge-Method: regex
X-VC-Purge-Host: osemain.opensourceecology.org
Connection: Close, close
X-Forwarded-For: 127.0.0.1
X-Varnish: 45279

--388c9d2b-F--
HTTP/1.1 403 Forbidden
Content-Length: 204
Connection: close
Content-Type: text/html; charset=iso-8859-1

--388c9d2b-E--

--388c9d2b-H--
Message: Access denied with code 403 (phase 1). Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf"] [line "31"] [id "960032"] [rev "2"] [msg "Method is not allowed by policy"] [data "PURGE"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"]
Action: Intercepted (phase 1)
Stopwatch: 1518136711479621 232 (- - -)
Stopwatch2: 1518136711479621 232; combined=97, p1=83, p2=0, p3=0, p4=0, p5=14, sr=18, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"

--388c9d2b-Z--

1. added id = 960032 to the httpd config whitelist
2. varnish log alerts showed that the purge request was being sent to obi, so I added it to that apache config too
3. confirmed this works in seedhome.openbuildinginstitue.org
4. confirmed this works in fef.opensourceecology.org
5. ah, I think this fails because the varnish config is defined to use 'www.opensourceecology.org' it's temporarily 'osemain.opensourceecology.org' due to a name collision with the prod site. I'm confident this will work once the name is correct after cuttover. ignoring this issue. in the meantime, I'm pretty sure nothing is being cached at all (which is actually nice for testing the staging site).
the top-left logo shows up fine on my disposable vm, but not in my locked-down https-everywhere browser http://opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png
1. should probably change the url to use https instead in Appearance -> Theme Options -> Site Logo to use https instead (note I can't test this until after the name change)
discovered that the facebook iframe on the main page was missing
1. solution was to change 'http' to 'https'

[segment class="first-segment"]
<div class="center"><iframe style="border: none; overflow: hidden; width: 820px; height: 435px; background: white; float: center;" src="https://www.facebook.com/plugins/likebox.php?href=http%3A%2F%2Fwww.facebook.com%2FOpenSourceEcology&width=820&colorscheme=light&show_faces=true&border_color&stream=true&header=true&height=435" width="300" height="150" frameborder="0" scrolling="yes"></iframe></div>
[/segment]

confirmed that the old broken pages are still working
1. /contributors/
  1. the top slider that Catarina said we've permenantly lost is still throwing an error; Marcin will remove this from the prod site, so this will be a non-issue at cutover
  2. the timeline shows up properly
2. /community-true-fans/
  1. this page perfectly matches prod
3. /history-timeline/
  1. this page perfectly matches prod
4. /cnc-torch-table-workshop/
  1. Eventbright on left still working as desired
  2. videos embed issues are still present as expected, and a non-blocker to the migration
5. /about-videos-3/
  1. missing video content is still (not) present as expected, and a non-blocker to the migration
all looks good to me. sent an email to Marcin asking for validation.
awstats were not updating again
1. found a permissions issue

[root@hetzner2 conf.d]# ls -lah /var/log/nginx/
total 11M
drwx------  9 nginx nginx 4.0K Feb  8 07:21 .
drwxr-xr-x 13 root  root   12K Feb  8 07:21 ..
-rw-r--r--  1 nginx nginx 3.5M Feb  9 02:03 access.log
-rw-r--r--  1 nginx nginx  222 Dec 24 01:33 access.log.1.gz
-rw-r--r--  1 nginx nginx 143K Jan 30 05:02 access.log-20180130.gz
-rw-r--r--  1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz
-rw-r--r--  1 nginx nginx 186K Feb  1 06:41 access.log-20180201.gz
-rw-r--r--  1 nginx nginx 107K Feb  2 08:54 access.log-20180202.gz
-rw-r--r--  1 nginx nginx  83K Feb  3 05:49 access.log-20180203.gz
-rw-r--r--  1 nginx nginx  81K Feb  4 04:48 access.log-20180204.gz
-rw-r--r--  1 nginx nginx 265K Feb  5 09:12 access.log-20180205.gz
-rw-r--r--  1 nginx nginx 351K Feb  6 06:00 access.log-20180206.gz
-rw-r--r--  1 nginx nginx 458K Feb  7 07:35 access.log-20180207.gz
-rw-r--r--  1 nginx nginx 4.8M Feb  8 07:21 access.log-20180208
-rw-r--r--  1 nginx nginx  420 Dec 24 01:32 access.log.2.gz
-rw-r--r--  1 nginx nginx  207 Dec 24 01:29 access.log.3.gz
-rw-r--r--  1 nginx nginx  187 Dec 24 01:28 access.log.4.gz
-rw-r--r--  1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz
-rw-r--r--  1 nginx nginx  271 Dec 24 01:21 access.log.6.gz
-rw-r--r--  1 nginx nginx 323K Feb  9 02:02 error.log
-rw-r--r--  1 nginx nginx  211 Dec 24 01:33 error.log.1.gz
-rw-r--r--  1 nginx nginx 7.8K Jan 30 05:00 error.log-20180130.gz
-rw-r--r--  1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz
-rw-r--r--  1 nginx nginx 9.0K Feb  1 06:41 error.log-20180201.gz
-rw-r--r--  1 nginx nginx 7.8K Feb  2 08:52 error.log-20180202.gz
-rw-r--r--  1 nginx nginx 6.0K Feb  3 05:49 error.log-20180203.gz
-rw-r--r--  1 nginx nginx 6.2K Feb  4 04:48 error.log-20180204.gz
-rw-r--r--  1 nginx nginx 8.5K Feb  5 09:03 error.log-20180205.gz
-rw-r--r--  1 nginx nginx 9.6K Feb  6 06:00 error.log-20180206.gz
-rw-r--r--  1 nginx nginx  13K Feb  7 07:35 error.log-20180207.gz
-rw-r--r--  1 nginx nginx 433K Feb  8 07:21 error.log-20180208
-rw-r--r--  1 nginx nginx  592 Dec 24 01:33 error.log.2.gz
-rw-r--r--  1 nginx nginx  413 Dec 24 01:29 error.log.3.gz
-rw-r--r--  1 nginx nginx  210 Dec 24 01:28 error.log.4.gz
-rw-r--r--  1 nginx nginx  211 Dec 24 01:27 error.log.5.gz
-rw-r--r--  1 nginx nginx  506 Dec 24 01:24 error.log.6.gz
drwxr-xr-x  2 root  root  4.0K Feb  8 07:21 fef.opensourceecology.org
drwxr-xr-x  2 root  root  4.0K Feb  8 07:21 forum.opensourceecology.org
drwxr-xr-x  2 root  root  4.0K Feb  8 07:21 oswh.opensourceecology.org
drwxr-xr-x  2 root  root  4.0K Feb  8 07:21 seedhome.openbuildinginstitute.org
drwxr-xr-x  2 root  root  4.0K Feb  8 07:21 wiki.opensourceecology.org
drw-r--r--  2 nginx nginx 4.0K Feb  9 02:01 www.openbuildinginstitute.org
drwxr-xr-x  2 root  root  4.0K Feb  8 07:21 www.opensourceecology.org
[root@hetzner2 conf.d]#

1. those probably should all be owned by nginx

[root@hetzner2 nginx]# chown nginx:nginx fef.opensourceecology.org
[root@hetzner2 nginx]# chown -R nginx:nginx fef.opensourceecology.org
[root@hetzner2 nginx]# chown -R nginx:nginx forum.opensourceecology.org/
[root@hetzner2 nginx]# chown -R nginx:nginx oswh.opensourceecology.org/
[root@hetzner2 nginx]# chown -R nginx:nginx seedhome.openbuildinginstitute.org/
[root@hetzner2 nginx]# chown -R nginx:nginx wiki.opensourceecology.org/
[root@hetzner2 nginx]# chown -R nginx:nginx www.opensourceecology.org/
[root@hetzner2 nginx]# chown -R nginx:nginx www.openbuildinginstitute.org/
[root@hetzner2 nginx]# ls -lah
total 11M
drwx------  9 nginx nginx 4.0K Feb  8 07:21 .
drwxr-xr-x 13 root  root   12K Feb  8 07:21 ..
-rw-r--r--  1 nginx nginx 3.5M Feb  9 02:06 access.log
-rw-r--r--  1 nginx nginx  222 Dec 24 01:33 access.log.1.gz
-rw-r--r--  1 nginx nginx 143K Jan 30 05:02 access.log-20180130.gz
-rw-r--r--  1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz
-rw-r--r--  1 nginx nginx 186K Feb  1 06:41 access.log-20180201.gz
-rw-r--r--  1 nginx nginx 107K Feb  2 08:54 access.log-20180202.gz
-rw-r--r--  1 nginx nginx  83K Feb  3 05:49 access.log-20180203.gz
-rw-r--r--  1 nginx nginx  81K Feb  4 04:48 access.log-20180204.gz
-rw-r--r--  1 nginx nginx 265K Feb  5 09:12 access.log-20180205.gz
-rw-r--r--  1 nginx nginx 351K Feb  6 06:00 access.log-20180206.gz
-rw-r--r--  1 nginx nginx 458K Feb  7 07:35 access.log-20180207.gz
-rw-r--r--  1 nginx nginx 4.8M Feb  8 07:21 access.log-20180208
-rw-r--r--  1 nginx nginx  420 Dec 24 01:32 access.log.2.gz
-rw-r--r--  1 nginx nginx  207 Dec 24 01:29 access.log.3.gz
-rw-r--r--  1 nginx nginx  187 Dec 24 01:28 access.log.4.gz
-rw-r--r--  1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz
-rw-r--r--  1 nginx nginx  271 Dec 24 01:21 access.log.6.gz
-rw-r--r--  1 nginx nginx 324K Feb  9 02:06 error.log
-rw-r--r--  1 nginx nginx  211 Dec 24 01:33 error.log.1.gz
-rw-r--r--  1 nginx nginx 7.8K Jan 30 05:00 error.log-20180130.gz
-rw-r--r--  1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz
-rw-r--r--  1 nginx nginx 9.0K Feb  1 06:41 error.log-20180201.gz
-rw-r--r--  1 nginx nginx 7.8K Feb  2 08:52 error.log-20180202.gz
-rw-r--r--  1 nginx nginx 6.0K Feb  3 05:49 error.log-20180203.gz
-rw-r--r--  1 nginx nginx 6.2K Feb  4 04:48 error.log-20180204.gz
-rw-r--r--  1 nginx nginx 8.5K Feb  5 09:03 error.log-20180205.gz
-rw-r--r--  1 nginx nginx 9.6K Feb  6 06:00 error.log-20180206.gz
-rw-r--r--  1 nginx nginx  13K Feb  7 07:35 error.log-20180207.gz
-rw-r--r--  1 nginx nginx 433K Feb  8 07:21 error.log-20180208
-rw-r--r--  1 nginx nginx  592 Dec 24 01:33 error.log.2.gz
-rw-r--r--  1 nginx nginx  413 Dec 24 01:29 error.log.3.gz
-rw-r--r--  1 nginx nginx  210 Dec 24 01:28 error.log.4.gz
-rw-r--r--  1 nginx nginx  211 Dec 24 01:27 error.log.5.gz
-rw-r--r--  1 nginx nginx  506 Dec 24 01:24 error.log.6.gz
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 fef.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 forum.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 oswh.opensourceecology.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 seedhome.openbuildinginstitute.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 wiki.opensourceecology.org
drw-r--r--  2 nginx nginx 4.0K Feb  9 02:01 www.openbuildinginstitute.org
drwxr-xr-x  2 nginx nginx 4.0K Feb  8 07:21 www.opensourceecology.org
[root@hetzner2 nginx]#

1. that appears to be working. Now I just have to wait another week or so..
added an 'awstats' record to opensourceecology.org in cloudflare.com

Wed Feb 07, 2018

Marcin responded to the osemain issues. Here's the status:
1. /
  1. Marcin confirmed that front-page content (the video) this was fixed
2. /contributors/
  1. Catarina will be looking into the "featured collaborators" slider issue on this page which is broken on production & staging
  2. I'll look into the "collaborators timeline" issue that is fine on prod, but missing on staging
  3. Marcin said the missing photos on the "collaborators timeline" should be dealt with later & not block this migration
3. /community/
  1. Marcin updated the site, removing the broken Flattr embed http://opensourceecology.org/community/
4. /community-true-fans/
  1. I need to investigate why these 2x sliders are broken in staging, but working in production
5. /history-timeline/
  1. I need to investigate why these 2x sliders are broken in staging, but working in production

/cnc-torch-table-workshop/

1. Eventbright is truncated on the left
2. The first youtube video shows up as a link, instead of as an embed = https://www.youtube.com/watch?v=JOH03vzDYQg
3. The first vemo video shows up as a link, instead of as an embed = https://vimeo.com/23785186

/about-videos-3/

1. Marcin said the videos are missing. I asked him to attempt to add them & if he encountered any issues, send me the errors so I can try to reproduce & fix it.
began debugging osemain staging slider issues on /contributors/
1. first, none of the plugins were enabled. This was a permission issue, solved by following the guide at Wordpress

vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"

chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

next "revolution slider" wasn't enabled not sure why. I enabled it and the one named "Revolution Slider Patch"
finally, I saw that the last slider wasn't actually a slider; it was just an iframe to an http site = http://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0ArpE5Y9PpJCXdFhleTktTkoyeHNFUXktUXJCMGVkbVE&font=Bevan-PotanoSans&maptype=toner&lang=en&start_at_end=true&hash_bookmark=true&height=650#16
replacing the above iframe's http protocol with https fixed the issue, but first I had to whitelist some modsec triggers to be able to update the page
1. 958057, XSS
2. 958056, XSS
3. 973327, XSS
began debugging issues on /community-true-fans/
1. same issue, there was an iframe linking to an http page on cdn.knightlab.com; changing to https fixed it https://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0ArpE5Y9PpJCXdEhYb1NkR1dGRnhPV2FGYndseUg2WkE&font=Merriweather-NewsCycle&maptype=toner&lang=en&start_at_end=true&start_at_slide=34&height=650
began debugging issues on /history-timeline/
1. same issue, there was an iframe linking to an http page on cdn.knightlab.com; changing to https fixed it https://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0AkNG-lv1ELQvdEMxdzRnU2VFbUllZ2Y0cnZPRld3SXc&font=Bevan-PotanoSans&maptype=toner&lang=en&hash_bookmark=true&start_at_slide=23&height=650
began troubleshooting /cnc-torch-table-workshop/
1. found a section on the page edit called "Choose Custom Sidebar" and a drop-down for "Primary Sidebar Choice" said "CNC Torch Table Workshop". So I opened Appearence -> Widgets in a new tab, and found the "CNC Torcn Table Workshop" widget with this contents:

 <a href="http://opensourceecology.org/CNC-torch-table-workshop/#overview">Overview</a>
 <a href="http://opensourceecology.org/CNC-torch-table-workshop/#learning">Learning Outcomes</a>
 <a href="http://opensourceecology.org/CNC-torch-table-workshop/#schedule">Schedule</a>
 <a href="http://opensourceecology.org/CNC-torch-table-workshop#registration">Registration</a>

<div style="width:195px; text-align:center;" ><iframe  src="https://www.eventbrite.com/countdown-widget?eid=38201763503" frameborder="0" height="322" width="195" marginheight="0" marginwidth="0" scrolling="no" allowtransparency="true"></iframe><div style="font-family:Helvetica, Arial; font-size:12px; padding:10px 0 5px; margin:2px; width:195px; text-align:center;" ><a class="powered-by-eb" style="color: #ADB0B6; text-decoration: none;" target="_blank" href="http://www.eventbrite.com/">Powered by Eventbrite</a></div></div>

1. I changed the height of "322" of the iframe to "999", pressed save, and cleared the varnish cache. That didn't help
2. I added ";height:999px" to the containing div, pressed save, and cleared the varnish cache.
3. I replaced the "CNC Torch Table Workshop" widget of type "Text" with one of the same contents of type "Custom HTML", and it worked.
  1. the "Custom HTML" widget was added in wordpress v4.8.1 https://wptavern.com/wordpress-4-8-1-adds-a-dedicated-custom-html-widget
    1. this makes sense, as this migration includes the update of core wp from v4.7.9 to v4.9.2
  2. I had to add some breaks to make the links on separate lines; I guess that's what "Text" does for you--and also breaks dynamic height? Here's the new contents:
    1. ugh, fucking hell. I can't copy from the new "Custom HTML widget?" It adds line numbers to my clipboard, breaking copy-and-paste to/from the window. This is horrible!
    2. I tried switching back to "text" so I could just try to uncheck the "automatically add paragraphs" checkbox, but that's only a "legacy" thing (as the link above points-out. Once you delete a text box & re-add it, that checkbox disappears :(
    3. it looks like this new "custom html" widget is powered by codemirror
    4. ok, I found the solution to the annoying codemirror issues, but it can't be fixed site-wide. It must be fixed on a per-user basis by going to the user's profile & checking the "Disable syntax highlighting when editing code" checkbox https://make.wordpress.org/core/tag/codemirror/
  3. ok, here's the new contents with line breaks:

 <a href="http://opensourceecology.org/CNC-torch-table-workshop/#overview">Overview</a>
<br/>

<a href="http://opensourceecology.org/CNC-torch-table-workshop/#learning">Learning Outcomes</a>
<br/>

<a href="http://opensourceecology.org/CNC-torch-table-workshop/#schedule">Schedule</a>
</>

<a href="http://opensourceecology.org/CNC-torch-table-workshop#registration">Registration</a><div style="width:195px; text-align:center;" >
<br/>
	
<div style="width:195px; text-align:center;" ><iframe  src="https://www.eventbrite.com/countdown-widget?eid=38201763503" frameborder="0" height="322" width="195" marginheight="0" marginwidth="0" scrolling="no" allowtransparency="true"></iframe><div style="font-family:Helvetica, Arial; font-size:12px; padding:10px 0 5px; margin:2px; width:195px; text-align:center;" ><a class="powered-by-eb" style="color: #ADB0B6; text-decoration: none;" target="_blank" href="http://www.eventbrite.com/">Powered by Eventbrite</a></div></div>

1. I emailed marcin; he said that the issue embedding the 2x videos on this page shouldn't be a blocker & confirmed that I should proceed with the migration with the content missing
began debugging the missing videos on /about-video-3/
1. there's literally nothing in the textarea for this page in wordpress. It's empty for both staging & production *shrug*
2. I emailed marcin; he said this shouldn't be a blocker & confirmed that I should proceed with the migration with the content missing

So the osemain issues that remain to be fixed are:
1. /contributors/
  1. Catarina will be looking into the "featured collaborators" slider issue on this page which is broken on production & staging

/cnc-torch-table-workshop/

Tue Feb 06, 2018

the osemain staging site was created on 2018-01-03

[maltfield@hetzner2 backups_for_migration_from_hetzner1]$ date
Tue Feb  6 15:53:00 UTC 2018
[maltfield@hetzner2 backups_for_migration_from_hetzner1]$ pwd
/var/tmp/backups_for_migration_from_hetzner1
[maltfield@hetzner2 backups_for_migration_from_hetzner1]$ ls -lah | grep -i osemain
drwxr-xr-x   4 root root 4.0K Jan  3 21:32 osemain_20180103
[maltfield@hetzner2 backups_for_migration_from_hetzner1]$

I refreshed it for today = 2018-02-06

# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180206"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
 dbUser_hetzner2="osemain_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

I updated the osemain site with a new wp core, but stopped short of installing our new security wp plugins or attempting to update the existing wp plugins. Here's a breakdown of what I found with the site:
1. /
  1. vid fixed w/ update
2. /contributors/
  1. revolution slider is broken on the production site with error message = "Revolution Slider Error: Slider with alias collabslider not found."
  2. collaborators timeline is missing entirely on the staging site
  3. many photos 404ing for the collaborators timeline on the production site
3. /community/

/community-true-fans/

1. 2x sliders missing on staging; working fine on production

/history-timeline/

1. 2x sliders missing on staging; working fine on production

/cnc-torch-table-workshop/

1. Eventbright is truncated on the left
2. The first youtube video shows up as a link, instead of as an embed = https://www.youtube.com/watch?v=JOH03vzDYQg
3. The first vemo video shows up as a link, instead of as an embed = https://vimeo.com/23785186

/about-videos-3/

1. Marcin said the videos are missing. I asked him to attempt to add them & if he encountered any issues, send me the errors so I can try to reproduce & fix it.

Mon Feb 05, 2018

DIdn't hear back from Marcin wrt my email about the osemigration, so I decided to delay the migration CHG-2018-02-05

Sun Feb 04, 2018

Marcin q confirmed that we can migrate the osemain wp site on Monday, Feb 5th.
I began documenting this CHG process at CHG-2018-02-05
I found that many of the plugins 3 of the active plugins had pending updates: fundraising, wpmdev-updates, and wp-smush-pro

[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning:  ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning:  file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------------------------------+----------+-----------+---------+
| name                                        | status   | update    | version |
+---------------------------------------------+----------+-----------+---------+
| akismet                                     | active   | none      | 3.3     |
| brankic-photostream-widget                  | active   | none      | 1.3.1   |
| cyclone-slider-2                            | inactive | none      | 2.12.4  |
| duplicate-post                              | active   | none      | 3.1.2   |
| flattr                                      | inactive | none      | 1.2.2   |
| force-strong-passwords                      | active   | none      | 1.8.0   |
| fundraising                                 | active   | available | 2.6.4.5 |
| google-authenticator                        | active   | none      | 0.48    |
| google-authenticator-encourage-user-activat | active   | none      | 0.2     |
| ion                                         |          |           |         |
| hello                                       | inactive | none      | 1.6     |
| insert-headers-and-footers                  | inactive | none      | 1.4.1   |
| jetpack                                     | active   | none      | 4.7.1   |
| ml-slider                                   | active   | none      | 3.5     |
| ml-slider-pro                               | active   | none      | 2.6.6   |
| open-in-new-window-plugin                   | inactive | none      | 2.4     |
| patch-for-revolution-slider                 | active   | none      | 2.4.1   |
| post-types-order                            | inactive | none      | 1.9.3   |
| really-simple-facebook-twitter-share-button | inactive | none      | 4.5     |
| s                                           |          |           |         |
| recent-tweets-widget                        | active   | none      | 1.6.6   |
| rename-wp-login                             | active   | none      | 2.5.5   |
| revision-control                            | inactive | none      | 2.3.2   |
| revslider                                   | active   | none      | 4.3.8   |
| shareaholic                                 | inactive | none      | 7.8.0.4 |
| share-on-diaspora                           | inactive | none      | 0.7.1   |
| ssl-insecure-content-fixer                  | active   | none      | 2.5.0   |
| vcaching                                    | active   | none      | 1.6.7   |
| w3-total-cache                              | inactive | none      | 0.9.5.2 |
| wp-memory-usage                             | inactive | none      | 1.2.2   |
| wp-optimize                                 | inactive | none      | 2.1.1   |
| wp-facebook-open-graph-protocol             | inactive | none      | 2.0.13  |
| wpmudev-updates                             | active   | available | 4.2     |
| wp-smushit                                  | inactive | none      | 2.6.1   |
| wp-smush-pro                                | active   | available | 2.4.5   |
+---------------------------------------------+----------+-----------+---------+
[WPMUDEV API Error] 4.2 | User has blocked requests through HTTP. ((unknown URL) [500]) 
[root@hetzner2 ~]#

that update attempt failed

[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning:  ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning:  file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Warning: Update package not available.
Warning: Update package not available.
Warning: Update package not available.
+-----------------+-------------+-------------+--------+
| name            | old_version | new_version | status |
+-----------------+-------------+-------------+--------+
| fundraising     | 2.6.4.5     | 2.6.4.9     | Error  |
| wpmudev-updates | 4.2         | 4.4         | Error  |
| wp-smush-pro    | 2.4.5       | 2.7.6       | Error  |
+-----------------+-------------+-------------+--------+
Success: Plugins already updated.
[root@hetzner2 ~]#

after commenting-out the WP_HTTP_BLOCK_EXTERNAL set to 'true' in wp-config.php, however, I see there were many more updates available

[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning:  ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning:  file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------------------------------+----------+-----------+---------+
| name                                        | status   | update    | version |
+---------------------------------------------+----------+-----------+---------+
| akismet                                     | active   | available | 3.3     |
| brankic-photostream-widget                  | active   | none      | 1.3.1   |
| cyclone-slider-2                            | inactive | available | 2.12.4  |
| duplicate-post                              | active   | available | 3.1.2   |
| flattr                                      | inactive | none      | 1.2.2   |
| force-strong-passwords                      | active   | none      | 1.8.0   |
| fundraising                                 | active   | available | 2.6.4.5 |
| google-authenticator                        | active   | none      | 0.48    |
| google-authenticator-encourage-user-activat | active   | none      | 0.2     |
| ion                                         |          |           |         |
| hello                                       | inactive | none      | 1.6     |
| insert-headers-and-footers                  | inactive | available | 1.4.1   |
| jetpack                                     | active   | available | 4.7.1   |
| ml-slider                                   | active   | available | 3.5     |
| ml-slider-pro                               | active   | none      | 2.6.6   |
| open-in-new-window-plugin                   | inactive | none      | 2.4     |
| patch-for-revolution-slider                 | active   | none      | 2.4.1   |
| post-types-order                            | inactive | available | 1.9.3   |
| really-simple-facebook-twitter-share-button | inactive | none      | 4.5     |
| s                                           |          |           |         |
| recent-tweets-widget                        | active   | available | 1.6.6   |
| rename-wp-login                             | active   | none      | 2.5.5   |
| revision-control                            | inactive | none      | 2.3.2   |
| revslider                                   | active   | none      | 4.3.8   |
| shareaholic                                 | inactive | available | 7.8.0.4 |
| share-on-diaspora                           | inactive | available | 0.7.1   |
| ssl-insecure-content-fixer                  | active   | none      | 2.5.0   |
| vcaching                                    | active   | none      | 1.6.7   |
| w3-total-cache                              | inactive | available | 0.9.5.2 |
| wp-memory-usage                             | inactive | none      | 1.2.2   |
| wp-optimize                                 | inactive | none      | 2.1.1   |
| wp-facebook-open-graph-protocol             | inactive | none      | 2.0.13  |
| wpmudev-updates                             | active   | available | 4.2     |
| wp-smushit                                  | inactive | available | 2.6.1   |
| wp-smush-pro                                | active   | available | 2.4.5   |
+---------------------------------------------+----------+-----------+---------+
[root@hetzner2 ~]#

then attempting updates again was partially successful

[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning:  ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning:  file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pea
r:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.
opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/va
r/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Downloading update from https://downloads.wordpress.org/plugin/akismet.4.0.2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/akismet-4.0.2.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/cyclone-slider-2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/cyclone-slider-2-3.2.0.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/duplicate-post.3.2.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
Downloading update from https://downloads.wordpress.org/plugin/insert-headers-and-footers.1.4.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/jetpack.5.7.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/ml-slider.3.6.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from http://wp-updates.com/api/1/download/plugin/sXpq1rDdVNBB4r8Tm-EPBg/ml-slider-pro...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/post-types-order.1.9.3.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/recent-tweets-widget.1.6.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/shareaholic.8.6.1.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/shareaholic-8.6.1.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/share-on-diaspora.0.7.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/w3-total-cache.0.9.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
Downloading update from https://downloads.wordpress.org/plugin/wp-smushit.2.7.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
+----------------------------+-------------+-------------+---------+
| name                       | old_version | new_version | status  |
+----------------------------+-------------+-------------+---------+
| akismet                    | 3.3         | 4.0.2       | Updated |
| cyclone-slider-2           | 2.12.4      | 3.2.0       | Updated |
| duplicate-post             | 3.1.2       | 3.2.1       | Updated |
| fundraising                | 2.6.4.5     | 2.6.4.9     | Updated |
| insert-headers-and-footers | 1.4.1       | 1.4.2       | Updated |
| jetpack                    | 4.7.1       | 5.7.1       | Updated |
| ml-slider                  | 3.5         | 3.6.8       | Updated |
| ml-slider-pro              | 2.6.6       | 2.7.1       | Updated |
| post-types-order           | 1.9.3       | 1.9.3.6     | Updated |
| recent-tweets-widget       | 1.6.6       | 1.6.8       | Updated |
| shareaholic                | 7.8.0.4     | 8.6.1       | Updated |
| share-on-diaspora          | 0.7.1       | 0.7.2       | Updated |
| w3-total-cache             | 0.9.5.2     | 0.9.6       | Updated |
| wpmudev-updates            | 4.2         | 4.4         | Updated |
| wp-smushit                 | 2.6.1       | 2.7.6       | Updated |
| wp-smush-pro               | 2.4.5       | 2.7.7       | Updated |
+----------------------------+-------------+-------------+---------+
Success: Updated 16 of 16 plugins.
[root@hetzner2 ~]#

what follows showed everything was updated, except the same 3 plugins that fail. So it's probably those specific plugins' fault. Let's move on..

[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
...
+---------------------------------------------+----------+-----------+---------+
| name                                        | status   | update    | version |
+---------------------------------------------+----------+-----------+---------+
| akismet                                     | active   | none      | 4.0.2   |
| brankic-photostream-widget                  | active   | none      | 1.3.1   |
| cyclone-slider-2                            | inactive | none      | 3.2.0   |
| duplicate-post                              | active   | none      | 3.2.1   |
| flattr                                      | inactive | none      | 1.2.2   |
| force-strong-passwords                      | active   | none      | 1.8.0   |
| fundraising                                 | active   | available | 2.6.4.5 |
| google-authenticator                        | active   | none      | 0.48    |
| google-authenticator-encourage-user-activat | active   | none      | 0.2     |
| ion                                         |          |           |         |
| hello                                       | inactive | none      | 1.6     |
| insert-headers-and-footers                  | inactive | none      | 1.4.2   |
| jetpack                                     | active   | none      | 5.7.1   |
| ml-slider                                   | active   | none      | 3.6.8   |
| ml-slider-pro                               | active   | none      | 2.7.1   |
| open-in-new-window-plugin                   | inactive | none      | 2.4     |
| patch-for-revolution-slider                 | active   | none      | 2.4.1   |
| post-types-order                            | inactive | none      | 1.9.3.6 |
| really-simple-facebook-twitter-share-button | inactive | none      | 4.5     |
| s                                           |          |           |         |
| recent-tweets-widget                        | active   | none      | 1.6.8   |
| rename-wp-login                             | active   | none      | 2.5.5   |
| revision-control                            | inactive | none      | 2.3.2   |
| revslider                                   | active   | none      | 4.3.8   |
| shareaholic                                 | inactive | none      | 8.6.1   |
| share-on-diaspora                           | inactive | none      | 0.7.2   |
| ssl-insecure-content-fixer                  | active   | none      | 2.5.0   |
| vcaching                                    | active   | none      | 1.6.7   |
| w3-total-cache                              | inactive | none      | 0.9.6   |
| wp-memory-usage                             | inactive | none      | 1.2.2   |
| wp-optimize                                 | inactive | none      | 2.1.1   |
| wp-facebook-open-graph-protocol             | inactive | none      | 2.0.13  |
| wpmudev-updates                             | active   | available | 4.2     |
| wp-smushit                                  | inactive | none      | 2.7.6   |
| wp-smush-pro                                | active   | available | 2.4.5   |
+---------------------------------------------+----------+-----------+---------+
[root@hetzner2 ~]#

Found a bunch of content issues on the site when doing a quick spot-check. Emailed Marcin for revalidation & a go/no-go decision for the cutover tomorrow
1. error "Revolution Slider Error: Slider with alias collabslider not found."

* http://opensourceecology.org/contributors/
* https://osemain.opensourceecology.org/contributors/

1. flattr embed is broken on donate section

* http://opensourceecology.org/community/
* https://osemain.opensourceecology.org/community/

1. slider for True Fans isn't appearing

* http://opensourceecology.org/community-true-fans/
* https://osemain.opensourceecology.org/community-true-fans/

added a CHG for the forum cutover CHG-2018-02-04
1. followed the above guide. We're now storing the dynamic (php) vanilla files outside the docroot, next to the static content. Of course, the static content takes up more space, but it's not too bad.

[root@hetzner2 forum.opensourceecology.org]# date
Sun Feb  4 18:13:00 UTC 2018
[root@hetzner2 forum.opensourceecology.org]# pwd
/var/www/html/forum.opensourceecology.org
[root@hetzner2 forum.opensourceecology.org]# du -sh *
2.7G    htdocs
173M    vanilla_docroot_backup.20180113
[root@hetzner2 forum.opensourceecology.org]#

updated dns of forum.opensourceecology.org to point to hetzner2 instead of hetzner1
1. deleted the CNAME entry for 'forum' pointing to dedi978.your-server.de
2. added an A entry for 'forum' pointing to 138.201.84.243
updated the name 'stagingforum' to 'forum' in /etc/httpd/conf.d/00-forum.opensourceecology.org.conf
updated the name 'stagingforum' to 'forum' in /etc/nginx/conf.d/forum.opensourceecology.org.conf
updated the opensourceecology.org certificate

[root@hetzner2 forum.opensourceecology.org]# certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org
...
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/opensourceecology.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/opensourceecology.org/privkey.pem
   Your cert will expire on 2018-05-05. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le
[root@hetzner2 forum.opensourceecology.org]# /bin/chmod 0400 /etc/letsencrypt/archive/*/pri*
[root@hetzner2 forum.opensourceecology.org]# nginx -t && service nginx reload 
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Redirecting to /bin/systemctl reload nginx.service
[root@hetzner2 forum.opensourceecology.org]#

Did an unsafe varnish cache clear

service varnish restart

confirmed that the static-content site was loading on the new url https://forum.opensourceecology.org
updated our wiki (where the forum links to for info on how to create an account) with a notice that the site was deprecated in 2018 Vanilla_Forums
began to test sending our encrypted ossec alerts to multiple recipients
discovered that our spf record is invalid. I looked up the proper way to do this on cloudflare https://support.cloudflare.com/hc/en-us/articles/200168626-How-do-I-add-a-SPF-record-
1. I had "spf" in the "Name" column, but it should have been "@". I updated this, and then Cloudflare automatically replaced the "@" that I typed with "opensourceecology.org"
2. after the change above, I got a change in the lookup tool; success! https://mxtoolbox.com/SuperTool.aspx?action=spf%3aopensourceecology.org&run=toolpage
updated the sent_encrypted_alarm.sh script to support multiple recipients

[root@hetzner2 ossec]# date
Mon Feb  5 01:39:24 UTC 2018
[root@hetzner2 ossec]# pwd
/var/ossec
[root@hetzner2 ossec]# cat sent_encrypted_alarm.sh 
#!/bin/bash

# store the would-be plaintext email body
plaintext=`/usr/bin/formail -I ""`

# loop through all recipients & send them individually-encrypted mail
recipients="michael@opensourceecology.org marcin@opensourceecology.org"
for recipient in $(echo $recipients); do
		# leave nothing unencrypted, including the subject!
		echo "${plaintext}" | /usr/bin/gpg --homedir /var/ossec/.gnupg --trust-model always -ear "${recipient}" | /usr/bin/mail -r noreply@opensourceecology.org -s "" "${recipient}"
done

exit 0

found & documented command to clear varnish cache without actually restarting the process Web_server_configuration

varnishadm 'ban req.url ~ "."'

Thr Feb 01, 2018

confirmed that the forum wget retry worked
rsync'd the wget into the document root, and a spot-checked showed that most pages worked
1. saw an issue (Forbidden) with all the tagged pages, ie https://stagingforum.opensourceecology.org/discussions/tagged/water.html

[Thu Feb 01 17:29:02.262787 2018] [core:crit] [pid 30854] (9)Bad file descriptor: [client 127.0.0.1:45488] AH00529: /var/www/html/forum.opensourceecology.org/htdocs/discussions/tagged/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/html/forum.opensourceecology.org/htdocs/discussions/tagged/' is executable

the issue appears to be with the directory named '.htaccess' for the tag of the same name

[root@hetzner2 htdocs]# ls -lah discussions/tagged/.htaccess
total 64K
drwxr-xr-x   2 root root 4.0K Feb  1 11:36 .
drwxr-xr-x 573 root root  36K Feb  1 17:27 ..
-rw-r--r--   1 root root 1.7K Feb  1 10:33 feed.rss
-rw-r--r--   1 root root  17K Feb  1 11:36 p1.html
[root@hetzner2 htdocs]#

removing the above dir worked. The consequence is we will get a 404 for any links to this tag, but that's an acceptable loss IMO; te discussions themselves will still be accessible
removed the forums dir from /etc/php.ini since we're now just static content
restarted httpd
confirmed that awstats has been updating, but--since today happens to be the first of the month--there's no data yet. I'll check again in a week or so.

Wed Jan 31, 2018

moved the wget'd forums into the docroot & found a lot of Forbidden responses to my requests
1. for some reason, the pages are named 'p1' instead of 'index.html' The "Forbidden" is caused by the server's refusal to do a directory listing.
2. solution: add a symlink to "p1" from "index.html" in every dir with a "p1" but no "index.html"

[root@hetzner2 htdocs]# for p1 in $(find . -name p1); do dir=`dirname $p1`; pushd $dir; ln -s "p1" "index.html"; popd; done

1. that helped, but there's still actually missing content (ie: https://stagingforum.opensourceecology.org/categories/other-languages/)

[root@hetzner2 htdocs]# ls -lah categories/other-languages/
total 12K
drwxrwxrwx  2 root root 4.0K Jan 25 16:23 .
drwxrwxrwx 55 root root 4.0K Jan 25 20:52 ..
-rwxrwxrwx  1 root root 3.5K Jan 25 16:23 feed.rss
[root@hetzner2 htdocs]#

1. attempting again with a different command from https://www.linuxjournal.com/content/downloading-entire-web-site-wget

[root@hetzner2 oseforum.20180201]# date
Thu Feb  1 03:39:34 UTC 2018
[root@hetzner2 oseforum.20180201]# pwd
/var/tmp/deleteMeIn2019/oseforum.20180201
[root@hetzner2 oseforum.20180201]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "forum.opensourceecology.org" "http://forum.opensourceecology.org"
<pre>
## if the above doesn't work, I'll try some of the options in the comments (ie: -m, --wait, --limit-rate, etc)

=Tue Jan 30, 2018=
# Trained Marcin on PGP

=Fri Jan 26, 2018=
# Emailed Marcin about setting up PGP
# Confirmed that the wget of the forums finished
<pre>
[root@hetzner2 wget]# pwd
/var/tmp/deleteMeIn2019/oseoforum.20180125/wget
[root@hetzner2 ~]# wget -r -p -e robots=off http://forum.opensourceecology.org
...
FINISHED --2018-01-26 00:53:29--
Total wall clock time: 8h 39m 3s
Downloaded: 96387 files, 2.2G in 1m 33s (24.5 MB/s)
[root@hetzner2 wget]# du -sh forum.opensourceecology.org/*
48K     forum.opensourceecology.org/activity
264K    forum.opensourceecology.org/applications
300K    forum.opensourceecology.org/cache
3.0M    forum.opensourceecology.org/categories
27M     forum.opensourceecology.org/dashboard
1.4G    forum.opensourceecology.org/discussion
15M     forum.opensourceecology.org/discussions
873M    forum.opensourceecology.org/entry
148K    forum.opensourceecology.org/index.html
744K    forum.opensourceecology.org/plugins
72M     forum.opensourceecology.org/profile
12K     forum.opensourceecology.org/search
28K     forum.opensourceecology.org/search?Search=#000000&Mode=like
28K     forum.opensourceecology.org/search?Search=#0&Mode=like
16K     forum.opensourceecology.org/search?Search=#13Lifetime&Mode=like
16K     forum.opensourceecology.org/search?Search=#197187&Mode=like
32K     forum.opensourceecology.org/search?Search=#1&Mode=like
28K     forum.opensourceecology.org/search?Search=#2&Mode=like
16K     forum.opensourceecology.org/search?Search=#363636&Mode=like
28K     forum.opensourceecology.org/search?Search=#3&Mode=like
16K     forum.opensourceecology.org/search?Search=#458&Mode=like
20K     forum.opensourceecology.org/search?Search=#4&Mode=like
28K     forum.opensourceecology.org/search?Search=#5&Mode=like
20K     forum.opensourceecology.org/search?Search=#6&Mode=like
16K     forum.opensourceecology.org/search?Search=#7&Mode=like
16K     forum.opensourceecology.org/search?Search=#8-High&Mode=like
28K     forum.opensourceecology.org/search?Search=#8&Mode=like
16K     forum.opensourceecology.org/search?Search=#9-Industrial&Mode=like
16K     forum.opensourceecology.org/search?Search=#9&Mode=like
16K     forum.opensourceecology.org/search?Search=#Another&Mode=like
16K     forum.opensourceecology.org/search?Search=#apollo&Mode=like
16K     forum.opensourceecology.org/search?Search=#Cloud&Mode=like
16K     forum.opensourceecology.org/search?Search=#Edit&Mode=like
16K     forum.opensourceecology.org/search?Search=#Fabiofranca&Mode=like
16K     forum.opensourceecology.org/search?Search=#freecad&Mode=like
16K     forum.opensourceecology.org/search?Search=#HUBCAMP&Mode=like
16K     forum.opensourceecology.org/search?Search=#openhardware&Mode=like
16K     forum.opensourceecology.org/search?Search=#opensourceecology&Mode=like
16K     forum.opensourceecology.org/search?Search=#qi-hardware&Mode=like
16K     forum.opensourceecology.org/search?Search=#qihardware&Mode=like
16K     forum.opensourceecology.org/search?Search=#REDIRECT&Mode=like
16K     forum.opensourceecology.org/search?Search=#toc&Mode=like
16K     forum.opensourceecology.org/search?Search=#toctitle&Mode=like
24K     forum.opensourceecology.org/themes
25M     forum.opensourceecology.org/uploads
764K    forum.opensourceecology.org/vanilla
[root@hetzner2 wget]#

Thr Jan 25, 2018

Marcin said that the forums initial validation is complete
discovered that there was an 'oseforum' directory inside the forums docroot that had a duplicate copy of all the files in the docroot.
1. I moved this director to
I began a quick investigation on the low-hanging-fruit to harden Vanilla
1. I didn't find any good guides. Lots of discussion about permissions, though. https://open.vanillaforums.com/discussion/748/security
2. the fucking installation guide tells us to make the conf directory inside the docroot *and* be writeable by the webserver. That's a foundational red-flag! https://github.com/vanilla/vanilla/blob/master/README.md
3. probably the most important step to securing the forums would be updating the core Vanilla version
  1. The version of Vanilla Forums that we're running is Version 2.0.18.1
  2. The current version of Vanilla Forums is 2.5
  3. The number of plugins that we're using tells me that an update of the Vanilla Forums core code probably going to be extremely non-trivial

[root@hetzner2 htdocs]# date Thu Jan 25 13:54:16 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/forum.opensourceecology.org/htdocs [root@hetzner2 htdocs]# grep 'EnabledPlugins' conf/config.php // EnabledPlugins $Configuration['EnabledPlugins']['HtmLawed'] = 'HtmLawed'; $Configuration['EnabledPlugins']['embedvanilla'] = 'embedvanilla'; $Configuration['EnabledPlugins']['Tagging'] = 'Tagging'; $Configuration['EnabledPlugins']['Flagging'] = 'Flagging'; $Configuration['EnabledPlugins']['Liked'] = 'Liked'; $Configuration['EnabledPlugins']['OpenID'] = 'OpenID'; $Configuration['EnabledPlugins']['Twitter'] = 'Twitter'; $Configuration['EnabledPlugins']['Sitemaps'] = 'Sitemaps'; $Configuration['EnabledPlugins']['GoogleSignIn'] = 'GoogleSignIn'; $Configuration['EnabledPlugins']['Facebook'] = 'Facebook'; $Configuration['EnabledPlugins']['FileUpload'] = 'FileUpload'; $Configuration['EnabledPlugins']['VanillaStats'] = 'VanillaStats'; $Configuration['EnabledPlugins']['EMailSubscribe'] = 'EMailSubscribe'; $Configuration['EnabledPlugins']['Emotify'] = 'Emotify'; $Configuration['EnabledPlugins']['VanillaInThisDiscussion'] = 'VanillaInThisDiscussion'; $Configuration['EnabledPlugins']['ProxyConnect'] = 'ProxyConnect'; $Configuration['EnabledPlugins']['ProxyConnectManual'] = 'ProxyConnectManualPlugin'; $Configuration['EnabledPlugins']['cleditor'] = 'cleditor'; $Configuration['EnabledPlugins']['TinyMCE'] = 'TinyMCE'; $Configuration['EnabledPlugins']['Categories2Menu'] = TRUE; $Configuration['EnabledPlugins']['SplitMerge'] = TRUE; $Configuration['EnabledPlugins']['Voting'] = TRUE; $Configuration['EnabledPlugins']['FeedDiscussions'] = TRUE; $Configuration['EnabledPlugins']['StopForumSpam'] = TRUE; $Configuration['EnabledPlugins']['Minify'] = TRUE; $Configuration['EnabledPlugins']['Cleanser'] = TRUE; $Configuration['EnabledPlugins']['RegistrationRestrictLogger'] = TRUE; [root@hetzner2 htdocs]#

I found no obvious way to make the forums read-only from the application-side
1. except an old extension that hasn't been updated since 2005 https://open.vanillaforums.com/addon/readonly-plugin
2. of course, we could achieve read-only by:
  1. revoking all access privlidges to the db user = ${dbUser_hetzner2} = 'oseforums_user' && granting them SELECT only access
  2. And making all of the files/directories read-only

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
 dbUser_hetzner2="oseforum_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

 time nice mysql -uroot -p${mysqlPass} -sNe "REVOKE ALL ON ${dbName_hetzner2}.* FROM '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"

# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0550 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0440 {} \;
chown apache:apache-admins "${docrootDir_hetzner2}/conf/config.php"
chmod 0440 "${docrootDir_hetzner2}/conf/config.php"

ran the above commands & confirmed that the permissions changed to SELECT-only in the db

[root@hetzner2 htdocs]#  nice mysql -uroot -p${mysqlPass} mysql -sNe "select * from db where Db = 'oseforum_db';"
localhost       oseforum_db     oseforum_user   Y       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N
[root@hetzner2 htdocs]#

attempted to login, but then I found that the fucking forums are leaking back to the user our server-side error message (in json), that includes the username & hostname of the db server in the response to our login query! https://stagingforum.opensourceecology.org/entry/signin

{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}

And then it created a popup window leaking all this info!

FATAL ERROR IN: Gdn_Database.Query();
"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'" update GDN_User User set DateLastActive = :DateLastActive, DateUpdated = :DateUpdated, UpdateIPAddress = :UpdateIPAddress where UserID = :UserID LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php > 276: > 277: if (!is_object($PDOStatement)) { > 278: trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR); > 279: } else if ($PDOStatement->execute($InputParameters) === FALSE) { >>> 280: trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR); > 281: } > 282: } else { > 283: $PDOStatement = $this->Connection()->query($Sql); > 284: } BACKTRACE: [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start(); [/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator(); [/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();

found the source of the logs being sent to the user as the line:

$Configuration['Garden']['Errors']['MasterView'] = 'deverror.master.php';

fixed the log leaking by commenting-out the above line & replacing it with:

$Configuration['Garden']['Errors']['LogEnabled']                = TRUE;                                                                                                     
$Configuration['Garden']['Errors']['LogFile']                   = '';

the above setting forced writes to the apache-defined error logfile at /var/log/httpd/forum.opensourceecology.org/error_log

[root@hetzner2 httpd]# date
Thu Jan 25 15:41:25 UTC 2018
[root@hetzner2 httpd]# pwd
/var/log/httpd
[root@hetzner2 httpd]# tail -f forum.opensourceecology.org/access_log forum.opensourceecology.org/error_log 
...
==> forum.opensourceecology.org/error_log <==
[Thu Jan 25 15:39:48.700738 2018] [:error] [pid 24269] [client 127.0.0.1:56072] [Garden] /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php, 280, Gdn_Database.Query(), UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User', update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID

ugh, but the response is actually still leaking info

<h1>FATAL ERROR IN: Gdn_Database.Query();</h1>
<div class="AjaxError">"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'"
update GDN_User User set 
 DateLastActive = :DateLastActive,
 DateUpdated = :DateUpdated,
 UpdateIPAddress = :UpdateIPAddress
where UserID = :UserID
LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php
> 276: 
> 277:          if (!is_object($PDOStatement)) {
> 278:             trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR);
> 279:          } else if ($PDOStatement->execute($InputParameters) === FALSE) {
>>> 280:             trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR);
> 281:          }
> 282:       } else {
> 283:          $PDOStatement = $this->Connection()->query($Sql);
> 284:       }
BACKTRACE:
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start();
[/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator();
[/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();
</div>

1. unfortunately, after a single login attempt, all future queries of any kind produce the same "Bonk" generic error message now. It appears that the fix is to clear the cookie named "Vanilla" for the domain on the client-side. By default, this cookie sticks-around for 1 month.

# I changed the LogEnabled flag to FALSE. That helped, but then the logs disappeared from the apache logfile & the response still contained the error (though a shorter one)!
<pre>
{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}

I'm pretty fucking over Vanilla. They seem to not give a shit about security.
1. Note: the leaks above are nothing more than we've openly disclosed in our specs & docs. However, error messages often contain very sensitive info. Sometimes they contain passwords. Hence, the fact that this is being leaked in general is a huge red flag. The fact that it's considered "normal" behaviour is terrifying. The fact that I googled around for a fix of this issue, and the only thing I found was a developer suggesting to use a browser-side debugger to find the server-side error messages is mortifying. Sending server-side errors to the client should be a opt-in feature. It should not be the default. But Vanilla seems to have it as the baked-in default with not only no way to disable it, but also no obvious way (if at all?) to disable it. This tells me that they're either incompetent or just don't care. That's not what we want from the developers of our tools.## phpbb looks a bit better, they have at least a few blog posts tagged "security" https://blog.phpbb.com/tag/security/
2. phpbb has a wiki with a guide recommending to move config.php out of the docroot. Not one of the guides I found for Vanilla mentioned that obvious requirement. They were occupied with permissions, and arguing if taking away the execute bit mattered to php files (read: it doesn't). https://wiki.phpbb.com/Hardening_Tips
began downloading a static-content snapshot of our forums with wget

[root@hetzner2 wget]# date
Thu Jan 25 16:49:39 UTC 2018
[root@hetzner2 wget]# pwd
/var/tmp/deleteMeIn2019/oseoforum.20180125/wget
[root@hetzner2 wget]# wget -r -p -e robots=off http://forum.opensourceecology.org

1. I'm concerned that the output above may be insanely long, since people can permalink to comments within discussions. So effectively we'll store a single thread N times where N is the number of comments in the thread :(
sent an email to Marcin asking if he was ready to fully archive our fourms to static content

I discovered that my defined hardened permissions for Wordpress should be improved such that
1. All of the files & directories that don't need rw permissions should not have write permissions. That's every file in a wordpress docroot except the folder "wp-content/uploads" and its subfiles/dirs.
2. World permissions (not-user && not-group) for all files & directories inside the docroot (and including the docroot dir itself!) should be set to 0 for all files & all directories.
3. Excluding 'wp-content/uploads/', these files should also not be owned by the user that runs a webserver (in cent, that's the 'apache' user). For even if the file is set to '0400', but it's owned by the 'apache' user, the 'apache' user can ignore the permissions & write to it anyway.
4. Excluding 'wp-content/uploads/', all directories in the docroot (including the docroot dir itself!) should be owned by a group that contains the user that runs our webserver (in cent, that's the apache user). The permissions for this group must be not include write access for files or directories. For even if a file is set to '0040', but the containing directory is '0060', any user in the group that owns the directory can delete the existing file and replace it with a new file, effectively ignoring the read-only permission set for the file.
5. currently, our documentation reads:

vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"

chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

1. but I believe the (untested) update should be:

vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"

chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0060 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0070 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"

1. ...such that:
  1. the 'not-apache' user is a new user that doesn't run any software (ie: a daemon such as a web server) and whose shell is "/sbin/nologin" and home is "/dev/null".
  2. the apache user is now in the apache-admins group
2. the result will now be that:
a compromised web server can no longer write data to a docroot (ie: adding malicious code to a php script) outside the 'wp-content/uploads/' directory
for anyone to make changes to any files in the docroot (other than 'wp-content/uploads/'), they must be the root user. I think this is fair if they don't have the skills necessary to become root, they probably shouldn't fuck with the wp core files anyway.
1. however, as with before, any user in the 'apache' group can read most files in the docroot. So, if we want an OSE Developer with ssh access to be able to access our server's files read-only, we should add them to the 'apache' group. If we trust them with passwords as well, we should additionally add them to the 'apache-admins' group.

Tue Jan 23, 2018

found the best way to browse the git sourcecode of the mediawiki core at this URL https://phabricator.wikimedia.org/source/mediawiki/tags/master/
1. for example, for v1.30.0 https://phabricator.wikimedia.org/source/mediawiki/browse/REL1_30/
2. or for the 'vendor' directory https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/
using composer to populate the 'vendor' directory does not seem like a practical options with our hardened php config.
I attempted to download the actual tarball, copy-in the 'vendor' directory, update permissions, and then run the update

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

mkdir /var/tmp/mediawiki-1.30.0
pushd /var/tmp/mediawiki-1.30.0

wget https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz
tar -xzvf mediawiki-1.30.0.tar.gz

cd mediawiki-1.30.0
rm -rf /var/www/html/wiki.opensourceecology.org/htdocs/vendor
cp -r vendor /var/www/html/wiki.opensourceecology.org/htdocs/

# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"

/var/www/html/wiki.opensourceecology.org/htdocs/maintenance
php update.php

popd

now I got a fatal php error

[Tue Jan 23 22:02:07.182288 2018] [:error] [pid 31837] [client 127.0.0.1:37506] PHP Fatal error:  Class 'Wikimedia\\WrappedString' not found in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Gadgets/GadgetHooks.php on line 217

1. I changed the require_once() to wfLoadExtension() in LocalSettings.php
2. I found that the directory path for the "WrappedString.php" file is distinct for the vendors dirs in [a] the git repo vs [b] the extracted tarball release.

# the tarball has an src dir with the 2 files (WrappedString.php & WrappedStringList.php)
[root@hetzner2 htdocs]# ls -lah /var/tmp/mediawiki-1.30.0/mediawiki-1.30.0/vendor/wikimedia/wrappedstring/src/
total 16K
drwxr-xr-x 2 501 games 4.0K Dec  8 23:20 .
drwxr-xr-x 3 501 games 4.0K Dec  8 23:20 ..
-rw-r--r-- 1 501 games 3.5K Dec  8 23:20 WrappedStringList.php
-rw-r--r-- 1 501 games 3.3K Dec  8 23:20 WrappedString.php

# but the git repo has an src dir with 2 dirs = WrappedString & Wikimedia. Both of *these* dirs have the 2 files https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/wikimedia/wrappedstring/src/
[root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/
total 16K
drwxr-xr-x 4 root root 4.0K Jan 22 21:22 .
drwxr-xr-x 3 root root 4.0K Jan 22 21:22 ..
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 Wikimedia
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 WrappedString
[root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/WrappedString/
total 16K
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 .
drwxr-xr-x 4 root root 4.0K Jan 22 21:22 ..
-rw-r--r-- 1 root root   98 Jan 22 21:22 WrappedStringList.php
-rw-r--r-- 1 root root   90 Jan 22 21:22 WrappedString.php
[root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/Wikimedia/
total 16K
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 .
drwxr-xr-x 4 root root 4.0K Jan 22 21:22 ..
-rw-r--r-- 1 root root 3.6K Jan 22 21:22 WrappedStringList.php
-rw-r--r-- 1 root root 3.4K Jan 22 21:22 WrappedString.php
[root@hetzner2 htdocs]#

Mon Jan 22, 2018

Marcin found that editing posts on the stagingforum failed with "You don't have permission to access /vanilla/post/editdiscussion/X on this server."
1. I coorelated this to a 403 error caused by modsecurity

--e381b03e-A--
[22/Jan/2018:14:02:11 +0000] WmXu49lQF2aUKXkcaav20AAAAAU 127.0.0.1 52050 127.0.0.1 8000
--e381b03e-B--
POST /vanilla/post/editdiscussion/237 HTTP/1.0
X-Real-IP: 173.234.159.236
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: stagingforum.opensourceecology.org
Content-Length: 3125
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: https://stagingforum.opensourceecology.org/vanilla/post/editdiscussion/237
Cookie: Vanilla=20082-1519221520%7C48d7a44f72d9d86b5e70a0da0e4b83c8%7C1516629520%7C20082%7C1519221520; Vanilla-Volatile=20082-1516802320%7C2271357054a693999ba33b7013e7c80e%7C1516629520%7C20082%7C1516802320
DNT: 1
X-Forwarded-For: 173.234.159.236, 127.0.0.1
X-Varnish: 34288

--e381b03e-C--
Discussion%2FTransientKey=82P0S34XB3Q0&Discussion%2Fhpt=&Discussion%2FDiscussionID=237&Discussion%2FDraftID=0&Discussion%2FName=Roof&Discussion%2FCategoryID=1&Discussion%2FBody=I+think+we+need+to+address+what+seems+to+be+a+hole+in+the+GVCS+-+constructing+a+good+roof.+Shelter+is+one+of+the+big+requirements+we+have+to+cover%2C+and+living+in+the+pacific+northwest%2C+I+am+painfully+aware+of+the+importance+and+costs+associated+with+a+roof.+Compressed+earth+blocks+cover%2C+to+a+decent+extent%2C+building+walls.+Floors+are+somewhat+covered+by+the+dimensional+saw+mill%2C+compressed+earth+blocks%2C+and+concrete.+However%2C+what+isn't+taken+into+account+yet+is+the+most+expensive+and+difficult+part+of+a+house+-+the+roof.%3Cbr%3E%3Cbr%3EI+think+figuring+out+a+good+roofing+solution+should+be+one+of+the+primary+focal+points+for+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2Fcomment%2F861%23Comment_861%22%3Efirst+wave%3C%2Fa%3E%3C%2Fu%3E+-+getting+the+basics+of+housing%2Ffood%2Fwater%2Fpower+up+and+running%2C+as+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F227%2Fbootstrapping-the-gvcs-priorities%22%3EGVCS+bootstrapping%3C%2Fa%3E%3C%2Fu%3E+thread.%3Cbr%3E%3Cbr%3EA+good+example+of+this+need+is+the+Factor+E+Farm+workshop+itself.+To+make+a+workshop+to+stage+the+development+of+the+GVCS%2C+they+were+able+to+make+part+of+the+structure+out+of+compressed+earth+block+columns.+However%2C+the+vast+majority+of+the+cost+of+the+structure+was+in+building+the+roof.+They+had+to+use+off-the-shelf+metal+roofing%2C+which+was+paid+for+through+donations%3A+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconstruction-time%2F%22%3Edonations+for+roofing+materials%3C%2Fa%3E%3C%2Fu%3E%2C+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%22%3Ebuilding+the+roof%3C%2Fa%3E%3C%2Fu%3E.%3Cbr%3E%3Cbr%3EAs+you+can+see%2C+building+a+roof+was+one+of+the+first+roadblocks%2C+and+was+solved+through+just+buying+one.%3Cbr%3E%26nbsp%3B%3Cbr%3EPerhaps+we+can+brainstorm+and+come+up+with+some+good+solutions.+I've+already+started+some+sections+and+given+some+background+requirements+on+the+wiki.+One+option+I+find+very+intriguing+myself+is+ferrocement.+You+can+see+it+being+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%23comments%22%3Ecomments%3C%2Fa%3E%3C%2Fu%3E+of+the+building+the+roof+blog+entry+for+the+farm.+I+made+a+thread+about+the+need+for+an+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F214%22%3Eopen+source+metal+lath+making+machine%3C%2Fa%3E%3C%2Fu%3E+with+roofing+as+one+of+its+applications.%3Cbr%3E%3Cbr%3EI've+started+a+wiki+entry+for+the+roof+here%3A+%3Cu%3Ehttp%3A%2F%2Fopenfarmtech.org%2Fwiki%2FRoof%3C%2Fu%3E+As+we+come+up+with+ideas+we+can+add+them.%3Cbr%3E%3Cbr%3EEdit%3A+This+is+Michael+attempting+to+edit+a+roof+post.%3Cbr%3E&Checkboxes%5B%5D=Announce&Checkboxes%5B%5D=Closed&Discussion%2FTags=&DeliveryType=VIEW&DeliveryMethod=JSON&Discussion/Save=Save
--e381b03e-F--
HTTP/1.1 403 Forbidden
Content-Length: 233
Connection: close
Content-Type: text/html; charset=iso-8859-1

--e381b03e-E--

--e381b03e-H--
Message: Access denied with code 403 (phase 2). Pattern match "\\W{4,}" at ARGS:Discussion/Body. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "37"] [id "960024"] [rev "2"] [msg "Meta-Character Anomaly Detection Alert - Repetative Non-Word Characters"] [data "Matched Data: > -  found within ARGS:Discussion/Body: I think we need to address what seems to be a hole in the GVCS - constructing a good roof. Shelter is one of the big requirements we have to cover, and living in the pacific northwest, I am painfully aware of the importance and costs associated with a roof. Compressed earth blocks cover, to a decent extent, building walls. Floors are somewhat covered by the dimensional saw mill, compressed earth blocks, and concrete. However, what isn't taken into acc..."] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"]
Action: Intercepted (phase 2)
Stopwatch: 1516629731152710 755 (- - -)
Stopwatch2: 1516629731152710 755; combined=355, p1=104, p2=236, p3=0, p4=0, p5=14, sr=20, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"

--e381b03e-Z--

1. realized that most of the whitelisting was done to just the 'wp-admin' dir, so I made it site-wide on the forums
Catarina reported successfully that she was able to upload 3M images to fef && also connect via sftp using filezilla.
1. marked fef migration as complete CHG-2018-01-03
moved LocalSettings.php out of the document root per these hardening guides
1. https://www.mediawiki.org/wiki/Manual:Security
2. https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening

[root@hetzner2 htdocs]# date
Mon Jan 22 20:26:16 UTC 2018
[root@hetzner2 htdocs]# pwd
/var/www/html/wiki.opensourceecology.org/htdocs
[root@hetzner2 htdocs]# cat LocalSettings.php
<?php
# including separate file that contains the database password so that it is not stored within the document root.
# For more info see:
#  * https://www.mediawiki.org/wiki/Manual:Security
#  * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening

$docRoot = dirname( FILE );
require_once "$docRoot/../LocalSettings.php";
?>
[root@hetzner2 htdocs]#

1. this also required me to change the original LocalSettings.php file (now outside the docroot in /var/www/html/wiki.opensourceecology.org) so that IP is inside of 'htdocs'

# If you customize your file layout, set $IP to the directory that contains                                                                                     
# the other MediaWiki files. It will be used as a base to locate files.                                                                                         
if( defined( 'MW_INSTALL_PATH' ) ) {                                                                                                                            
   $IP = MW_INSTALL_PATH;                                                                                                                                       
} else {                                                                                                                                                        
   $IP = dirname( FILE ) . "/htdocs" ;                                                                                                                      
}

decided to move the logo to 'images/' instead of just the base docroot to simplify future updates
installed composer per the upgrade guide https://www.mediawiki.org/wiki/Manual:Upgrading
1. which linked to https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries
2. else, I got this error

MediaWiki 1.30 internal error

Installing some external dependencies (e.g. via composer) is required.
External dependencies

MediaWiki now also has some external dependencies that need to be installed via composer or from a separate git repo. Please see mediawiki.org for help on installing the required components.

1. and the actual install is

yum install composer

1. and I had to add /usr/share/php/ to open_basedir in /etc/php.ini
2. the composer steps themselves are included in the code block below
found that several of our MediaWiki extensions are no longer maintained
1. TreeAndMenu https://www.mediawiki.org/wiki/Extension:TreeAndMenu
2. RSSReader https://www.mediawiki.org/wiki/Extension:RSS_Reader
3. google-coop https://www.mediawiki.org/wiki/Extension_talk:Google_Custom_Search_Engine
4. Mtag https://www.mediawiki.org/wiki/MediaWiki_and_LaTeX_on_a_host_with_shell_access
5. PayPal https://www.mediawiki.org/wiki/Extension:Paypal
6. Flattr https://www.mediawiki.org/wiki/Extension:Flattr
7. JSWikiGnatt https://www.mediawiki.org/wiki/Extension:JSWikiGantt
8. ProxyConnect https://www.mediawiki.org/wiki/Extension:ProxyConnect
9. FreeMind https://www.mediawiki.org/wiki/Extension:FreeMind
commented-out the entire function fnAddPersonalUrls() a section from LocalSettings.php that caused the following error

[Mon Jan 22 22:25:45.130218 2018] [:error] [pid 26913] [client 127.0.0.1:50368] PHP Fatal error:  Call to undefined method User::getSkin() in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 421

upgraded mediawiki install, including a switch to a git-backed core download to make future updates easier
removed googleAnalytics require from LocalSettings.php, because privacy matters && awstats works.

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# clone the latest stable version of the Mediawiki core code from git
pushd ${backupDir_hetzner2}/current
time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git
pushd core
latestStableMediawikiVersion=`git tag -l | sort -V | grep -E '^[0-9\.]*$' | tail -n1`
git checkout "${latestStableMediawikiVersion}"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git

# extensions
pushd extensions
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git

git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd

#cp -r ../../../old/htdocs/extensions/TreeAndMenu .
#cp -r ../../../old/htdocs/extensions/RSSReader .
#cp -r ../../../old/htdocs/extensions/google-coop.php
#cp -r ../../../old/htdocs/extensions/Mtag.php
#cp -r ../../../old/htdocs/extensions/PayPal.php
#cp -r ../../../old/htdocs/extensions/Flattr
#cp -r ../../../old/htdocs/extensions/JSWikiGantt
#cp -r ../../../old/htdocs/extensions/ProxyConnect
popd

# skins
pushd skins
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd

popd

# copy core into the docroot
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
mkdir "${docrootDir_hetzner2}"
time nice rsync -av --progress core/ "${docrootDir_hetzner2}/"

# copy back essential content
rsync -av --progress "../old/LocalSettings.php" "${vhostDir_hetzner2}/"
rsync -av --progress "../old/htdocs/LocalSettings.php" "${docrootDir_hetzner2}/"
rsync -av --progress "../old/htdocs/images" "${docrootDir_hetzner2}/"

# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"

having some 404 issues with images

<td><a href="http://www.shuttleworthfoundation.org/"><img alt="Shuttleworth funded-02---web.jpg" src="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg" width="200" height="55" srcset="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/300px-Shuttleworth_funded-02---web.jpg 1.5x, /wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/400px-Shuttleworth_funded-02---web.jpg 2x" /></a>

1. the above links to (400) https://wiki.opensourceecology.org/wiki/Images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
2. but it should link to (200) https://wiki.opensourceecology.org/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
I logged into the hetzner1 konsleH config & tried to pull up the apache config there (since we can't see it in ssh on this annoying shared server). In the wui, click 'opensourceecology.org' -> Services -> Server Configuration -> Click the yellow wrench next to public_html -> Enhanced View:

Options All -Indexes

# Site-wide redirects
#Redirect 301 /community http://community.opensourceecology.org
#Redirect 301 /forum http://forum.opensourceecology.org
#Redirect 301 /weblog http://blog.opensourceecology.org
#RedirectMatch 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us
# Redirect 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us

# http://openfarmtech.org/w/index.php?title=CiviCRM_tech_notes&action=edit&redlink=1
# RedirectMatch 301 /CiviCRM http://openfarmtech.org/civicrm

RewriteEngine On

RewriteCond %{HTTP_HOST} ^www\.
RewriteRule ^(.*)$ http://opensourceecology.org/$1 [R=301,L,QSA]

#RewriteRule ^civicrm/(.*)$ /community/civicrm/$1 [PT,L,QSA]

RewriteRule ^wiki/(.*)$ /w/index.php?title=$1 [PT,L,QSA]
#RewriteRule ^wiki/(.*:.*)$  wiki/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/(.*:.*)$  /w/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/*$ /w/index.php [L,QSA,T=text/html]
RewriteRule ^index.php/(.*)$ /wiki/$1?old-url=slash [R=permanent,L,QSA]
# RewriteRule ^/*$ /w/index.php [L,QSA]

# RewriteLog "/home/marcin_ose/openfarmtech.org/rewrite.log" 
# RewriteLogLevel 3

# See http://docs.jboss.org/jbossweb/3.0.x/proxy-howto.html
# ProxyPass /OpenKM  http://localhost:8080/OpenKM
# ProxyPassReverse /OpenKM  http://localhost:8080/OpenKM

# RewriteRule ^OpenKM/(.*)$ http://localhost:8080/OpenKM/$1 [P]



# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

1. actually, just setting `$wgScriptPath = ""` in LocalSettings.php fixed everything!!
attempting to login produces an error message

Database error
A database query error has occurred. This may indicate a bug in the software.[WmZ1UeWiTZqK5j1xUV@YaQAAAAo] 2018-01-22 23:35:46: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"

1. err, I found my password leaked in cleartext to /var/log/httpd/modsec_audit.log
  1. added "wpPassword" to sanitiseArg list in /etc/httpd/modexecurity.d/do_not_log_passwords.conf

[root@hetzner2 modsecurity.d]# date
Mon Jan 22 23:46:06 UTC 2018
[root@hetzner2 modsecurity.d]# cd /etc/httpd
[root@hetzner2 httpd]# cat modsecurity.d/do_not_log_passwords.conf 
################################################################################
# File:    modsecurity.d/do_not_log_passwords.conf
# Version: 0.2
# Purpose: Defines a list of POST arguments that contain passwords and instructs
#          modsecurity to sanitise-out the values of these variables (with **s)
#          when logging to files (ie: /var/log/httpd/modsec_audit.log)
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-12-16
# Updated: 2018-01-22
################################################################################
SecAction "nolog,phase:2,id:131,sanitiseArg:password,sanitiseArg:Password,sanitiseArg:wpPassword,sanitiseArg:newPassword,sanitiseArg:oldPassword,sanitiseArg:pwd"
[root@hetzner2 httpd]#

it's possible that the login issue is due to a pending update.php execution required, but I'm getting an error when I attempt to run it

[root@hetzner2 maintenance]# php update.php 
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Warning:  putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53
PHP Warning:  is_dir(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/forum.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fsfile/TempFSFile.php on line 90
PHP Notice:  Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127
PHP Notice:  Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
MediaWiki 1.30.0 Updater

oojs/oojs-ui: 0.25.1 installed, 0.23.0 required.
wikimedia/ip-set: 1.2.0 installed, 1.1.0 required.
wikimedia/relpath: 2.1.1 installed, 2.0.0 required.
wikimedia/remex-html: 1.0.2 installed, 1.0.1 required.
wikimedia/running-stat: 1.2.1 installed, 1.1.0 required.
wikimedia/wrappedstring: 2.3.0 installed, 2.2.0 required.
Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies
[root@hetzner2 maintenance]#

1. I can't run composer, since it requires some unsafe, disabled php functions. That's why I chose to grab them via git (`git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git`). But apparently it gave me old versions?
2. re-reading the error message, it says I have too new of versions!
3. looks like the branches aren't maintained; we want REL1_30 but it isn't there

[root@hetzner2 vendor]# date
Tue Jan 23 00:01:57 UTC 2018
[root@hetzner2 vendor]# pwd
/var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor
[root@hetzner2 vendor]# git branch -r
  origin/HEAD -> origin/master
  origin/fundraising/REL1_27
  origin/master
  origin/wmf/1.31.0-wmf.15
  origin/wmf/1.31.0-wmf.16
  origin/wmf/1.31.0-wmf.17
[root@hetzner2 vendor]#

Sat Jan 20, 2018

the db backup & file backup of the wiki finished in a little over an hour

...
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$ time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
...
real    71m2.031s
user    17m36.700s
sys     1m38.868s
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$

I initiated an scp of the data to hetnzer2

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

transfer of hetzner1 backups to hetzner2 finished after 8 minutes
added wiki.opensourceecology.org A record in cloudflare DNS to 138.201.84.243
disabled CDN/DNS proxy in cloudflare for the 'network' subdomain
decreased TTL of all DNS entries to lowest time possible = 2 minutes for all entries except 'www', 'opensourceecology.org', and 'blog' (all 3 use the CDN for now, and are thus required to be set to TTL = 'automatic')
created/updated necessary config files for forum.opensourceecology.org
1. /etc/httpd/conf.d/00-wiki.opensourceecology.org.conf
2. /etc/varnish/sites-enabled/wiki.opensourceecology.org
3. /etc/varnish/all-vhosts.vcl
4. /etc/nginx/conf.d/wiki.opensourceecology.org.conf
created necessary dirs
1. /var /www/html/wiki.opensourceecology.org/htdocs
2. /var/log/httpd/wiki.opensourceecology.org
3. /var/log/nginx/wiki.opensourceecology.org
updated /etc/php.ini to include "/var/www/html/forum.opensourceecology.org/" in /etc/php.in's "open_basedir"
added wiki SAN to opensourceecology.org cert

certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d stagingforum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org
/bin/chmod 0400 /etc/letsencrypt/archive/*/pri*
nginx -t && service nginx reload

created osewiki_db mysql database on hetzner2 from dump from hetzner1
copied htdocs files on hetzner2 from backup on hetzner1
updated LocalSettings.php with new db name & credentials
updated permissions of docroot files
commented-out $wgScriptPath & $wgStylePath to remove the "/w/" subdir in LocalSettings.php
removed the "/w/" subdir prefix from $wgLogo in LocalSettings.php
commented-out $wgServer & $wgFavicon to prevent a 301 redirect back to the naked domain opensourceecology.org
commented-out $wgCacheDirectory & $wgUseFileCache. And `rm -rf cache`. This directory is *supposed* to be located outside the docroot for security reasons. But we won't need this feature as we use varnish https://www.mediawiki.org/wiki/Manual:Security#File_permissions
added "Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php" to /etc/httpd/conf.d/00-wiki.opensourceecology.org
commented-out debugging lines in LocalSettings.php

################################################################                                                                                            
# Debugging                                                                                                                                                 
# error_reporting(E_ALL | E_STRICT);                                                                                                                        
# error_reporting(E_ALL);                                                                                                                                   
# ini_set("display_errors", 1);                                                                                                                             
																																							
# $wgShowExceptionDetails = true; ## Verbose output to user                                                                                                 
																																							
#$wgShowSQLErrors = true;                                                                                                                                   
#$wgDebugDumpSql  = true;                                                                                                                                   
																																							
#$wgDebugLogFile = "/usr/home/osemain/public_html/logs/wiki-error.log";                                                                                     
#$wgDebugLogFile = "/home/oft_site/logs/wiki-error.log";                                                                                                    
# $wgDebugRawPage = true;                                                                                                                                   
# $wgDebugComments = true;                                                                                                                                  
################################################################

saw some errors for generating temporary thumbnails

Error creating thumbnail: Unable to save thumbnail to destination

1. attempted setting the image directory to be writeable, to no avail

find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;

began investigating the guide to install Mediawiki via git https://www.mediawiki.org/wiki/Download_from_Git

time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git

pushd ${backupDir_hetzner2}/current

determined that the latest version of Mediawiki is v1.30.0, and that we're currently running v1.24.2 https://wiki.opensourceecology.org/wiki/Special:Version

Thr Jan 18, 2018

fixed awstats cron job & config files
confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
1. did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site

* https://core.trac.wordpress.org/ticket/17048

there's no good, robust solution.

* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress
* https://wordpress.org/plugins/relative-url/
* https://wordpress.org/plugins/root-relative-urls/
* https://deluxeblogtips.com/relative-urls/
* http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/

take away is:
1. Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
2. However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.
attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
1. using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf

[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf
		# allow large posts for image uploads
		#client_max_body_size 1k;
		#client_max_body_size 900k;
		client_max_body_size 10M;

1. next, I got a 403 error from /wp-admin/async-upload.php
  1. /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:

==> /var/log/httpd/fef.opensourceecology.org/error_log <==
[Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"]

1. 1. as above, whitelisted rule IDs:
    1. 960915, multipart_unmatched_boundary
    2. 200003, multipart_unmatched_boundary
moved '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org' to '/usr/home/osemain/noBackup/deleteMeIn2019/oswh_olddocroot'
added a 301 redirect from 'http://opensourcewarehouse.org' to 'https://oswh.opensourceecology.org' in new file = '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org/index.php'

Sat Jan 13, 2018

Meeting with Marcin

Fri Jan 12, 2018

finished configuring oswh wp plugins
determined that the oswh hack was responsible for injecting pop-ups for a "windows defender" phishing site on some subset of page loads
gained access to oseforums, oseblog, osecivi users via ssh on our non-dedicated server (hetzner1)
checked /etc/passwd & found another 8x org-specific users with home directories that I couldn't access still
emailed hetzner for advice on how to gain ssh access to these users' home directories

addon:x:1011:1011:addontest.opensourceecology.org:/usr/home/addon:bin/false
oseirc:x:1018:1018:irc.opensourceecology.org:/usr/home/oseirc:bin/false
oseholla:x:1019:1019:holla.opensourceecology.org:/usr/home/oseholla:bin/false
osesurv:x:1020:1020:survey.opensourceecology.org:/usr/home/osesurv:/bin/bash
sandbox:x:1021:1021:sandbox.opensourceecology.org:/usr/home/sandbox:/bin/false
microft:x:1022:1022:test.opensourceecology.org:/usr/home/microft:/bin/bash
zabbix:x:118:121::/var/run/zabbix:/bin/false
openswh:x:1023:1023:opensourcewarehouse.org:/usr/home/openswh:/bin/false

created backup of oseforum's docroot (58M) & db (44M). Both sizes are bzip2'd.

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
 dbUser_hetzner2="oseforum_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

Mon Jan 08, 2018

tried installing the fresh version of the Eventor theme to the website running the old wp core, but it was still broken
reverted to the docroot I backed-up before attempting to try the outdated wp core && installed _that_ Eventor theme to the fresh version 1.7 that was just downloaded, and the site actually worked!
finally, attempted the wp-cli commands to update themes & plugins && install our minimal set of sec plugins. The site was still functional after

Fri Jan 05, 2018

investigation of minor fef issues

Thr Jan 04, 2018

downloaded the Eventor theme v 1.7, thanks to Simone's contact with Themes Kingdom
Hetzner responded saying we can use WebFTP to uplaod to $HOME by clicking "the server at the top"
Marcin responded with some issues with osemain's ephemeral clone
Catarina found some linking issues in fef
1. I brought the site down & did a string replacement for all occurrences of 'http://opensourceecology.org/fef' to '/', brought the site back up, and asked Catarina to check again
2. updated documentation at Wordpress#replace_strings_everywhere_in_wp_database_backend

Wed Jan 03, 2018

migrated fef to hetzner2 CHG-2018-01-03
updated statuscake for obi to hit 'https://www.openbuildinginstitute.org'
updated statuscake for fef to hit 'https://fef.opensourceecology.org'
ensured that ssh was activated for all domains/users on our (apparently dedicated, per hetzner support) hetzner1 server (but without root access) via the konsoleh site -> click on the server -> Account Management -> SSH access -> Select domain (for each) -> Next
the kosoleh wui only allowed editing files in the docroot, not the user's home-dir, which prevented me from actually adding my ssh pubic key to $HOME/.ssh/authorized_keys file
I emailed hetzner support back asking if [a] they could just add my pub key to all our user account's authorized_keys files or [b] tell me how I could reset all the user's passwords
oswh was cannibalized by a virus & is awaiting a fresh version of the theme. the forums is awaiting access to the user account. I'm now going to work on beginning the migration of osemain
1. it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
  1. archive
  2. w
  3. logs
  4. mediawiki-1.24.2.extra
2. the entire dir is 23G. Excluding the above, it's ~ 0.7G

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
 dbUser_hetzner1="${mysqlUser_osemain}"
 dbPass_hetzner1="${mysqlPass_osemain}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}

1. the gz-compressed tarball generated from above was 353M.

# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180103"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
 dbUser_hetzner2="osemain_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

created domain name 'osemain.opensourceecology.org' for testing the osemain site on hetzner2
using above vars, I followed the guide to migrate the files & db data from hetzner1 to hetzner2 Wordpress#migrate_site_from_hetzner1_to_hetzner2
created necessary files & dirs:
1. /etc/httpd/conf.d/00-www.opensourceecology.org.conf
2. /etc/varnish/sites-enabled/www.opensourceecology.org
3. /etc/nginx/conf.d/www.opensourceecology.org.conf
  1. this file has a temporary override for the 'Host' header passed to varnish, since the staging url is going to be 'osemain.opensourceecology.org' but the prod site will be 'opensourceecology.org'
4. /var/log/httpd/www.opensourceecology.org
5. /var/log/nginx/www.opensourceecology.org
updated necessary files
1. /etc/varnish/all-vhosts.vcl
2. /etc/php.ini
finished setting up ephemeral clone of osemain at https://osemain.opensourceecology.org
1. sent email to Marcin & Catarina for validation

Tue Jan 02, 2018

got an email from Simone Cicero stating that she emailed Themes Kingdom for a clean copy of Eventor 1.7
emailed back-and-forth with hetzner
1. learned that the forums are in /usr/www/users/oseforum/
2. learned that we have a bunch of users on this box, and it might even be dedicated just for us (though without root access)

osemain@dedi978:~$ grep 'ose' /etc/group
users:x:100:osemain,addon,osecivi,oseblog,oseforum,oseirc,oseholla,osesurv,sandbox,microft,openswh
osemain:x:1010:
osecivi:x:1014:
oseblog:x:1015:
oseforum:x:1016:
oseirc:x:1018:
oseholla:x:1019:
osesurv:x:1020:

1. but I couldn't actually access the home dirs of the other users through 'osemain'

osemain@dedi978:~$ date
Tue Jan  2 16:21:13 CET 2018
osemain@dedi978:~$ ls -lah /usr/home/
ls: cannot open directory /usr/home/: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/addon
ls: cannot open directory /usr/home/addon: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osecivi
ls: cannot open directory /usr/home/osecivi: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseblog
ls: cannot open directory /usr/home/oseblog: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseirc
ls: cannot open directory /usr/home/oseirc: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseforum
ls: cannot open directory /usr/home/oseforum: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osesurv
ls: cannot open directory /usr/home/osesurv: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/openswh
ls: cannot open directory /usr/home/openswh: Permission denied

1. so I asked hetzner support to add the 'osemain' user to all the other users groups listed above, and I asked them to find any other accounts that we own that I may have missed

Maltfield Log/2018 Q1: Difference between revisions

Revision as of 20:08, 12 February 2018

Contents

See Also

Thr Feb 09, 2018

Thr Feb 08, 2018

Wed Feb 07, 2018

Tue Feb 06, 2018

Mon Feb 05, 2018

Sun Feb 04, 2018

Thr Feb 01, 2018

Wed Jan 31, 2018

Thr Jan 25, 2018

Tue Jan 23, 2018

Mon Jan 22, 2018

Sat Jan 20, 2018

Thr Jan 18, 2018

Sat Jan 13, 2018

Fri Jan 12, 2018

Mon Jan 08, 2018

Fri Jan 05, 2018

Thr Jan 04, 2018

Wed Jan 03, 2018

Tue Jan 02, 2018

Navigation menu