Maltfield Log/2018 Q1: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 5: | Line 5: | ||
# [[User:Maltfield]] | # [[User:Maltfield]] | ||
# [[Special:Contributions/Maltfield]] | # [[Special:Contributions/Maltfield]] | ||
=Thr Apr 26, 2018= | |||
# Meeting with Marcin | |||
## Wiki validation so far | |||
### 2 outstanding issues | |||
#### We have a superfluous link at the top-right = Create Account and Request Account we should delete the "Create Account" link | |||
#### Marcin can't login because his old password was <20 characters, and the new wiki has a minimum password requirement for Administrators to have >=20 character passwords | |||
#### | |||
## Test Plan | |||
### marcin will make a first-draft of a goolgle doc with an exhaustive list of things to test on the wiki (50-100-ish items) & send it to me by next week. it should include | |||
#### Both simple & complex daily or routine wiki tasks | |||
#### OSE-specific workflow tasks on the wiki that have broken in the past per your experiences that pre-date me | |||
#### other wiki functions that we discovered were broken in the past few months when validating the staging wiki | |||
### we may migrate the wiki together in-person when I visit FeF for a few days in Mid-May | |||
# Jitsi | |||
## I had some questions about this (see below), but we didn't get to it (we focused mostly on the wiki & backups) | |||
### what are the reqs for September? Same as before? | |||
### thoughts on hipchat? | |||
### maybe rockchat. | |||
### or maybe jangouts powered by janus (C > Node) | |||
# I reset Marcin's password on the wiki to be >=20 characters using 'htdocs/maintenance/changePassword.php' | |||
</pre> | |||
[root@hetzner2 maintenance]# php changePassword.php --user=Marcin --password='fake567890example890' | |||
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 | |||
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674 | |||
PHP Notice: Undefined index: HTTP_USER_AGENT in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5 | |||
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 | |||
PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 | |||
PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 | |||
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693 | |||
Password set for Marcin | |||
[root@hetzner2 maintenance]# | |||
</pre> | |||
=Wed Apr 25, 2018= | |||
# atlassian got back to me giving me a free Hipchat Community License | |||
# it's not well advertised, but I managed to find that the limit (as of 2018-01) the limit of participants in a single Stride call are 25 participants https://community.atlassian.com/t5/Stride-questions/How-many-participants-can-join-a-stride-video-call/qaq-p/694918 | |||
# while Hipchat has a limit of 20 participants https://confluence.atlassian.com/hipchat/video-chat-with-your-team-838548935.html | |||
# updated our Videoconferencing article to include Rocket Chat and Hipchat | |||
=Tue Apr 24, 2018= | |||
# I sent an email to Atlassian requesting a community license request for Stride https://www.atlassian.com/software/views/community-license-request | |||
# Atlassian automatically mailed me a service desk ticket for this request https://getsupport.atlassian.com/servicedesk/customer/portal/35/CA-452135 | |||
# I found a great article describing what Jitsi gives us as an SFU https://webrtchacks.com/atlassian-sfu-qa/ | |||
## and this video comparing mesh vs sfu vs mcu https://webrtcglossary.com/mcu/ | |||
## and this great article https://bloggeek.me/how-many-users-webrtc-call[[https://bloggeek.me/how-many-users-webrtc-call/|/]] | |||
## we might actually want to look into open source MCUs instead of an SFU. The downside is more processing on our server, but I think that'd be less of a bottleneck than each participant downloading n streams. | |||
# responded to Marcin's validation follow-up email | |||
## (3) Chrome ERR_BLOCKED_BY_XSS_AUDITOR on Preview | |||
### marcin found another permission denied issue when attempting to view the preview of the FAQ page https://wiki.opensourceecology.org/index.php?title=FAQ&action=submit | |||
#### this was a red herring; it's just another mod_security false-positive that's distinct from the chrome ERR_BLOCKED_BY_XSS_AUDITOR issue | |||
#### I whitelisted 959071 = sqli, and I confirmed that fixed the issue | |||
### Marcin said he's running chromium Version 64.0.3282.167 (Official Build) Built on Ubuntu , running on Ubuntu 16.04 (64-bit) | |||
### My chromium is Version 57.0.2987.98 Built on 8.7, running on Debian 8.10 (64-bit) | |||
### I went ahead and updated my ose VM to debian 9, but chromium in debian 9 is still Version 57.0.2987.98 Built on 8.7, running on Debian 9.4 (64-bit) | |||
### the preview bug was supposed to be fixed in v58, but maybe the bug was just that it was blocking the page *and* the debugging. In any case, I can't see what's causing chrome to issue the ERR_BLOCKED_BY_XSS_AUDITOR | |||
### I spun up a disposable vm & installed the latest version of google chrome, but that failed | |||
<pre> | |||
[user@fedora-23-dvm ~]$ wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm | |||
--2018-04-24 13:38:31-- https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm | |||
Resolving dl.google.com (dl.google.com)... 172.217.23.174, 2a00:1450:4001:81f::200e | |||
Connecting to dl.google.com (dl.google.com)|172.217.23.174|:443... connected. | |||
HTTP request sent, awaiting response... 200 OK | |||
Length: 52364199 (50M) [application/x-rpm] | |||
Saving to: ‘google-chrome-stable_current_x86_64.rpm’ | |||
google-chrome-stable_current_x86 100%[=========================================================>] 49.94M 1.19MB/s in 47s | |||
2018-04-24 13:39:24 (1.06 MB/s) - ‘google-chrome-stable_current_x86_64.rpm’ saved [52364199/52364199] | |||
[user@fedora-23-dvm ~]$ sudo dnf install google-chrome-stable_current_x86_64.rpm | |||
Last metadata expiration check: 0:09:47 ago on Tue Apr 24 13:37:39 2018. | |||
Error: nothing provides libssl3.so(NSS_3.28)(64bit) needed by google-chrome-stable-66.0.3359.117-1.x86_64 | |||
(try to add '--allowerasing' to command line to replace conflicting packages) | |||
[user@fedora-23-dvm ~]$ sudo dnf --allowerasing install google-chrome-stable_current_x86_64.rpm | |||
Last metadata expiration check: 0:10:35 ago on Tue Apr 24 13:37:39 2018. | |||
Error: nothing provides libssl3.so(NSS_3.28)(64bit) needed by google-chrome-stable-66.0.3359.117-1.x86_64 | |||
</pre> | |||
# fedora does have chromium in its repos; I installed them, but this was even more out-of-date @ Version 54.0.2840.90 Fedora Project (64-bit) | |||
### I found & installed the google-chrome depend (openssl-devel), but the same error occured. I found a fix of installing fedora 27 :| https://stackoverflow.com/questions/48839199/install-google-chrome-in-fedora-23 | |||
### giving up on fedora 27, I created a new VM from debian-9 for installing google-chrome https://unix.stackexchange.com/questions/20614/how-do-i-install-the-latest-version-of-chromium-in-debian-squeeze | |||
<pre> | |||
root@google-chrome:~# vim /etc/apt/apt.conf.d/ | |||
00notify-hook 20auto-upgrades 50appstream 70debconf | |||
01autoremove 20listchanges 50unattended-upgrades 70no-unattended | |||
01autoremove-kernels 20packagekit 60gnome-software | |||
root@google-chrome:~# vim /etc/apt/apt.conf.d/^C | |||
root@google-chrome:~# wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb | |||
--2018-04-24 14:29:16-- https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb | |||
Resolving dl.google.com (dl.google.com)... 172.217.23.174, 2a00:1450:4001:81f::200e | |||
Connecting to dl.google.com (dl.google.com)|172.217.23.174|:443... connected. | |||
HTTP request sent, awaiting response... 200 OK | |||
Length: 52231002 (50M) [application/x-debian-package] | |||
Saving to: ‘google-chrome-stable_current_amd64.deb’ | |||
google-chrome-stable_curre 100%[========================================>] 49.81M 1.19MB/s in 57s | |||
2018-04-24 14:30:24 (895 KB/s) - ‘google-chrome-stable_current_amd64.deb’ saved [52231002/52231002] | |||
root@google-chrome:~# dpkg -i google-chrome-stable_current_amd64.deb | |||
Selecting previously unselected package google-chrome-stable. | |||
(Reading database ... 123078 files and directories currently installed.) | |||
Preparing to unpack google-chrome-stable_current_amd64.deb ... | |||
Unpacking google-chrome-stable (66.0.3359.117-1) ... | |||
dpkg: dependency problems prevent configuration of google-chrome-stable: | |||
google-chrome-stable depends on libappindicator3-1; however: | |||
Package libappindicator3-1 is not installed. | |||
dpkg: error processing package google-chrome-stable (--install): | |||
dependency problems - leaving unconfigured | |||
Processing triggers for man-db (2.7.6.1-2) ... | |||
Processing triggers for qubes-core-agent (3.2.28-1+deb9u1) ... | |||
Processing triggers for desktop-file-utils (0.23-1) ... | |||
Processing triggers for mime-support (3.60) ... | |||
Errors were encountered while processing: | |||
google-chrome-stable | |||
root@google-chrome:~# | |||
</pre> | |||
### I installed it with fix broken | |||
<pre> | |||
apt-get --fix-broken install | |||
</pre> | |||
### now I'm running Google Chrome Version 66.0.3359.117 (Official Build) (64-bit) | |||
### I loaded Marcin's log in this very-new-version of Google Chrome, logged in, clicked to edit the page, and to view the preview. I got the ERR_BLOCKED_BY_XSS_AUDITOR error again, and the console was still blank :( | |||
### As a test, I tried the same thing on our old site, and I got the issue again! That means this is an issue that's not relevant to our migration. If it hasn't been a major blocker before the migration, I think we should just mark this as "won't fix" with solution being "use firefox" for the few times this occurs | |||
=Mon Apr 23, 2018= | |||
# verified that our ec2 charges are still $0 in the aws console | |||
# I created a dns entry for jitsi.opensourceecology.org on cloudflare pointing to 34.210.153.174 = our ec2 instance's public IP | |||
# continuing with the jitsi manual install on our free ec2 t2.micro dev instance, I went to install jitsi meet; here's the easy part | |||
<pre> | |||
# download jitsi meet | |||
pushd /var/www/html/jitsi.opensourceecology.org | |||
git clone https://github.com/jitsi/jitsi-meet.git | |||
mv htdocs htdocs.`date "+%Y%m%d_%H%M%S"`.old | |||
mv "jitsi-meet" "htdocs" | |||
pushd htdocs | |||
</pre> | |||
# but all the npm stuff fails, as Christian found. There's some mentions of this in the github, suggesting that some have made it work https://github.com/jitsi/jitsi-meet/search?o=desc&q=centos&s=created&type=Issues | |||
<pre> | |||
[root@ip-172-31-28-115 htdocs]# yum install -y npm nodejs | |||
... | |||
[root@ip-172-31-28-115 htdocs]# npm install | |||
npm WARN deprecated nomnom@1.6.2: Package no longer supported. Contact support@npmjs.com for more info. | |||
npm WARN deprecated connect@2.30.2: connect 2.x series is deprecated | |||
Killed ..] / preinstall:url-polyfill: sill doParallel preinstall 855 | |||
[root@ip-172-31-28-115 htdocs]# make | |||
./node_modules/.bin/webpack -p | |||
module.js:478 | |||
throw err; | |||
^ | |||
Error: Cannot find module 'babel-preset-react' | |||
at Function.Module._resolveFilename (module.js:476:15) | |||
at Function.resolve (internal/module.js:27:19) | |||
at Object.<anonymous> (/var/www/html/jitsi.opensourceecology.org/htdocs/webpack.config.js:80:29) | |||
at Module._compile (module.js:577:32) | |||
at Object.Module._extensions..js (module.js:586:10) | |||
at Module.load (module.js:494:32) | |||
at tryModuleLoad (module.js:453:12) | |||
at Function.Module._load (module.js:445:3) | |||
at Module.require (module.js:504:17) | |||
at require (internal/module.js:20:19) | |||
make: *** [compile] Error 1 | |||
[root@ip-172-31-28-115 htdocs]# | |||
</pre> | |||
# the source is line 80 of webpack.config.js, which has a line = "require.resolve('babel-preset-react')" | |||
<pre> | |||
[root@ip-172-31-28-115 htdocs]# sed -n 70,90p /var/www/html/jitsi.opensourceecology.org/htdocs/webpack.config.js | |||
// jitsi-meet. The require.resolve, of course, mandates the use | |||
// of the prefix babel-preset- in the preset names. | |||
presets: [ | |||
[ | |||
require.resolve('babel-preset-env'), | |||
// Tell babel to avoid compiling imports into CommonJS | |||
// so that webpack may do tree shaking. | |||
{ modules: false } | |||
], | |||
require.resolve('babel-preset-react'), | |||
require.resolve('babel-preset-stage-1') | |||
] | |||
}, | |||
test: /\.jsx?$/ | |||
}, { | |||
// Expose jquery as the globals $ and jQuery because it is expected | |||
// to be available in such a form by multiple jitsi-meet | |||
// dependencies including lib-jitsi-meet. | |||
loader: 'expose-loader?$!expose-loader?jQuery', | |||
[root@ip-172-31-28-115 htdocs]# | |||
</pre> | |||
# these issues were specifically mentioned in this issue https://github.com/jitsi/jitsi-meet/issues/446 | |||
# I fixed this by manually installing the packages called-out | |||
<pre> | |||
npm install babel-preset-react | |||
npm install babel-preset-stage-1 | |||
</pre> | |||
# but then the next make had a different issue | |||
<pre> | |||
[root@ip-172-31-28-115 htdocs]# make | |||
./node_modules/.bin/webpack -p | |||
Hash: f38346765ad69b5229dc9fb40aa6056b410b3138 | |||
Version: webpack 3.9.1 | |||
Child | |||
Hash: f38346765ad69b5229dc | |||
Time: 20725ms | |||
Asset Size Chunks Chunk Names | |||
dial_in_info_bundle.min.js 90.5 kB 0 [emitted] dial_in_info_bundle | |||
app.bundle.min.js 90.3 kB 1 [emitted] app.bundle | |||
dial_in_info_bundle.min.map 735 kB 0 [emitted] dial_in_info_bundle | |||
app.bundle.min.map 735 kB 1 [emitted] app.bundle | |||
[90] (webpack)/buildin/global.js 509 bytes {0} {1} [built] | |||
[327] multi babel-polyfill whatwg-fetch ./app.js 52 bytes {1} [built] | |||
[328] multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page 64 bytes {0} [built] | |||
+ 326 hidden modules | |||
ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
ERROR in multi babel-polyfill whatwg-fetch ./app.js | |||
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
@ multi babel-polyfill whatwg-fetch ./app.js | |||
ERROR in multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page | |||
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
@ multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page | |||
ERROR in multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page | |||
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
@ multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page | |||
ERROR in multi babel-polyfill whatwg-fetch ./app.js | |||
Module not found: Error: Can't resolve 'whatwg-fetch' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
@ multi babel-polyfill whatwg-fetch ./app.js | |||
ERROR in multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page | |||
Module not found: Error: Can't resolve 'whatwg-fetch' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
@ multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page | |||
Child | |||
Hash: 9fb40aa6056b410b3138 | |||
Time: 20699ms | |||
Asset Size Chunks Chunk Names | |||
external_api.min.js 90.5 kB 0 [emitted] external_api | |||
external_api.min.map 735 kB 0 [emitted] external_api | |||
[90] (webpack)/buildin/global.js 509 bytes {0} [built] | |||
[125] multi babel-polyfill ./modules/API/external/index.js 40 bytes {0} [built] | |||
+ 326 hidden modules | |||
ERROR in multi babel-polyfill ./modules/API/external/index.js | |||
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs' | |||
@ multi babel-polyfill ./modules/API/external/index.js | |||
make: *** [compile] Error 2 | |||
[root@ip-172-31-28-115 htdocs]# | |||
</pre> | |||
# fixed by installing babel-loader | |||
<pre> | |||
npm install babel-loader | |||
make | |||
</pre> | |||
# the next failure lacked info.. | |||
<pre> | |||
[root@ip-172-31-28-115 htdocs]# make | |||
./node_modules/.bin/webpack -p | |||
make: *** [compile] Killed | |||
[root@ip-172-31-28-115 htdocs]# echo $? | |||
2 | |||
[root@ip-172-31-28-115 htdocs]# | |||
</pre> | |||
# I opened up port 80 & 443 on the server's security group in the aws console's ec2 service | |||
# to simplify debugging, I changed the nginx config to use port 80, not 443. | |||
# selinux had to be disabled to prevent "permission denied" errors as popped up in the logs: | |||
<pre> | |||
==> /var/log/nginx/error.log <== | |||
2018/04/24 02:57:58 [error] 3101#0: *1 open() "/var/www/html/jitsi.opensourceecology.org/htdocs/index.html" failed (13: Permission denied), client: 76.97.223.185, server: jitsi.opensourceecology.org, request: "GET /index.html HTTP/1.1", host: "ec2-34-210-153-174.us-west-2.compute.amazonaws.com" | |||
==> /var/log/nginx/access.log <== | |||
76.97.223.185 - - [24/Apr/2018:02:57:58 +0000] "GET /index.html HTTP/1.1" 403 180 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-" | |||
</pre> | |||
# I disabled selinux by setting it to Permissive & restarting nginx | |||
<pre> | |||
[root@ip-172-31-28-115 jitsi.opensourceecology.org]# getenforce | |||
Enforcing | |||
[root@ip-172-31-28-115 jitsi.opensourceecology.org]# setenforce Permissive | |||
[root@ip-172-31-28-115 jitsi.opensourceecology.org]# service nginx restart | |||
Redirecting to /bin/systemctl restart nginx.service | |||
[root@ip-172-31-28-115 jitsi.opensourceecology.org]# | |||
</pre> | |||
# ok, this got a page load, but with an error | |||
<pre> | |||
Uh oh! We couldn't fully download everything we needed :( | |||
We will try again shortly. In the mean time, check for problems with your Internet connection! | |||
Missing http://ec2-34-210-153-174.us-west-2.compute.amazonaws.com/libs/lib-jitsi-meet.min.js?v=139 | |||
show less reload now | |||
</pre> | |||
# ah, the "Killed" issue is because linux is running out of memory | |||
<pre> | |||
[root@ip-172-31-28-115 ~]# tail -n 100 /var/log/messages | |||
Apr 24 03:00:35 ip-172-31-28-115 dbus[519]: avc: received setenforce notice (enforcing=0) | |||
Apr 24 03:00:35 ip-172-31-28-115 dbus-daemon: dbus[519]: avc: received setenforce notice (enforcing=0) | |||
Apr 24 03:00:42 ip-172-31-28-115 systemd: Stopping The nginx HTTP and reverse proxy server... | |||
Apr 24 03:00:42 ip-172-31-28-115 systemd: Starting The nginx HTTP and reverse proxy server... | |||
Apr 24 03:00:42 ip-172-31-28-115 nginx: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok | |||
Apr 24 03:00:42 ip-172-31-28-115 nginx: nginx: configuration file /etc/nginx/nginx.conf test is successful | |||
Apr 24 03:00:42 ip-172-31-28-115 systemd: Failed to read PID from file /run/nginx.pid: Invalid argument | |||
Apr 24 03:00:42 ip-172-31-28-115 systemd: Started The nginx HTTP and reverse proxy server. | |||
Apr 24 03:01:01 ip-172-31-28-115 systemd: Started Session 88 of user root. | |||
Apr 24 03:01:01 ip-172-31-28-115 systemd: Starting Session 88 of user root. | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: systemd invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: systemd cpuset=/ mems_allowed=0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: CPU: 0 PID: 1 Comm: systemd Not tainted 3.10.0-693.11.6.el7.x86_64 #1 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Call Trace: | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a5ea1>] dump_stack+0x19/0x1b | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a1296>] dump_header+0x90/0x229 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff812b9dfb>] ? cred_has_capability+0x6b/0x120 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff81188094>] oom_kill_process+0x254/0x3d0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff812b9fde>] ? selinux_capable+0x2e/0x40 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff811888d6>] out_of_memory+0x4b6/0x4f0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a1d9a>] __alloc_pages_slowpath+0x5d6/0x724 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8118eaa5>] __alloc_pages_nodemask+0x405/0x420 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff811d6075>] alloc_pages_vma+0xb5/0x200 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a924e>] ? __wait_on_bit+0x7e/0x90 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff811b42a0>] handle_mm_fault+0xb60/0xfa0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8132f4b3>] ? number.isra.2+0x323/0x360 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816b37e4>] __do_page_fault+0x154/0x450 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816b3b15>] do_page_fault+0x35/0x90 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816af8f8>] page_fault+0x28/0x30 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff81332459>] ? copy_user_enhanced_fast_string+0x9/0x20 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8122854b>] ? seq_read+0x2ab/0x3b0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8120295c>] vfs_read+0x9c/0x170 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8120381f>] SyS_read+0x7f/0xe0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Mem-Info: | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: active_anon:214589 inactive_anon:4438 isolated_anon:0#012 active_file:36 inactive_file:783 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:4921 slab_unreclaimable:8167#012 mapped:879 shmem:6524 pagetables:2371 bounce:0#012 free:12232 free_pcp:118 free_cma:0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA free:4588kB min:704kB low:880kB high:1056kB active_anon:10344kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:360kB slab_reclaimable:268kB slab_unreclaimable:476kB kernel_stack:48kB pagetables:140kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: lowmem_reserve[]: 0 973 973 973 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA32 free:44340kB min:44348kB low:55432kB high:66520kB active_anon:848012kB inactive_anon:17748kB active_file:144kB inactive_file:3132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032192kB managed:998676kB mlocked:0kB dirty:0kB writeback:0kB mapped:3516kB shmem:25736kB slab_reclaimable:19416kB slab_unreclaimable:32192kB kernel_stack:2352kB pagetables:9344kB unstable:0kB bounce:0kB free_pcp:472kB local_pcp:472kB free_cma:0kB writeback_tmp:0kB pages_scanned:711 all_unreclaimable? yes | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: lowmem_reserve[]: 0 0 0 0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA: 25*4kB (UEM) 19*8kB (UE) 19*16kB (UEM) 12*32kB (UE) 5*64kB (UE) 12*128kB (UEM) 3*256kB (M) 0*512kB 1*1024kB (E) 0*2048kB 0*4096kB = 4588kB | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA32: 1281*4kB (UEM) 1040*8kB (UE) 593*16kB (UEM) 251*32kB (UE) 67*64kB (UE) 41*128kB (UEM) 9*256kB (UEM) 3*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 44340kB | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: 7345 total pagecache pages | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: 0 pages in swap cache | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Swap cache stats: add 0, delete 0, find 0/0 | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Free swap = 0kB | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Total swap = 0kB | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: 262045 pages RAM | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: 0 pages HighMem/MovableOnly | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: 8400 pages reserved | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 321] 0 321 8841 803 24 0 0 systemd-journal | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 519] 81 519 24607 167 18 0 -900 dbus-daemon | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 524] 0 524 135883 1481 84 0 0 NetworkManager | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 531] 0 531 6051 82 16 0 0 systemd-logind | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 562] 0 562 28343 3126 58 0 0 dhclient | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1031] 0 1031 22386 260 43 0 0 master | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1033] 89 1033 22429 253 46 0 0 qmgr | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1139] 0 1139 27511 33 12 0 0 agetty | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1140] 0 1140 27511 33 10 0 0 agetty | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 8587] 0 8587 11692 394 27 0 -1000 systemd-udevd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9607] 0 9607 28198 256 58 0 -1000 sshd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9722] 0 9722 13877 111 29 0 -1000 auditd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9760] 0 9760 58390 946 48 0 0 rsyslogd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9781] 0 9781 31571 159 18 0 0 crond | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9826] 999 9826 134633 1647 60 0 0 polkitd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9903] 0 9903 26991 38 9 0 0 rhnsd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9914] 0 9914 143438 3265 99 0 0 tuned | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9985] 998 9985 25120 91 20 0 0 chronyd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [11651] 0 11651 32007 188 17 0 0 screen | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [11652] 0 11652 28891 135 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [12264] 1001 12264 1291983 15415 88 0 0 java | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [12516] 0 12516 47972 155 49 0 0 su | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [12517] 1001 12517 28859 103 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [12611] 0 12611 28859 105 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [27867] 0 27867 39164 336 80 0 0 sshd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [27870] 1000 27870 39361 552 78 0 0 sshd | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [27871] 1000 27871 28859 95 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [27892] 1000 27892 31908 66 18 0 0 screen | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [27893] 1000 27893 32113 296 16 0 0 screen | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [27894] 1000 27894 28859 115 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30326] 0 30326 54628 261 63 0 0 sudo | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30327] 0 30327 47972 155 49 0 0 su | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30328] 0 30328 28892 125 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30369] 1000 30369 28859 105 13 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30385] 0 30385 54628 261 61 0 0 sudo | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30386] 0 30386 47972 155 49 0 0 su | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [30387] 0 30387 28892 141 14 0 0 bash | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3017] 997 3017 24190 5474 48 0 0 lua | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3164] 0 3164 30201 523 54 0 0 nginx | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3165] 996 3165 30317 649 56 0 0 nginx | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3177] 0 3177 30814 57 15 0 0 anacron | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3180] 89 3180 22428 276 46 0 0 pickup | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3185] 0 3185 27066 60 9 0 0 make | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3186] 0 3186 469819 174655 759 0 0 node | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Out of memory: Kill process 3186 (node) score 670 or sacrifice child | |||
Apr 24 03:07:57 ip-172-31-28-115 kernel: Killed process 3186 (node) total-vm:1879276kB, anon-rss:698620kB, file-rss:0kB, shmem-rss:0kB | |||
Apr 24 03:10:36 ip-172-31-28-115 su: (to root) ec2-user on pts/5 | |||
[root@ip-172-31-28-115 ~]# | |||
</pre> | |||
# looks like we need a bigger box :\ or swap? | |||
<pre> | |||
[root@ip-172-31-28-115 ~]# free -m | |||
total used free shared buff/cache available | |||
Mem: 990 179 689 25 121 652 | |||
Swap: 0 0 0 | |||
[root@ip-172-31-28-115 ~]# df -h | |||
Filesystem Size Used Avail Use% Mounted on | |||
/dev/xvda2 10G 3.8G 6.3G 38% / | |||
devtmpfs 476M 0 476M 0% /dev | |||
tmpfs 496M 0 496M 0% /dev/shm | |||
tmpfs 496M 26M 470M 6% /run | |||
tmpfs 496M 0 496M 0% /sys/fs/cgroup | |||
tmpfs 100M 0 100M 0% /run/user/0 | |||
tmpfs 100M 0 100M 0% /run/user/1000 | |||
[root@ip-172-31-28-115 ~]# | |||
</pre> | |||
# we have 1G of RAM and 6G of free disk space. I created a 2G swap file & enabled it | |||
<pre> | |||
[root@ip-172-31-28-115 ~]# dd if=/dev/zero of=/swap1 bs=1M count=2048 | |||
2048+0 records in | |||
2048+0 records out | |||
2147483648 bytes (2.1 GB) copied, 30.2187 s, 71.1 MB/s | |||
[root@ip-172-31-28-115 ~]# mkswap /swap1 | |||
Setting up swapspace version 1, size = 2097148 KiB | |||
no label, UUID=a401ef50-ccb8-4bca-abeb-9de5a63b107c | |||
[root@ip-172-31-28-115 ~]# chmod 0600 /swap1 | |||
[root@ip-172-31-28-115 ~]# swapon /swap1 | |||
[root@ip-172-31-28-115 ~]# free -m | |||
total used free shared buff/cache available | |||
Mem: 990 180 69 25 740 608 | |||
Swap: 2047 0 2047 | |||
[root@ip-172-31-28-115 ~]# | |||
</pre> | |||
# the next `make` run didn't get killed by oom, but it had a bunch of complaints of missing modules: | |||
<pre> | |||
./node_modules/.bin/webpack -p | |||
Hash: 63ec7d579fb8d7f2444256d53b2e2b409f23ea8a | |||
Version: webpack 3.9.1 | |||
Child | |||
Hash: 63ec7d579fb8d7f24442 | |||
Time: 72401ms | |||
Asset Size Chunks Chunk Names | |||
app.bundle.min.js 317 kB 0 [emitted] [big] app.bundle | |||
dial_in_info_bundle.min.js 116 kB 1 [emitted] dial_in_info_bundle | |||
device_selection_popup_bundle.min.js 101 kB 2 [emitted] device_selection_popup_bundle | |||
do_external_connect.min.js 2 kB 3 [emitted] do_external_connect | |||
alwaysontop.min.js 731 bytes 4 [emitted] alwaysontop | |||
app.bundle.min.map 4.21 MB 0 [emitted] app.bundle | |||
dial_in_info_bundle.min.map 912 kB 1 [emitted] dial_in_info_bundle | |||
device_selection_popup_bundle.min.map 2.11 MB 2 [emitted] device_selection_popup_bundle | |||
do_external_connect.min.map 19.4 kB 3 [emitted] do_external_connect | |||
alwaysontop.min.map 39.2 kB 4 [emitted] alwaysontop | |||
[43] ./react/features/base/config/index.js + 4 modules 13.2 kB {0} {1} {2} [built] | |||
[70] ./react/features/base/config/parseURLParams.js 1.51 kB {0} {1} {2} {3} [built] | |||
[122] ./react/features/base/config/getRoomName.js 761 bytes {0} {1} {2} {3} [built] | |||
[176] ./react/features/base/react/prop-types-polyfill.js 227 bytes {0} {1} {2} [built] | |||
[444] ./modules/settings/Settings.js 6.12 kB {0} [built] | |||
[450] multi babel-polyfill whatwg-fetch ./app.js 52 bytes {0} [built] | |||
[451] ./app.js + 2 modules 4.67 kB {0} [built] | |||
[465] ./modules/UI/UI.js 35 kB {0} [built] | |||
[480] ./react/index.web.js 1.06 kB {0} [built] | |||
[481] ./react/features/device-selection/popup.js 476 bytes {2} [built] | |||
[482] ./react/features/device-selection/DeviceSelectionPopup.js 12.3 kB {2} [built] | |||
[483] ./react/features/always-on-top/index.js + 2 modules 20.3 kB {4} [built] | |||
[484] multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page 64 bytes {1} [built] | |||
[485] ./react/features/invite/components/dial-in-info-page/index.js + 2 modules 4 kB {1} [built] | |||
[486] ./connection_optimization/do_external_connect.js 2.51 kB {3} [built] | |||
+ 472 hidden modules | |||
ERROR in ./react/features/invite/components/AddPeopleDialog.web.js | |||
Module not found: Error: Can't resolve '@atlaskit/avatar' in '/var/www/html/jitsi.opensourceecology.org/htdocs/react/features/invite/components' | |||
@ ./react/features/invite/components/AddPeopleDialog.web.js 15:0-38 | |||
@ ./react/features/invite/components/index.js | |||
@ ./react/features/invite/index.js | |||
@ ./react/features/toolbox/components/Toolbox.web.js | |||
@ ./react/features/toolbox/components/index.js | |||
@ ./react/features/toolbox/index.js | |||
@ ./conference.js | |||
@ ./app.js | |||
@ multi babel-polyfill whatwg-fetch ./app.js | |||
ERROR in ./react/features/base/dialog/components/StatelessDialog.web.js | |||
Module not found: Error: Can't resolve '@atlaskit/button' in '/var/www/html/jitsi.opensourceecology.org/htdocs/react/features/base/dialog/components' | |||
@ ./react/features/base/dialog/components/StatelessDialog.web.js 9:0-55 | |||
@ ./react/features/base/dialog/components/index.js | |||
@ ./react/features/base/dialog/index.js | |||
@ ./modules/keyboardshortcut/keyboardshortcut.js | |||
@ ./app.js | |||
@ multi babel-polyfill whatwg-fetch ./app.js | |||
... | |||
</pre> | |||
# the log file above was huge; here's a grep for just the missing module bit | |||
<pre> | |||
[root@ip-172-31-28-115 htdocs]# grep -i 'module not found' make.log | cut -d' ' -f1-11 | sort -u | |||
Module not found: Error: Can't resolve '@atlaskit/avatar' | |||
Module not found: Error: Can't resolve '@atlaskit/button' | |||
Module not found: Error: Can't resolve '@atlaskit/dropdown-menu' | |||
Module not found: Error: Can't resolve '@atlaskit/field-text' | |||
Module not found: Error: Can't resolve '@atlaskit/field-text-area' | |||
Module not found: Error: Can't resolve '@atlaskit/flag' | |||
Module not found: Error: Can't resolve '@atlaskit/icon/glyph/chevron-down' | |||
Module not found: Error: Can't resolve '@atlaskit/icon/glyph/editor/info' | |||
Module not found: Error: Can't resolve '@atlaskit/icon/glyph/error' | |||
Module not found: Error: Can't resolve '@atlaskit/icon/glyph/star' | |||
Module not found: Error: Can't resolve '@atlaskit/icon/glyph/star-filled' | |||
Module not found: Error: Can't resolve '@atlaskit/icon/glyph/warning' | |||
Module not found: Error: Can't resolve '@atlaskit/inline-dialog' | |||
Module not found: Error: Can't resolve '@atlaskit/inline-message' | |||
Module not found: Error: Can't resolve '@atlaskit/layer-manager' | |||
Module not found: Error: Can't resolve '@atlaskit/lozenge' | |||
Module not found: Error: Can't resolve '@atlaskit/modal-dialog' | |||
Module not found: Error: Can't resolve '@atlaskit/multi-select' | |||
Module not found: Error: Can't resolve '@atlaskit/spinner' | |||
Module not found: Error: Can't resolve '@atlaskit/tabs' | |||
Module not found: Error: Can't resolve '@atlaskit/theme' | |||
Module not found: Error: Can't resolve '@atlaskit/tooltip' | |||
Module not found: Error: Can't resolve 'autosize' | |||
Module not found: Error: Can't resolve 'i18next' | |||
Module not found: Error: Can't resolve 'i18next-browser-languagedetector' | |||
Module not found: Error: Can't resolve 'i18next-xhr-backend' | |||
Module not found: Error: Can't resolve 'jitsi-meet-logger' | |||
Module not found: Error: Can't resolve 'jquery' | |||
Module not found: Error: Can't resolve 'jquery-contextmenu' | |||
Module not found: Error: Can't resolve 'jquery-i18next' | |||
Module not found: Error: Can't resolve 'jQuery-Impromptu' | |||
Module not found: Error: Can't resolve 'js-md5' | |||
Module not found: Error: Can't resolve 'jwt-decode' | |||
Module not found: Error: Can't resolve 'lodash' | |||
Module not found: Error: Can't resolve 'lodash/debounce' | |||
Module not found: Error: Can't resolve 'lodash/throttle' | |||
Module not found: Error: Can't resolve 'moment' | |||
Module not found: Error: Can't resolve 'moment/locale/bg' | |||
Module not found: Error: Can't resolve 'moment/locale/de' | |||
Module not found: Error: Can't resolve 'moment/locale/eo' | |||
Module not found: Error: Can't resolve 'moment/locale/es' | |||
Module not found: Error: Can't resolve 'moment/locale/fr' | |||
Module not found: Error: Can't resolve 'moment/locale/hy-am' | |||
Module not found: Error: Can't resolve 'moment/locale/it' | |||
Module not found: Error: Can't resolve 'moment/locale/nb' | |||
Module not found: Error: Can't resolve 'moment/locale/pl' | |||
Module not found: Error: Can't resolve 'moment/locale/pt' | |||
Module not found: Error: Can't resolve 'moment/locale/pt-br' | |||
Module not found: Error: Can't resolve 'moment/locale/ru' | |||
Module not found: Error: Can't resolve 'moment/locale/sk' | |||
Module not found: Error: Can't resolve 'moment/locale/sl' | |||
Module not found: Error: Can't resolve 'moment/locale/sv' | |||
Module not found: Error: Can't resolve 'moment/locale/tr' | |||
Module not found: Error: Can't resolve 'moment/locale/zh-cn' | |||
Module not found: Error: Can't resolve 'postis' | |||
Module not found: Error: Can't resolve 'prop-types' | |||
Module not found: Error: Can't resolve 'react' | |||
Module not found: Error: Can't resolve 'react-dom' | |||
Module not found: Error: Can't resolve 'react-i18next' | |||
Module not found: Error: Can't resolve 'react-redux' | |||
Module not found: Error: Can't resolve 'redux' | |||
Module not found: Error: Can't resolve 'redux-thunk' | |||
Module not found: Error: Can't resolve 'url-polyfill' | |||
Module not found: Error: Can't resolve 'whatwg-fetch' | |||
[root@ip-172-31-28-115 htdocs]# | |||
</pre> | |||
=Fri Apr 20, 2018= | =Fri Apr 20, 2018= | ||
Revision as of 19:04, 1 May 2018
My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.
See Also
Thr Apr 26, 2018
- Meeting with Marcin
- Wiki validation so far
- 2 outstanding issues
- We have a superfluous link at the top-right = Create Account and Request Account we should delete the "Create Account" link
- Marcin can't login because his old password was <20 characters, and the new wiki has a minimum password requirement for Administrators to have >=20 character passwords
- 2 outstanding issues
- Test Plan
- marcin will make a first-draft of a goolgle doc with an exhaustive list of things to test on the wiki (50-100-ish items) & send it to me by next week. it should include
- Both simple & complex daily or routine wiki tasks
- OSE-specific workflow tasks on the wiki that have broken in the past per your experiences that pre-date me
- other wiki functions that we discovered were broken in the past few months when validating the staging wiki
- we may migrate the wiki together in-person when I visit FeF for a few days in Mid-May
- marcin will make a first-draft of a goolgle doc with an exhaustive list of things to test on the wiki (50-100-ish items) & send it to me by next week. it should include
- Wiki validation so far
- Jitsi
- I had some questions about this (see below), but we didn't get to it (we focused mostly on the wiki & backups)
- what are the reqs for September? Same as before?
- thoughts on hipchat?
- maybe rockchat.
- or maybe jangouts powered by janus (C > Node)
- I had some questions about this (see below), but we didn't get to it (we focused mostly on the wiki & backups)
- I reset Marcin's password on the wiki to be >=20 characters using 'htdocs/maintenance/changePassword.php'
[root@hetzner2 maintenance]# php changePassword.php --user=Marcin --password='fake567890example890' PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674 PHP Notice: Undefined index: HTTP_USER_AGENT in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693 Password set for Marcin [root@hetzner2 maintenance]#
Wed Apr 25, 2018
- atlassian got back to me giving me a free Hipchat Community License
- it's not well advertised, but I managed to find that the limit (as of 2018-01) the limit of participants in a single Stride call are 25 participants https://community.atlassian.com/t5/Stride-questions/How-many-participants-can-join-a-stride-video-call/qaq-p/694918
- while Hipchat has a limit of 20 participants https://confluence.atlassian.com/hipchat/video-chat-with-your-team-838548935.html
- updated our Videoconferencing article to include Rocket Chat and Hipchat
Tue Apr 24, 2018
- I sent an email to Atlassian requesting a community license request for Stride https://www.atlassian.com/software/views/community-license-request
- Atlassian automatically mailed me a service desk ticket for this request https://getsupport.atlassian.com/servicedesk/customer/portal/35/CA-452135
- I found a great article describing what Jitsi gives us as an SFU https://webrtchacks.com/atlassian-sfu-qa/
- and this video comparing mesh vs sfu vs mcu https://webrtcglossary.com/mcu/
- and this great article https://bloggeek.me/how-many-users-webrtc-call[[1]]
- we might actually want to look into open source MCUs instead of an SFU. The downside is more processing on our server, but I think that'd be less of a bottleneck than each participant downloading n streams.
- responded to Marcin's validation follow-up email
- (3) Chrome ERR_BLOCKED_BY_XSS_AUDITOR on Preview
- marcin found another permission denied issue when attempting to view the preview of the FAQ page https://wiki.opensourceecology.org/index.php?title=FAQ&action=submit
- this was a red herring; it's just another mod_security false-positive that's distinct from the chrome ERR_BLOCKED_BY_XSS_AUDITOR issue
- I whitelisted 959071 = sqli, and I confirmed that fixed the issue
- Marcin said he's running chromium Version 64.0.3282.167 (Official Build) Built on Ubuntu , running on Ubuntu 16.04 (64-bit)
- My chromium is Version 57.0.2987.98 Built on 8.7, running on Debian 8.10 (64-bit)
- I went ahead and updated my ose VM to debian 9, but chromium in debian 9 is still Version 57.0.2987.98 Built on 8.7, running on Debian 9.4 (64-bit)
- the preview bug was supposed to be fixed in v58, but maybe the bug was just that it was blocking the page *and* the debugging. In any case, I can't see what's causing chrome to issue the ERR_BLOCKED_BY_XSS_AUDITOR
- I spun up a disposable vm & installed the latest version of google chrome, but that failed
- marcin found another permission denied issue when attempting to view the preview of the FAQ page https://wiki.opensourceecology.org/index.php?title=FAQ&action=submit
- (3) Chrome ERR_BLOCKED_BY_XSS_AUDITOR on Preview
[user@fedora-23-dvm ~]$ wget https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm --2018-04-24 13:38:31-- https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm Resolving dl.google.com (dl.google.com)... 172.217.23.174, 2a00:1450:4001:81f::200e Connecting to dl.google.com (dl.google.com)|172.217.23.174|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 52364199 (50M) [application/x-rpm] Saving to: ‘google-chrome-stable_current_x86_64.rpm’ google-chrome-stable_current_x86 100%[=========================================================>] 49.94M 1.19MB/s in 47s 2018-04-24 13:39:24 (1.06 MB/s) - ‘google-chrome-stable_current_x86_64.rpm’ saved [52364199/52364199] [user@fedora-23-dvm ~]$ sudo dnf install google-chrome-stable_current_x86_64.rpm Last metadata expiration check: 0:09:47 ago on Tue Apr 24 13:37:39 2018. Error: nothing provides libssl3.so(NSS_3.28)(64bit) needed by google-chrome-stable-66.0.3359.117-1.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages) [user@fedora-23-dvm ~]$ sudo dnf --allowerasing install google-chrome-stable_current_x86_64.rpm Last metadata expiration check: 0:10:35 ago on Tue Apr 24 13:37:39 2018. Error: nothing provides libssl3.so(NSS_3.28)(64bit) needed by google-chrome-stable-66.0.3359.117-1.x86_64
- fedora does have chromium in its repos; I installed them, but this was even more out-of-date @ Version 54.0.2840.90 Fedora Project (64-bit)
- I found & installed the google-chrome depend (openssl-devel), but the same error occured. I found a fix of installing fedora 27 :| https://stackoverflow.com/questions/48839199/install-google-chrome-in-fedora-23
- giving up on fedora 27, I created a new VM from debian-9 for installing google-chrome https://unix.stackexchange.com/questions/20614/how-do-i-install-the-latest-version-of-chromium-in-debian-squeeze
root@google-chrome:~# vim /etc/apt/apt.conf.d/ 00notify-hook 20auto-upgrades 50appstream 70debconf 01autoremove 20listchanges 50unattended-upgrades 70no-unattended 01autoremove-kernels 20packagekit 60gnome-software root@google-chrome:~# vim /etc/apt/apt.conf.d/^C root@google-chrome:~# wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb --2018-04-24 14:29:16-- https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb Resolving dl.google.com (dl.google.com)... 172.217.23.174, 2a00:1450:4001:81f::200e Connecting to dl.google.com (dl.google.com)|172.217.23.174|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 52231002 (50M) [application/x-debian-package] Saving to: ‘google-chrome-stable_current_amd64.deb’ google-chrome-stable_curre 100%[========================================>] 49.81M 1.19MB/s in 57s 2018-04-24 14:30:24 (895 KB/s) - ‘google-chrome-stable_current_amd64.deb’ saved [52231002/52231002] root@google-chrome:~# dpkg -i google-chrome-stable_current_amd64.deb Selecting previously unselected package google-chrome-stable. (Reading database ... 123078 files and directories currently installed.) Preparing to unpack google-chrome-stable_current_amd64.deb ... Unpacking google-chrome-stable (66.0.3359.117-1) ... dpkg: dependency problems prevent configuration of google-chrome-stable: google-chrome-stable depends on libappindicator3-1; however: Package libappindicator3-1 is not installed. dpkg: error processing package google-chrome-stable (--install): dependency problems - leaving unconfigured Processing triggers for man-db (2.7.6.1-2) ... Processing triggers for qubes-core-agent (3.2.28-1+deb9u1) ... Processing triggers for desktop-file-utils (0.23-1) ... Processing triggers for mime-support (3.60) ... Errors were encountered while processing: google-chrome-stable root@google-chrome:~#
- I installed it with fix broken
apt-get --fix-broken install
- now I'm running Google Chrome Version 66.0.3359.117 (Official Build) (64-bit)
- I loaded Marcin's log in this very-new-version of Google Chrome, logged in, clicked to edit the page, and to view the preview. I got the ERR_BLOCKED_BY_XSS_AUDITOR error again, and the console was still blank :(
- As a test, I tried the same thing on our old site, and I got the issue again! That means this is an issue that's not relevant to our migration. If it hasn't been a major blocker before the migration, I think we should just mark this as "won't fix" with solution being "use firefox" for the few times this occurs
Mon Apr 23, 2018
- verified that our ec2 charges are still $0 in the aws console
- I created a dns entry for jitsi.opensourceecology.org on cloudflare pointing to 34.210.153.174 = our ec2 instance's public IP
- continuing with the jitsi manual install on our free ec2 t2.micro dev instance, I went to install jitsi meet; here's the easy part
# download jitsi meet pushd /var/www/html/jitsi.opensourceecology.org git clone https://github.com/jitsi/jitsi-meet.git mv htdocs htdocs.`date "+%Y%m%d_%H%M%S"`.old mv "jitsi-meet" "htdocs" pushd htdocs
- but all the npm stuff fails, as Christian found. There's some mentions of this in the github, suggesting that some have made it work https://github.com/jitsi/jitsi-meet/search?o=desc&q=centos&s=created&type=Issues
[root@ip-172-31-28-115 htdocs]# yum install -y npm nodejs ... [root@ip-172-31-28-115 htdocs]# npm install npm WARN deprecated nomnom@1.6.2: Package no longer supported. Contact support@npmjs.com for more info. npm WARN deprecated connect@2.30.2: connect 2.x series is deprecated Killed ..] / preinstall:url-polyfill: sill doParallel preinstall 855 [root@ip-172-31-28-115 htdocs]# make ./node_modules/.bin/webpack -p module.js:478 throw err; ^ Error: Cannot find module 'babel-preset-react' at Function.Module._resolveFilename (module.js:476:15) at Function.resolve (internal/module.js:27:19) at Object.<anonymous> (/var/www/html/jitsi.opensourceecology.org/htdocs/webpack.config.js:80:29) at Module._compile (module.js:577:32) at Object.Module._extensions..js (module.js:586:10) at Module.load (module.js:494:32) at tryModuleLoad (module.js:453:12) at Function.Module._load (module.js:445:3) at Module.require (module.js:504:17) at require (internal/module.js:20:19) make: *** [compile] Error 1 [root@ip-172-31-28-115 htdocs]#
- the source is line 80 of webpack.config.js, which has a line = "require.resolve('babel-preset-react')"
[root@ip-172-31-28-115 htdocs]# sed -n 70,90p /var/www/html/jitsi.opensourceecology.org/htdocs/webpack.config.js
// jitsi-meet. The require.resolve, of course, mandates the use
// of the prefix babel-preset- in the preset names.
presets: [
[
require.resolve('babel-preset-env'),
// Tell babel to avoid compiling imports into CommonJS
// so that webpack may do tree shaking.
{ modules: false }
],
require.resolve('babel-preset-react'),
require.resolve('babel-preset-stage-1')
]
},
test: /\.jsx?$/
}, {
// Expose jquery as the globals $ and jQuery because it is expected
// to be available in such a form by multiple jitsi-meet
// dependencies including lib-jitsi-meet.
loader: 'expose-loader?$!expose-loader?jQuery',
[root@ip-172-31-28-115 htdocs]#
- these issues were specifically mentioned in this issue https://github.com/jitsi/jitsi-meet/issues/446
- I fixed this by manually installing the packages called-out
npm install babel-preset-react npm install babel-preset-stage-1
- but then the next make had a different issue
[root@ip-172-31-28-115 htdocs]# make
./node_modules/.bin/webpack -p
Hash: f38346765ad69b5229dc9fb40aa6056b410b3138
Version: webpack 3.9.1
Child
Hash: f38346765ad69b5229dc
Time: 20725ms
Asset Size Chunks Chunk Names
dial_in_info_bundle.min.js 90.5 kB 0 [emitted] dial_in_info_bundle
app.bundle.min.js 90.3 kB 1 [emitted] app.bundle
dial_in_info_bundle.min.map 735 kB 0 [emitted] dial_in_info_bundle
app.bundle.min.map 735 kB 1 [emitted] app.bundle
[90] (webpack)/buildin/global.js 509 bytes {0} {1} [built]
[327] multi babel-polyfill whatwg-fetch ./app.js 52 bytes {1} [built]
[328] multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page 64 bytes {0} [built]
+ 326 hidden modules
ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
ERROR in multi babel-polyfill whatwg-fetch ./app.js
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
@ multi babel-polyfill whatwg-fetch ./app.js
ERROR in multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
@ multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page
ERROR in multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
@ multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page
ERROR in multi babel-polyfill whatwg-fetch ./app.js
Module not found: Error: Can't resolve 'whatwg-fetch' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
@ multi babel-polyfill whatwg-fetch ./app.js
ERROR in multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page
Module not found: Error: Can't resolve 'whatwg-fetch' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
@ multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page
Child
Hash: 9fb40aa6056b410b3138
Time: 20699ms
Asset Size Chunks Chunk Names
external_api.min.js 90.5 kB 0 [emitted] external_api
external_api.min.map 735 kB 0 [emitted] external_api
[90] (webpack)/buildin/global.js 509 bytes {0} [built]
[125] multi babel-polyfill ./modules/API/external/index.js 40 bytes {0} [built]
+ 326 hidden modules
ERROR in multi babel-polyfill ./modules/API/external/index.js
Module not found: Error: Can't resolve 'babel-loader' in '/var/www/html/jitsi.opensourceecology.org/htdocs'
@ multi babel-polyfill ./modules/API/external/index.js
make: *** [compile] Error 2
[root@ip-172-31-28-115 htdocs]#
- fixed by installing babel-loader
npm install babel-loader make
- the next failure lacked info..
[root@ip-172-31-28-115 htdocs]# make ./node_modules/.bin/webpack -p make: *** [compile] Killed [root@ip-172-31-28-115 htdocs]# echo $? 2 [root@ip-172-31-28-115 htdocs]#
- I opened up port 80 & 443 on the server's security group in the aws console's ec2 service
- to simplify debugging, I changed the nginx config to use port 80, not 443.
- selinux had to be disabled to prevent "permission denied" errors as popped up in the logs:
==> /var/log/nginx/error.log <== 2018/04/24 02:57:58 [error] 3101#0: *1 open() "/var/www/html/jitsi.opensourceecology.org/htdocs/index.html" failed (13: Permission denied), client: 76.97.223.185, server: jitsi.opensourceecology.org, request: "GET /index.html HTTP/1.1", host: "ec2-34-210-153-174.us-west-2.compute.amazonaws.com" ==> /var/log/nginx/access.log <== 76.97.223.185 - - [24/Apr/2018:02:57:58 +0000] "GET /index.html HTTP/1.1" 403 180 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-"
- I disabled selinux by setting it to Permissive & restarting nginx
[root@ip-172-31-28-115 jitsi.opensourceecology.org]# getenforce Enforcing [root@ip-172-31-28-115 jitsi.opensourceecology.org]# setenforce Permissive [root@ip-172-31-28-115 jitsi.opensourceecology.org]# service nginx restart Redirecting to /bin/systemctl restart nginx.service [root@ip-172-31-28-115 jitsi.opensourceecology.org]#
- ok, this got a page load, but with an error
Uh oh! We couldn't fully download everything we needed :( We will try again shortly. In the mean time, check for problems with your Internet connection! Missing http://ec2-34-210-153-174.us-west-2.compute.amazonaws.com/libs/lib-jitsi-meet.min.js?v=139 show less reload now
- ah, the "Killed" issue is because linux is running out of memory
[root@ip-172-31-28-115 ~]# tail -n 100 /var/log/messages Apr 24 03:00:35 ip-172-31-28-115 dbus[519]: avc: received setenforce notice (enforcing=0) Apr 24 03:00:35 ip-172-31-28-115 dbus-daemon: dbus[519]: avc: received setenforce notice (enforcing=0) Apr 24 03:00:42 ip-172-31-28-115 systemd: Stopping The nginx HTTP and reverse proxy server... Apr 24 03:00:42 ip-172-31-28-115 systemd: Starting The nginx HTTP and reverse proxy server... Apr 24 03:00:42 ip-172-31-28-115 nginx: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok Apr 24 03:00:42 ip-172-31-28-115 nginx: nginx: configuration file /etc/nginx/nginx.conf test is successful Apr 24 03:00:42 ip-172-31-28-115 systemd: Failed to read PID from file /run/nginx.pid: Invalid argument Apr 24 03:00:42 ip-172-31-28-115 systemd: Started The nginx HTTP and reverse proxy server. Apr 24 03:01:01 ip-172-31-28-115 systemd: Started Session 88 of user root. Apr 24 03:01:01 ip-172-31-28-115 systemd: Starting Session 88 of user root. Apr 24 03:07:57 ip-172-31-28-115 kernel: systemd invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 Apr 24 03:07:57 ip-172-31-28-115 kernel: systemd cpuset=/ mems_allowed=0 Apr 24 03:07:57 ip-172-31-28-115 kernel: CPU: 0 PID: 1 Comm: systemd Not tainted 3.10.0-693.11.6.el7.x86_64 #1 Apr 24 03:07:57 ip-172-31-28-115 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 Apr 24 03:07:57 ip-172-31-28-115 kernel: Call Trace: Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a5ea1>] dump_stack+0x19/0x1b Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a1296>] dump_header+0x90/0x229 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff812b9dfb>] ? cred_has_capability+0x6b/0x120 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff81188094>] oom_kill_process+0x254/0x3d0 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff812b9fde>] ? selinux_capable+0x2e/0x40 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff811888d6>] out_of_memory+0x4b6/0x4f0 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a1d9a>] __alloc_pages_slowpath+0x5d6/0x724 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8118eaa5>] __alloc_pages_nodemask+0x405/0x420 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff811d6075>] alloc_pages_vma+0xb5/0x200 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816a924e>] ? __wait_on_bit+0x7e/0x90 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff811b42a0>] handle_mm_fault+0xb60/0xfa0 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8132f4b3>] ? number.isra.2+0x323/0x360 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816b37e4>] __do_page_fault+0x154/0x450 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816b3b15>] do_page_fault+0x35/0x90 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816af8f8>] page_fault+0x28/0x30 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff81332459>] ? copy_user_enhanced_fast_string+0x9/0x20 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8122854b>] ? seq_read+0x2ab/0x3b0 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8120295c>] vfs_read+0x9c/0x170 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff8120381f>] SyS_read+0x7f/0xe0 Apr 24 03:07:57 ip-172-31-28-115 kernel: [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b Apr 24 03:07:57 ip-172-31-28-115 kernel: Mem-Info: Apr 24 03:07:57 ip-172-31-28-115 kernel: active_anon:214589 inactive_anon:4438 isolated_anon:0#012 active_file:36 inactive_file:783 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:4921 slab_unreclaimable:8167#012 mapped:879 shmem:6524 pagetables:2371 bounce:0#012 free:12232 free_pcp:118 free_cma:0 Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA free:4588kB min:704kB low:880kB high:1056kB active_anon:10344kB inactive_anon:4kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:360kB slab_reclaimable:268kB slab_unreclaimable:476kB kernel_stack:48kB pagetables:140kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Apr 24 03:07:57 ip-172-31-28-115 kernel: lowmem_reserve[]: 0 973 973 973 Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA32 free:44340kB min:44348kB low:55432kB high:66520kB active_anon:848012kB inactive_anon:17748kB active_file:144kB inactive_file:3132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032192kB managed:998676kB mlocked:0kB dirty:0kB writeback:0kB mapped:3516kB shmem:25736kB slab_reclaimable:19416kB slab_unreclaimable:32192kB kernel_stack:2352kB pagetables:9344kB unstable:0kB bounce:0kB free_pcp:472kB local_pcp:472kB free_cma:0kB writeback_tmp:0kB pages_scanned:711 all_unreclaimable? yes Apr 24 03:07:57 ip-172-31-28-115 kernel: lowmem_reserve[]: 0 0 0 0 Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA: 25*4kB (UEM) 19*8kB (UE) 19*16kB (UEM) 12*32kB (UE) 5*64kB (UE) 12*128kB (UEM) 3*256kB (M) 0*512kB 1*1024kB (E) 0*2048kB 0*4096kB = 4588kB Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 DMA32: 1281*4kB (UEM) 1040*8kB (UE) 593*16kB (UEM) 251*32kB (UE) 67*64kB (UE) 41*128kB (UEM) 9*256kB (UEM) 3*512kB (M) 0*1024kB 0*2048kB 0*4096kB = 44340kB Apr 24 03:07:57 ip-172-31-28-115 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Apr 24 03:07:57 ip-172-31-28-115 kernel: 7345 total pagecache pages Apr 24 03:07:57 ip-172-31-28-115 kernel: 0 pages in swap cache Apr 24 03:07:57 ip-172-31-28-115 kernel: Swap cache stats: add 0, delete 0, find 0/0 Apr 24 03:07:57 ip-172-31-28-115 kernel: Free swap = 0kB Apr 24 03:07:57 ip-172-31-28-115 kernel: Total swap = 0kB Apr 24 03:07:57 ip-172-31-28-115 kernel: 262045 pages RAM Apr 24 03:07:57 ip-172-31-28-115 kernel: 0 pages HighMem/MovableOnly Apr 24 03:07:57 ip-172-31-28-115 kernel: 8400 pages reserved Apr 24 03:07:57 ip-172-31-28-115 kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 321] 0 321 8841 803 24 0 0 systemd-journal Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 519] 81 519 24607 167 18 0 -900 dbus-daemon Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 524] 0 524 135883 1481 84 0 0 NetworkManager Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 531] 0 531 6051 82 16 0 0 systemd-logind Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 562] 0 562 28343 3126 58 0 0 dhclient Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1031] 0 1031 22386 260 43 0 0 master Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1033] 89 1033 22429 253 46 0 0 qmgr Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1139] 0 1139 27511 33 12 0 0 agetty Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 1140] 0 1140 27511 33 10 0 0 agetty Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 8587] 0 8587 11692 394 27 0 -1000 systemd-udevd Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9607] 0 9607 28198 256 58 0 -1000 sshd Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9722] 0 9722 13877 111 29 0 -1000 auditd Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9760] 0 9760 58390 946 48 0 0 rsyslogd Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9781] 0 9781 31571 159 18 0 0 crond Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9826] 999 9826 134633 1647 60 0 0 polkitd Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9903] 0 9903 26991 38 9 0 0 rhnsd Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9914] 0 9914 143438 3265 99 0 0 tuned Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 9985] 998 9985 25120 91 20 0 0 chronyd Apr 24 03:07:57 ip-172-31-28-115 kernel: [11651] 0 11651 32007 188 17 0 0 screen Apr 24 03:07:57 ip-172-31-28-115 kernel: [11652] 0 11652 28891 135 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [12264] 1001 12264 1291983 15415 88 0 0 java Apr 24 03:07:57 ip-172-31-28-115 kernel: [12516] 0 12516 47972 155 49 0 0 su Apr 24 03:07:57 ip-172-31-28-115 kernel: [12517] 1001 12517 28859 103 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [12611] 0 12611 28859 105 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [27867] 0 27867 39164 336 80 0 0 sshd Apr 24 03:07:57 ip-172-31-28-115 kernel: [27870] 1000 27870 39361 552 78 0 0 sshd Apr 24 03:07:57 ip-172-31-28-115 kernel: [27871] 1000 27871 28859 95 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [27892] 1000 27892 31908 66 18 0 0 screen Apr 24 03:07:57 ip-172-31-28-115 kernel: [27893] 1000 27893 32113 296 16 0 0 screen Apr 24 03:07:57 ip-172-31-28-115 kernel: [27894] 1000 27894 28859 115 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [30326] 0 30326 54628 261 63 0 0 sudo Apr 24 03:07:57 ip-172-31-28-115 kernel: [30327] 0 30327 47972 155 49 0 0 su Apr 24 03:07:57 ip-172-31-28-115 kernel: [30328] 0 30328 28892 125 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [30369] 1000 30369 28859 105 13 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [30385] 0 30385 54628 261 61 0 0 sudo Apr 24 03:07:57 ip-172-31-28-115 kernel: [30386] 0 30386 47972 155 49 0 0 su Apr 24 03:07:57 ip-172-31-28-115 kernel: [30387] 0 30387 28892 141 14 0 0 bash Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3017] 997 3017 24190 5474 48 0 0 lua Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3164] 0 3164 30201 523 54 0 0 nginx Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3165] 996 3165 30317 649 56 0 0 nginx Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3177] 0 3177 30814 57 15 0 0 anacron Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3180] 89 3180 22428 276 46 0 0 pickup Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3185] 0 3185 27066 60 9 0 0 make Apr 24 03:07:57 ip-172-31-28-115 kernel: [ 3186] 0 3186 469819 174655 759 0 0 node Apr 24 03:07:57 ip-172-31-28-115 kernel: Out of memory: Kill process 3186 (node) score 670 or sacrifice child Apr 24 03:07:57 ip-172-31-28-115 kernel: Killed process 3186 (node) total-vm:1879276kB, anon-rss:698620kB, file-rss:0kB, shmem-rss:0kB Apr 24 03:10:36 ip-172-31-28-115 su: (to root) ec2-user on pts/5 [root@ip-172-31-28-115 ~]#
- looks like we need a bigger box :\ or swap?
[root@ip-172-31-28-115 ~]# free -m total used free shared buff/cache available Mem: 990 179 689 25 121 652 Swap: 0 0 0 [root@ip-172-31-28-115 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda2 10G 3.8G 6.3G 38% / devtmpfs 476M 0 476M 0% /dev tmpfs 496M 0 496M 0% /dev/shm tmpfs 496M 26M 470M 6% /run tmpfs 496M 0 496M 0% /sys/fs/cgroup tmpfs 100M 0 100M 0% /run/user/0 tmpfs 100M 0 100M 0% /run/user/1000 [root@ip-172-31-28-115 ~]#
- we have 1G of RAM and 6G of free disk space. I created a 2G swap file & enabled it
[root@ip-172-31-28-115 ~]# dd if=/dev/zero of=/swap1 bs=1M count=2048 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 30.2187 s, 71.1 MB/s [root@ip-172-31-28-115 ~]# mkswap /swap1 Setting up swapspace version 1, size = 2097148 KiB no label, UUID=a401ef50-ccb8-4bca-abeb-9de5a63b107c [root@ip-172-31-28-115 ~]# chmod 0600 /swap1 [root@ip-172-31-28-115 ~]# swapon /swap1 [root@ip-172-31-28-115 ~]# free -m total used free shared buff/cache available Mem: 990 180 69 25 740 608 Swap: 2047 0 2047 [root@ip-172-31-28-115 ~]#
- the next `make` run didn't get killed by oom, but it had a bunch of complaints of missing modules:
./node_modules/.bin/webpack -p
Hash: 63ec7d579fb8d7f2444256d53b2e2b409f23ea8a
Version: webpack 3.9.1
Child
Hash: 63ec7d579fb8d7f24442
Time: 72401ms
Asset Size Chunks Chunk Names
app.bundle.min.js 317 kB 0 [emitted] [big] app.bundle
dial_in_info_bundle.min.js 116 kB 1 [emitted] dial_in_info_bundle
device_selection_popup_bundle.min.js 101 kB 2 [emitted] device_selection_popup_bundle
do_external_connect.min.js 2 kB 3 [emitted] do_external_connect
alwaysontop.min.js 731 bytes 4 [emitted] alwaysontop
app.bundle.min.map 4.21 MB 0 [emitted] app.bundle
dial_in_info_bundle.min.map 912 kB 1 [emitted] dial_in_info_bundle
device_selection_popup_bundle.min.map 2.11 MB 2 [emitted] device_selection_popup_bundle
do_external_connect.min.map 19.4 kB 3 [emitted] do_external_connect
alwaysontop.min.map 39.2 kB 4 [emitted] alwaysontop
[43] ./react/features/base/config/index.js + 4 modules 13.2 kB {0} {1} {2} [built]
[70] ./react/features/base/config/parseURLParams.js 1.51 kB {0} {1} {2} {3} [built]
[122] ./react/features/base/config/getRoomName.js 761 bytes {0} {1} {2} {3} [built]
[176] ./react/features/base/react/prop-types-polyfill.js 227 bytes {0} {1} {2} [built]
[444] ./modules/settings/Settings.js 6.12 kB {0} [built]
[450] multi babel-polyfill whatwg-fetch ./app.js 52 bytes {0} [built]
[451] ./app.js + 2 modules 4.67 kB {0} [built]
[465] ./modules/UI/UI.js 35 kB {0} [built]
[480] ./react/index.web.js 1.06 kB {0} [built]
[481] ./react/features/device-selection/popup.js 476 bytes {2} [built]
[482] ./react/features/device-selection/DeviceSelectionPopup.js 12.3 kB {2} [built]
[483] ./react/features/always-on-top/index.js + 2 modules 20.3 kB {4} [built]
[484] multi babel-polyfill whatwg-fetch ./react/features/base/react/prop-types-polyfill.js ./react/features/invite/components/dial-in-info-page 64 bytes {1} [built]
[485] ./react/features/invite/components/dial-in-info-page/index.js + 2 modules 4 kB {1} [built]
[486] ./connection_optimization/do_external_connect.js 2.51 kB {3} [built]
+ 472 hidden modules
ERROR in ./react/features/invite/components/AddPeopleDialog.web.js
Module not found: Error: Can't resolve '@atlaskit/avatar' in '/var/www/html/jitsi.opensourceecology.org/htdocs/react/features/invite/components'
@ ./react/features/invite/components/AddPeopleDialog.web.js 15:0-38
@ ./react/features/invite/components/index.js
@ ./react/features/invite/index.js
@ ./react/features/toolbox/components/Toolbox.web.js
@ ./react/features/toolbox/components/index.js
@ ./react/features/toolbox/index.js
@ ./conference.js
@ ./app.js
@ multi babel-polyfill whatwg-fetch ./app.js
ERROR in ./react/features/base/dialog/components/StatelessDialog.web.js
Module not found: Error: Can't resolve '@atlaskit/button' in '/var/www/html/jitsi.opensourceecology.org/htdocs/react/features/base/dialog/components'
@ ./react/features/base/dialog/components/StatelessDialog.web.js 9:0-55
@ ./react/features/base/dialog/components/index.js
@ ./react/features/base/dialog/index.js
@ ./modules/keyboardshortcut/keyboardshortcut.js
@ ./app.js
@ multi babel-polyfill whatwg-fetch ./app.js
...
- the log file above was huge; here's a grep for just the missing module bit
[root@ip-172-31-28-115 htdocs]# grep -i 'module not found' make.log | cut -d' ' -f1-11 | sort -u Module not found: Error: Can't resolve '@atlaskit/avatar' Module not found: Error: Can't resolve '@atlaskit/button' Module not found: Error: Can't resolve '@atlaskit/dropdown-menu' Module not found: Error: Can't resolve '@atlaskit/field-text' Module not found: Error: Can't resolve '@atlaskit/field-text-area' Module not found: Error: Can't resolve '@atlaskit/flag' Module not found: Error: Can't resolve '@atlaskit/icon/glyph/chevron-down' Module not found: Error: Can't resolve '@atlaskit/icon/glyph/editor/info' Module not found: Error: Can't resolve '@atlaskit/icon/glyph/error' Module not found: Error: Can't resolve '@atlaskit/icon/glyph/star' Module not found: Error: Can't resolve '@atlaskit/icon/glyph/star-filled' Module not found: Error: Can't resolve '@atlaskit/icon/glyph/warning' Module not found: Error: Can't resolve '@atlaskit/inline-dialog' Module not found: Error: Can't resolve '@atlaskit/inline-message' Module not found: Error: Can't resolve '@atlaskit/layer-manager' Module not found: Error: Can't resolve '@atlaskit/lozenge' Module not found: Error: Can't resolve '@atlaskit/modal-dialog' Module not found: Error: Can't resolve '@atlaskit/multi-select' Module not found: Error: Can't resolve '@atlaskit/spinner' Module not found: Error: Can't resolve '@atlaskit/tabs' Module not found: Error: Can't resolve '@atlaskit/theme' Module not found: Error: Can't resolve '@atlaskit/tooltip' Module not found: Error: Can't resolve 'autosize' Module not found: Error: Can't resolve 'i18next' Module not found: Error: Can't resolve 'i18next-browser-languagedetector' Module not found: Error: Can't resolve 'i18next-xhr-backend' Module not found: Error: Can't resolve 'jitsi-meet-logger' Module not found: Error: Can't resolve 'jquery' Module not found: Error: Can't resolve 'jquery-contextmenu' Module not found: Error: Can't resolve 'jquery-i18next' Module not found: Error: Can't resolve 'jQuery-Impromptu' Module not found: Error: Can't resolve 'js-md5' Module not found: Error: Can't resolve 'jwt-decode' Module not found: Error: Can't resolve 'lodash' Module not found: Error: Can't resolve 'lodash/debounce' Module not found: Error: Can't resolve 'lodash/throttle' Module not found: Error: Can't resolve 'moment' Module not found: Error: Can't resolve 'moment/locale/bg' Module not found: Error: Can't resolve 'moment/locale/de' Module not found: Error: Can't resolve 'moment/locale/eo' Module not found: Error: Can't resolve 'moment/locale/es' Module not found: Error: Can't resolve 'moment/locale/fr' Module not found: Error: Can't resolve 'moment/locale/hy-am' Module not found: Error: Can't resolve 'moment/locale/it' Module not found: Error: Can't resolve 'moment/locale/nb' Module not found: Error: Can't resolve 'moment/locale/pl' Module not found: Error: Can't resolve 'moment/locale/pt' Module not found: Error: Can't resolve 'moment/locale/pt-br' Module not found: Error: Can't resolve 'moment/locale/ru' Module not found: Error: Can't resolve 'moment/locale/sk' Module not found: Error: Can't resolve 'moment/locale/sl' Module not found: Error: Can't resolve 'moment/locale/sv' Module not found: Error: Can't resolve 'moment/locale/tr' Module not found: Error: Can't resolve 'moment/locale/zh-cn' Module not found: Error: Can't resolve 'postis' Module not found: Error: Can't resolve 'prop-types' Module not found: Error: Can't resolve 'react' Module not found: Error: Can't resolve 'react-dom' Module not found: Error: Can't resolve 'react-i18next' Module not found: Error: Can't resolve 'react-redux' Module not found: Error: Can't resolve 'redux' Module not found: Error: Can't resolve 'redux-thunk' Module not found: Error: Can't resolve 'url-polyfill' Module not found: Error: Can't resolve 'whatwg-fetch' [root@ip-172-31-28-115 htdocs]#
Fri Apr 20, 2018
- I terminated my free-tier ec2 instance last night so I could ensure that it cost $0 before leaving it running. I checked the bill this morning, and I saw ec2 pop into our bill. The total was $0 for 4 hours of RHEL t2.micro instance-hours under monthly free-tier + $0 for 0.040 GB-Mo of EBS General Purpose SSD provisioned storage under monthly free tier. The only other fee I could imagine us getting is for general Datom
a Transfer. Currently we're at 348.233G (mostly from huge files into glacier). All of that is $0 under free tier so far.
- relaunched the instance with higher confidence that it's totally $0 = i-05a5af8b75bb5a0d9
- connected to the new instance over ssh
user@ose:~$ ssh -p 22 -i .ssh/id_rsa.ose ec2-user@ec2-34-210-153-174.us-west-2.compute.amazonaws.com The authenticity of host 'ec2-34-210-153-174.us-west-2.compute.amazonaws.com (34.210.153.174)' can't be established. ECDSA key fingerprint is 89:74:84:57:64:c4:9a:71:fd:8d:9d:22:59:3c:d2:4d. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ec2-34-210-153-174.us-west-2.compute.amazonaws.com,34.210.153.174' (ECDSA) to the list of known hosts. [ec2-user@ip-172-31-28-115 ~]$
- fixed various issues existing jitsi install documentation Jitis
- added nginx to jitsi install documentation
# install it from the repos
yum install -y nginx
# create config file for jitsi.opensourceecology.org
mkdir -p /var/www/html/jitsi.opensourceecology.org/htdocs
cat << EOF > /etc/nginx/conf.d/jitsi.opensourceecology.org
server_names_hash_bucket_size 64;
server {
listen 443;
# tls configuration that is not covered in this guide
# we recommend the use of https://certbot.eff.org/
server_name jitsi.opensourceecology.org;
# set the root
root /var/www/html/jitsi.opensourceecology.org/htdocs;
index index.html;
location ~ ^/([a-zA-Z0-9=\?]+)$ {
rewrite ^/(.*)$ / break;
}
location / {
ssi on;
}
# BOSH
location /http-bind {
proxy_pass http://localhost:5280/http-bind;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
}
}
EOF
- successfully installed maven from source https://xmodulo.com/how-to-install-maven-on-centos.html
wget http://mirror.metrocast.net/apache/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz tar -xzvf apache-maven-*.tar.gz -C /usr/local pushd /usr/local ln -s apache-maven-* maven
- version of maven was verified as 3.5.3
[root@ip-172-31-28-115 local]# /usr/local/maven/bin/mvn -v Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) Maven home: /usr/local/maven Java version: 1.8.0_171, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.171-7.b10.el7.x86_64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: "amd64", family: "unix" [root@ip-172-31-28-115 local]#
Thr Apr 19, 2018
- I'm still waiting for Marcin to validate the staging wiki & send me the first draft of our migration-day test plan google doc
- I still don't want to do backups to s3 yet, as I want to migrate the wiki first so we reduce our backup size by ~12G per day once hetzner1 becomes near-0.
- I checked our usage on dreamhost, we're at 132G. Hopefully that flies under the radar until we finish the wiki migration
- After those blocked items, jitsi is probably my most important task
- I started this item by building a local VM running centos7 http://www.gtlib.gatech.edu/pub/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1708.iso
- I looked into software that's built-on jitsi to see if maybe there was a project that bundled what we need into stable builds designed for centos (because everything produced by jitsi is intended for debian) https://jitsi.org/built-on-jitsi/
- I found an Administrator's guide to "Jitis Video Bridge" for the Rocket Chat project that had instructions for enabling Video Bridge because their video chat uses p2p WebRTC, which won't scale beyond 3 users or so. But then there's a seperate section to "set up yur own Jitsi Video Bridge" which indicates that when you enable the video bridge function in Rocket Chat, it just uses jitsi.org's infrastructure. To run the Video Bridge yourself, they simply link you back to the Jitsi source/docs, which are best for Debian https://rocket.chat/docs/administrator-guides/jitsi-video-bridge/
- The Matrix project appears to be designed for 1:1 calls, not conference calls https://matrix.org/docs/guides/faq.html
- Atlassian's not-free Stride platform apparently uses Jitsi, and it states "Stride video conferencing is free for unlimited teammates." Now, idk how far it scales, but I do know that Atlassian does provide some their products to non-profits for free. I emailed Marcin asking for a scanned copy of our income tax exemption letter for proof so I can ask Atlassian if they offer Stride for free to non-profits https://www.stride.com/conferencing#
- found the quick install guide for debian https://github.com/jitsi/jitsi-meet/blob/master/doc/quick-install.md
- found the server install from source for other OSs https://github.com/jitsi/jitsi-meet/blob/master/doc/manual-install.md
- found the mailing list for jitsi users http://lists.jitsi.org/pipermail/users/
- found the mailing list for jitsi devs http://lists.jitsi.org/pipermail/dev/
- the search function of the lists.jitsi.org site sucks; google is better https://www.google.com/search?q="centos"+site%3Alists.jitsi.org&oq="centos"+site%3Alists.jitsi.org
- I only got 1 result regarding the 'speex' dependency, which isn't in the yum repos; the solution was given http://lists.jitsi.org/pipermail/dev/2017-November/035909.html
- I installed my VM as "Server with GUI"
- The Template VM install failed; apparently CentOS isn't supported, and the guide to using a non-supported OS (written for Arch) shows this process is very non-trivial
- I installed the "Server with GUI" on an HVM, but that didn't come up after rebooting after the install
NMI watchdog: BUG: soft lockup - CPU#X stuck for Ys!
- I installed another HVM as minimal, but it had the same issue!
- I checked our aws account, and we should get 750 hours of t2.micro instances per month. That's enough to run it 24/7. This item expires 12 months from our signup date, so it's use it or loose it. I'm going to use it.
- I added my public key to the aws ec2 service from the aws console & named it 'maltfield'
- I changed the default security group to only allow port 22 TCP inbound. outbound is completely open.
- I launched a t2.micro node (i-065bfc806b4f39923) with this security group
- I used the default 10G EBS, but I confirmed that for 12 months we get 30G of EBS storage for free.
- confirmed that I could log in via ssh
user@ose:~$ ssh -p 22 -i .ssh/id_rsa.ose ec2-user@ec2-52-32-28-0.us-west-2.compute.amazonaws.com Enter passphrase for key '.ssh/id_rsa.ose': user@ose:~$ ssh -p 22 -i .ssh/id_rsa.ose ec2-user@ec2-52-32-28-0.us-west-2.compute.amazonaws.com Last login: Fri Apr 20 00:40:02 2018 from c-76-97-223-185.hsd1.ga.comcast.net [ec2-user@ip-172-31-29-174 ~]$ hostname ip-172-31-29-174.us-west-2.compute.internal [ec2-user@ip-172-31-29-174 ~]$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP qlen 1000 link/ether 02:55:43:fa:bf:b2 brd ff:ff:ff:ff:ff:ff inet 172.31.29.174/20 brd 172.31.31.255 scope global dynamic eth0 valid_lft 3015sec preferred_lft 3015sec inet6 fe80::55:43ff:fefa:bfb2/64 scope link valid_lft forever preferred_lft forever [ec2-user@ip-172-31-29-174 ~]$ date Fri Apr 20 00:40:38 UTC 2018 [ec2-user@ip-172-31-29-174 ~]$ pwd /home/ec2-user [ec2-user@ip-172-31-29-174 ~]$
- created a wiki page titled "Jitsi" here's what I have for the install so farl
# become root
sudo su -
# first, update software
yum update
# install my prereqs
yum install -y vim screen wget unzip
# enable epel
cat << EOF > /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - \$basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/\$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=\$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - \$basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/\$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=\$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-source]
name=Extra Packages for Enterprise Linux 7 - \$basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=\$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
EOF
# and epel key
cat << EOF > /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1.4.11 (GNU/Linux)
mQINBFKuaIQBEAC1UphXwMqCAarPUH/ZsOFslabeTVO2pDk5YnO96f+rgZB7xArB
OSeQk7B90iqSJ85/c72OAn4OXYvT63gfCeXpJs5M7emXkPsNQWWSju99lW+AqSNm
jYWhmRlLRGl0OO7gIwj776dIXvcMNFlzSPj00N2xAqjMbjlnV2n2abAE5gq6VpqP
vFXVyfrVa/ualogDVmf6h2t4Rdpifq8qTHsHFU3xpCz+T6/dGWKGQ42ZQfTaLnDM
jToAsmY0AyevkIbX6iZVtzGvanYpPcWW4X0RDPcpqfFNZk643xI4lsZ+Y2Er9Yu5
S/8x0ly+tmmIokaE0wwbdUu740YTZjCesroYWiRg5zuQ2xfKxJoV5E+Eh+tYwGDJ
n6HfWhRgnudRRwvuJ45ztYVtKulKw8QQpd2STWrcQQDJaRWmnMooX/PATTjCBExB
9dkz38Druvk7IkHMtsIqlkAOQMdsX1d3Tov6BE2XDjIG0zFxLduJGbVwc/6rIc95
T055j36Ez0HrjxdpTGOOHxRqMK5m9flFbaxxtDnS7w77WqzW7HjFrD0VeTx2vnjj
GqchHEQpfDpFOzb8LTFhgYidyRNUflQY35WLOzLNV+pV3eQ3Jg11UFwelSNLqfQf
uFRGc+zcwkNjHh5yPvm9odR1BIfqJ6sKGPGbtPNXo7ERMRypWyRz0zi0twARAQAB
tChGZWRvcmEgRVBFTCAoNykgPGVwZWxAZmVkb3JhcHJvamVjdC5vcmc+iQI4BBMB
AgAiBQJSrmiEAhsPBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAKCRBqL66iNSxk
5cfGD/4spqpsTjtDM7qpytKLHKruZtvuWiqt5RfvT9ww9GUUFMZ4ZZGX4nUXg49q
ixDLayWR8ddG/s5kyOi3C0uX/6inzaYyRg+Bh70brqKUK14F1BrrPi29eaKfG+Gu
MFtXdBG2a7OtPmw3yuKmq9Epv6B0mP6E5KSdvSRSqJWtGcA6wRS/wDzXJENHp5re
9Ism3CYydpy0GLRA5wo4fPB5uLdUhLEUDvh2KK//fMjja3o0L+SNz8N0aDZyn5Ax
CU9RB3EHcTecFgoy5umRj99BZrebR1NO+4gBrivIfdvD4fJNfNBHXwhSH9ACGCNv
HnXVjHQF9iHWApKkRIeh8Fr2n5dtfJEF7SEX8GbX7FbsWo29kXMrVgNqHNyDnfAB
VoPubgQdtJZJkVZAkaHrMu8AytwT62Q4eNqmJI1aWbZQNI5jWYqc6RKuCK6/F99q
thFT9gJO17+yRuL6Uv2/vgzVR1RGdwVLKwlUjGPAjYflpCQwWMAASxiv9uPyYPHc
ErSrbRG0wjIfAR3vus1OSOx3xZHZpXFfmQTsDP7zVROLzV98R3JwFAxJ4/xqeON4
vCPFU6OsT3lWQ8w7il5ohY95wmujfr6lk89kEzJdOTzcn7DBbUru33CQMGKZ3Evt
RjsC7FDbL017qxS+ZVA/HGkyfiu4cpgV8VUnbql5eAZ+1Ll6Dw==
=hdPa
-----END PGP PUBLIC KEY BLOCK-----
EOF
# update again
yum update
#######
# prosody #
# install jitsi prereqs
yum install -y prosody
# configure prosody
mkdir -p /etc/prosody/conf.avail/
cat << EOF > /etc/prosody/conf.avail/jitsi.opensourceecology.org.cfg.lua
VirtualHost "jitsi.opensourceecology.org"
authentication = "anonymous"
ssl = {
key = "/var/lib/prosody/jitsi.opensourceecology.org.key";
certificate = "/var/lib/prosody/jitsi.opensourceecology.org.crt";
}
modules_enabled = {
"bosh";
"pubsub";
}
c2s_require_encryption = false
VirtualHost "auth.jitsi.opensourceecology.org"
ssl = {
key = "/var/lib/prosody/auth.jitsi.opensourceecology.org.key";
certificate = "/var/lib/prosody/auth.jitsi.opensourceecology.org.crt";
}
authentication = "internal_plain"
admins = { "focus@auth.jitsi.opensourceecology.org" }
Component "conference.jitsi.example.com" "muc"
Component "jitsi-videobridge.jitsi.opensourceecology.org"
component_secret = "YOURSECRET1"
Component "focus.jitsi.opensourceecology.org"
component_secret = "YOURSECRET2"
EOF
ln -s /etc/prosody/conf.avail/jitsi.opensourceecology.org.cfg.lua /etc/prosody/conf.d/jitsi.opensourceecology.org.cfg.lua
prosodyctl cert generate jitsi.opensourceecology.org
prosodyctl cert generate auth.jitsi.opensourceecology.org
mkdir -p /usr/local/share/ca-certificates
ln -sf /var/lib/prosody/auth.jitsi.opensourceecology.org.crt /usr/local/share/ca-certificates/auth.jitsi.opensourceecology.org.crt
# this binary doesn't exist; TODO: find out if it's necessary?
update-ca-certificates -f
prosodyctl register focus auth.jitsi.opensourceecology.org YOURSECRET3
#########
# NGINX #
#########
TODO
#####################
# Jitsi Videobridge #
#####################
# install depends
yum install -y java-1.8.0-openjdk
wget https://download.jitsi.org/jitsi-videobridge/linux/jitsi-videobridge-linux-x64-1053.zip
unzip jitsi-videobridge-linux-x64-1053.zip
cat << EOF > /home/ec2-user/.sip-communicator
org.jitsi.impl.neomedia.transform.srtp.SRTPCryptoContext.checkReplay=false
EOF
chown ec2-user:ec2-user /home/ec2-user/.sip-communicator
./jvb.sh --host=localhost --domain=jitsi.opensourceecology.org --port=5347 --secret=YOURSECRET1
- terminated the instance; I'll check to see what the budget says in a couple days to make sure today's testing was totally free
Mon Apr 16, 2018
- fixed a false-positive mod_security issue that Catarina hit while trying to update obi. id = 981318, sqli
- updated my log & the current meeting doc
Sun Apr 15, 2018
- Marcin got back to me pointing out a few differences between the staging & prod wikis:
1. Color on second graph of https://wiki.opensourceecology.org/wiki/Marcin_Log are different than original. Not sure if that matters. 2. Spacing is added after every main heading section in ephemeral. For example, https://wiki.opensourceecology.org/wiki/AbeAnd_Log. Or my log. 3. Error upon saving changes to my log. https://wiki.opensourceecology.org/index.php?title=Marcin_Log&action=submit . See screenshot, Point3.png. 4. I tried 3 again. It works. But Preview doesn't work - see screenshot. When I hit show preview, it gives me an error. See 2 screenshots Point4.png. 5. Picture upload doesn't work - see Point5 which I tried to upload from my log.
- my responses below for each
- (1) graph color
- the color appears to change on every refresh; this appears to be how Lex coded it
- (2) header spacing
- the element picker in the firefox debugger shows that there's an element defined as ".mw-body-content h1" with a "margin-top: 1em;" on our new wiki, but not the old wiki. When I check for the same thing on wikipedia, I see they also have a "margin-top: 1em;"
- I found the actual line to be in htdocs/skins/Vector/components/common.less
- (1) graph color
.mw-body-content {
position: relative;
line-height: @content-line-height;
font-size: @content-font-size;
z-index: 0;
p {
line-height: inherit;
margin: 0.5em 0;
}
h1 {
margin-top: 1em;
}
h2 {
font-size: 1.5em;
margin-top: 1em;
}
- the oldest commit of this file in gerrit (from 2014-08-07) does *not* have this margin, so we should be able to isolate when it was added https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/skins/Vector/+/d28f09df312edbb72b19ac6ac5d124f11007a4ba/components/common.less
- ok, I isolated it to this change on 2015-05-23 https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/skins/Vector/+/35ca341ed4e9acfa00505e216d2416f73c253948%5E%21/#F0
Minor header fixes for Typography Refresh
This fixes top-margin for H1 inside .mw-body-content, and increase font-size for H3 from 1.17em to 1.2em (bumping to 17px default).
Final patch for this bug.
Bug: T66653
- the bug referenced is this one https://phabricator.wikimedia.org/T66653
Change-Id: I1e75bc4fc3e04ca6c9238d4ce116136e9bafacd1
- I recommended to Marcin that we just keep the theme defaults the same as what Wikipedia uses
- (3) Request Entity Too Large
- I confirmed that I got this error when attempting to edit a very long article = my log
- the response I got was an error 413 Request Entity Too Large
Request Entity Too Large The requested resource /index.php does not allow request data with POST requests, or the amount of data provided in the request exceeds the capacity limit.
- at the same time, this popped into my logs
[root@hetzner2 httpd]# tail -f wiki.opensourceecology.org/access_log wiki.opensourceecology.org/error_log error_log ... => wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:18:02.842032 2018] [:error] [pid 32271] [client 127.0.0.1] ModSecurity: Request body no files data length is larger than the configured limit (131072).. Deny with code (413) [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNfGhEne29xOSpIJVtJbQAAAAg"] ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [15/Apr/2018:14:18:02 +0000] "POST /index.php?title=Maltfield_log_2018&action=submit HTTP/1.0" 413 338 "https://wiki.opensourceecology.org/index.php?title=Maltfield_log_2018&action=submit" "Mozilla/5.0 (X11; OpenBSD amd64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36"
- I fixed this by setting the SecRequestBodyNoFilesLimit to 1MB in "/etc/httpd/conf.d/00-wiki.opensourceecology.org.conf"
# disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <IfModule security2_module> SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246 960915 200003 981173 981318 981260 950911 973302 973324 973317 981255 958057 958056 973327 950018 950001 958008 973329 # set the (sans file) POST size limit to 1M (default is 128K) SecRequestBodyNoFilesLimit 1000000 </IfModule>
- but now I got a 403 forbidden false-positive generic attack; it saw "wget" in my own log file :\
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:38:12.390804 2018] [:error] [pid 20103] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:[\\\\;\\\\|\\\\`]\\\\W*?\\\\bcc|\\\\b(wget|curl))\\\\b|\\\\/cc(?:[\\\\'\\"\\\\|\\\\;\\\\`\\\\-\\\\s]|$))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "25"] [id "950907"] [rev "2"] [msg "System Command Injection"] [data "Matched Data: wget found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I checked again just a..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/COMMAND_INJECTION"] [tag "WASCTC/WASC-31"] [tag "OWASP_TOP_10/A1"] [tag "PCI/6.5. [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNj1GTPX8kx9Zp-CaA3zQAAAAM"] ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [15/Apr/2018:14:38:12 +0000] "POST /index.php?title=Maltfield_log_2018&action=submit HTTP/1.0" 403 211 "https://wiki.opensourceecology.org/index.php?title=Maltfield_log_2018&action=submit" "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:48.0) Gecko/20100101 Firefox/48.0"
- I whitelisted "950907", but I got another. I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:39:55.629972 2018] [:error] [pid 20239] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "[\\\\n\\\\r](?:content-(type|length)|set-cookie|location):" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "134"] [id "950910"] [rev "2"] [msg "HTTP Response Splitting Attack"] [data "Matched Data: \\x0acontent-type: found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0amy work log from the year 2018. i intentionally made this verbose to make future admin's work easier when troubleshooting. the more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future ose sysadmin.\\x0d\\x0a\\x0d\\x0a=see also=\\x0d\\x0a# maltfield_log\\x0d\\x0a# user:maltfield\\x0d\\x0a# special:contributions/maltfield\\x0d\\x0a\\x0d\\x0a=sun apr 08, 2018=\\x0d\\x0a# i checked..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNkO8vleiEPKc6tRLX2PgAAAAU"]
- next I got a false-positive from "950005"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:46:24.788412 2018] [:error] [pid 21166] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?:\\\\b(?:\\\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\\\.asa|httpd\\\\.conf|boot\\\\.ini)\\\\b|\\\\/etc\\\\/)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "205"] [id "950005"] [rev "3"] [msg "Remote File Access Attempt"] [data "Matched Data: /etc/ found within ARGS:wpTextbox1: test1 my work log from the year 2018. i intentionally made this verbose to make future admins work easier when troubleshooting. the more keywords error messages etc that are listed in this log the more helpful it will be for the future ose sysadmin. =see also= # maltfield_log # user:maltfield # special:contributions/maltfield =sun apr 08 2018= # i checked again just after midnight the retry appears to have worked pretty great. just 2 archives ..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10 [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNlwEB2obt3oWmzHguGRAAAAAU"]
- next I got a false-positive from "950006"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <==
[Sun Apr 15 14:48:52.180898 2018] [:error] [pid 21264] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?:\\\\b(?:(?:n(?:et(?:\\\\b\\\\W+?\\\\blocalgroup|\\\\.exe)|(?:map|c)\\\\.exe)|t(?:racer(?:oute|t)|elnet\\\\.exe|clsh8?|ftp)|(?:w(?:guest|sh)|rcmd|ftp)\\\\.exe|echo\\\\b\\\\W*?\\\\by+)\\\\b|c(?:md(?:(?:\\\\.exe|32)\\\\b|\\\\b\\\\W*?\\\\/c)|d(?:\\\\b\\\\W*?[\\\\/]|\\\\W*?\\\\.\\\\.)|hmod.{0,40}?\\\\ ..." at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "221"] [id "950006"] [rev "3"] [msg "System Command Injection"] [data "Matched Data: `echo found within ARGS:wpTextbox1: test1 my work log from the year 2018. i intentionally made this verbose to make future admins work easier when troubleshooting. the more keywords error messages etc that are listed in this log the more helpful it will be for the future ose sysadmin. =see also= # maltfield_log # user:maltfield # special:contributions/maltfield =sun apr 08 2018= # i checked again just after midnight the retry appears to have worked pretty great. just 2 archives ..."] [severity [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNmVJDQehGtAwf9cZFxVQAAAAE"]
- next I got a false-positive from "959151"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:50:23.449483 2018] [:error] [pid 21758] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "<\\\\?(?!xml)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "230"] [id "959151"] [rev "2"] [msg "PHP Injection Attack"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/WEB_ATTACK/PHP_INJECTION"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "PCI/6.5.2"] [tag "WASCTC/WASC-25"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE4"] [tag "PCI/6.5.2"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNmr7Npwd4Zn01cn3kYrgAAAAg"]
- next I got a false-positive from "958976"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:51:47.116321 2018] [:error] [pid 21825] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i)(?:\\\\b(?:f(?:tp_(?:nb_)?f?(?:ge|pu)t|get(?:s?s|c)|scanf|write|open|read)|gz(?:(?:encod|writ)e|compress|open|read)|s(?:ession_start|candir)|read(?:(?:gz)?file|dir)|move_uploaded_file|(?:proc_|bz)open|call_user_func)|\\\\$_(?:(?:pos|ge)t|session))\\\\b" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "233"] [id "958976"] [rev "2"] [msg "PHP Injection Attack"] [data "Matched Data: $_GET found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I checked again just ..."] [severity "CRITICAL [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNnAwxMy3-QXaGk8S8DZAAAAAg"]
- next I got a false-positive from "950007"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <==
[Sun Apr 15 14:53:13.101976 2018] [:error] [pid 21880] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:\\\\b(?:(?:s(?:ys\\\\.(?:user_(?:(?:t(?:ab(?:_column|le)|rigger)|object|view)s|c(?:onstraints|atalog))|all_tables|tab)|elect\\\\b.{0,40}\\\\b(?:substring|users?|ascii))|m(?:sys(?:(?:queri|ac)e|relationship|column|object)s|ysql\\\\.(db|user))|c(?:onstraint ..." at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "116"] [id "950007"] [rev "2"] [msg "Blind SQL Injection Attack"] [data "Matched Data: substring found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I checked again j..."] [ [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNnWSnNDDq2JC5w2P06-AAAAAk"]
- next I got a false-positive from "959070"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <==
[Sun Apr 15 14:54:27.832922 2018] [:error] [pid 21954] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "\\\\b(?i:having)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')\\\\s*?[=<>]|(?i:\\\\bexecute(\\\\s{1,5}[\\\\w\\\\.$]{1,5}\\\\s{0,3})?\\\\()|\\\\bhaving\\\\b ?(?:\\\\d{1,10}|[\\\\'\\"][^=]{1,10}[\\\\'\\"]) ?[=<>]+|(?i:\\\\bcreate\\\\s+?table.{0,20}?\\\\()|(?i:\\\\blike\\\\W*?char\\\\W*?\\\\()|(?i:(?:(select(.* ..." at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "130"] [id "959070"] [rev "2"] [msg "SQL Injection Attack"] [data "Matched Data: from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I checked again just after midnight; the retry appears to have worked pretty great. Just 2..."] [severi [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNno1tQGkBXlULnvmvpaAAAAAU"]
- next I got a false-positive from "950908"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:55:32.054195 2018] [:error] [pid 22398] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:\\\\b(?:coalesce\\\\b|root\\\\@))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "140"] [id "950908"] [rev "2"] [msg "SQL Injection Attack."] [data "Matched Data: root@ found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I checked again just ..."] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNn44JlXq26uhw4uNUXCAAAAAU"]
- next I got a false-positive from "981250"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:57:24.386324 2018] [:error] [pid 22465] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:(select|;)\\\\s+(?:benchmark|if|sleep)\\\\s*?\\\\(\\\\s*?\\\\(?\\\\s*?\\\\w+))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "215"] [id "981250"] [msg "Detects SQL benchmark and sleep injection attempts including conditional queries"] [data "Matched Data: ; \\x0d\\x0a\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09\\x09 \\x0d\\x0a\\x09 if (beresp found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier w..."] [severity "CRITICAL"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNoVNnaOuebwb3@ICNKAwAAAAg"]
- next I got a false-positive from "981241"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:58:25.610212 2018] [:error] [pid 22500] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:[\\\\s()]case\\\\s*?\\\\()|(?:\\\\)\\\\s*?like\\\\s*?\\\\()|(?:having\\\\s*?[^\\\\s]+\\\\s*?[^\\\\w\\\\s])|(?:if\\\\s?\\\\([\\\\d\\\\w]\\\\s*?[=<>~]))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "217"] [id "981241"] [msg "Detects conditional SQL injection attempts"] [data "Matched Data: having 'fundraiser' found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I check..."] [severity "CRITICAL"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNokaA9VhPOcZIEe5Ri-AAAAAU"]
- next I got a false-positive from "981252"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 14:59:16.861059 2018] [:error] [pid 22538] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:alter\\\\s*?\\\\w+.*?character\\\\s+set\\\\s+\\\\w+)|([\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98];\\\\s*?waitfor\\\\s+time\\\\s+[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98])|(?:[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98];.*?:\\\\s*?goto))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "219"] [id "981252"] [msg "Detects MySQL charset switch and MSSQL DoS attempts"] [data "Matched Data: alternative. Other supported op code caches are: mmTurck, WinCache, XCache.\\x0d\\x0a\\x0d\\x0aOpcode caches store the compiled output of PHP scripts, greatly reducing the amount of time needed to run a script multiple times. MediaWiki does not need to be configured to do PHP bytecode caching and will \\x22just work\\x22 once installed and enabled them. \\x0d\\x0a</blockquote>\\x0d\\x0a## but we can't use OPcache for the mediawiki caching (ie: message caching) since it is only a opcode cache, not an ..."] [severity "CRITICAL"] [tag "OWA [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNoxMBiBJrTpPno-1d0JgAAAAU"]
- next I got a false-positive from "981256"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:00:06.400039 2018] [:error] [pid 22593] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:merge.*?using\\\\s*?\\\\()|(execute\\\\s*?immediate\\\\s*?[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98])|(?:\\\\W+\\\\d*?\\\\s*?having\\\\s*?[^\\\\s\\\\-])|(?:match\\\\s*?[\\\\w(),+-]+\\\\s*?against\\\\s*?\\\\())" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "221"] [id "981256"] [msg "Detects MATCH AGAINST, MERGE, EXECUTE IMMEDIATE and HAVING injections"] [data "Matched Data: having 7 found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I checked again j..."] [severity "CRITICAL"] [tag "OWASP_CRS/WEB_ [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNo9sgDr4oS7e@QketkjwAAAAU"]
- next I got a false-positive from "981249"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:01:37.432935 2018] [:error] [pid 23080] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\s+and\\\\s*?=\\\\W)|(?:\\\\(\\\\s*?select\\\\s*?\\\\w+\\\\s*?\\\\()|(?:\\\\*\\\\/from)|(?:\\\\+\\\\s*?\\\\d+\\\\s*?\\\\+\\\\s*?@)|(?:\\\\w[\\"'`\\xc2\\xb4\\xe2\\x80\\x99\\xe2\\x80\\x98]\\\\s*?(?:[-+=|@]+\\\\s*?)+[\\\\d(])|(?:coalesce\\\\s*?\\\\(|@@\\\\w+\\\\s*?[ ..." at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "233"] [id "981249"] [msg "Detects chained SQL injection attempts 2/2"] [data "Matched Data: case my gpg/tar manipulations deletes the originals and I don't want to pay to download them from Glacier again!\\x0d\\x0a<pre>\\x0d\\x0a[root@hetzner2 glacier-cli]# mkdir ../orig\\x0d\\x0a[root@hetzner2 glacier-cli]# cp hetzner1_20170901-052001.fileList.txt.bz2.gpg\\x5c:\\x5c this\\x5c is\\x5c a\\x5c metadata\\x5c file\\x5c showing\\x5c the\\x5c file\\x5c and\\x5c dir\\x5c list\\x5c contents\\x5c of\\x5c the\\x5c archive\\x5c of\\x5c the\\x5c same\\x5c prefix\\x5c name ../orig/\\x0d\\x0a[root@hetzner2 glacier-cli]# c. [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNpUcmL3wtkV-YO1ihZDAAAAAU"]
- next I got a false-positive from "981251"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <==
[Sun Apr 15 15:05:19.598269 2018] [:error] [pid 23148] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:create\\\\s+function\\\\s+\\\\w+\\\\s+returns)|(?:;\\\\s*?(?:select|create|rename|truncate|load|alter|delete|update|insert|desc)\\\\s*?[\\\\[(]?\\\\w{2,}))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "241"] [id "981251"] [msg "Detects MySQL UDF injection and other data/structure manipulation attempts"] [data "Matched Data: ;\\x0d\\x0aCREATE USER found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0aMy work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.\\x0d\\x0a\\x0d\\x0a=See Also=\\x0d\\x0a# Maltfield_Log\\x0d\\x0a# User:Maltfield\\x0d\\x0a# Special:Contributions/Maltfield\\x0d\\x0a\\x0d\\x0a=Sun Apr 08, 2018=\\x0d\\x0a# I chec..."] [severity "CRITICAL"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNqL8Ru5tp4rXAvCPJSGgAAAAQ"]
- next I got a false-positive from "973336"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <==
[Sun Apr 15 15:06:20.052955 2018] [:error] [pid 23683] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i)(<script[^>]*>[\\\\s\\\\S]*?<\\\\/script[^>]*>|<script[^>]*>[\\\\s\\\\S]*?<\\\\/script\\\\s\\\\S*[\\\\s\\\\S]|<script[^>]*>[\\\\s\\\\S]*?<\\\\/script[\\\\s]*[\\\\s]|<script[^>]*>[\\\\s\\\\S]*?<\\\\/script|<script[^>]*>[\\\\s\\\\S]*?)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "14"] [id "973336"] [rev "1"] [msg "XSS Filter - Category 1: Script Tag Vector"] [data "Matched Data: <script>\\x0d\\x0a (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){\\x0d\\x0a (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),\\x0d\\x0a m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)\\x0d\\x0a })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');\\x0d\\x0a\\x0d\\x0a ga('create', 'UA-58526017-1', 'auto');\\x0d\\x0a ga('send', 'pageview');\\x0d\\x0a\\x0d\\x0a</script> found within ARGS..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [matu [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNqa27f2r5Ci7rKo19IZgAAAAE"]
- next I got a false-positive from "958006"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:07:37.678507 2018] [:error] [pid 23747] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "<body\\\\b.*?\\\\bbackground\\\\b" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "115"] [id "958006"] [rev "2"] [msg "Cross-site Scripting (XSS) Attack"] [data "Matched Data: <body> <h1>not found</h1> <p\\x22. skipping. all renewal attempts failed. the following certs could not be renewed: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (failure) ------------------------------------------------------------------------------- processing /etc/letsencrypt/renewal/opensourceecology.org.conf ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- proce..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNquZaUhFQqxRu6UgZZrAAAAAE"]
- next I got a false-positive from "958049"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:08:36.257521 2018] [:error] [pid 23804] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "\\\\< ?meta\\\\b" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "169"] [id "958049"] [rev "2"] [msg "Cross-site Scripting (XSS) Attack"] [data "Matched Data: <meta found within ARGS:wpTextbox1: test1 my work log from the year 2018. i intentionally made this verbose to make future admin's work easier when troubleshooting. the more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future ose sysadmin. =see also= # maltfield_log # user:maltfield # special:contributions/maltfield =sun apr 08, 2018= # i checked again just after midnight; the retry appears to have worked pretty great. just 2 arc..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNq8w21sJT73GKTS-bkXgAAAAU"]
- next I got a false-positive from "958051"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:09:43.732355 2018] [:error] [pid 23864] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "\\\\< ?script\\\\b" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "211"] [id "958051"] [rev "2"] [msg "Cross-site Scripting (XSS) Attack"] [data "Matched Data: <script found within ARGS:wpTextbox1: test1 my work log from the year 2018. i intentionally made this verbose to make future admin's work easier when troubleshooting. the more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future ose sysadmin. =see also= # maltfield_log # user:maltfield # special:contributions/maltfield =sun apr 08, 2018= # i checked again just after midnight; the retry appears to have worked pretty great. just 2 a..."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNrN-L3I7mwD9Dr@YNrLAAAAAE"]
- next I got a false-positive from "973305"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:12:17.682638 2018] [:error] [pid 24338] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(asfunction|javascript|vbscript|data|mocha|livescript):" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "351"] [id "973305"] [rev "2"] [msg "XSS Attack Detected"] [data "Matched Data: data: found within ARGS:wpTextbox1: test1myworklogfromtheyear2018.iintentionallymadethisverbosetomakefutureadmin'sworkeasierwhentroubleshooting.themorekeywords,errormessages,etcthatarelistedinthislog,themorehelpfulitwillbeforthefutureosesysadmin.=seealso=#maltfield_log#user:maltfield#special:contributions/maltfield=sunapr08,2018=#icheckedagainjustaftermidnight;theretryappearstohaveworkedprettygreat.just2archivesfailedonthisrun<pre>hancock%datesatapr722:14:46pdt2018hancock%pwd/ho..."] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNr0e4ceIffaD1NC6xG1AAAAAU"]
- next I got a false-positive from "973314"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:13:51.802124 2018] [:error] [pid 24463] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "<!(doctype|entity)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "464"] [id "973314"] [rev "2"] [msg "XSS Attack Detected"] [data "Matched Data: <!doctype found within ARGS:wpTextbox1: test1\\x0d\\x0a\\x0d\\x0amy work log from the year 2018. i intentionally made this verbose to make future admin's work easier when troubleshooting. the more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future ose sysadmin.\\x0d\\x0a\\x0d\\x0a=see also=\\x0d\\x0a# maltfield_log\\x0d\\x0a# user:maltfield\\x0d\\x0a# special:contributions/maltfield\\x0d\\x0a\\x0d\\x0a=sun apr 08, 2018=\\x0d\\x0a# i checked again j..."] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNsL2emkLMGeDo3MiZaagAAAAU"]
- next I got a false-positive from "973331"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:14:45.386955 2018] [:error] [pid 24511] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:<script.*?>)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "472"] [id "973331"] [rev "2"] [msg "IE XSS Filters - Attack Detected."] [data "Matched Data: <script> found within ARGS:wpTextbox1: test1 My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin. =See Also= # Maltfield_Log # User:Maltfield # Special:Contributions/Maltfield =Sun Apr 08, 2018= # I checked again just after midnight; the retry appears to have worked pretty great. Just 2 ..."] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNsZEtxjbRbHLbCcH0csAAAAAU"]
- next I got a false-positive from "973330"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <==
[Sun Apr 15 15:15:55.146051 2018] [:error] [pid 24972] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:<script.*?[ /+\\\\t]*?((src)|(xlink:href)|(href))[ /+\\\\t]*=)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "476"] [id "973330"] [rev "2"] [msg "IE XSS Filters - Attack Detected."] [data "Matched Data: <script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src= found within ARGS:wpTextbox1: test1 My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin..."] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5. [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNsqmLfl9DexEr1vNbnJwAAAAA"]
- next I got a false-positive from "973348"; I whitelisted it too
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:16:58.419220 2018] [:error] [pid 25067] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:<META[ /+\\\\t].*?charset[ /+\\\\t]*=)" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_xss_attacks.conf"] [line "492"] [id "973348"] [rev "2"] [msg "IE XSS Filters - Attack Detected."] [data "Matched Data: <meta charset= found within ARGS:wpTextbox1: test1 My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin. =See Also= # Maltfield_Log # User:Maltfield # Special:Contributions/Maltfield =Sun Apr 08, 2018= # I checked again just after midnight; the retry appears to have worked pretty great. J..."] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A2"] [tag "OWASP_AppSensor/IE1"] [tag "PCI/6.5.1"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNs5z8-aFiK3LAiiWdNlQAAAAU"]
- finally, the edit went through. Here's what the apache conf file looks like now:
[root@hetzner2 conf.d]# cat 00-wiki.opensourceecology.org.conf ... # disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <IfModule security2_module> SecRuleRemoveById 960015 960024 960904 960015 960017 970901 950109 981172 981231 981245 973338 973306 950901 981317 959072 981257 981243 958030 973300 973304 973335 973333 973316 200004 973347 981319 981240 973301 973344 960335 960020 950120 959073 981244 981248 981253 973334 973332 981242 981246 960915 200003 981173 981318 981260 950911 973302 973324 973317 981255 958057 958056 973327 950018 950001 958008 973329 950907 950910 950005 950006 959151 958976 950007 959070 950908 981250 981241 981252 981256 981249 981251 973336 958006 958049 958051 973305 973314 973331 973330 973348 # set the (sans file) POST size limit to 1M (default is 128K) SecRequestBodyNoFilesLimit 1000000 </IfModule> ... [root@hetzner2 conf.d]#
- I tried to edit Marcin's log, but I got another Forbidden; I whitelisted "981276". Then it worked fine
==> wiki.opensourceecology.org/error_log <== [Sun Apr 15 15:21:28.339555 2018] [:error] [pid 25157] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i:(?:(union(.*?)select(.*?)from)))" at ARGS:wpTextbox1. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "225"] [id "981276"] [msg "Looking for basic sql injection. Common attack string for mysql, oracle and others."] [data "Matched Data: Unions in St. Joseph]]. How to Find a Good Local Bank. Seed Eco-Home Utilities. How to Glue PVC and ABS.\\x0d\\x0a\\x0d\\x0a=Tue Sep 12, 2017=\\x0d\\x0aOSE HeroX - The Open Source Microfactory Challenge. Tiny Homes.\\x0d\\x0a=Mon Sep 11, 2017=\\x0d\\x0aComparison of CNC Milling to 3D Printing in Metal. Putin Interviews. Track Construction Set. Unauthorized ACH. \\x0d\\x0a\\x0d\\x0a=Sat Sep 9, 2017=\\x0d\\x0a2\\x22 Universal Axis. [[The Monetary System Visually Explain..."] [severity "CRITICAL"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [hostname "wiki.opensourceecology.org"] [uri "/index.php"] [unique_id "WtNt@GQFLmh587VJM0wsSgAAAAM"]
- I'll ask Marcin to try again
- (3) Chrome ERR_BLOCKED_BY_XSS_AUDITOR on Preview
- I was able to reproduce this issue in Chrome only on Preview of Marcin's log file. A quick google search suggests that it's a bug in Chrome v57 & fixed in v58. I confirmed that I have Chromium v57.0.2987.98.
- I asked Marcin to just use Firefox until the bug in Chromium is fixed
- (4) thumbnail generation error
- so it does appear that the image uploaded https://wiki.opensourceecology.org/images/e/e9/Joehaas.jpg
- but the thumbnail generation is throwing an error
- for some reason this doesn't happen to the image I uploaded https://wiki.opensourceecology.org/wiki/File:NewFile.jpg
Error creating thumbnail: Unable to run external programs, proc_open() is disabled. Error code: 1
- this appears to be because MediaWiki was configured to use image magick for thumbnail generation. I disabled this in LocalSettings.php, and the page refresh showed the thumbnail instead of the error.
# we disable using image magick because we intentionally don't grant php exec() # or proc_open() permissions $wgUseImageMagick = false; #$wgImageMagickConvertCommand = "/usr/bin/convert";
- I confirmed that the thumbnails all exist as files on the system, so at least these thumbnails don't need to be generated at every page load
[root@hetzner2 wiki.opensourceecology.org]# ls -lah htdocs/images/thumb/e/e9/Joehaas.jpg/ total 804K drwxr-xr-x 2 apache apache 4.0K Apr 15 16:11 . drwxrwx--- 36 apache apache 4.0K Apr 15 16:11 .. -rw-r--r-- 1 apache apache 242K Apr 15 16:11 1200px-Joehaas.jpg -rw-r--r-- 1 apache apache 6.2K Apr 15 16:11 120px-Joehaas.jpg -rw-r--r-- 1 apache apache 383K Apr 15 16:11 1600px-Joehaas.jpg -rw-r--r-- 1 apache apache 29K Apr 15 16:11 320px-Joehaas.jpg -rw-r--r-- 1 apache apache 127K Apr 15 16:11 800px-Joehaas.jpg [root@hetzner2 wiki.opensourceecology.org]#
- I sent an email with these findings back to Marcin. I'm still waiting for the test plan.
Thr Apr 12, 2018
- ok, returning to the wiki. last items I changed was to fix the caching to use APCU (CACHE_ACCEL) instead of the db to prevent the cpPosTime cookie from causing varnish to hit-for-pass, which was rendering our varnish cache useless until the change to APCU. Now that's fixed, I need to test that updating a page's content necessarily includes a call to varnish to purge the cache for the given page.
- welll...the wiki site is inaccessible because I moved it out of the docroot to reduce our backup sizes of hetzner2 on dreamhost. The content was super stale anyway, so I'll just follow my guide to do a fresh fork of the site
- I updated the wiki guide to migrating the wiki to use the new tmp dir for the data dumps @ "/usr/home/osemain/noBackup/tmp/" instead of "/usr/home/osemain/tmp/". This prevents the redundant data from being archived int he daily backup.
- The data dump of the wiki took 1 hour to complete on hetzner1.
# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/noBackup/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/w'
dbName_hetzner1='osewiki'
dbUser_hetzner1="${mysqlUser_wiki}"
dbPass_hetzner1="${mysqlPass_wiki}"
# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases --single-transaction | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}
# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
...
/usr/www/users/osemain/w/maintenance/testRunner.ora.sql
real 60m16.755s
user 20m29.404s
sys 1m55.104s
osemain@dedi978:~/noBackup/tmp/backups_for_migration_to_hetzner2/wiki_20180412/current$
- declared variables for this dump, note the timestamp is hardcoded here for future reference & reuse. I also double-checked that mediawiki 1.30.0 is still the latest stable version.
# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180412"
backupDir_hetzner1="/usr/home/osemain/noBackup/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
newMediawikiSourceUrl='https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz'
- fixed an issue with the rsync command in [Mediawiki#migrate_site_from_hetzner1_to_hetzner2]
- discovered that the htdocs/.htaccess file doesn't actually exist; hmm
[root@hetzner2 current]# find /var/www/html/wiki.opensourceecology.org/htdocs/ | grep -i htaccess /var/www/html/wiki.opensourceecology.org/htdocs/images/deleted/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/images/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/archives/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/includes/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/includes/composer/ComposerVendorHtaccessCreator.php /var/www/html/wiki.opensourceecology.org/htdocs/languages/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/serialized/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Widgets/compiled_templates/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/tests/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/tests/qunit/.htaccess /var/www/html/wiki.opensourceecology.org/htdocs/cache/.htaccess [root@hetzner2 current]# [root@hetzner2 current]# find mediawiki-1.30.0 | grep -i htaccess mediawiki-1.30.0/images/.htaccess mediawiki-1.30.0/maintenance/archives/.htaccess mediawiki-1.30.0/maintenance/.htaccess mediawiki-1.30.0/includes/.htaccess mediawiki-1.30.0/includes/composer/ComposerVendorHtaccessCreator.php mediawiki-1.30.0/languages/.htaccess mediawiki-1.30.0/serialized/.htaccess mediawiki-1.30.0/tests/.htaccess mediawiki-1.30.0/tests/qunit/.htaccess mediawiki-1.30.0/cache/.htaccess [root@hetzner2 current]#
- attempting to run the maintenance/update.php script failed
[root@hetzner2 current]# pushd ${docrootDir_hetzner2}/maintenance
/var/www/html/wiki.opensourceecology.org/htdocs/maintenance /var/tmp/backups_for_migration_from_hetzner1/wiki_20180412/current /var/www/html
[root@hetzner2 maintenance]# php update.php
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674
PHP Notice: Undefined index: HTTP_USER_AGENT in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
MediaWiki 1.30.0 Updater
Your composer.lock file is up to date with current dependencies!
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693
Set $wgShowExceptionDetails = true; and $wgShowDBErrorBacktrace = true; at the bottom of LocalSettings.php to show detailed debugging information.
[root@hetzner2 maintenance]#
- but when I attempt to load the page, I get the following response
<!DOCTYPE html>
<html><head><title>Internal error - Open Source Ecology</title><style>body { font-family: sans-serif; margin: 0; padding: 0.5em 2em; }</style></head><body>
<div class="errorbox">[Ws@Ikz3LkjSWMQ6sfECfMQAAAAo] 2018-04-12 16:25:55: Fatal exception of type MWException</div>
<!-- Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. --></body></html>
- oh, duh, I left the dbPass_hetzner2 at "CHANGEME" I re-did the DB commands with this fixed, and then tried again
- but I wanted to first make sure that I deleting the db also deleted the associated users, so before I deleted the db I did a dump of the users & the DBs they have access to
[root@hetzner2 sites-enabled]# mysql -uroot -p${mysqlPass} mysql -sNe "select Host,Db,User from db;"
% test
% test\\_%
127.0.0.1 cacti_db cacti_user
localhost cacti_db cacti_user
localhost fef_db fef_user
localhost obi2_db obi2_user
localhost obi3_db obi3_user
localhost obi_db obi_user
localhost obi_staging_db obi_staging_user
localhost oseforum_db oseforum_user
localhost osemain_db osemain_user
localhost osemain_s_db osemain_s_user
localhost osewiki_db osewiki_user
localhost oswh_db oswh_user
localhost piwik_obi_db piwik_obi_user
localhost seedhome_db seedhome_user
[root@hetzner2 sites-enabled]#
- then I dropped the db
[root@hetzner2 current]# time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
real 0m0.165s
user 0m0.004s
sys 0m0.000s
[root@hetzner2 current]#
- then I checked again
[root@hetzner2 sites-enabled]# mysql -uroot -p${mysqlPass} mysql -sNe "select Host,Db,User from db;"
% test
% test\\_%
127.0.0.1 cacti_db cacti_user
localhost cacti_db cacti_user
localhost fef_db fef_user
localhost obi2_db obi2_user
localhost obi3_db obi3_user
localhost obi_db obi_user
localhost obi_staging_db obi_staging_user
localhost oseforum_db oseforum_user
localhost osemain_db osemain_user
localhost osemain_s_db osemain_s_user
localhost osewiki_db osewiki_user
localhost oswh_db oswh_user
localhost piwik_obi_db piwik_obi_user
localhost seedhome_db seedhome_user
[root@hetzner2 sites-enabled]#
- well that sucks; the user is still there! This concurs with their documentation https://dev.mysql.com/doc/refman/5.7/en/drop-database.html
Important: When a database is dropped, privileges granted specifically for the database are not automatically dropped. They must be dropped manually. See Section 13.7.1.4, “GRANT Syntax”.
- so here's the user:
[root@hetzner2 sites-enabled]# mysql -uroot -p${mysqlPass} mysql -sNe "select Host,Db,User from db where db = 'osewiki_db';"
localhost osewiki_db osewiki_user
[root@hetzner2 sites-enabled]#
- I had issues dropping the user, but the REVOKE worked.
[root@hetzner2 sites-enabled]# mysql -uroot -p${mysqlPass} mysql -sNe "REVOKE ALL PRIVILEGES ON osewiki_db.* FROM 'osewiki_user'@'localhost'; DROP USER 'osewiki_db'@'localhost'; FLUSH PRIVILEGES;"
ERROR 1396 (HY000) at line 1: Operation DROP USER failed for 'osewiki_db'@'localhost'
[root@hetzner2 sites-enabled]# mysql -uroot -p${mysqlPass} mysql -sNe "select Host,Db,User from db where db = 'osewiki_db';"
[root@hetzner2 sites-enabled]#
[root@hetzner2 sites-enabled]# mysql -uroot -p${mysqlPass} mysql -sNe "select Host,Db,User from db;"
% test
% test\\_%
127.0.0.1 cacti_db cacti_user
localhost cacti_db cacti_user
localhost fef_db fef_user
localhost obi2_db obi2_user
localhost obi3_db obi3_user
localhost obi_db obi_user
localhost obi_staging_db obi_staging_user
localhost oseforum_db oseforum_user
localhost osemain_db osemain_user
localhost osemain_s_db osemain_s_user
localhost oswh_db oswh_user
localhost piwik_obi_db piwik_obi_user
localhost seedhome_db seedhome_user
[root@hetzner2 sites-enabled]#
- ok, I created the db & user again.
[root@hetzner2 current]# time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
real 0m0.004s
user 0m0.000s
sys 0m0.003s
[root@hetzner2 current]# time nice mysql -uroot -p${mysqlPass} < "db.sql"
real 2m18.618s
user 0m9.201s
sys 0m0.429s
[root@hetzner2 current]# time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT, INSERT, UPDATE, DELETE ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
real 0m0.004s
user 0m0.002s
sys 0m0.001s
[root@hetzner2 current]#
- I ran the maintenance/update.php script again; this time it did something
[root@hetzner2 current]# pushd ${docrootDir_hetzner2}/maintenance
/var/www/html/wiki.opensourceecology.org/htdocs/maintenance /var/tmp/backups_for_migration_from_hetzner1/wiki_20180412/current /var/www/html
[root@hetzner2 maintenance]# php update.php
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674
PHP Notice: Undefined index: HTTP_USER_AGENT in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 5
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
MediaWiki 1.30.0 Updater
Your composer.lock file is up to date with current dependencies!
PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693
Going to run database updates for osewiki_db-wiki_
Depending on the size of your database this may take a while!
Abort with control-c in the next five seconds (skip this countdown with --quick) ... 0
Turning off Content Handler DB fields for this part of upgrade.
...have ipb_id field in ipblocks table.
...have ipb_expiry field in ipblocks table.
...already have interwiki table
...indexes seem up to 20031107 standards.
...have rc_type field in recentchanges table.
...index new_name_timestamp already set on recentchanges table.
...have user_real_name field in user table.
...querycache table already exists.
...objectcache table already exists.
...categorylinks table already exists.
...have pagelinks; skipping old links table updates
...il_from OK
...have rc_ip field in recentchanges table.
...index PRIMARY already set on image table.
...have rc_id field in recentchanges table.
...have rc_patrolled field in recentchanges table.
...logging table already exists.
...have user_token field in user table.
...have wl_notificationtimestamp field in watchlist table.
...watchlist talk page rows already present.
...user table does not contain user_emailauthenticationtimestamp field.
...page table already exists.
...have log_params field in logging table.
...logging table has correct log_title encoding.
...have ar_rev_id field in archive table.
...have page_len field in page table.
...revision table does not contain inverse_timestamp field.
...have rev_text_id field in revision table.
...have rev_deleted field in revision table.
...have img_width field in image table.
...have img_metadata field in image table.
...have user_email_token field in user table.
...have ar_text_id field in archive table.
...page_namespace is already a full int (int(11)).
...ar_namespace is already a full int (int(11)).
...rc_namespace is already a full int (int(11)).
...wl_namespace is already a full int (int(11)).
...qc_namespace is already a full int (int(11)).
...log_namespace is already a full int (int(11)).
...have img_media_type field in image table.
...already have pagelinks table.
...image table does not contain img_type field.
...already have unique user_name index.
...user_groups table exists and is in current format.
...have ss_total_pages field in site_stats table.
...user_newtalk table already exists.
...transcache table already exists.
...have iw_trans field in interwiki table.
...wl_notificationtimestamp is already nullable.
...index times already set on logging table.
...have ipb_range_start field in ipblocks table.
...no page_random rows needed to be set
...have user_registration field in user table.
...templatelinks table already exists
...externallinks table already exists.
...job table already exists.
...have ss_images field in site_stats table.
...langlinks table already exists.
...querycache_info table already exists.
...filearchive table already exists.
...have ipb_anon_only field in ipblocks table.
...index rc_ns_usertext already set on recentchanges table.
...index rc_user_text already set on recentchanges table.
...have user_newpass_time field in user table.
...redirect table already exists.
...querycachetwo table already exists.
...have ipb_enable_autoblock field in ipblocks table.
...index pl_namespace on table pagelinks includes field pl_from.
...index tl_namespace on table templatelinks includes field tl_from.
...index il_to on table imagelinks includes field il_from.
...have rc_old_len field in recentchanges table.
...have user_editcount field in user table.
...page_restrictions table already exists.
...have log_id field in logging table.
...have rev_parent_id field in revision table.
...have pr_id field in page_restrictions table.
...have rev_len field in revision table.
...have rc_deleted field in recentchanges table.
...have log_deleted field in logging table.
...have ar_deleted field in archive table.
...have ipb_deleted field in ipblocks table.
...have fa_deleted field in filearchive table.
...have ar_len field in archive table.
...have ipb_block_email field in ipblocks table.
...index cl_sortkey on table categorylinks includes field cl_from.
...have oi_metadata field in oldimage table.
...index usertext_timestamp already set on archive table.
...index img_usertext_timestamp already set on image table.
...index oi_usertext_timestamp already set on oldimage table.
...have ar_page_id field in archive table.
...have img_sha1 field in image table.
...protected_titles table already exists.
...have ipb_by_text field in ipblocks table.
...page_props table already exists.
...updatelog table already exists.
...category table already exists.
...category table already populated.
...have ar_parent_id field in archive table.
...have user_last_timestamp field in user_newtalk table.
...protected_titles table has correct pt_title encoding.
...have ss_active_users field in site_stats table.
...ss_active_users user count set...
...have ipb_allow_usertalk field in ipblocks table.
...change_tag table already exists.
...tag_summary table already exists.
...valid_tag table already exists.
...user_properties table already exists.
...log_search table already exists.
...have log_user_text field in logging table.
...l10n_cache table already exists.
...index change_tag_rc_tag already set on change_tag table.
...have rd_interwiki field in redirect table.
...transcache tc_time already converted.
...*_mime_minor fields are already long enough.
...iwlinks table already exists.
...index iwl_prefix_title_from already set on iwlinks table.
...have ul_value field in updatelog table.
...have iw_api field in interwiki table.
...iwl_prefix key doesn't exist.
...have cl_collation field in categorylinks table.
...categorylinks up-to-date.
...module_deps table already exists.
...ar_page_revid key doesn't exist.
...index ar_revid already set on archive table.
...ll_lang is up-to-date.
...user_last_timestamp is already nullable.
...index user_email already set on user table.
...up_property in table user_properties already modified by patch patch-up_property.sql.
...uploadstash table already exists.
...user_former_groups table already exists.
...index type_action already set on logging table.
...have rev_sha1 field in revision table.
...batch conversion of user_options: nothing to migrate. done.
...user table does not contain user_options field.
...have ar_sha1 field in archive table.
...index page_redirect_namespace_len already set on page table.
...have us_chunk_inx field in uploadstash table.
...have job_timestamp field in job table.
...index page_user_timestamp already set on revision table.
...have ipb_parent_block_id field in ipblocks table.
...index ipb_parent_block_id already set on ipblocks table.
...category table does not contain cat_hidden field.
...have rev_content_format field in revision table.
...have rev_content_model field in revision table.
...have ar_content_format field in archive table.
...have ar_content_model field in archive table.
...have page_content_model field in page table.
Content Handler DB fields should be usable now.
...site_stats table does not contain ss_admins field.
...recentchanges table does not contain rc_moved_to_title field.
...sites table already exists.
...have fa_sha1 field in filearchive table.
...have job_token field in job table.
...have job_attempts field in job table.
...have us_props field in uploadstash table.
...ug_group in table user_groups already modified by patch patch-ug_group-length-increase-255.sql.
...ufg_group in table user_former_groups already modified by patch patch-ufg_group-length-increase-255.sql.
...index pp_propname_page already set on page_props table.
...index img_media_mime already set on image table.
...iwl_prefix_title_from index is already non-UNIQUE.
...index iwl_prefix_from_title already set on iwlinks table.
...have ar_id field in archive table.
...have el_id field in externallinks table.
...have rc_source field in recentchanges table.
...index log_user_text_type_time already set on logging table.
...index log_user_text_time already set on logging table.
...have page_links_updated field in page table.
...have user_password_expires field in user table.
...have pp_sortkey field in page_props table.
...recentchanges table does not contain rc_cur_time field.
...index wl_user_notificationtimestamp already set on watchlist table.
...have page_lang field in page table.
...have pl_from_namespace field in pagelinks table.
...have tl_from_namespace field in templatelinks table.
...have il_from_namespace field in imagelinks table.
...img_major_mime in table image already modified by patch patch-img_major_mime-chemical.sql.
...oi_major_mime in table oldimage already modified by patch patch-oi_major_mime-chemical.sql.
...fa_major_mime in table filearchive already modified by patch patch-fa_major_mime-chemical.sql.
Extending edit summary lengths (and setting defaults) ...Set $wgShowExceptionDetails = true; and $wgShowDBErrorBacktrace = true; at the bottom of LocalSettings.php to show detailed debugging information.
[root@hetzner2 maintenance]#
- but I'm still getting an error when trying to load it
user@personal:~$ curl -i "https://wiki.opensourceecology.org/"
HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Thu, 12 Apr 2018 17:05:38 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 421
Connection: keep-alive
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Varnish: 98392 32786
Age: 86
Via: 1.1 varnish-v4
<!DOCTYPE html>
<html><head><title>Internal error - Open Source Ecology</title><style>body { font-family: sans-serif; margin: 0; padding: 0.5em 2em; }</style></head><body>
<div class="errorbox">[Ws@RjNRic8mf4rYA2bqP2AAAAAg] 2018-04-12 17:04:12: Fatal exception of type MWException</div>
<!-- Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information. --></body></html>
user@personal:~$
- unfortunately 'wiki-error.log' is not showing any content. So I took the error's advice & added $wgShowExceptionDetails, even though this will leak the error to the user :(
- silly, there was still no content sent to 'wiki-error.log', but the curl gave me better info
user@personal:~$ curl -i "https://wiki.opensourceecology.org/"
HTTP/1.1 500 Internal Server Error
Server: nginx
Date: Thu, 12 Apr 2018 17:10:01 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 3184
Connection: keep-alive
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Varnish: 196610 65728
Age: 55
Via: 1.1 varnish-v4
<!DOCTYPE html>
<html><head><title>Internal error - Open Source Ecology</title><style>body { font-family: sans-serif; margin: 0; padding: 0.5em 2em; }</style></head><body>
<p>[Ws@SsmBHAg3J1XRaFStUtgAAAAQ] / MWException from line 108 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/localisation/LCStoreCDB.php: Unable to open CDB file "/var/www/html/wiki.opensourceecology.org/htdocs/../cache/l10n_cache-en.cdb.tmp.956119238" for write.</p><p>Backtrace:</p><p>#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/localisation/LocalisationCache.php(1013): LCStoreCDB->startWrite(string)<br />
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/localisation/LocalisationCache.php(459): LocalisationCache->recache(string)<br />
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/localisation/LocalisationCache.php(376): LocalisationCache->initLanguage(string)<br />
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/localisation/LocalisationCache.php(291): LocalisationCache->loadSubitem(string, string, string)<br />
#4 /var/www/html/wiki.opensourceecology.org/htdocs/languages/Language.php(2587): LocalisationCache->getSubitem(string, string, string)<br />
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(933): Language->getMessage(string)<br />
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(LanguageEn, string, boolean, array)<br />
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(LanguageEn, string, boolean)<br />
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get(string, boolean, LanguageEn)<br />
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()<br />
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString(string)<br />
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/title/MalformedTitleException.php(49): Message->text()<br />
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/title/MediaWikiTitleCodec.php(311): MalformedTitleException->__construct(string, string)<br />
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(3526): MediaWikiTitleCodec->splitTitleString(string, integer)<br />
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Title.php(361): Title->secureAndSplit()<br />
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(84): Title::newFromURL(NULL)<br />
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(140): MediaWiki->parseTitle()<br />
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(767): MediaWiki->getTitle()<br />
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()<br />
#19 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()<br />
#20 {main}</p>
</body></html>
user@personal:~$
- ok, so this is an issue with the 'cache' dir outside the docroot, which is needed for storing interface messages to files per Aaron Schulz's guide. I actually already created this, but the permissions were wrong! I updated the documentation in the migration guide to include creating this dir & set its permissions correctly
[root@hetzner2 wiki.opensourceecology.org]# ls -lah /var/www/html/wiki.opensourceecology.org/cache
total 1.1M
d---r-x--- 2 not-apache apache 4.0K Mar 16 23:55 .
d---r-x--- 4 not-apache apache 4.0K Apr 12 17:08 ..
----r----- 1 not-apache apache 1.1M Mar 16 23:55 l10n_cache-en.cdb
[root@hetzner2 wiki.opensourceecology.org]# [ -d "${vhostDir_hetzner2}/cache" ] || mkdir "${vhostDir_hetzner2}/cache"
[root@hetzner2 wiki.opensourceecology.org]# chown -R apache:apache "${vhostDir_hetzner2}/cache"
[root@hetzner2 wiki.opensourceecology.org]# find "${vhostDir_hetzner2}/cache" -type f -exec chmod 0660 {} \;
[root@hetzner2 wiki.opensourceecology.org]# find "${vhostDir_hetzner2}/cache" -type d -exec chmod 0770 {} \;
[root@hetzner2 wiki.opensourceecology.org]#
[root@hetzner2 wiki.opensourceecology.org]#
[root@hetzner2 wiki.opensourceecology.org]#
[root@hetzner2 wiki.opensourceecology.org]# ls -lah /var/www/html/wiki.opensourceecology.org/cache
total 1.1M
drwxrwx--- 2 apache apache 4.0K Mar 16 23:55 .
d---r-x--- 4 not-apache apache 4.0K Apr 12 17:08 ..
-rw-rw---- 1 apache apache 1.1M Mar 16 23:55 l10n_cache-en.cdb
[root@hetzner2 wiki.opensourceecology.org]#
- I manually purged the varnish cache, reloaded, and it worked!
[root@hetzner2 wiki.opensourceecology.org]# varnishadm 'ban req.url ~ "."' [root@hetzner2 wiki.opensourceecology.org]# user@personal:~$ curl -i "https://wiki.opensourceecology.org/" HTTP/1.1 301 Moved Permanently Server: nginx Date: Thu, 12 Apr 2018 17:28:07 GMT Content-Type: text/html; charset=utf-8 Content-Length: 0 Connection: keep-alive X-Content-Type-Options: nosniff Vary: Accept-Encoding,Cookie Cache-Control: s-maxage=1200, must-revalidate, max-age=0 Last-Modified: Thu, 12 Apr 2018 17:28:07 GMT Location: https://wiki.opensourceecology.org/wiki/Main_Page X-XSS-Protection: 1; mode=block X-Varnish: 625 Age: 0 Via: 1.1 varnish-v4 Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" user@personal:~$ curl -i "https://wiki.opensourceecology.org/wiki/Main_Page" ... </body> </html> user@personal:~$
- ...but when I went to login, I got an error:
[Ws@XjG9Z0eot@07Oyosq6gAAAAc] 2018-04-12 17:29:48: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- I apparently encountered this in the past, but the issue was that I needed to fix an ini_set that I mangled with a sed & re-run the maintenance scripts [Maltfield_log_2018#Tue_Feb_27.2C_2018]
- I gave the maintenance scripts another 2x taps, cleared the varnish cache, and tried to login again; I got the same error
[Ws@Zy9Ric8mf4rYA2bqQDgAAAAg] cu-04-12 17:39:23: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- I noticed that the 'wiki-error.log' file was populating with output from the update.php run, and it looks like the issue was that the db user doesn't have the ALTER permission to the db
[error] [75060eb56a79742c1c46e7a5] [no req] ErrorException from line 1507 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php: PHP Notice: Undefined index: SERVER_NAME
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(1507): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/db/MWLBFactory.php(60): wfHostname()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/ServiceWiring.php(54): MWLBFactory::applyDefaultConfig(array, GlobalVarConfig, ConfiguredReadOnlyMode)
#3 [internal function]: MediaWiki\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/services/ServiceContainer.php(361): call_user_func_array(Closure, array)
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/services/ServiceContainer.php(344): MediaWiki\Services\ServiceContainer->createService(string)
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWikiServices.php(503): MediaWiki\Services\ServiceContainer->getService(string)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(664): MediaWiki\MediaWikiServices->getDBLoadBalancerFactory()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(79): require_once(string)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(249): require_once(string)
#10 {main}
IP: 127.0.0.1
Start command line script update.php
[caches] cluster: APCBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: APCBagOStuff, session: APCBagOStuff
[caches] LocalisationCache: using store LCStoreNull
[error] [75060eb56a79742c1c46e7a5] [no req] ErrorException from line 1507 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php: PHP Notice: Undefined index: SERVER_NAME
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(1507): MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php(565): wfHostname()
#2 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(89): Maintenance->setAgentAndTriggers()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(249): require_once(string)
#4 {main}
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": false,
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[error] [75060eb56a79742c1c46e7a5] [no req] ErrorException from line 693 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php: PHP Warning: ini_set() has been disabled for security reasons
#0 [internal function]: MWExceptionHandler::handleError(integer, string, string, integer, array)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(693): ini_set(string, string)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DatabaseMysqlBase.php(129): Wikimedia\Rdbms\Database->installErrorHandler()
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(285): Wikimedia\Rdbms\DatabaseMysqlBase->open(string, string, string, string)
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DatabaseMysqlBase.php(102): Wikimedia\Rdbms\Database->__construct(array)
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(415): Wikimedia\Rdbms\DatabaseMysqlBase->__construct(array)
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/loadbalancer/LoadBalancer.php(985): Wikimedia\Rdbms\Database::factory(string, array)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/loadbalancer/LoadBalancer.php(801): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(array, boolean)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/loadbalancer/LoadBalancer.php(667): Wikimedia\Rdbms\LoadBalancer->openConnection(integer, boolean, integer)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php(2858): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, boolean)
#10 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php(1253): wfGetDB(integer, array, boolean)
#11 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(146): Maintenance->getDB(integer)
#12 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(92): UpdateMediaWiki->execute()
#13 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(249): require_once(string)
#14 {main}
[DBConnection] Connected to database 0 at 'localhost'.
[DBQuery] SQL ERROR: ALTER command denied to user 'osewiki_user'@'localhost' for table 'wiki_revision' (localhost)
[exception] [75060eb56a79742c1c46e7a5] [no req] Wikimedia\Rdbms\DBQueryError from line 1149 of /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading?
Query: ALTER TABLE `wiki_revision` MODIFY rev_comment varbinary(767) NOT NULL default ''
Function: Wikimedia\Rdbms\Database::sourceFile( /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/archives/patch-editsummary-length.sql )
Error: 1142 ALTER command denied to user 'osewiki_user'@'localhost' for table 'wiki_revision' (localhost)
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(979): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(3325): Wikimedia\Rdbms\Database->query(string, string)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(3274): Wikimedia\Rdbms\Database->sourceStream(unknown type, NULL, NULL, string, NULL)
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(673): Wikimedia\Rdbms\Database->sourceFile(string)
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/MysqlUpdater.php(1194): DatabaseUpdater->applyPatch(string, boolean, string)
#5 [internal function]: MysqlUpdater->doExtendCommentLengths()
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(472): call_user_func_array(array, array)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/installer/DatabaseUpdater.php(436): DatabaseUpdater->runUpdates(array, boolean)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(204): DatabaseUpdater->doUpdates(array)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/doMaintenance.php(92): UpdateMediaWiki->execute()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/update.php(249): require_once(string)
#11 {main}
[DBConnection] Closing connection to database 'localhost'.
- this issue was somewhat anticipated as it was described in the Mediawiki Security docs https://www.mediawiki.org/wiki/Manual:Security#General_MySQL_and_MariaDB_recommendations
- I created a new db "superuser" & granted it all permissions. note that I had to use "osewiki_superusr" instead of "osewiki_superuser" due to string lenth limits of the username *sigh*
[root@hetzner2 maintenance]# dbSuperUser_hetzner2="osewiki_superuser"
[root@hetzner2 maintenance]# dbSuperPass_hetzner2="CHANGEME"
[root@hetzner2 maintenance]# time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbSuperUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbSuperPass_hetzner2}'; FLUSH PRIVILEGES;"
ERROR 1470 (HY000) at line 1: String 'osewiki_superuser' is too long for user name (should be no longer than 16)
real 0m0.004s
user 0m0.002s
sys 0m0.002s
[root@hetzner2 maintenance]#
[root@hetzner2 maintenance]# dbSuperUser_hetzner2="osewiki_superusr"
[root@hetzner2 maintenance]# time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbSuperUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbSuperPass_hetzner2}'; FLUSH PRIVILEGES;"
real 0m0.004s
user 0m0.000s
sys 0m0.003s
[root@hetzner2 maintenance]#
- I ran the maintenance script again, giving it a distinct db user's credentials via arguments. It worked.
[root@hetzner2 maintenance]# php update.php --dbuser "${dbSuperUser_hetzner2}" --dbpass "${dbSuperPass_hetzner2}"
...
Attempted to insert 685 IP revisions, 685 actually done.
Set the local repo temp zone container to be private.
Purging caches...done.
Done in 26 s.
[root@hetzner2 maintenance]#
- I refreshed the login attempt, and I logged-in successfully. I also updated the documentation to include these arguments in the call to execute update.php.
- made an edit to a page, and it appeared to work fine. Then I went to a distinct ephemeral browser, loaded the page, and the edit didn't show.
- I got a command to check for varnish purges; I tested it by triggering a purge from the wordpress page on osemain
[root@hetzner2 ~]# varnishlog | grep -EC20 "ReqMethod\s*PURGE" - Timestamp Process: 1523562967.680578 0.341190 0.000040 - Debug "RES_MODE 4" - RespHeader Connection: close - Timestamp Resp: 1523562967.691852 0.352464 0.011274 - Debug "XXX REF 2" - ReqAcct 339 0 339 393 91587 91980 - End * << Session >> 99573 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 57242 127.0.0.1:6081 127.0.0.1 6081 1523562967.339361 12 - Link req 99574 rxreq - SessClose RESP_CLOSE 0.353 - End * << Request >> 329175 - Begin req 329174 rxreq - Timestamp Start: 1523562967.712793 0.000000 0.000000 - Timestamp Req: 1523562967.712793 0.000000 0.000000 - ReqStart 127.0.0.1 57244 - ReqMethod PURGE - ReqURL /.* - ReqProtocol HTTP/1.1 - ReqHeader User-Agent: WordPress/4.9.4; https://www.opensourceecology.org - ReqHeader Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5 - ReqHeader host: www.opensourceecology.org - ReqHeader X-VC-Purge-Method: regex - ReqHeader X-VC-Purge-Host: www.opensourceecology.org - ReqHeader Connection: Close - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1, 127.0.0.1 - ReqHeader X-VC-My-Purge-Key: JOaSAn72IJzrykJp1pfEWaECvUU8KvxZJnxSue3repId3qV8wJOHexjtuhi9r6Wv4FH9y9eFfiMjXX6hvxRrVOEWr2IaBVZMZ7ToEz8nLFdRyjyMkUGMANd6MHOzxiTJ - ReqHeader X-VC-Purge-Key-Auth: false - VCL_acl MATCH purge "localhost" - Debug "VCL_error(200, Purged /.* www.opensourceecology.org)" - VCL_return synth - ReqUnset Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5 - ReqHeader Accept-Encoding: gzip - VCL_call HASH
- I just noticed the "X-VC-My-Purge-Key", in my logs, which matches /etc/varnish/secret. Well, I already logged it on the public internet so it's not a secret anymore. Our purges *should* work by ACL limited to IP address, so I went ahead and changed the contents of /etc/varnish/secret to a new, random 128 character string & restarted varnish
- I went to update this purge key in wordpress, but I didn't see it set anywhere
- I updated a minor change to the workshops page & saved it. My browser showed the change, but a distinct/epehermal browser refresh did not show it. I triggered a purge of the page from the wp wui, and I got a log in my grep to pop-up. I refreshed it in the distinct/ephemeral browser, and the change was now visible. That confirms that purging still works despite the change the the purge key. Note that the output below still shows the old purge key. *shrug*
* << Request >> 458860 - Begin req 458859 rxreq - Timestamp Start: 1523563588.895393 0.000000 0.000000 - Timestamp Req: 1523563588.895393 0.000000 0.000000 - ReqStart 127.0.0.1 59502 - ReqMethod PURGE - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader User-Agent: WordPress/4.9.4; https://www.opensourceecology.org - ReqHeader Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5 - ReqHeader host: www.opensourceecology.org - ReqHeader X-VC-Purge-Method: default - ReqHeader X-VC-Purge-Host: www.opensourceecology.org - ReqHeader Connection: Close - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1, 127.0.0.1 - ReqHeader X-VC-My-Purge-Key: JOaSAn72IJzrykJp1pfEWaECvUU8KvxZJnxSue3repId3qV8wJOHexjtuhi9r6Wv4FH9y9eFfiMjXX6hvxRrVOEWr2IaBVZMZ7ToEz8nLFdRyjyMkUGMANd6MHOzxiTJ - ReqHeader X-VC-Purge-Key-Auth: false - VCL_acl MATCH purge "localhost" - ReqURL / - Debug "VCL_error(200, Purged / www.opensourceecology.org)" - VCL_return synth - ReqUnset Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5 - ReqHeader Accept-Encoding: gzip
- I checked my varnish config for the wiki site, and I realized that I started at the vcl_recv() function. But, it does get defined in conf/acl.vcl, which is included by the main vcl file = default.vcl
[root@hetzner2 varnish]# cat default.vcl
################################################################################
# File: default.vcl
# Version: 0.1
# Purpose: Main config file for varnish cache. Note that it's intentionally
# mostly bare to allow robust vhost-specific logic. Please see this
# for more info:
# * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2017-11-12
################################################################################
vcl 4.0;
############
# INCLUDES #
############
#
import std;
include "conf/acl.vcl";
include "lib/purge.vcl";
include "all-vhosts.vcl";
include "catch-all.vcl";
[root@hetzner2 varnish]# cat conf/acl.vcl
acl purge {
"localhost";
"127.0.0.1";
}
[root@hetzner2 varnish]#
- I tested an edit again, but no output came from my grep of the varnishlog for purge requests. hmm.
- I dug down to a tcpdump, here's a positive coming from the purge in the worpdress wui from osemain
[root@hetzner2 varnish]# tcpdump -i lo -nX dst port 6081 ... 22:55:00.501715 IP 127.0.0.1.53826 > 127.0.0.1.6081: Flags [P.], seq 0:1252, ack 1, win 342, options [nop,nop,TS val 3519411272 ecr 3519411272], length 1252 0x0000: 4500 0518 ddd4 4000 4006 5a09 7f00 0001 E.....@.@.Z..... 0x0010: 7f00 0001 d242 17c1 9300 299b 26a2 4143 .....B....).&.AC 0x0020: 8018 0156 030d 0000 0101 080a d1c5 f448 ...V...........H 0x0030: d1c5 f448 4745 5420 2f77 702d 6164 6d69 ...HGET./wp-admi 0x0040: 6e2f 3f70 7572 6765 5f76 6172 6e69 7368 n/?purge_varnish 0x0050: 5f63 6163 6865 3d31 265f 7770 6e6f 6e63 _cache=1&_wpnonc 0x0060: 653d 6661 3862 6565 6264 6566 2048 5454 e=fa8beebdef.HTT 0x0070: 502f 312e 300d 0a58 2d52 6561 6c2d 4950 P/1.0..X-Real-IP 0x0080: 3a20 3736 2e39 372e 3232 332e 3138 350d :.76.97.223.185. 0x0090: 0a58 2d46 6f72 7761 7264 6564 2d46 6f72 .X-Forwarded-For 0x00a0: 3a20 3736 2e39 372e 3232 332e 3138 350d :.76.97.223.185. 0x00b0: 0a58 2d46 6f72 7761 7264 6564 2d50 726f .X-Forwarded-Pro 0x00c0: 746f 3a20 6874 7470 730d 0a58 2d46 6f72 to:.https..X-For 0x00d0: 7761 7264 6564 2d50 6f72 743a 2034 3433 warded-Port:.443 0x00e0: 0d0a 486f 7374 3a20 7777 772e 6f70 656e ..Host:.www.open 0x00f0: 736f 7572 6365 6563 6f6c 6f67 792e 6f72 sourceecology.or 0x0100: 670d 0a43 6f6e 6e65 6374 696f 6e3a 2063 g..Connection:.c 0x0110: 6c6f 7365 0d0a 5573 6572 2d41 6765 6e74 lose..User-Agent ...
- I did a page update in mediawiki while running this tcpdump; I saw the page update come through varnish, but there was no purge.
[root@hetzner2 varnish]# tcpdump -i lo -nX dst port 6081 ... 23:05:17.973341 IP 127.0.0.1.54964 > 127.0.0.1.6081: Flags [P.], seq 0:4047, ack 1, win 342, options [nop,nop,TS val 3520028743 ecr 3520028743], length 4047 0x0000: 4500 1003 96c7 4000 4006 962b 7f00 0001 E.....@.@..+.... 0x0010: 7f00 0001 d6b4 17c1 fba0 f683 3a84 17b3 ............:... 0x0020: 8018 0156 0df8 0000 0101 080a d1cf 6047 ...V..........`G 0x0030: d1cf 6047 504f 5354 202f 696e 6465 782e ..`GPOST./index. 0x0040: 7068 703f 7469 746c 653d 5573 6572 3a4d php?title=User:M 0x0050: 616c 7466 6965 6c64 2661 6374 696f 6e3d altfield&action= 0x0060: 7375 626d 6974 2048 5454 502f 312e 300d submit.HTTP/1.0. 0x0070: 0a58 2d52 6561 6c2d 4950 3a20 3736 2e39 .X-Real-IP:.76.9 0x0080: 372e 3232 332e 3138 350d 0a58 2d46 6f72 7.223.185..X-For 0x0090: 7761 7264 6564 2d46 6f72 3a20 3736 2e39 warded-For:.76.9 0x00a0: 372e 3232 332e 3138 350d 0a58 2d46 6f72 7.223.185..X-For 0x00b0: 7761 7264 6564 2d50 726f 746f 3a20 6874 warded-Proto:.ht 0x00c0: 7470 730d 0a58 2d46 6f72 7761 7264 6564 tps..X-Forwarded 0x00d0: 2d50 6f72 743a 2034 3433 0d0a 486f 7374 -Port:.443..Host 0x00e0: 3a20 7769 6b69 2e6f 7065 6e73 6f75 7263 :.wiki.opensourc 0x00f0: 6565 636f 6c6f 6779 2e6f 7267 0d0a 436f eecology.org..Co ...
- I learned that Mediawiki defaults to sending the purge requests over port 80. I changed that to the default varnish port that we're using = 6081 by setting this line in LocalSettings.php
$wgUseSquid = true; $wgSquidServers = array( '127.0.0.1:6081'); $wgUsePrivateIPs = true;
- then I did a page update in MediaWiki, and confirmed the PURGE came in via `varnishlog`
[root@hetzner2 varnish]# tcpdump -i lo -nX dst port 6081 ... * << Request >> 331532 - Begin req 331530 rxreq - Timestamp Start: 1523574861.741242 0.000000 0.000000 - Timestamp Req: 1523574861.741242 0.000000 0.000000 - ReqStart 127.0.0.1 55936 - ReqMethod PURGE - ReqURL /index.php?title=User:Maltfield&action=history - ReqProtocol HTTP/1.1 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: Keep-Alive - ReqHeader Proxy-Connection: Keep-Alive - ReqHeader User-Agent: MediaWiki/1.30.0 SquidPurgeClient - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - Debug "VCL_error(200, Purged)" - VCL_return synth - Timestamp Process: 1523574861.741277 0.000035 0.000035 - RespHeader Date: Thu, 12 Apr 2018 23:14:21 GMT - RespHeader Server: Varnish
- I loaded a wiki page in an ephemeral browser, updated it in my other logged-in browser, then reloaded it back in the distinct/ephemeral brower. I confirmed that the change came through in the distinct/ephemeral browser. So purging is working!
- I launched a fresh disposable vm ephemeral browser, loaded the page, got a miss, loaded the page again, got a hit. So caching is working for the Main_Page at least
- unfortunately, the page load included several GET requests that were not HITs
- /load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector
- /load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector
- /load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector
- /load.php?debug=false&lang=en&modules=jquery%2Cmediawiki&only=scripts&skin=vector&version=1ubqa9r
- /load.php?debug=false&lang=en&modules=jquery.accessKeyLabel%2CcheckboxShiftClick%2Cclient%2Ccookie%2CgetAttrs%2ChighlightText%2Cmw-jump%2Csuggestions%2CtabIndex%2Cthrottle-debounce%7Cmediawiki.RegExp%2Capi%2Ccookie%2Cnotify%2CsearchSuggest%2Cstorage%2Cto
- //load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector
- I did another refresh I caught the MISSES
- /load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector
- /load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector
- /load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector
- /load.php?debug=false&lang=en&modules=jquery.accessKeyLabel%2CcheckboxShiftClick%2Cclient%2Ccookie%2CgetAttrs%2ChighlightText%2Cmw-jump%2Csuggestions%2CtabIndex%2Cthrottle-debounce%7Cmediawiki.RegExp%2Capi%2Ccookie%2Cnotify%2CsearchSuggest%2Cstorage%2Cto
- so many of those are the same; let's isolate to the first one = "/load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector"
- umm, but a call from curl yielded a HIT
user@personal:~$ curl -i "https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector"
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 13 Apr 2018 00:16:49 GMT
Content-Type: text/css; charset=utf-8
Content-Length: 921
Connection: keep-alive
X-Content-Type-Options: nosniff
Access-Control-Allow-Origin: *
ETag: W/"0vstmhv"
Cache-Control: public, max-age=300, s-maxage=300
Expires: Thu, 12 Apr 2018 23:21:50 GMT
X-XSS-Protection: 1; mode=block
X-Varnish: 297412 426598
Age: 3599
Via: 1.1 varnish-v4
Accept-Ranges: bytes
Strict-Transport-Security: max-age=15552001
Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report"
.lang{background:#F9F9F9;border:1px solid #E9E9E9;font-size:smaller;margin:0 0 1em 0;padding:0.5em 1em;text-align:left}.lang ul{display:inline;margin-left:0;padding-left:0}.lang ul li{border-left:1px solid #E4E4E4;display:inline;list-style:none;margin-left:0;padding:0 0.5em}.lang ul li.lang_main{border-left:none;display:inline;list-style:none;margin-left:0;padding-left:0}.lang ul a.external{background:none ! important;padding-right:0 ! important}.lang ul li.lang_title{display:none}.dtree{font-family:Verdana,Geneva,Arial,Helvetica,sans-serif;font-size:11px;color:#666;white-space:nowrap}.dtree img{border:0px;vertical-align:middle}.dtree a{color:#333;text-decoration:none}.dtree a.node,.dtree a.nodeSel{white-space:nowrap;padding:1px 2px 1px 2px}.dtree a.node:hover,.dtree a.nodeSel:hover{color:#333;text-decoration:underline}.dtree a.nodeSel{background-color:#c0d2ec}.dtree .clip{overflow:hidden;padding-bottom:1px}user@personal:~$
* << Request >> 297412 - Begin req 297411 rxreq - Timestamp Start: 1523578609.269812 0.000000 0.000000 - Timestamp Req: 1523578609.269812 0.000000 0.000000 - ReqStart 127.0.0.1 36238 - ReqMethod GET - ReqURL /load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 76.97.223.185 - ReqHeader X-Forwarded-For: 76.97.223.185 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.38.0 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 76.97.223.185 - ReqHeader X-Forwarded-For: 76.97.223.185, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 76.97.223.185, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 426598 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Thu, 12 Apr 2018 23:16:50 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Access-Control-Allow-Origin: * - RespHeader ETag: W/"0vstmhv" - RespHeader Cache-Control: public, max-age=300, s-maxage=300 - RespHeader Expires: Thu, 12 Apr 2018 23:21:50 GMT - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Content-Length: 921 - RespHeader Content-Type: text/css; charset=utf-8 - RespHeader X-Varnish: 297412 426598 - RespHeader Age: 3599 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1523578609.269849 0.000036 0.000036 - Debug "RES_MODE 2" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1523578609.269870 0.000057 0.000021 - Debug "XXX REF 2" - ReqAcct 288 0 288 438 921 1359 - End
- but when I call the same thing from my browser, I get a PASS & fetch
* << Request >> 201266 - Begin req 201265 rxreq - Timestamp Start: 1523578751.880694 0.000000 0.000000 - Timestamp Req: 1523578751.880694 0.000000 0.000000 - ReqStart 127.0.0.1 36486 - ReqMethod GET - ReqURL /load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 76.97.223.185 - ReqHeader X-Forwarded-For: 76.97.223.185 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate, br - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-None-Match: W/"1a4906v" - ReqUnset X-Forwarded-For: 76.97.223.185 - ReqHeader X-Forwarded-For: 76.97.223.185, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 76.97.223.185, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return pass - VCL_call HASH - VCL_return lookup - VCL_call PASS - VCL_return fetch - Link bereq 201267 pass - Timestamp Fetch: 1523578751.993003 0.112309 0.112309 - RespProtocol HTTP/1.0 - RespStatus 304 - RespReason Not Modified - RespHeader Date: Fri, 13 Apr 2018 00:19:11 GMT - RespHeader Server: Apache - RespHeader ETag: W/"1a4906v" - RespHeader Expires: Fri, 13 Apr 2018 00:24:11 GMT - RespHeader Cache-Control: public, max-age=300, s-maxage=300 - RespProtocol HTTP/1.1 - RespHeader X-Varnish: 201266 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1523578751.993043 0.112350 0.000040 - Debug "RES_MODE 0" - RespHeader Connection: close - Timestamp Resp: 1523578751.993091 0.112397 0.000048 - Debug "XXX REF 1" - ReqAcct 540 0 540 258 0 258 - End
- I think I just crossed the line of diminished return. The pages themselves are definitely being cached. Images are being cached. Maybe some minification resources are not being cached. I'm confident that the site will run fine after cutover. Then, once it's live in prod, I'll get munin graphs & varnish logs to show me which requests are *not* hits, and I can optimize from there (as was done with osemain by removing the Fundraising addon post migration ).
Wed Apr 11, 2018
- I haven't heard back from dreamhost, and they haven't deleted what remains of our backup data yet.
- checked the state of storage on dreamhost, and found our old backups dir has 52G. Our new backups dirs have 44G.
hancock% date Wed Apr 11 19:40:17 PDT 2018 hancock% pwd /home/marcin_ose hancock% du -sh hetzner1/* 12G hetzner1/20180409-052001 12G hetzner1/20180410-052001 12G hetzner1/20180411-052001 hancock% du -sh hetzner2/* 2.8G hetzner2/20180409-072001 2.8G hetzner2/20180410-072001 2.8G hetzner2/20180411-072001 hancock% du -sh backups/hetzner1/* 248M backups/hetzner1/20180402-052001 0 backups/hetzner1/20180406-052001 12G backups/hetzner1/20180407-052001 12G backups/hetzner1/20180408-052001 hancock% du -sh backups/hetzner2/* 0 backups/hetzner2/20180406-072001 14G backups/hetzner2/20180407-072001 14G backups/hetzner2/20180408-072001 hancock%
- since we have 3x copies of daily backups in the new dir, I just went ahead and deleted the 52G remaining from the old daily backup dir
hancock% date Wed Apr 11 19:43:20 PDT 2018 hancock% pwd /home/marcin_ose/backups hancock% du -sh hetzner1/* 248M hetzner1/20180402-052001 0 hetzner1/20180406-052001 12G hetzner1/20180407-052001 12G hetzner1/20180408-052001 hancock% du -sh hetzner2/* 0 hetzner2/20180406-072001 14G hetzner2/20180407-072001 14G hetzner2/20180408-072001 hancock% rm -rf hetzner1/* zsh: sure you want to delete all the files in /home/marcin_ose/backups/hetzner1 [yn]? y hancock% rm -rf hetzner2/* zsh: sure you want to delete all the files in /home/marcin_ose/backups/hetzner2 [yn]? y hancock% ls -lah hetzner1/ total 4.0K drwxr-xr-x 2 marcin_ose pg1589252 10 Apr 11 19:43 . drwxr-xr-x 4 marcin_ose pg1589252 4.0K Apr 9 13:20 .. hancock% ls -lah hetzner2/ total 4.0K drwxr-xr-x 2 marcin_ose pg1589252 10 Apr 11 19:43 . drwxr-xr-x 4 marcin_ose pg1589252 4.0K Apr 9 13:20 .. hancock%
- now our entire home dir's usage is 47G!
hancock% date Wed Apr 11 19:46:23 PDT 2018 hancock% pwd /home/marcin_ose hancock% du -sh 47G . hancock%
- I expect that once I start working on the wiki again, backups will grow to probably 12*4 + 20*4 = ~128G. Then, after the wiki is migrated, the hetzner1 backups will become negligible (it's mostly just the wiki), so we'll only have ~20*4 = 80G of backups to store. The long-term solution is to migrate to s3 with lifecycle policies on the first-of-the-month daily going into glacier and all other backups getting automatically deleted a few days after upload. The cost difference of this s3 solution between 128G in s3 vs 80G in s3 may be such that I should focus on getting this wiki migrated before recoding our backup scripts to go to s3. While hoping that dreamhost doesn't notice our daily backups (which are significantly smaller than their >500G usage before) have just moved to another dir.
- wow, within a few minutes from deleting the big directories in the 'backups' dir, I got an email from "Jin K" at dreamhost stating that our "Action Required: Disk usage warning - acceptable use policy violation" ticket was "RESOLVED!", thanking us for taking care of it. Either that response was automated or I got it just before they deleted it for us
Hi there! It looks like we've applied more time for you previously regarding the backup location that was left, but it looks like you've cleared that up since then as the location below is now empty: hancock:/home/marcin_ose/backups# 96K . Thanks for getting that done! This notice is just to let you know that we're all set and this matter is now closed. Please give us a shout at any time if you have any questions or concerns at all moving forward. We'd be happy to help! Thank you kindly, Jin K.
Mon Apr 09, 2018
- I confirmed that the backups from last night came into their new location
hancock% du -sh ../hetzner1/* 12G ../hetzner1/20180409-052001 hancock% du -sh ../hetzner2/* 2.8G ../hetzner2/20180409-072001 hancock%
- I deleted the encryption key from dreamhost's server. Future backups can be done on hetzner's servers directly.
hancock% chmod 0700 ose-backups-cron.key hancock% shred -u ose-backups-cron.key hancock%
- now the our home dir's entire usage is currently 121G
hancock% date Mon Apr 9 13:18:36 PDT 2018 hancock% pwd /home/marcin_ose hancock% du -sh 121G . hancock%
- 104G of that is going to be automatically deleted by the cron over the next week as the dailys become stale
hancock% du -sh backups/* 4.0K backups/getGlacierJob.sh 48G backups/hetzner1 56G backups/hetzner2 4.0K backups/output.json 4.0K backups/readme.txt 64K backups/retryUploadToGlacier.log 4.0K backups/retryUploadToGlacier.sh 28M backups/uploadToGlacier 4.0K backups/uploadToGlacier.py 8.0K backups/uploadToGlacier.sh hancock% du -sh backups 104G backups hancock%
- I deleted the entire uploadToGlacier directory, which only had fileLists that failed to delete due to a minor bug in my script
hancock% du -sh uploadToGlacier/* 2.4M uploadToGlacier/hetzner1_20170701-052001.fileList.txt.bz2 2.4M uploadToGlacier/hetzner1_20170801-052001.fileList.txt.bz2 2.3M uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2 2.3M uploadToGlacier/hetzner1_20180101-062001.fileList.txt.bz2 2.4M uploadToGlacier/hetzner1_20180201-062001.fileList.txt.bz2 2.4M uploadToGlacier/hetzner1_20180301-062002.fileList.txt.bz2 2.2M uploadToGlacier/hetzner1_20180401-052001.fileList.txt.bz2 2.0M uploadToGlacier/hetzner2_20170702-052001.fileList.txt.bz2 196K uploadToGlacier/hetzner2_20170801-072001.fileList.txt.bz2 284K uploadToGlacier/hetzner2_20170901-072001.fileList.txt.bz2 648K uploadToGlacier/hetzner2_20171001-072001.fileList.txt.bz2 276K uploadToGlacier/hetzner2_20171101-072001.fileList.txt.bz2 308K uploadToGlacier/hetzner2_20171202-072001.fileList.txt.bz2 488K uploadToGlacier/hetzner2_20180102-072001.fileList.txt.bz2 2.4M uploadToGlacier/hetzner2_20180202-072001.fileList.txt.bz2 3.4M uploadToGlacier/hetzner2_20180302-072001.fileList.txt.bz2 1.6M uploadToGlacier/hetzner2_20180401-072001.fileList.txt.bz2 hancock% rm -rf uploadToGlacier hancock%
- I updated the crontab to cleanBackups from the new backup dir as well
hancock% crontab -l ###--- BEGIN DREAMHOST BLOCK ###--- Changes made to this part of the file WILL be destroyed! # Backup site-root MAILTO="elifarley@gmail.com" @weekly /usr/local/bin/setlock -n /tmp/cronlock.2671804.96324 sh -c $'. \176/altroot/init.sh \012\043 \012\176/bin/mbkp.sh site-root' # Backup MONTHLY MAILTO="elifarley@gmail.com" @monthly /usr/local/bin/setlock -n /tmp/cronlock.2671804.96354 sh -c $'. \176/altroot/init.sh \012\043 \012\176/bin/mbkp.sh home \012\176/bin/mbkp.sh altroot \012\043\176/bin/mbkp.sh blog-cache \012' # delete older backup files 20 22 * * * /usr/bin/perl /home/marcin_ose/bin/cleanLocal.pl -l /home/marcin_ose/backups/hetzner1 -d 3 &>> /home/marcin_ose/logs/cleanBackups.log 20 22 * * * /usr/bin/perl /home/marcin_ose/bin/cleanLocal.pl -l /home/marcin_ose/backups/hetzner2 -d 3 &>> /home/marcin_ose/logs/cleanBackups.log ###--- You can make changes below the next line and they will be preserved! ###--- END DREAMHOST BLOCK hancock% hancock% crontab -e ... hancock% crontab -l ###--- BEGIN DREAMHOST BLOCK ###--- Changes made to this part of the file WILL be destroyed! # Backup site-root MAILTO="elifarley@gmail.com" @weekly /usr/local/bin/setlock -n /tmp/cronlock.2671804.96324 sh -c $'. \176/altroot/init.sh \012\043 \012\176/bin/mbkp.sh site-root' # Backup MONTHLY MAILTO="elifarley@gmail.com" @monthly /usr/local/bin/setlock -n /tmp/cronlock.2671804.96354 sh -c $'. \176/altroot/init.sh \012\043 \012\176/bin/mbkp.sh home \012\176/bin/mbkp.sh altroot \012\043\176/bin/mbkp.sh blog-cache \012' # delete older backup files 20 22 * * * /usr/bin/perl /home/marcin_ose/bin/cleanLocal.pl -l /home/marcin_ose/backups/hetzner1 -d 3 &>> /home/marcin_ose/logs/cleanBackups.log 20 22 * * * /usr/bin/perl /home/marcin_ose/bin/cleanLocal.pl -l /home/marcin_ose/backups/hetzner2 -d 3 &>> /home/marcin_ose/logs/cleanBackups.log 20 22 * * * /usr/bin/perl /home/marcin_ose/bin/cleanLocal.pl -l /home/marcin_ose/hetzner1 -d 3 &>> /home/marcin_ose/logs/cleanBackups.log 20 22 * * * /usr/bin/perl /home/marcin_ose/bin/cleanLocal.pl -l /home/marcin_ose/hetzner2 -d 3 &>> /home/marcin_ose/logs/cleanBackups.log ###--- You can make changes below the next line and they will be preserved! ###--- END DREAMHOST BLOCK hancock%
Sun Apr 08, 2018
- I checked again just after midnight; the retry appears to have worked pretty great. Just 2 archives failed on this run
hancock% date Sat Apr 7 22:14:46 PDT 2018 hancock% pwd /home/marcin_ose/backups hancock% du -sh uploadToGlacier/*.gpg 39G uploadToGlacier/hetzner1_20170801-052001.tar.gpg 12G uploadToGlacier/hetzner1_20180101-062001.tar.gpg hancock%
- the archive list on hetzner2 looks pretty great too
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170701-052001.fileList.txt.bz2.gpg hetzner1_20170801-052001.fileList.txt.bz2.gpg hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates hetzner1_20171001-052001.fileList.txt.bz2.gpg hetzner1_20171001-052001.tar.gpg hetzner1_20171101-062001.fileList.txt.bz2.gpg hetzner1_20171101-062001.tar.gpg hetzner1_20171201-062001.fileList.txt.bz2.gpg hetzner1_20180101-062001.fileList.txt.bz2.gpg hetzner1_20180201-062001.fileList.txt.bz2.gpg hetzner1_20180201-062001.tar.gpg hetzner1_20180301-062002.fileList.txt.bz2.gpg hetzner1_20180301-062002.tar.gpg hetzner1_20180401-052001.fileList.txt.bz2.gpg hetzner1_20180401-052001.tar.gpg hetzner2_20170702-052001.fileList.txt.bz2.gpg hetzner2_20170702-052001.tar.gpg hetzner2_20170801-072001.fileList.txt.bz2.gpg hetzner2_20170801-072001.tar.gpg hetzner2_20170901-072001.fileList.txt.bz2.gpg hetzner2_20170901-072001.tar.gpg hetzner2_20171001-072001.fileList.txt.bz2.gpg hetzner2_20171001-072001.tar.gpg hetzner2_20171101-072001.fileList.txt.bz2.gpg hetzner2_20171101-072001.tar.gpg hetzner2_20171202-072001.fileList.txt.bz2.gpg hetzner2_20171202-072001.tar.gpg hetzner2_20180102-072001.fileList.txt.bz2.gpg hetzner2_20180102-072001.tar.gpg hetzner2_20180202-072001.fileList.txt.bz2.gpg hetzner2_20180302-072001.fileList.txt.bz2.gpg hetzner2_20180401-072001.fileList.txt.bz2.gpg hetzner2_20180401-072001.tar.gpg id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- I kicked off a fresh inventory fetch; hopefully that'll get the ones that I just uploaded on retry
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync --max-age=0 --wait deleteMeIn2020
- meanwhile, back on dreamhost's hancock, I kicked off the re-upload attempt for the remaining 2x archives
- I also checked the aws console; the bill for April so far is $3.32. Not bad!
- most of that is $3.25 for 65,038 requests.
- Unfortunately, I discovered that the AWS Budget service is itself not free. We apparently have $0 charges because there's 62 days of free Budget service in the Free Tier.
- according to their docs, the first 2x budgets are free of charge. Additional budgets are $0.02/day. I'll leave our $10 budget email. If we get charged >$0 for it ever, I'll delete it.
- I changed the existing budget from $1 to $8 (so $96/yr). I added an alert for both exceeds actual & foretasted amounts. fwiw, we're currently only being charged $2.28, but the forecast is $9.25. I assume that's expecting us to keep uploading at our current rate all month, which won't happen..
- ...
- when I woke up, the 2x remaining uploads completed successfully!
- I checked on the archive list, but it still didn't show the complete list; so I kicked off another inventory refresh
- after the inventory shows all the archives, I'll delete them from dreamhost. Tomorrow is the deadline, so hopefully this can be done today.
- I got an email that the projected budget was to exceed the $8 budget. The actual budget is still $3.97.
- ...
- a few hours later, the inventory sync was complete, and many of the archives were now listed
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170701-052001.fileList.txt.bz2.gpg hetzner1_20170701-052001.tar.gpg hetzner1_20170801-052001.fileList.txt.bz2.gpg hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates hetzner1_20171001-052001.fileList.txt.bz2.gpg hetzner1_20171001-052001.tar.gpg hetzner1_20171101-062001.fileList.txt.bz2.gpg hetzner1_20171101-062001.tar.gpg id:lryfyQFE4NbtWg5Q6uTq8Qqyc-y9il9WYe7lHs8H2lzFSBADOJQmCIgp6FxrkiaCcwnSMIReJPWyWcR4UOnurxwONhw8fojEHQTTeOpkf6fgfWBAPP9P6GOZZ0v8d8Jz_-QFVaV6Bw hetzner1_20171201-062001.fileList.txt.bz2.gpg id:NR3Z9zdD2rW0NG1y3QW735TzykIivP_cnFDMCNX6RcIPh0mRb_6QiC5qy1GrBTIoroorfzaGDIKQ0BY18jbcR3XfEzfcmrZ1FiT1YvQw-c1ag6vT46-noPvmddZ_zyy2O1ItIygI6Q hetzner1_20171201-062001.fileList.txt.bz2.gpg hetzner1_20171201-062001.tar.gpg hetzner1_20180101-062001.fileList.txt.bz2.gpg hetzner1_20180201-062001.fileList.txt.bz2.gpg hetzner1_20180201-062001.tar.gpg hetzner1_20180301-062002.fileList.txt.bz2.gpg hetzner1_20180301-062002.tar.gpg hetzner1_20180401-052001.fileList.txt.bz2.gpg hetzner1_20180401-052001.tar.gpg hetzner2_20170702-052001.fileList.txt.bz2.gpg hetzner2_20170702-052001.tar.gpg hetzner2_20170801-072001.fileList.txt.bz2.gpg hetzner2_20170801-072001.tar.gpg hetzner2_20170901-072001.fileList.txt.bz2.gpg hetzner2_20170901-072001.tar.gpg hetzner2_20171001-072001.fileList.txt.bz2.gpg hetzner2_20171001-072001.tar.gpg hetzner2_20171101-072001.fileList.txt.bz2.gpg hetzner2_20171101-072001.tar.gpg hetzner2_20171202-072001.fileList.txt.bz2.gpg hetzner2_20171202-072001.tar.gpg hetzner2_20180102-072001.fileList.txt.bz2.gpg hetzner2_20180102-072001.tar.gpg hetzner2_20180202-072001.fileList.txt.bz2.gpg hetzner2_20180202-072001.tar.gpg hetzner2_20180302-072001.fileList.txt.bz2.gpg hetzner2_20180302-072001.tar.gpg hetzner2_20180401-072001.fileList.txt.bz2.gpg hetzner2_20180401-072001.tar.gpg id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- we can see that these archives appear obviously to be in the glacier vault (both encrypted tarballs are present & named as exptected)
hetzner1_20170701-052001 hetzner1_20171001-052001 hetzner1_20171101-062001 hetzner1_20180201-062001 hetzner1_20180301-062002 hetzner1_20180401-052001 hetzner2_20170702-052001 hetzner2_20170801-072001 hetzner2_20170901-072001 hetzner2_20171001-072001 hetzner2_20171101-072001 hetzner2_20171202-072001 hetzner2_20180102-072001 hetzner2_20180202-072001 hetzner2_20180302-072001 hetzner2_20180401-072001
- but there's a couple more that have an odd naming convention and/or duplicate archives from when I was testing the upload script; these also appear to be in the glacier vault:
hetzner1_20170901-052001 hetzner1_20171201-062001
- finally, there's a couple where I didn't include the file name in the archive description, from my very early testing--back before I realized the archive name wouldn't be remembered by glacier, and that it would only be given a reference uid. from my notes, I know that these archives correspond to:
hetzner1_20171001-052001
- therefore, the following archives have been fully uploaded into glacier, and they can be deleted from dreamhost:
hetzner1_20170701-052001 hetzner1_20170901-052001 hetzner1_20171001-052001 hetzner1_20171101-062001 hetzner1_20171201-062001 hetzner1_20180201-062001 hetzner1_20180301-062002 hetzner1_20180401-052001 hetzner2_20170702-052001 hetzner2_20170801-072001 hetzner2_20170901-072001 hetzner2_20171001-072001 hetzner2_20171101-072001 hetzner2_20171202-072001 hetzner2_20180102-072001 hetzner2_20180202-072001 hetzner2_20180302-072001 hetzner2_20180401-072001
- I kicked-off the deletions
hancock% rm -rf hetzner1/20170701-052001 hancock% rm -rf hetzner1/20170701-052001 hancock% rm -rf hetzner1/20171001-052001 hancock% rm -rf hetzner1/20171101-062001 hancock% rm -rf hetzner1/20180201-062001 hancock% rm -rf hetzner1/20180301-062002 hancock% rm -rf hetzner1/20180401-052001 hancock% rm -rf hetzner2/20170702-052001 hancock% rm -rf hetzner2/20170801-072001 hancock% rm -rf hetzner2/20170901-072001 hancock% rm -rf hetzner2/20171001-072001 hancock% rm -rf hetzner2/20171101-072001 hancock% rm -rf hetzner2/20171202-072001 hancock% rm -rf hetzner2/20180102-072001 hancock% rm -rf hetzner2/20180202-072001 hancock% rm -rf hetzner2/20180302-072001 hancock% rm -rf hetzner2/20180401-072001 hancock% rm -rf hetzner1/20170901-052001 hancock% rm -rf hetzner1/20171201-062001 hancock% rm -rf hetzner1/20171001-052001 hancock%
- and here's what remains
hancock% du -sh hetzner1/* 39G hetzner1/20170801-052001 12G hetzner1/20180101-062001 248M hetzner1/20180402-052001 0 hetzner1/20180403-052001 12G hetzner1/20180404-052001 12G hetzner1/20180405-052001 12G hetzner1/20180406-052001 12G hetzner1/20180407-052001 12G hetzner1/20180408-052001 hancock% du -sh hetzner2/* 0 hetzner2/20180403-072001 14G hetzner2/20180404-072001 14G hetzner2/20180405-072001 14G hetzner2/20180406-072001 14G hetzner2/20180407-072001 14G hetzner2/20180408-072001 hancock%
- so that finishes off hetzner2. The backups that are present are just the recent few days worth (eventually dailys may have to go to s3, but for now I'm primarily focused on shipping our historical monthlies off to somewhere safe = glacier)
- hetzner1 has 2x remaining archives that need to be confirmed in glacier, then deleted from dreamhost:
hancock% du -sh hetzner1/* 39G hetzner1/20170801-052001 12G hetzner1/20180101-062001
- unfortunately, the hetzner2 backups have exploded to 14G again; they should be ~3G each. Looks like I still had an 'orig' copy of the archives I restored when testing glacier in the /root/glacierRestore directory. The '/root'/ directory is itself backed-up.
- I updated the "restore from glacier" documentation on the wiki so that the 'glacier-cli' dir is placed in '/root/sandbox/', the 'glacier.py' binary is linked to by '/root/bin/glacier.py' (/root/bin is already in $PATH), and that the restores themselves get done in a temporary directory in /var/tmp/
- I updated the upload path in '/root/backups/backup.settings' to be '/home/marcin_ose/hetzner2' instead of '/home/marcin_ose/backups/ hetzner2'
- I updated the upload path in '/usr/home/osemain/backups/backup.settings' to be '/home/marcin_ose/hetzner1' instead of '/home/marcin_ose/backups/hetzner1'
- ...
- I checked again just before midnight, and here's the new listing
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170701-052001.fileList.txt.bz2.gpg hetzner1_20170701-052001.tar.gpg hetzner1_20170801-052001.fileList.txt.bz2.gpg hetzner1_20170801-052001.tar.gpg hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates hetzner1_20171001-052001.fileList.txt.bz2.gpg hetzner1_20171001-052001.tar.gpg hetzner1_20171101-062001.fileList.txt.bz2.gpg hetzner1_20171101-062001.tar.gpg id:lryfyQFE4NbtWg5Q6uTq8Qqyc-y9il9WYe7lHs8H2lzFSBADOJQmCIgp6FxrkiaCcwnSMIReJPWyWcR4UOnurxwONhw8fojEHQTTeOpkf6fgfWBAPP9P6GOZZ0v8d8Jz_-QFVaV6Bw hetzner1_20171201-062001.fileList.txt.bz2.gpg id:NR3Z9zdD2rW0NG1y3QW735TzykIivP_cnFDMCNX6RcIPh0mRb_6QiC5qy1GrBTIoroorfzaGDIKQ0BY18jbcR3XfEzfcmrZ1FiT1YvQw-c1ag6vT46-noPvmddZ_zyy2O1ItIygI6Q hetzner1_20171201-062001.fileList.txt.bz2.gpg hetzner1_20171201-062001.tar.gpg hetzner1_20180101-062001.fileList.txt.bz2.gpg hetzner1_20180101-062001.tar.gpg hetzner1_20180201-062001.fileList.txt.bz2.gpg hetzner1_20180201-062001.tar.gpg hetzner1_20180301-062002.fileList.txt.bz2.gpg hetzner1_20180301-062002.tar.gpg hetzner1_20180401-052001.fileList.txt.bz2.gpg hetzner1_20180401-052001.tar.gpg hetzner2_20170702-052001.fileList.txt.bz2.gpg hetzner2_20170702-052001.tar.gpg hetzner2_20170801-072001.fileList.txt.bz2.gpg hetzner2_20170801-072001.tar.gpg hetzner2_20170901-072001.fileList.txt.bz2.gpg hetzner2_20170901-072001.tar.gpg hetzner2_20171001-072001.fileList.txt.bz2.gpg hetzner2_20171001-072001.tar.gpg hetzner2_20171101-072001.fileList.txt.bz2.gpg hetzner2_20171101-072001.tar.gpg hetzner2_20171202-072001.fileList.txt.bz2.gpg hetzner2_20171202-072001.tar.gpg hetzner2_20180102-072001.fileList.txt.bz2.gpg hetzner2_20180102-072001.tar.gpg hetzner2_20180202-072001.fileList.txt.bz2.gpg hetzner2_20180202-072001.tar.gpg hetzner2_20180302-072001.fileList.txt.bz2.gpg hetzner2_20180302-072001.tar.gpg hetzner2_20180401-072001.fileList.txt.bz2.gpg hetzner2_20180401-072001.tar.gpg id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- that includes the following, which are the 2x archives were previously absent
hetzner1/20170801-052001 hetzner1/20180101-062001
- I deleted those archives from dreamhost
rm -rf hetzner1/20170801-052001 rm -rf hetzner1/20180101-062001
- that leaves only recent daily archives
hancock% du -sh hetzner1/* 248M hetzner1/20180402-052001 0 hetzner1/20180403-052001 12G hetzner1/20180404-052001 12G hetzner1/20180405-052001 12G hetzner1/20180406-052001 12G hetzner1/20180407-052001 12G hetzner1/20180408-052001 hancock% du -sh hetzner2/* 0 hetzner2/20180403-072001 14G hetzner2/20180404-072001 14G hetzner2/20180405-072001 14G hetzner2/20180406-072001 14G hetzner2/20180407-072001 14G hetzner2/20180408-072001 hancock%
Sat Apr 07, 2018
- checked dreamhost; the screen died (damn dreamhost), so I can't see the last command's output (shoulda sent it to a log file..)
- anyway, I can tell which files appeared to have failed from the gpg files in the dir
hancock% date Sat Apr 7 08:27:09 PDT 2018 hancock% pwd /home/marcin_ose/backups/uploadToGlacier hancock% ls -lah *.gpg -rw-r--r-- 1 marcin_ose pg1589252 39G Apr 4 18:59 hetzner1_20170701-052001.tar.gpg -rw-r--r-- 1 marcin_ose pg1589252 39G Apr 4 22:13 hetzner1_20170801-052001.tar.gpg -rw-r--r-- 1 marcin_ose pg1589252 2.3M Apr 3 16:15 hetzner1_20171201-062001.fileList.txt.bz2.gpg -rw-r--r-- 1 marcin_ose pg1589252 12G Apr 3 16:37 hetzner1_20171201-062001.tar.gpg -rw-r--r-- 1 marcin_ose pg1589252 12G Apr 5 00:51 hetzner1_20180101-062001.tar.gpg -rw-r--r-- 1 marcin_ose pg1589252 14G Apr 4 11:00 hetzner2_20180202-072001.tar.gpg -rw-r--r-- 1 marcin_ose pg1589252 25G Apr 4 12:39 hetzner2_20180302-072001.tar.gpg hancock%
- I can't confirm as the inventory is stale, but I kicked-off a sync
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates hetzner1_20171001-052001.fileList.txt.bz2.gpg hetzner1_20171001-052001.tar.gpg hetzner1_20171101-062001.fileList.txt.bz2.gpg hetzner1_20171101-062001.tar.gpg hetzner1_20171201-062001.fileList.txt.bz2.gpg id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]# vault sync deleteMeIn2020 glacier: queued inventory job for u'deleteMeIn2020' [root@hetzner2 glacier-cli]#
- so this is the list of failed backups that I need to retry. It's ~150G, which is too big to stash on hetzner2; hopefully this retry knocks it below 90G, which is how much free space we have on hetzner2
hancock% du -sh *.gpg 39G hetzner1_20170701-052001.tar.gpg 39G hetzner1_20170801-052001.tar.gpg 2.3M hetzner1_20171201-062001.fileList.txt.bz2.gpg 12G hetzner1_20171201-062001.tar.gpg 12G hetzner1_20180101-062001.tar.gpg 14G hetzner2_20180202-072001.tar.gpg 25G hetzner2_20180302-072001.tar.gpg hancock%
- the most concerning is the first backups I made on the hetzner1 server on 201707. I made that backup just before I deleted anything, so we really, really need that in glacier.
- I copied the uploadToGlacier.sh script to a modified new script named retryUploadToGlaicer.sh
hancock% cat retryUploadToGlacier.sh
#!/bin/bash -x
############
# SETTINGS #
############
backupArchives="uploadToGlacier/hetzner1_20170701-052001.tar.gpg uploadToGlacier/hetzner1_20170801-052001.tar.gpg uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2.gpg uploadToGlacier/hetzner1_20171201-062001.tar.gpg uploadToGlacier/hetzner1_20180101-062001.tar.gpg uploadToGlacier/hetzner2_20180202-072001.tar.gpg uploadToGlacier/hetzner2_20180302-072001.tar.gpg"
export AWS_ACCESS_KEY_ID='CHANGEME'
export AWS_SECRET_ACCESS_KEY='CHANGEME'
##############
# DO UPLOADS #
##############
for archive in $(echo $backupArchives); do
# upload it
/home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 "${archive}"
if $? -eq 0 ; then
rm -f "${archive}"
fi
done
hancock%
- kicked off the script, this time logging to a file
hancock% ./retryUploadToGlacier.sh &> retryUploadToGlacier.log
- it's uploading the 201707 archive to hetzner1
hancock% tail -f retryUploadToGlacier.log + backupArchives='uploadToGlacier/hetzner1_20170701-052001.tar.gpg uploadToGlacier/hetzner1_20170801-052001.tar.gpg uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2.gpg uploadToGlacier/hetzner1_20171201-062001.tar.gpg uploadToGlacier/hetzner1_20180101-062001.tar.gpg uploadToGlacier/hetzner2_20180202-072001.tar.gpg uploadToGlacier/hetzner2_20180302-072001.tar.gpg' + export AWS_ACCESS_KEY_ID=CHANGEME + AWS_ACCESS_KEY_ID=CHANGEME + export AWS_SECRET_ACCESS_KEY=CHANGEME + AWS_SECRET_ACCESS_KEY=CHANGEME ++ echo uploadToGlacier/hetzner1_20170701-052001.tar.gpg uploadToGlacier/hetzner1_20170801-052001.tar.gpg uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2.gpg uploadToGlacier/hetzner1_20171201-062001.tar.gpg uploadToGlacier/hetzner1_20180101-062001.tar.gpg uploadToGlacier/hetzner2_20180202-072001.tar.gpg uploadToGlacier/hetzner2_20180302-072001.tar.gpg + for archive in '$(echo $backupArchives)' + /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 uploadToGlacier/hetzner1_20170701-052001.tar.gpg
- I'll check on this in the evening
Thr Apr 05, 2018
- the backup script is still running on the dreamhost server
- It's currently on 'hetzner1_20180401-052001', which is the last one!
- It looks like some uploads have succeeded & some have failed. the staging dir = 'uploadToGlacier/', which has all the encrypted archives (those that failed intentionally don't get deleted so I can re-try them later) has swelled to 150G. I already deleted ~250G, so I think dreamhost can deal with that (at least for 4 more days, which is our deadline)
- I got an email saying that our budget of $1/mo was exceeded with our aws account. Good to know that works! I'll increase that to $10 soon
- I'll come back to this tomorrow, after the script has finished running. Then I'll try uploading the remaining files sequentially in a loop, and hopefully they'll all be done by our deadline
- Marcin emailed me regarding broken images on the workshops page of osemain https://www.opensourceecology.org/workshops-and-programs/ .
- I can't fix this now as my phone is broken & I have to rebuild it + restore my 2fa tokens to login, but I told him the fix was to replace the absolute ulrs with the wrong protocol (http) with just a relative path to the image which is more robust
Wed Apr 04, 2018
- the upload of the archive 'hetzner1_20171201-062001.tar.gpg' failed again last night!
+ tar -cvf /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar hetzner1/20171201-062001/ hetzner1/20171201-062001/ hetzner1/20171201-062001/public_html/ hetzner1/20171201-062001/public_html/public_html.20171201-062001.tar.bz2 + gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar Reading passphrase from file descriptor 3 + rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar + /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar.gpg Traceback (most recent call last): File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 736, in <module> main() File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 732, in main App().main() File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 718, in main self.args.func() File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 500, in archive_upload file_obj=self.args.file, description=name) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file writer.write(data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 219, in write self.partitioner.write(data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 61, in write self._send_part() File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part self.send_fn(part) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part self.uploader.upload_part(self.next_part_index, part_data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part content_range, part_data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 637, in upload_part response_headers=response_headers) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 84, in make_request raise UnexpectedHTTPResponseError(ok_responses, response) boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.) + 1 -eq 0 marcin_ose@hancock:~/backups$
- the good news is that my new if statement prevented it from being deleted
hancock% ls -lah uploadToGlacier total 12G drwxr-xr-x 2 marcin_ose pg1589252 4.0K Apr 3 16:37 . drwxr-xr-x 5 marcin_ose pg1589252 4.0K Apr 3 15:37 .. -rw-r--r-- 1 marcin_ose pg1589252 2.3M Apr 3 16:14 hetzner1_20171201-062001.fileList.txt.bz2 -rw-r--r-- 1 marcin_ose pg1589252 2.3M Apr 3 16:15 hetzner1_20171201-062001.fileList.txt.bz2.gpg -rw-r--r-- 1 marcin_ose pg1589252 12G Apr 3 16:37 hetzner1_20171201-062001.tar.gpg hancock%
- I found a lot of people complain about this timeout issue. One post mentioned that we could increase the boto num_retries from the default (None) to some small number, but someone responded saying it didn't help them.. https://github.com/uskudnik/amazon-glacier-cmd-interface/issues/171
- err, I checked the documentation, and it shows the default is actually '5', not '0' http://docs.pythonboto.org/en/latest/boto_config_tut.html#boto
- I am concerned that these 408 errors & all our retries will result in higher bills. Even if only because of higher requests because--say--an upload that would require 1,000 requests now requires 1,500 requests because of the retry.
- I checked the aws console, and our bill for March is $0.41.
- Not bad considering we did a restore of 12G of data. It lists a $0.01 fee for restoring 1.187G of data. Indeed, I restored ~12G of data, but it looks like aws free tier permits 10G per month of data retrievals. So we got that restore test for essentially free; awesome.
- our largest spend is 7,797 requests at $0.039. I'm afraid that increasing the chunk size to decrease the # of requests may increase the timeout risk? I think I'll just stick with the default & cross my fingers
- and $0.01 for 2.456 GB-Mo storage fee
- I did a sync & listed the archives on hetzner2, and now it lists all the files I'd expect, except the one that keeps failing with timeout issues
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync deleteMeIn2020 [root@hetzner2 glacier-cli]# ./glacier.py archive list deleteMeIn2020 hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates hetzner1_20171001-052001.fileList.txt.bz2.gpg hetzner1_20171001-052001.tar.gpg hetzner1_20171101-062001.fileList.txt.bz2.gpg hetzner1_20171101-062001.tar.gpg hetzner1_20171201-062001.fileList.txt.bz2.gpg id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- except for the timeout issues, this process appears to work; we have only 5 days to delete all this data, so I'll kick-off the upload for all remaining archives using the script
backupDirs="hetzner2/20170702-052001 hetzner2/20170801-072001 hetzner2/20170901-072001 hetzner2/20171001-072001 hetzner2/20171101-072001 hetzner2/20171202-072001 hetzner2/20180102-072001 hetzner2/20180202-072001 hetzner2/20180302-072001 hetzner2/20180401-072001 hetzner1/20170701-052001 hetzner1/20170801-052001 hetzner1/20180101-062001 hetzner1/20180201-062001 hetzner1/20180301-062002 hetzner1/20180401-052001"
- and the manual re-upload attempt of 'hetzner1_20171201-062001.tar.gpg' that failed with the timeout failed again with timeout; I'll hold-off on the reupload until the other script finishes; hopefully it will just be a small subset of archives that I need to retry
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar.gpg Traceback (most recent call last): File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 736, in <module> main() File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 732, in main App().main() File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 718, in main self.args.func() File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 500, in archive_upload file_obj=self.args.file, description=name) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file writer.write(data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 219, in write self.partitioner.write(data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 61, in write self._send_part() File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part self.send_fn(part) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part self.uploader.upload_part(self.next_part_index, part_data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part content_range, part_data) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 637, in upload_part response_headers=response_headers) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 84, in make_request raise UnexpectedHTTPResponseError(ok_responses, response) boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.) hancock%
- I just discovered that there's 2x versions of glacier-cli; one was forked from the other https://github.com/basak/glacier-cli/pull/24
- determined that our dreamhost glacier-cli sandbox was checked-out from basak's repo, which was last updated on feb 6. this is the original repo.
hancock% pwd /home/marcin_ose/sandbox/glacier-cli hancock% git remote show origin * remote origin Fetch URL: git://github.com/basak/glacier-cli.git Push URL: git://github.com/basak/glacier-cli.git HEAD branch: master Remote branches: annex-hook-script tracked master tracked Local branch configured for 'git pull': master merges with remote master Local ref configured for 'git push': master pushes to master (up to date) hancock%
- determined that the hetzner2 repo uses the same origin
[root@hetzner2 glacier-cli]# pwd /root/backups/glacierRestore/glacier-cli [root@hetzner2 glacier-cli]# git remote show origin * remote origin Fetch URL: git://github.com/basak/glacier-cli.git Push URL: git://github.com/basak/glacier-cli.git HEAD branch: master Remote branches: annex-hook-script tracked master tracked Local branch configured for 'git pull': master merges with remote master Local ref configured for 'git push': master pushes to master (up to date) [root@hetzner2 glacier-cli]#
- important to note that the 2x repos document different commands for upload.
- the 'basak' repo is more basic
glacier vault list glacier vault create vault-name glacier vault sync [--wait] [--fix] [--max-age hours] vault-name glacier archive list vault-name glacier archive upload [--name archive-name] vault-name filename glacier archive retrieve [--wait] [-o filename] [--multipart-size bytes] vault-name archive-name glacier archive retrieve [--wait] [--multipart-size bytes] vault-name archive-name [archive-name...] glacier archive delete vault-name archive-name glacier job list
- but the 'pkaleta' repo has options for the part size & thread count for the upload
glacier vault list glacier vault create vault-name glacier vault sync [--wait] [--fix] [--max-age hours] vault-name glacier archive list vault-name glacier archive upload [--encrypt] [--concurrent [--part-size size] [--num-threads count]] [--name archive-name] vault-name filename glacier archive retrieve [--wait] [--decrypt] [-o filename] [--part-size bytes] vault-name archive-name [archive-name...] glacier archive delete vault-name archive-name glacier job list
- so if I have horrible timeout issues from the script as-is, I'll try manual re-uploads using the 'pkaleta' repo to see if I have better results
Tue Apr 03, 2018
- Marcin pointed out some 'disk full' sql issues on our wordpress site (why is our site leaking error logs anyway?!?). I accidentally filled the disk when testing the glacier restore (12G down + 12G copying backup file + 12G decrypted + >12G uncompressed .. it adds up!). Because we're using varnish, it didn't take the whole site down, just the cache misses. I deleted my glacier restore files & banned the whole varnish cache for all sites to fix it.
- checked the inventory, but the backups I uploaded last night were not listed yet
[root@hetzner2 glacier-cli]# ./glacier.py archive list deleteMeIn2020 hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]# ./glacier.py -h
- kicked-off a sync; I'll check on this tomorrow
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync deleteMeIn2020 glacier: queued inventory job for u'deleteMeIn2020' [root@hetzner2 glacier-cli]#
- the upload script output shows a traceback dump due to an exception: boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)
marcin_ose@hancock:~/backups$ ./uploadToGlacier.sh
+ backupDirs='hetzner1/20171101-062001 hetzner1/20171201-062001'
+ syncDir=/home/marcin_ose/backups/uploadToGlacier
+ encryptionKeyFilePath=/home/marcin_ose/backups/ose-backups-cron.key
+ export AWS_ACCESS_KEY_ID=CHANGEME
+ AWS_ACCESS_KEY_ID=CHANGEME
+ export AWS_SECRET_ACCESS_KEY=CHANGEME
+ AWS_SECRET_ACCESS_KEY=CHANGEME
++ echo hetzner1/20171101-062001 hetzner1/20171201-062001
+ for dir in '$(echo $backupDirs)'
++ echo hetzner1/20171101-062001
++ tr / _
+ archiveName=hetzner1_20171101-062001
++ date -u --rfc-3339=seconds
+ timestamp='2018-04-02 18:15:42+00:00'
+ fileListFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20171101-062001.fileList.txt
+ archiveFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20171101-062001.tar
+ echo ================================================================================
+ echo 'This file is metadata for the archive '\hetzner1_20171101-062001'\. In it, we list all the files included in the
compressed/encrypted archive (produced using '\ls -lahR hetzner1/20171101-062001'\), including the files within the tarba
lls within the archive (produced using '\find hetzner1/20171101-062001 -type f -exec tar -tvf '\{}'\ \; '\)'
+ echo ''
+ echo ' - Michael Altfield <maltfield@opensourceecology.org>'
+ echo ''
+ echo ' Note: this file was generated at 2018-04-02 18:15:42+00:00'
+ echo ================================================================================
+ echo '#############################'
+ echo '# '\ls -lahR'\ output follows #'
+ echo '#############################'
+ ls -lahR hetzner1/20171101-062001
+ echo ================================================================================
+ echo '############################'
+ echo '# tarball contents follows #'
+ echo '############################'
+ find hetzner1/20171201-062001 -type f -exec tar -tvf '{}' ';'
+ echo ================================================================================
+ bzip2 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt
rm: cannot remove ‘/home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt’: No such file or directory
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2.gpg
+ tar -cvf /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar hetzner1/20171201-062001/
hetzner1/20171201-062001/
hetzner1/20171201-062001/public_html/
hetzner1/20171201-062001/public_html/public_html.20171201-062001.tar.bz2
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar.gpg
Traceback (most recent call last):
File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 736, in <module>
main()
File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 732, in main
App().main()
File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 718, in main
self.args.func()
File "/home/marcin_ose/sandbox/glacier-cli/glacier.py", line 500, in archive_upload
file_obj=self.args.file, description=name)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
writer.write(data)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 219, in write
self.partitioner.write(data)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 61, in write
self._send_part()
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part
self.send_fn(part)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part
self.uploader.upload_part(self.next_part_index, part_data)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part
content_range, part_data)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 637, in upload_part
response_headers=response_headers)
File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 84, in make_request
raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)
+ rm -rf /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.fileList.txt.bz2.gpg /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171201-062001.tar.gpg
marcin_ose@hancock:~/backups$
- so it looks like all were successful except the last actual data archive for 'hetzner1_20171201-062001.tar.gpg'
- that's pretty awful if we were to loop it. I could add some retry logic, but the dollar cost of accidental uploading it twice is pretty high :\ for now I'll just run the script again for this second upload only for the archive
- I also updated the script to only delete a file if $? was 0 after the upload attempt. Then, if it fails, I can manually trigger the upload after reviewing the script output and the contents of the directory..
Mon Apr 02, 2018
- dreamhost got back to me on the extension. It wasn't Elizabeth, but Jin. Jin said we could have until Apr 9th. So 3 more days. Hopefully I can get all the data onto glacier by then which is 7 days.
- the backup of the 'hetzner1_20171001-052001' backup data issued last night was complete
marcin_ose@hancock:~/backups$ ./uploadToGlacier.sh
+ backupDirs=hetzner1/20171001-052001
+ syncDir=/home/marcin_ose/backups/uploadToGlacier
+ encryptionKeyFilePath=/home/marcin_ose/backups/ose-backups-cron.key
+ export AWS_ACCESS_KEY_ID=CHANGEME
+ AWS_ACCESS_KEY_ID=CHANGEME
+ export AWS_SECRET_ACCESS_KEY=CHANGEME
+ AWS_SECRET_ACCESS_KEY=CHANGEME
++ echo hetzner1/20171001-052001
+ for dir in '$(echo $backupDirs)'
++ echo hetzner1/20171001-052001
++ tr / _
+ archiveName=hetzner1_20171001-052001
++ date -u --rfc-3339=seconds
+ timestamp='2018-04-01 19:44:13+00:00'
+ fileListFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.fileList.txt
+ archiveFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.tar
+ echo ================================================================================
+ echo 'This file is metadata for the archive '\hetzner1_20171001-052001'\. In it, we list all the files included in the compressed/encrypted archive (produced using '\ls -lahR hetzner1/20171001-052001'\), including the files within the tarballs within the archive (produced using '\find hetzner1/20171001-052001 -type f -exec tar -tvf '\{}'\ \; '\)'
+ echo ''
+ echo ' - Michael Altfield <maltfield@opensourceecology.org>'
+ echo ''
+ echo ' Note: this file was generated at 2018-04-01 19:44:13+00:00'
+ echo ================================================================================
+ echo '#############################'
+ echo '# '\ls -lahR'\ output follows #'
+ echo '#############################'
+ ls -lahR hetzner1/20171001-052001
+ echo ================================================================================
+ echo '############################'
+ echo '# tarball contents follows #'
+ echo '############################'
+ find hetzner1/20171001-052001 -type f -exec tar -tvf '{}' ';'
+ echo ================================================================================
+ bzip2 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.fileList.txt
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.fileList.txt.bz2
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.fileList.txt
rm: cannot remove ‘/home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.fileList.txt’: No such file or directory
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.fileList.txt.bz2.gpg
+ tar -cvf /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.tar hetzner1/20171001-052001/
hetzner1/20171001-052001/
hetzner1/20171001-052001/public_html/
hetzner1/20171001-052001/public_html/public_html.20171001-052001.tar.bz2
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.tar
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.tar
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20171001-052001.tar.gpg
marcin_ose@hancock:~/backups$
- the inventory still doesn't show the above backups, so I issued a sync
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]# [root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync deleteMeIn2020 glacier: queued inventory job for u'deleteMeIn2020' [root@hetzner2 glacier-cli]# [root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- the restore of the 'hetzner1_20170901-052001' backup data from Glacier that I kicked off yesterday appears to have finished
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive retrieve --wait deleteMeIn2020 'hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name' 'hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates' [root@hetzner2 glacier-cli]# timed out waiting for input: auto-logout [maltfield@hetzner2 ~]$
- yep, the files got dropped into the cwd with the archive's description as the filename. This should be more sane going forward now that I've updated the uploadToGlacier.sh script to *not* specify a '--name', which means that glacier-cli will set the archive description to the filename
- before I do anything, I'm going to make copies of these files in case my gpg/tar manipulations deletes the originals and I don't want to pay to download them from Glacier again!
[root@hetzner2 glacier-cli]# mkdir ../orig [root@hetzner2 glacier-cli]# cp hetzner1_20170901-052001.fileList.txt.bz2.gpg\:\ this\ is\ a\ metadata\ file\ showing\ the\ file\ and\ dir\ list\ contents\ of\ the\ archive\ of\ the\ same\ prefix\ name ../orig/ [root@hetzner2 glacier-cli]# cp hetzner1_20170901-052001.tar.gpg\:\ this\ is\ an\ encrypted\ tarball\ of\ a\ backup\ from\ our\ ose\ server\ taken\ at\ the\ time\ that\ the\ archive\ description\ prefix\ indicates ../orig [root@hetzner2 glacier-cli]#
- I renamed the other files to just the filename, removing the description starting with the colon (:), and decrypted them
[root@hetzner2 glacierRestore]# gpg --batch --passphrase-file /root/backups/ose-backups-cron.key --output hetzner1_20170901-052001.fileList.txt.bz2 --decrypt hetzner1_20170901-052001.fileList.txt.bz2.gpg gpg: AES encrypted data gpg: encrypted with 1 passphrase [root@hetzner2 glacierRestore]# ls glacier-cli hetzner1_20170901-052001.fileList.txt.bz2.gpg orig hetzner1_20170901-052001.fileList.txt.bz2 hetzner1_20170901-052001.hetzner1_20170901-052001.tar.gpg [root@hetzner2 glacierRestore]# [root@hetzner2 glacierRestore]# ls glacier-cli hetzner1_20170901-052001.fileList.txt.bz2.gpg hetzner1_20170901-052001.tar.gpg hetzner1_20170901-052001.fileList.txt.bz2 hetzner1_20170901-052001.tar orig [root@hetzner2 glacierRestore]#
- extracted the decrypted archive that was downloaded from glacier
[root@hetzner2 glacierRestore]# tar -xf hetzner1_20170901-052001.tar [root@hetzner2 glacierRestore]# [root@hetzner2 glacierRestore]# ls glacier-cli hetzner1_20170901-052001.fileList.txt.bz2 hetzner1_20170901-052001.tar orig hetzner1 hetzner1_20170901-052001.fileList.txt.bz2.gpg hetzner1_20170901-052001.tar.gpg [root@hetzner2 glacierRestore]#
- extracted the compressed public_html contents of the wrapper, uncompressed tarball
[root@hetzner2 public_html]# tar -xjf public_html.20170901-052001.tar.bz2 [root@hetzner2 public_html]# [root@hetzner2 public_html]# du -sh * 12G public_html.20170901-052001.tar.bz2 20G usr [root@hetzner2 public_html]#
- confirmed that I could read the contents of one of the files after the archive was downloaded from glacier, decrypted, and extract. This completes our end-to-end test of a restore from glacier
[root@hetzner2 public_html]# head usr/www/users/osemain/w/README ##### MediaWiki MediaWiki is a popular and free, open-source wiki software package written in PHP. It serves as the platform for Wikipedia and the other projects of the Wikimedia Foundation, which deliver content in over 280 languages to more than half a billion people each month. MediaWiki's reliability and robust feature set have earned it a large and vibrant community of third-party users and developers. MediaWiki is: [root@hetzner2 public_html]#
Sun Apr 01, 2018
- the reinventory of vault archive metadata finished last night
root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync --max-age=0 --wait deleteMeIn2020 [root@hetzner2 glacier-cli]# timed out waiting for input: auto-logout [maltfield@hetzner2 ~]$
- but the contents is still stale! My best guess is that there is some delay after an archive is uploaded before it's even available to be listed in an inventory
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- I logged into the aws console, which shows the deleteMeIn2020 vault as having 7 archives @ a total size of 13.1 GB. This inventory was last updated at "Apr 1, 2018 5:41:24 AM". The help alt text says the timezone is my system's time. So that's much later than I initiated the inventory last night.
- I also checked the aws console billing while I was in. The month just cutover (it's April 1st--but no jokes here), so I can see the entire bill for March came to a total of $0.15.
- unfortunately, the majority of the fee was in request: 2,872 requests for a total of $0.14. The storage fee itself was just $0.01. Therefore, I should probably look into how to configure boto to increase the chunk size for 'glacier-cli' uploads. Even still, the total would be ~20x that for all our backup dumps to glacier = 0.14*20 = $2.8. Ok, that's cheap enough. Even $5 should be fine.
- I tried a refresh again; same result
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- playing around with the glacier-cli command, I came across this job listing
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 job list i/d 2018-04-01T03:07:35.470Z deleteMeIn2020 i/d 2018-03-31T19:47:45.511Z deleteMeIn2020 [root@hetzner2 glacier-cli]#
- I kicked-off another max-age=0 sync while I poke around (might as well--it takes 4 hours!)
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync --max-age=0 --wait deleteMeIn2020
- then I checked the job list again, and it grew!
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 job list i/p 2018-04-01T13:43:13.901Z deleteMeIn2020 i/d 2018-04-01T03:07:35.470Z deleteMeIn2020 i/d 2018-03-31T19:47:45.511Z deleteMeIn2020 [root@hetzner2 glacier-cli]#
- the top one is the one I just initiated. My best guess is that the 'i' means "Inventory", the 'p' means "inProgress" and the 'd' means "Done"
- hopped over to the aws-cli command to investigate this further
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-jobs --account-id - --vault-name deleteMeIn2020
{
"JobList": [
{
"InventoryRetrievalParameters": {
"Format": "JSON"
},
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"Completed": false,
"JobId": "7Q31uhwAGJ6TmzC-s4nif9ldkoiZYrMh6V1xHE9QoaMGvGcf0qSp7xo76LtrwsVDE5-CIeW1a3UzwnwCiOcZSngCGV1V",
"Action": "InventoryRetrieval",
"CreationDate": "2018-04-01T13:43:13.901Z",
"StatusCode": "InProgress"
},
{
"CompletionDate": "2018-04-01T03:07:35.470Z",
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"InventoryRetrievalParameters": {
"Format": "JSON"
},
"Completed": true,
"InventorySizeInBytes": 2232,
"JobId": "gHpKcv0KXVmfoMOa_TrqeVzLFAzZzpCdwsJl-9FeEbQHQFr6LEwzspwE6nZqrEi1HgmeDixjtWbw1JciInf5QxHc9dFe",
"Action": "InventoryRetrieval",
"CreationDate": "2018-03-31T23:21:19.873Z",
"StatusMessage": "Succeeded",
"StatusCode": "Succeeded"
},
{
"CompletionDate": "2018-03-31T19:47:45.511Z",
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"InventoryRetrievalParameters": {
"Format": "JSON"
},
"Completed": true,
"InventorySizeInBytes": 2232,
"JobId": "5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0",
"Action": "InventoryRetrieval",
"CreationDate": "2018-03-31T16:01:15.869Z",
"StatusMessage": "Succeeded",
"StatusCode": "Succeeded"
}
]
}
hancock%
- so that confirms that all 3x jobs are "InventoryRetrieval" jobs. 2x are "Succeeded" and 1x (the one I just initiated) is "InProgress". The finished ones say the size in bytes is '2232' in both cases. That's 2K, which is probably the size of the inventory itself (ie: the metadata report)--not the size of the vault's archives.
- got the output of the oldest inventory job
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier get-job-output --account-id - --vault-name deleteMeIn2020 --job-id 5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0 output.json
{
"status": 200,
"acceptRanges": "bytes",
"contentType": "application/json"
}
hancock% cat output.json
{"VaultARN":"arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020","InventoryDate":"2018-03-31T15:25:52Z","ArchiveList":[{"ArchiveId":"qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:35:48Z","Size":380236,"SHA256TreeHash":"a1301459044fa4680af11d3e2d60b33a49de7e091491bd02d497bfd74945e40b"},{"ArchiveId":"lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:50:36Z","Size":280709,"SHA256TreeHash":"3f79016e6157ff3e1c9c853337b7a3e7359a9183ae9b26f1d03c1d1c594e45ab"},{"ArchiveId":"fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:53:00Z","Size":280718,"SHA256TreeHash":"6ba4c8a93163b2d3978ae2d87f26c5ad571330ecaa9da3b6161b95074558cef4"},{"ArchiveId":"zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:55:04Z","Size":1187682789,"SHA256TreeHash":"c90c696931ed1dc7cd587dc1820ddb0567a4835bd46db76c9a326215d9950c8f"},{"ArchiveId":"Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:57:50Z","Size":877058000,"SHA256TreeHash":"fdefdad19e585df8324ed25f2f52f7d98bcc368929f84dafa9a4462333af095b"}]}% hancock%
- got the output of the next newest inventory, which I guess is the one I generated last night
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier get-job-output --account-id - --vault-name deleteMeIn2020 --job-id 'gHpKcv0KXVmfoMOa_TrqeVzLFAzZzpCdwsJl-9FeEbQHQFr6LEwzspwE6nZqrEi1HgmeDixjtWbw1JciInf5QxHc9dFe' output.json
{
"status": 200,
"acceptRanges": "bytes",
"contentType": "application/json"
}
hancock% cat output.json
{"VaultARN":"arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020","InventoryDate":"2018-03-31T15:25:52Z","ArchiveList":[{"ArchiveId":"qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:35:48Z","Size":380236,"SHA256TreeHash":"a1301459044fa4680af11d3e2d60b33a49de7e091491bd02d497bfd74945e40b"},{"ArchiveId":"lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:50:36Z","Size":280709,"SHA256TreeHash":"3f79016e6157ff3e1c9c853337b7a3e7359a9183ae9b26f1d03c1d1c594e45ab"},{"ArchiveId":"fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:53:00Z","Size":280718,"SHA256TreeHash":"6ba4c8a93163b2d3978ae2d87f26c5ad571330ecaa9da3b6161b95074558cef4"},{"ArchiveId":"zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:55:04Z","Size":1187682789,"SHA256TreeHash":"c90c696931ed1dc7cd587dc1820ddb0567a4835bd46db76c9a326215d9950c8f"},{"ArchiveId":"Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:57:50Z","Size":877058000,"SHA256TreeHash":"fdefdad19e585df8324ed25f2f52f7d98bcc368929f84dafa9a4462333af095b"}]}% hancock%
- so both of the most recently completed inventory results show only 5 archives in the deleteMeIn2020 vault that's stale. We want the metadata for the 'hetzner1/20170901-052001' backup so we can test to make sure that a >4G archive restoration works before proceeding with dumping the rest of the bacukups into glacier.
- I sent an email to Elizabeth at Dreahost telling her that we already reduced our usage by ~250G, and I asked for an additional 2 weeks so we could validate our Glacier POC before uploading the data before they delete it.
...
- I biked 40km, ate lunch, and checked again; the inventory was complete & now listed the backups that I uploaded last night = 'hetzner1_20170901-052001'
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync --max-age=0 --wait deleteMeIn2020 [root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive list deleteMeIn2020 hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name [root@hetzner2 glacier-cli]#
- as nice as glacier-cli is for uploading, the list tries to simplify things (treating archvies as files), so it excludes the actual archive id. And it lacks the hash, size, & creation timestamp. To get this, I got the json job output using the aws-cli
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier get-job-output --account-id - --vault-name deleteMeIn2020 --job-id '7Q31uhwAGJ6TmzC-s4nif9ldkoiZYrMh6V1xHE9QoaMGvGcf0qSp7xo76LtrwsVDE5-CIeW1a3UzwnwCiOcZSngCGV1V' output.json
{
"status": 200,
"acceptRanges": "bytes",
"contentType": "application/json"
}
hancock% cat output.json
{"VaultARN":"arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020","InventoryDate":"2018-04-01T09:41:24Z","ArchiveList":[{"ArchiveId":"qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:35:48Z","Size":380236,"SHA256TreeHash":"a1301459044fa4680af11d3e2d60b33a49de7e091491bd02d497bfd74945e40b"},{"ArchiveId":"lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:50:36Z","Size":280709,"SHA256TreeHash":"3f79016e6157ff3e1c9c853337b7a3e7359a9183ae9b26f1d03c1d1c594e45ab"},{"ArchiveId":"fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:53:00Z","Size":280718,"SHA256TreeHash":"6ba4c8a93163b2d3978ae2d87f26c5ad571330ecaa9da3b6161b95074558cef4"},{"ArchiveId":"zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:55:04Z","Size":1187682789,"SHA256TreeHash":"c90c696931ed1dc7cd587dc1820ddb0567a4835bd46db76c9a326215d9950c8f"},{"ArchiveId":"Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:57:50Z","Size":877058000,"SHA256TreeHash":"fdefdad19e585df8324ed25f2f52f7d98bcc368929f84dafa9a4462333af095b"},{"ArchiveId":"P9wIGNBbLaAoz7xGht6Y4k7j33nGgPmg0RQ4sesN2tImQLjFN1dtkooVGrBnQqbPt8YhgvwUXv8eO_N72KRjS3RrZQYvkGxAQ9uPcJ-zaDOG8kII7l4p7UzGfaroO63ZreHItIW4GA","ArchiveDescription":"hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name","CreationDate":"2018-03-31T22:46:18Z","Size":2299038,"SHA256TreeHash":"2e789c8c99f08d338f8c1c2440afd76c23f76124c3dbdd33cbfa9f46f5c6b2aa"},{"ArchiveId":"o-naX0m4kQde-2i-8JZbEESi7r8OlFjIoDjgbQSXT_zt9L_e7qOH3HQ1R7ViQC3i7M0lVLbODsGZm9w9HfI3tHYKb2R1T_WWBwMxFuC_OhYiPX8uepTvvBg2Mg6KysP9H3zNzwGSZw","ArchiveDescription":"hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates","CreationDate":"2018-03-31T23:47:51Z","Size":12009829896,"SHA256TreeHash":"022f088abcfadefe7df5ac770f45f315ddee708f2470133ebd027ce988e1a45d"}]}% hancock%
- so I want to restore these 2x archives:
- P9wIGNBbLaAoz7xGht6Y4k7j33nGgPmg0RQ4sesN2tImQLjFN1dtkooVGrBnQqbPt8YhgvwUXv8eO_N72KRjS3RrZQYvkGxAQ9uPcJ-zaDOG8kII7l4p7UzGfaroO63ZreHItIW4GA
- o-naX0m4kQde-2i-8JZbEESi7r8OlFjIoDjgbQSXT_zt9L_e7qOH3HQ1R7ViQC3i7M0lVLbODsGZm9w9HfI3tHYKb2R1T_WWBwMxFuC_OhYiPX8uepTvvBg2Mg6KysP9H3zNzwGSZw
- unfortunately, it appears that the 'glacier-cli' tool also doesn't allow you to specify the id for restoring. I think I'm going to have to just make the archive description a filename at upload to play nice with glacier-cli. in the meantime, I'll use my insane "name" which is actually a human-readable description for the archive (as amazon intended for this field)
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive retrieve --wait deleteMeIn2020 'P9wIGNBbLaAoz7xGht6Y4k7j33nGgPmg0RQ4sesN2tImQLjFN1dtkooVGrBnQqbPt8YhgvwUXv8eO_N72KRjS3RrZQYvkGxAQ9uPcJ-zaDOG8kII7l4p7UzGfaroO63ZreHItIW4GA' 'o-naX0m4kQde-2i-8JZbEESi7r8OlFjIoDjgbQSXT_zt9L_e7qOH3HQ1R7ViQC3i7M0lVLbODsGZm9w9HfI3tHYKb2R1T_WWBwMxFuC_OhYiPX8uepTvvBg2Mg6KysP9H3zNzwGSZw' glacier: archive 'P9wIGNBbLaAoz7xGht6Y4k7j33nGgPmg0RQ4sesN2tImQLjFN1dtkooVGrBnQqbPt8YhgvwUXv8eO_N72KRjS3RrZQYvkGxAQ9uPcJ-zaDOG8kII7l4p7UzGfaroO63ZreHItIW4GA' not found [root@hetzner2 glacier-cli]#
- but first I tried a substring; that failed too
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive retrieve --wait deleteMeIn2020 'hetzner1_20170901-052001.fileList.txt.bz2.gpg' 'hetzner1_20170901-052001.tar.gpg' glacier: archive 'hetzner1_20170901-052001.fileList.txt.bz2.gpg' not found [root@hetzner2 glacier-cli]#
- ok, using the whole description as the name
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 archive retrieve --wait deleteMeIn2020 'hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name' 'hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates'
- that command appears to be waiting; I'll check on that tomorrow
- in the meantime, on the dreamhost server, I'll fix the script to just use the filename as the description & upload the next archive using this setting
- note that it's a new month, so dreamhost has our first-of-the-month backups for 2018-04-01. Therefore, here's the updated list I'll be sending to this deleteMeIn2020 vault (total 276G)
20G hetzner2/20170702-052001 1.7G hetzner2/20170801-072001 1.7G hetzner2/20170901-072001 2.5G hetzner2/20171001-072001 838M hetzner2/20171101-072001 997M hetzner2/20171202-072001 1.1G hetzner2/20180102-072001 14G hetzner2/20180202-072001 25G hetzner2/20180302-072001 2.8G hetzner2/20180401-072001 39G hetzner1/20170701-052001 39G hetzner1/20170801-052001 12G hetzner1/20170901-052001 12G hetzner1/20171001-052001 12G hetzner1/20171101-062001 12G hetzner1/20171201-062001 12G hetzner1/20180101-062001 27G hetzner1/20180201-062001 28G hetzner1/20180301-062002 12G hetzner1/20180401-052001
- the last backup was ' hetzner1/20170901-052001', so for this test I'll use the following month that's the same size (12G, which is our smallest size that exceeds the 4G limit) = 'hetzner1/20171001-052001'
- I added a line to delete the contents of the 'uploadToGlacier/' directory after the upload to glacier
- I updated the uploadToGlacier.sh script to attempt to upload the next 2x backups by setting 'backupDirs="hetzner1/20171101-062001 hetzner1/20171201-062001"'
- I kicked-off the uploadToGlacier.sh script. If the next sync in ~48 hours shows all 3x backups using the description as the file name (per glacier-cli's desire), then I think I can execute this script for the remaining backups.
Sat Mar 31, 2018
- the run from last night finished; here's the output:
hancock% ./uploadToGlacier.sh
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors
Reading passphrase from file descriptor 3
rm: cannot remove ‘/home/marcin_ose/backups/uploadToGlacier/hetzner2_20171101-072001.fileList.txt’: No such file or directory
Error parsing parameter '--body': Blob values must be a path to a file.
{
"archiveId": "fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw",
"checksum": "6ba4c8a93163b2d3978ae2d87f26c5ad571330ecaa9da3b6161b95074558cef4",
"location": "/099400651767/vaults/deleteMeIn2020/archives/fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCp
YkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw"
}
hetzner2/20171101-072001/
hetzner2/20171101-072001/etc/
hetzner2/20171101-072001/etc/etc.20171101-072001.tar.gz
hetzner2/20171101-072001/home/
hetzner2/20171101-072001/home/home.20171101-072001.tar.gz
hetzner2/20171101-072001/log/
hetzner2/20171101-072001/log/log.20171101-072001.tar.gz
hetzner2/20171101-072001/mysqldump/
hetzner2/20171101-072001/mysqldump/mysqldump.20171101-072001.sql.gz
hetzner2/20171101-072001/root/
hetzner2/20171101-072001/root/root.20171101-072001.tar.gz
hetzner2/20171101-072001/www/
hetzner2/20171101-072001/www/www.20171101-072001.tar.gz
Reading passphrase from file descriptor 3
Reading passphrase from file descriptor 3
{
"archiveId": "zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA",
"checksum": "c90c696931ed1dc7cd587dc1820ddb0567a4835bd46db76c9a326215d9950c8f",
"location": "/099400651767/vaults/deleteMeIn2020/archives/zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA"
}
{
"archiveId": "Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw",
"checksum": "fdefdad19e585df8324ed25f2f52f7d98bcc368929f84dafa9a4462333af095b",
"location": "/099400651767/vaults/deleteMeIn2020/archives/Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw"
}
hancock%
- logged into the aws console to check the status of the Glacier uploads I did last night. It nows shows 1 archive, though there should be a few. It shows the "Inventory Last Updated" at "Mar 31, 2018 5:22:27 AM"
- I checked our billing page in the aws console, and it now shows $0.01 due to 5 requests to Glacier in usw2. That fee is $0.050 per 1,000 requests.
- did some research on the glacier docs, and found that archives apparently don't have names; they're given a 138-byte uid by glacier. Therefore, I should include the name of the backup (which includes the very important timestamp that the backup was taken & the server it was taken on) in the description. I also learned that I can't force an inventory to be taken of a vault, and that ti's done "approximately daily" https://docs.aws.amazon.com/amazonglacier/latest/dev/working-with-archives.html#client-side-key-map-concept
Except for the optional archive description, Amazon Glacier does not support any additional metadata for the archives. When you upload an archive Amazon Glacier assigns an ID, an opaque sequence of characters, from which you cannot infer any meaning about the archive. You might maintain metadata about the archives on the client-side. The metadata can include archive name and some other meaningful information about the archive.
Note: If you are an Amazon Simple Storage Service (Amazon S3) customer, you know that when you upload an object to a bucket, you can assign the object an object key such as MyDocument.txt or SomePhoto.jpg. In Amazon Glacier, you cannot assign a key name to the archives you upload.
If you maintain client-side archive metadata, note that Amazon Glacier maintains a vault inventory that includes archive IDs and any descriptions you provided during the archive upload. You might occasionally download the vault inventory to reconcile any issues in your client-side database you maintain for the archive metadata. However, Amazon Glacier takes vault inventory approximately daily. When you request a vault inventory, Amazon Glacier returns the last inventory it prepared, a point in time snapshot.
- there is no way to list all archvies. Archive operations are: upload, download, and delete. No list!
- there is a 'describe-vault' argument, which gives apparently more up-to-date results than the aws console. Here it shows we have 5 archives, whereas the console still shows 1 archive.
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier describe-vault --account-id - --vault-name deleteMeIn2020
{
"SizeInBytes": 2065846292,
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"LastInventoryDate": "2018-03-31T15:25:52.860Z",
"NumberOfArchives": 5,
"CreationDate": "2018-03-29T21:36:06.041Z",
"VaultName": "deleteMeIn2020"
}
hancock%
- I clicked the 'refresh" circle button in the aws console in the glacier Service, and it updated to list 5 archives & a size of 1.9 GB for the deleteMeIn2020 vault. It does not appear to show the archives ids & descriptions only the number & total size.
- for future reference, here's the objects that I have documented uploading
- qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA
- ??
- fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw
- zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA
- Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw
- I also read that you cannot use the upload method I used yesterday for files >4G. Instead, you have to use "multipart upload" for archives >4G (which is most of our backups). https://docs.aws.amazon.com/amazonglacier/latest/dev/uploading-an-archive.html
Options for Uploading an Archive to Amazon Glacier
Depending on the size of the data you are uploading, Amazon Glacier offers the following options:
Upload archives in a single operation In a single operation, you can upload archives from 1 byte to up to 4 GB in size. However, we encourage Amazon Glacier customers to use multipart upload to upload archives greater than 100 MB. For more information, see Uploading an Archive in a Single Operation.
Upload archives in parts Using the multipart upload API, you can upload large archives, up to about 40,000 GB (10,000 * 4 GB).
The multipart upload API call is designed to improve the upload experience for larger archives. You can upload archives in parts. These parts can be uploaded independently, in any order, and in parallel. If a part upload fails, you only need to upload that part again and not the entire archive. You can use multipart upload for archives from 1 byte to about 40,000 GB in size. For more information, see Uploading Large Archives in Parts (Multipart Upload).
- found more info on multipart uploads here, but I'll need to test this https://docs.aws.amazon.com/amazonglacier/latest/dev/uploading-archive-mpu.html
- apparently the way to view the contents of a vault is to use the initiate-job subcommand with inventory-retrieval type https://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html
- I initiated a inventory-retrieval job for the deleteMeIn2020 vault
hancock% aws glacier initiate-job --account-id - --vault-name deleteMeIn2020 --job-parameters '{"Type": "inventory-retrieval"}'
zsh: command not found: aws
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier initiate-job --account-id - --vault-name deleteMeIn2020 --job-parameters '{"Type": "inventory-retrieval"}'
{
"location": "/099400651767/vaults/deleteMeIn2020/jobs/5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0",
"jobId": "5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0"
}
hancock%
- and a list-jobs shows the job is InProgress
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-jobs --account-id - --vault-name deleteMeIn2020
{
"JobList": [
{
"InventoryRetrievalParameters": {
"Format": "JSON"
},
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"Completed": false,
"JobId": "5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0",
"Action": "InventoryRetrieval",
"CreationDate": "2018-03-31T16:01:15.869Z",
"StatusCode": "InProgress"
}
]
}
hancock%
- updated the script to uploadToGlacier.sh script to include the $archiveName in the archive-description, since there is no archive name--it's just identified by a uid.
- I've read that if we use a lifecycle rule in s3 to offload to glacier, then the metadata stays in s3, making retrievals much easier
- this could take up to 4 hours. when the job is available, I think it will only be available for 24 hours. I got the command to download it, and got an error that it's not ready for download yet
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier get-job-output --account-id - --vault-name deleteMeIn2020 --job-id '5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0' output.json An error occurred (InvalidParameterValueException) when calling the GetJobOutput operation: The job is not currently available for download: 5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0 hancock%
- also tested describe-job, which showed the same output as list-jobs, though not in an array
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier describe-job --account-id - --vault-name deleteMeIn2020 --job-id '5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0'
{
"InventoryRetrievalParameters": {
"Format": "JSON"
},
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"Completed": false,
"JobId": "5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0",
"Action": "InventoryRetrieval",
"CreationDate": "2018-03-31T16:01:15.869Z",
"StatusCode": "InProgress"
}
hancock%
- there was no good way to wait on the job & automatically download the output, so I wrote a simple script. I think this should take >4 hours
hancock% cat getGlacierJob.sh
#!/bin/bash
############
# SETTINGS #
############
jobId='5COrR-wLYeA8ZTyhlBI50Pq4Egnx5G11OmTZ2lVwpuuJTgdvwbEeC1rY1dzST0fCPRm1-D_pvHH5wyg1fJpIhgHJ4ii0'
#################
# QUERY FOR JOB #
#################
loop=true;
while $loop -eq true; do
echo "INFO: attempting to get job status"
date
jobStatus=`/home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier describe-job --account-id - --vault-name deleteMeIn2020 --job-id "${jobId}" | grep "StatusCode" | cut -d: -f2 | xargs echo -n`
if grep "InProgress"` ; then
# job is still InProgress; wait 30 minutes before querying again
echo "INFO: job status was InProgress; sleeping for 30 minutes before trying again"
sleep 1800
else
# job is not InProgress; exit this wait loop so we can get the output
echo "INFO: job status was not InProgress! Exiting wait loop."
loop=false;
fi
echo ''
done
##################
# GET JOB OUTPUT #
##################
/home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier get-job-output --account-id - --vault-name deleteMeIn2020 --job-id "${jobId}" output.json
exit 0
hancock%
- when this finally finished, the output was just a status = 200
hancock% ./getGlacierJob.sh
...
INFO: attempting to get job status
Sat Mar 31 13:08:23 PDT 2018
INFO: job status was not InProgress! Exiting wait loop.
{
"status": 200,
"acceptRanges": "bytes",
"contentType": "application/json"
}
hancock%
- and the output stored to a json file:
hancock% cat output.json
{"VaultARN":"arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020","InventoryDate":"2018-03-31T15:25:52Z","ArchiveList":[{"ArchiveId":"qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:35:48Z","Size":380236,"SHA256TreeHash":"a1301459044fa4680af11d3e2d60b33a49de7e091491bd02d497bfd74945e40b"},{"ArchiveId":"lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:50:36Z","Size":280709,"SHA256TreeHash":"3f79016e6157ff3e1c9c853337b7a3e7359a9183ae9b26f1d03c1d1c594e45ab"},{"ArchiveId":"fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw","ArchiveDescription":"this is a metadata file showing the file and dir list contents of the archive of the same name","CreationDate":"2018-03-31T02:53:00Z","Size":280718,"SHA256TreeHash":"6ba4c8a93163b2d3978ae2d87f26c5ad571330ecaa9da3b6161b95074558cef4"},{"ArchiveId":"zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:55:04Z","Size":1187682789,"SHA256TreeHash":"c90c696931ed1dc7cd587dc1820ddb0567a4835bd46db76c9a326215d9950c8f"},{"ArchiveId":"Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw","ArchiveDescription":"this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates","CreationDate":"2018-03-31T02:57:50Z","Size":877058000,"SHA256TreeHash":"fdefdad19e585df8324ed25f2f52f7d98bcc368929f84dafa9a4462333af095b"}]}% hancock%
- while I'm waiting for the inventory, I decided to dig into how to upload archives >4G
- the smallest backup we have that's >4G is 12G. I'll test with this 12G backup: 'hetzner1/20170901-052001'
- I updated the uploadToGlacier.sh script with the -x builtin & changed backupDirs to = 'hetzner1/20170901-052001'. I ran it, expecting it to fail because it would attempt to upload a file >4G. When/if it fails, I'll have the .gpg file prepared & can attempt to manually upload it
- this failed as follows
hancock% ./uploadToGlacier.sh
+ backupDirs=hetzner1/20170901-052001
+ syncDir=/home/marcin_ose/backups/uploadToGlacier
+ encryptionKeyFilePath=/home/marcin_ose/backups/ose-backups-cron.key
++ echo hetzner1/20170901-052001
+ for dir in '$(echo $backupDirs)'
++ echo hetzner1/20170901-052001
++ tr / _
+ archiveName=hetzner1_20170901-052001
++ date -u --rfc-3339=seconds
+ timestamp='2018-03-31 19:08:48+00:00'
+ fileListFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt
+ archiveFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gz
+ echo ================================================================================
+ echo 'This file is metadata for the archive '\hetzner1_20170901-052001'\. In it, we list all the files included in the compressed/encrypted archive (produced using '\ls -lahR hetzner1/20170901-052001'\), including the files within the tarballs within the archive (produced using '\find hetzner1/20170901-052001 -type f -exec tar -tvf '\{}'\ \; '\)'
+ echo ''
+ echo ' - Michael Altfield <maltfield@opensourceecology.org>'
+ echo ''
+ echo ' Note: this file was generated at 2018-03-31 19:08:48+00:00'
+ echo ================================================================================
+ echo '#############################'
+ echo '# '\ls -lahR'\ output follows #'
+ echo '#############################'
+ ls -lahR hetzner1/20170901-052001
+ echo ================================================================================
+ echo '############################'
+ echo '# tarball contents follows #'
+ echo '############################'
+ find hetzner1/20170901-052001 -type f -exec tar -tvf '{}' ';'
+ echo ================================================================================
+ bzip2 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt.bz2
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt
rm: cannot remove ‘/home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt’: No such file or directory
+ tar -czvf /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gz hetzner1/20170901-052001/
hetzner1/20170901-052001/
hetzner1/20170901-052001/public_html/
hetzner1/20170901-052001/public_html/public_html.20170901-052001.tar.bz2
+ gpg --symmetric --cipher-algo aes --armor --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gz
Reading passphrase from file descriptor 3
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gz
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gz
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier upload-archive --account-id - --vault-name deleteMeIn2020 --archive-description 'hetzner1_20170901-052001: this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates' --body /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gz.gpg
An error occurred (InvalidParameterValueException) when calling the UploadArchive operation: Invalid Content-Length: 11989358711
hancock%
- it's also important to note the size differences between encrypting the archive using ascii-armor (16G) vs not (12G). We should def skip '--armor' if it doesn't break anything
hancock% while true; do date; du -sh uploadToGlacier/*; sleep 300; echo; done Sat Mar 31 13:51:20 PDT 2018 2.2M uploadToGlacier/hetzner1_20170901-052001.fileList.txt.bz2 2.2M uploadToGlacier/hetzner1_20170901-052001.fileList.txt.bz2.gpg 16G uploadToGlacier/hetzner1_20170901-052001.tar.gz.asc 12G uploadToGlacier/hetzner1_20170901-052001.tar.gz.gpg 276K uploadToGlacier/hetzner2_20171101-072001.fileList.txt.bz2 276K uploadToGlacier/hetzner2_20171101-072001.fileList.txt.bz2.gpg 1.2G uploadToGlacier/hetzner2_20171101-072001.tar.gz.asc 837M uploadToGlacier/hetzner2_20171101-072001.tar.gz.gpg hancock%
- I found 4 relevant subcommands for multiplart uploads
- list-multipart-uploads
- initiate-multipart-upload
- upload-multipart-part
- complete-multipart-upload
- tested list-multipart-uploads
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-multipart-uploads --account-id - --vault-name deleteMeIn2020
{
"UploadsList": []
}
hancock%
- initiate-multipart-upload appears to just return a multipart upload resource ID, which is needed to be specified when uploading the parts using upload-multipart-part
- it looks like aws-cli will *not* split our file into parts; we have to do that. I found an official aws guide that suggests using the `split` command, which I confirmed is present on hancock https://docs.aws.amazon.com/cli/latest/userguide/cli-using-glacier.html
hancock% split --help Usage: split [OPTION]... [INPUT [PREFIX]] Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default size is 1000 lines, and default PREFIX is 'x'. With no INPUT, or when INPUT is -, read standard input. Mandatory arguments to long options are mandatory for short options too. -a, --suffix-length=N generate suffixes of length N (default 2) --additional-suffix=SUFFIX append an additional SUFFIX to file names. -b, --bytes=SIZE put SIZE bytes per output file -C, --line-bytes=SIZE put at most SIZE bytes of lines per output file -d, --numeric-suffixes[=FROM] use numeric suffixes instead of alphabetic. FROM changes the start value (default 0). -e, --elide-empty-files do not generate empty output files with '-n' --filter=COMMAND write to shell COMMAND; file name is $FILE -l, --lines=NUMBER put NUMBER lines per output file -n, --number=CHUNKS generate CHUNKS output files. See below -u, --unbuffered immediately copy input to output with '-n r/...' --verbose print a diagnostic just before each output file is opened --help display this help and exit --version output version information and exit SIZE is an integer and optional unit (example: 10M is 10*1024*1024). Units are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (powers of 1000). CHUNKS may be: N split into N files based on size of input K/N output Kth of N to stdout l/N split into N files without splitting lines l/K/N output Kth of N to stdout without splitting lines r/N like 'l' but use round robin distribution r/K/N likewise but only output Kth of N to stdout Report split bugs to bug-coreutils@gnu.org GNU coreutils home page: <http://www.gnu.org/software/coreutils/> General help using GNU software: <http://www.gnu.org/gethelp/> For complete documentation, run: info coreutils 'split invocation' hancock%
- the document also describes how to generate the treehash using openssl & cat. It's documented, but not trivial. And the iteration process is so slow, it may be better to use a 3rd party tool that does this.
- the best tool from previous research appears to be glacier-cli, so I'll attempt to install it on hancock
cd mkdir sandbox cd sandbox git clone git://github.com/basak/glacier-cli.git cd glacier-cli
- execution directly fails with a missing iso8601 module
hancock% ./glacier.py Traceback (most recent call last): File "./glacier.py", line 37, in <module> import iso8601 ImportError: No module named iso8601 hancock%
- execution with the python binary using aws-cli also fails with a missing boto library
hancock% /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py Traceback (most recent call last): File "./glacier.py", line 36, in <module> import boto.glacier ImportError: No module named boto.glacier hancock%
- this was shocking; I'd expect aws-cli to use boto! I did find it, but it's called botocore
hancock% find /home/marcin_ose | grep -i boto ... /home/marcin_ose/.local/lib/aws/lib/python2.7/site-packages/botocore ... hancock%
- I went to the boto github & followed their pip install instructions, but it said it's already installed https://github.com/boto/boto/
hancock% pip install boto Requirement already satisfied (use --upgrade to upgrade): boto in /usr/lib/python2.7/dist-packages Cleaning up... hancock%
- checking the sourcecode of glacier-cli shows that the boto import occurs before the iso8601 import, so probably we're better off using pip to get that for the system-python, if possible
hancock% grep -C 2 'boto.glacier' glacier.py | head -n 4 import time import boto.glacier import iso8601 hancock%
- attempted to install iso8601 with pip, but zsh killed it for "excessive resource usage" ??
hancock% pip install ios8601 Downloading/unpacking ios8601 Yikes! One of your processes (pip, pid 10498) was just killed for excessive resource usage. Please contact DreamHost Support for details. zsh: killed pip install ios8601 hancock%
- attempted to install using bash; same result
hancock% bash marcin_ose@hancock:~/sandbox/glacier-cli$ pip install ios8601 Downloading/unpacking ios8601 Yikes! One of your processes (pip, pid 11572) was just killed for excessive resource usage. Please contact DreamHost Support for details. Killed marcin_ose@hancock:~/sandbox/glacier-cli$
- attempted to install again, prefixing with nice. same result :(
marcin_ose@hancock:~/sandbox/glacier-cli$ nice pip install ios8601 Downloading/unpacking ios8601 Yikes! One of your processes (pip, pid 12348) was just killed for excessive resource usage. Please contact DreamHost Support for details. Killed marcin_ose@hancock:~/sandbox/glacier-cli$
- got a list of the PATH dirs for the built-in python
marcin_ose@hancock:~/sandbox/glacier-cli$ python Python 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> print sys.path ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/pymodules/python2.7'] >>> marcin_ose@hancock:~/sandbox/glacier-cli$
- got a list of the PATH dirs for the aws-cli python
marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python Python 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> print sys.path; ['', '/home/marcin_ose/.local/lib/aws/lib/python2.7', '/home/marcin_ose/.local/lib/aws/lib/python2.7/plat-x86_64-linux-gnu', '/home/marcin_ose/.local/lib/aws/lib/python2.7/lib-tk', '/home/marcin_ose/.local/lib/aws/lib/python2.7/lib-old', '/home/marcin_ose/.local/lib/aws/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages', '/home/marcin_ose/.local/lib/aws/lib/python2.7/site-packages'] >>> marcin_ose@hancock:~/sandbox/glacier-cli$
- found boto in /usr/lib/python2.7/dist-packages/boto/
marcin_ose@hancock:~/sandbox/glacier-cli$ find /usr/lib | grep -i boto ... /usr/lib/python2.7/dist-packages/boto ... marcin_ose@hancock:~/sandbox/glacier-cli$
- copied the directory from the system library dir into our user-specific library dir
marcin_ose@hancock:~/sandbox/glacier-cli$ cp -r /usr/lib/python2.7/dist-packages/boto /home/marcin_ose/.local/lib/aws/lib/python2.7/site-packages/ marcin_ose@hancock:~/sandbox/glacier-cli$
- better; now the local one fails on iso8601
hancock% /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py Traceback (most recent call last): File "./glacier.py", line 37, in <module> import iso8601 ImportError: No module named iso8601 hancock%
- fuck pip on this box without root. trying from source
cd mkdir src cd src wget "https://pypi.python.org/packages/45/13/3db24895497345fb44c4248c08b16da34a9eb02643cea2754b21b5ed08b0/iso8601-0.1.12.tar.gz" tar -xzvf iso8601-0.1.12.tar.gz cd iso8601-0.1.12
- attempted to install
hancock% /home/marcin_ose/.local/lib/aws/bin/python setup.py install running install running bdist_egg running egg_info writing iso8601.egg-info/PKG-INFO writing top-level names to iso8601.egg-info/top_level.txt writing dependency_links to iso8601.egg-info/dependency_links.txt reading manifest file 'iso8601.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'iso8601.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/iso8601 copying iso8601/init.py -> build/lib.linux-x86_64-2.7/iso8601 copying iso8601/iso8601.py -> build/lib.linux-x86_64-2.7/iso8601 copying iso8601/test_iso8601.py -> build/lib.linux-x86_64-2.7/iso8601 creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/iso8601 copying build/lib.linux-x86_64-2.7/iso8601/init.py -> build/bdist.linux-x86_64/egg/iso8601 copying build/lib.linux-x86_64-2.7/iso8601/iso8601.py -> build/bdist.linux-x86_64/egg/iso8601 copying build/lib.linux-x86_64-2.7/iso8601/test_iso8601.py -> build/bdist.linux-x86_64/egg/iso8601 byte-compiling build/bdist.linux-x86_64/egg/iso8601/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/iso8601/iso8601.py to iso8601.pyc byte-compiling build/bdist.linux-x86_64/egg/iso8601/test_iso8601.py to test_iso8601.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying iso8601.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying iso8601.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying iso8601.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying iso8601.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/iso8601-0.1.12-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing iso8601-0.1.12-py2.7.egg Copying iso8601-0.1.12-py2.7.egg to /home/marcin_ose/.local/lib/aws/lib/python2.7/site-packages Adding iso8601 0.1.12 to easy-install.pth file Installed /home/marcin_ose/.local/lib/aws/lib/python2.7/site-packages/iso8601-0.1.12-py2.7.egg Processing dependencies for iso8601==0.1.12 Finished processing dependencies for iso8601==0.1.12 hancock%
- that worked, next fail @ sqlalchemy
marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py Traceback (most recent call last): File "./glacier.py", line 38, in <module> import sqlalchemy ImportError: No module named sqlalchemy marcin_ose@hancock:~/sandbox/glacier-cli$
- followed the same procedure for sqlalchemy
wget "https://pypi.python.org/packages/da/ef/f10a6892f8ff3c1fec1c25699a7379d1f72f291c8fa40b71c31cab3f779e/SQLAlchemy-1.2.6.tar.gz" tar -xzvf SQLAlchemy-1.2.6.tar.gz cd SQLAlchemy-1.2.6 /home/marcin_ose/.local/lib/aws/bin/python setup.py install
- that worked!
marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py
usage: glacier.py [-h] [--region REGION] {vault,archive,job} ...
glacier.py: error: too few arguments
marcin_ose@hancock:~/sandbox/glacier-cli$
- glacier-cli uses a client-side metadata cache for the archives to solve the issues I encountered above when doing this shit manually, so the first step is to initiate a sync. Unfortunately, that failed; it needs my access keys & stuff
marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py vault sync --wait deleteMeIn2020 Traceback (most recent call last): File "./glacier.py", line 736, in <module> main() File "./glacier.py", line 732, in main App().main() File "./glacier.py", line 707, in init connection = boto.glacier.connect_to_region(args.region) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/init.py", line 59, in connect_to_region return region.connect(**kw_params) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/regioninfo.py", line 63, in connect return self.connection_cls(region=self, **kw_params) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer2.py", line 38, in init self.layer1 = Layer1(*args, **kwargs) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/glacier/layer1.py", line 63, in init suppress_consec_slashes) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/connection.py", line 559, in init host, config, self.provider, self._required_auth_capability()) File "/home/marcin_ose/.local/lib/aws/local/lib/python2.7/site-packages/boto/auth.py", line 732, in get_auth_handler 'Check your credentials' % (len(names), str(names))) boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials marcin_ose@hancock:~/sandbox/glacier-cli$
- there is no documentation on how to add creds in the glacier-cli project, but an issue ticket suggests to use env vars. boto suggests that it can pull from my ~/.aws/credentials file. I tried setting the env vars too, but got the same result https://boto3.readthedocs.io/en/latest/guide/configuration.html
- err, I missed the 'export' prefix for the vars. adding that worked
marcin_ose@hancock:~/sandbox/glacier-cli$ export AWS_ACCESS_KEY_ID='CHANGEME' marcin_ose@hancock:~/sandbox/glacier-cli$ export AWS_SECRET_ACCESS_KEY='CHANGEME' marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py --region us-west-2 vault sync --wait deleteMeIn2020 marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py --region us-west-2 vault list deleteMeIn2020 marcin_ose@hancock:~/sandbox/glacier-cli$
- this is fucking awesome! the archive list works wonders, and stores it locally
marcin_ose@hancock:~/sandbox/glacier-cli$ /home/marcin_ose/.local/lib/aws/bin/python ./glacier.py archive list deleteMeIn2020 id:zr-OjFat_oTJ4k_bMRdczuqDL_GNBpbgTVcHYSg6N-vTWvCe9FNgxJXrFeT26eL2LiXMEpijzaretHvFdyFYQarfZZzcFr0GEEB2O4rVEjtslkGuhbHfWMIGFbQZXQgmjE9aKl4EpA this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:Rb3XtAMEDXlx4KSEZ_-OdA121VJ4jHPEPHIGr33GUJ7wbixaxIzSa5gXV-2i_7-AH-_KUCuLMQbmMPxRN7an7xmMr3PHlzdZMXQj1YTFlJC0g2BT2_F1HJf8h6IocDcR-7EJQeFTqw this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates id:qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA this is a metadata file showing the file and dir list contents of the archive of the same name id:lEJNkWsTF-zZ1fj_2XDVrgbTFGhthkMo0FsLyCb7EM18JrQ-SimUAhAi7HtkrTZMT-wuYSDupFGDVzh87cZlzxRXrex_9NHtTkQyp93A2gICb9zOLDViUr8gHJO6AcyN-R9j2yiIDw this is a metadata file showing the file and dir list contents of the archive of the same name id:fOeCrDHiQUrbvZoyT-jkSQH_euCAhtRcy8wetvONgUWyJBYzxM7AMmbc4YJzRuroL57hVmIUDQRHS-deAo3WG0esgBU52W2qes-47L1-VkczCpYkeGQjlNFGXaKE7ZeZ6jgZ3hBnpw this is a metadata file showing the file and dir list contents of the archive of the same name marcin_ose@hancock:~/sandbox/glacier-cli$
- the archive metadata itself is stored locally in ~/.cache/glacier-cli/db. It's only a few K.
marcin_ose@hancock:~/sandbox/glacier-cli$ ls -lah ~/.cache/glacier-cli/db -rw-r--r-- 1 marcin_ose pg1589252 5.0K Mar 31 14:19 /home/marcin_ose/.cache/glacier-cli/db
- removed the gz compression of the tarball creation for the archive; it shouldn't buy us anything for these files that are already compressed..
- updated the uploadToGlacier.sh script to use glacer-cli
- ran the uploadToGlacier.sh script again; hopefully this should fully do the upload for 'hetzner1/20170901-052001'
- while I waited for the uploadToGlacier.sh script to finish for 'hetzner1/20170901-052001', I began to document why we don't use provisioning tool on our wiki (config rot) http://opensourceecology.org/wiki/OSE_Server#Provisioning
- the uploadToGlacier.sh script finished
marcin_ose@hancock:~/backups$ ./uploadToGlacier.sh
+ backupDirs=hetzner1/20170901-052001
+ syncDir=/home/marcin_ose/backups/uploadToGlacier
+ encryptionKeyFilePath=/home/marcin_ose/backups/ose-backups-cron.key
++ echo hetzner1/20170901-052001
+ for dir in '$(echo $backupDirs)'
++ echo hetzner1/20170901-052001
++ tr / _
+ archiveName=hetzner1_20170901-052001
++ date -u --rfc-3339=seconds
+ timestamp='2018-03-31 21:38:39+00:00'
+ fileListFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt
+ archiveFilePath=/home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar
+ echo ================================================================================
+ echo 'This file is metadata for the archive '\hetzner1_20170901-052001'\. In it, we list all the files included in the compressed/encrypted archive (produced using '\ls -lahR hetzner1/20170901-052001'\), including the files within the tarballs within the archive (produced using '\find hetzner1/20170901-052001 -type f -exec tar -tvf '\{}'\ \; '\)'
+ echo ''
+ echo ' - Michael Altfield <maltfield@opensourceecology.org>'
+ echo ''
+ echo ' Note: this file was generated at 2018-03-31 21:38:39+00:00'
+ echo ================================================================================
+ echo '#############################'
+ echo '# '\ls -lahR'\ output follows #'
+ echo '#############################'
+ ls -lahR hetzner1/20170901-052001
+ echo ================================================================================
+ echo '############################'
+ echo '# tarball contents follows #'
+ echo '############################'
+ find hetzner1/20170901-052001 -type f -exec tar -tvf '{}' ';'
+ echo ================================================================================
+ bzip2 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt.bz2
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt
rm: cannot remove ‘/home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt’: No such file or directory
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload --name 'hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name' /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt.bz2.gpg
usage: glacier.py archive upload [-h] [--name NAME] vault file
glacier.py archive upload: error: too few arguments
+ tar -cvf /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar hetzner1/20170901-052001/
hetzner1/20170901-052001/
hetzner1/20170901-052001/public_html/
hetzner1/20170901-052001/public_html/public_html.20170901-052001.tar.bz2
+ gpg --symmetric --cipher-algo aes --passphrase-file /home/marcin_ose/backups/ose-backups-cron.key /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar
Reading passphrase from file descriptor 3
+ rm /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar
+ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload --name 'hetzner1_20170901-052001.tar.gpg: this is an encrypted tarball of a backup from our ose server taken at the time that the archive description prefix indicates' /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.tar.gpg
usage: glacier.py archive upload [-h] [--name NAME] vault file
glacier.py archive upload: error: too few arguments
marcin_ose@hancock:~/backups$
- damn, I missed the vault name in the command. did it manually & updated the script
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/sandbox/glacier-cli/glacier.py --region us-west-2 archive upload --name 'hetzner1_20170901-052001.fileList.txt.bz2.gpg: this is a metadata file showing the file and dir list contents of the archive of the same prefix name' deleteMeIn2020 /home/marcin_ose/backups/uploadToGlacier/hetzner1_20170901-052001.fileList.txt.bz2.gpg hancock%
- a backup isn't a backup until a restore has been tested, so I'm going to fully test a restore of this 'hetzner1/20170901-052001' from a distinct machine (hetzner2) and verify that I can download the archive, decrypt it, extract it, and--finally--cat a config file (as an end-to-end test of the restore procedure)
- I successfully installed glacier-cli on hetzner2 & documented how on the wiki OSE_Server#Restore_from_Glacier
- I kicked-off an inventory update on hetnzer2; this will take ~4 hours. I'll have to wait until tomorrow to kick-off the job to download the 'hetzner1/20170901-052001' backup from Glacier onto hetzner2.
[root@hetzner2 glacier-cli]# ./glacier.py --region us-west-2 vault sync --max-age=0 --wait deleteMeIn2020
- checked the aws console for the billing dashboard; it now shows $0.02. $0.01 for 6 requests to glacier & $0.01 for 0.062 GB-Mo for "USW2-TimedStorage-ByteHrs"
- for some reason S3 popped-up as showing 3 requests. We're well within the free-tier for these requests, but I guess that might be glacier-cli doing some s3 requests?
Fri Mar 30, 2018
- tried to install aws-cli on dreamhost, just for our user
# create temporary directory tmpdir=`mktemp -d` pushd "$tmpdir" /usr/bin/wget "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" /usr/bin/unzip awscli-bundle.zip ./awscli-bundle/install -b ~/bin/aws # cleanly exit popd /bin/rm -rf "$tmpdir" exit 0
- unfortunately, we still got a "permission denied"
hancock% ./awscli-bundle/install -b ~/bin/aws zsh: permission denied: ./awscli-bundle/install hancock%
- I tried switching from the default shell (zsh) to bash & ensured I had execute permissions, but the same issue occurred
- I checked the shebang, and I saw it was '/usr/bin/env/python'. So I just tried to execute it with the python shell, and that seemed to work:
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ python ./awscli-bundle/install -b ~/bin/aws Running cmd: /usr/bin/python virtualenv.py --no-download --python /usr/bin/python /home/marcin_ose/.local/lib/aws Running cmd: /home/marcin_ose/.local/lib/aws/bin/pip install --no-index --find-links file:///tmp/tmp.zfN3S0BJAC/awscli-bundle/packages awscli-1.14.67.tar.gz You can now run: /home/marcin_ose/bin/aws --version marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /usr/bin/env python
- execution of the now "installed" aws-cli package still failed
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ ls -lah /home/marcin_ose/bin/aws lrwxrwxrwx 1 marcin_ose pg1589252 39 Mar 30 09:28 /home/marcin_ose/bin/aws -> /home/marcin_ose/.local/lib/aws/bin/aws marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /home/marcin_ose/bin/aws --version bash: /home/marcin_ose/bin/aws: Permission denied marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /home/marcin_ose/.local/lib/aws/bin/aws --version bash: /home/marcin_ose/.local/lib/aws/bin/aws: Permission denied marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$
- calling this with the built-in python does not work
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /usr/bin/python /home/marcin_ose/.local/lib/aws/bin/aws --version Traceback (most recent call last): File "/home/marcin_ose/.local/lib/aws/bin/aws", line 19, in <module> import awscli.clidriver ImportError: No module named awscli.clidriver marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$
- confirmed it's python, but a different path as the built-in python
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ head -n1 /home/marcin_ose/.local/lib/aws/bin/aws #!/home/marcin_ose/.local/lib/aws/bin/python marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$
- calling this through python does work
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/.local/lib/aws/bin/aws --version aws-cli/1.14.67 Python/2.7.6 Linux/3.2.61-grsec-modsign botocore/1.9.20 marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$
- calling this with the short-hand symlink in "~/bin/" also does work
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws --version aws-cli/1.14.67 Python/2.7.6 Linux/3.2.61-grsec-modsign botocore/1.9.20 marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$
- got the command to test the aws-cli & creds
marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$ /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-vaults usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters] To see help text, you can run: aws help aws <command> help aws <command> <subcommand> help aws: error: argument --account-id is required marcin_ose@hancock:/tmp/tmp.zfN3S0BJAC$
- saved the creds for the 'backup-cron' user's access keys to the configuration on our shared dreamhost's server (hancock)
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws configure AWS Access Key ID [None]: CHANGEME AWS Secret Access Key [None]: CHANEME Default region name [None]: us-west-2 Default output format [None]: hancock%
- this produced the following config files. note that we use 'us-west-2' for the default region. Not 'usw2'. Not 'us-west-2a'.
hancock% cat ~/.aws/credentials [default] aws_access_key_id = CHANGEME aws_secret_access_key = CHANGEME hancock% cat ~/.aws/config [default] region = us-west-2 hancock%
- successfully queried for the vaults, and got the vault I created in the console yesterday
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-vaults --account-id '099400651767'
{
"VaultList": [
{
"SizeInBytes": 0,
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"CreationDate": "2018-03-29T21:36:06.041Z",
"VaultName": "deleteMeIn2020",
"NumberOfArchives": 0
}
]
}
- note that this fails if I leave off the '--account-id' argument
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-vaults usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters] To see help text, you can run: aws help aws <command> help aws <command> <subcommand> help aws: error: argument --account-id is required hancock%
- but also note that I can let it default to the account associated with my access key by using the hyphen (-)
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier list-vaults --account-id '-'
{
"VaultList": [
{
"SizeInBytes": 0,
"VaultARN": "arn:aws:glacier:us-west-2:099400651767:vaults/deleteMeIn2020",
"CreationDate": "2018-03-29T21:36:06.041Z",
"VaultName": "deleteMeIn2020",
"NumberOfArchives": 0
}
]
}
- found the command to upload 'archives' into 'vaults'. Note that I should also use '--archive-description' to describe what the backup is, to reduce the risk of costly, superfluous future retrievals
aws glacier upload-archive --account-id - --vault-name 'deleteMeIn2020' --archive-description 'blah blah' --body archive.tar.bz2.gpg
- began further cleanup of the hetzner1 & hetzner2 directories on dreamhost's hancock. We definitely want to keep the oldest backup that I made, but then we only need one from the first of every month. Here's what it was before cleanup (total of 410G)
hancock% du -sh hetzner1/* 39G hetzner1/20170701-052001 519M hetzner1/20170718-052001 39G hetzner1/20170801-052001 217M hetzner1/20170802-052001 12G hetzner1/20170901-052001 2.3G hetzner1/20170902-052001 12G hetzner1/20171001-052001 2.4G hetzner1/20171002-052001 12G hetzner1/20171101-062001 2.5G hetzner1/20171102-062001 12G hetzner1/20171201-062001 15G hetzner1/20171201-214116 2.9G hetzner1/20171202-062001 12G hetzner1/20180101-062001 3.1G hetzner1/20180102-062001 27G hetzner1/20180201-062001 241M hetzner1/20180202-062002 28G hetzner1/20180301-062002 28G hetzner1/20180301-150405 254M hetzner1/20180302-062002 0 hetzner1/20180325-052001 12G hetzner1/20180326-052001 12G hetzner1/20180327-052001 12G hetzner1/20180328-052001 12G hetzner1/20180329-052001 12G hetzner1/20180330-052001 hancock% du -sh hetzner2/* 20G hetzner2/20170702-052001 52K hetzner2/20170729-072001 1.7G hetzner2/20170801-072001 1.7G hetzner2/20170901-072001 2.5G hetzner2/20171001-072001 838M hetzner2/20171101-072001 840M hetzner2/20171202-010653 997M hetzner2/20171202-072001 1.1G hetzner2/20180102-072001 14G hetzner2/20180202-072001 26G hetzner2/20180301-150533 25G hetzner2/20180302-072001 0 hetzner2/20180325-072001 2.8G hetzner2/20180326-072001 2.8G hetzner2/20180327-072001 2.8G hetzner2/20180328-072001 2.8G hetzner2/20180329-072001 2.8G hetzner2/20180330-072001 hancock%
- deleted superfluous backups
rm -rf hetzner1/20170718-052001 rm -rf hetzner1/20170802-052001 rm -rf hetzner1/20170902-052001 rm -rf hetzner1/20171002-052001 rm -rf hetzner1/20171102-062001 rm -rf hetzner1/20171201-214116 rm -rf hetzner1/20171202-062001 rm -rf hetzner1/20180102-062001 rm -rf hetzner1/20180202-062002 rm -rf hetzner1/20180301-150405 rm -rf hetzner1/20180302-062002 rm -rf hetzner2/20170729-072001 rm -rf hetzner2/20171202-010653 rm -rf hetzner2/20180301-150533
- And after cleanup (total of 328G)
hancock% du -sh * 247G hetzner1 81G hetzner2 4.0K readme.txt 4.0K uploadToGlacier.py hancock% du -sh hetzner1/* 39G hetzner1/20170701-052001 15M hetzner1/20170718-052001 39G hetzner1/20170801-052001 12G hetzner1/20170901-052001 12G hetzner1/20171001-052001 12G hetzner1/20171101-062001 12G hetzner1/20171201-062001 12G hetzner1/20180101-062001 27G hetzner1/20180201-062001 28G hetzner1/20180301-062002 0 hetzner1/20180325-052001 12G hetzner1/20180326-052001 12G hetzner1/20180327-052001 12G hetzner1/20180328-052001 12G hetzner1/20180329-052001 12G hetzner1/20180330-052001 hancock% du -sh hetzner2/* 20G hetzner2/20170702-052001 1.7G hetzner2/20170801-072001 1.7G hetzner2/20170901-072001 2.5G hetzner2/20171001-072001 838M hetzner2/20171101-072001 997M hetzner2/20171202-072001 1.1G hetzner2/20180102-072001 14G hetzner2/20180202-072001 25G hetzner2/20180302-072001 0 hetzner2/20180325-072001 2.8G hetzner2/20180326-072001 2.8G hetzner2/20180327-072001 2.8G hetzner2/20180328-072001 2.8G hetzner2/20180329-072001 2.8G hetzner2/20180330-072001 hancock%
- I didn't delete the recent few days backups, but those aren't going to glacier. Only these are going to glacier (total 261G):
39G hetzner1/20170701-052001 39G hetzner1/20170801-052001 12G hetzner1/20170901-052001 12G hetzner1/20171001-052001 12G hetzner1/20171101-062001 12G hetzner1/20171201-062001 12G hetzner1/20180101-062001 27G hetzner1/20180201-062001 28G hetzner1/20180301-062002 20G hetzner2/20170702-052001 1.7G hetzner2/20170801-072001 1.7G hetzner2/20170901-072001 2.5G hetzner2/20171001-072001 838M hetzner2/20171101-072001 997M hetzner2/20171202-072001 1.1G hetzner2/20180102-072001 14G hetzner2/20180202-072001 25G hetzner2/20180302-072001
- amazon claims that they encrypt the data on their end...and they hold the encryption keys. That's not good. This data holds many config files with many of our passwords. It's extremely important that it's encrypted before being shipped to a 3rd party cloud provider.
- It would be nice if there were some mechanism for easily retrieving a directory list for a given archive in a glacier vault, without having to download the entire encrypted archive, decrypt it, extract it, then list it. That could easily cost $20 for a given archive. So, if someone needs to restore something, but they don't know exactly which monthly archive they need to restore, an (encrypted) directory list for each archive could save some serious $$.
- I think the best option is, for each of the once-a-month backups listed above, we upload a pair of archives: (1) the compressed & encrypted archive itself and (b) a small compressed & encrypted dir/file list. For the compression, I'll use bz2, but I won't convert existing gz-compressed tarballs to bz2. For encryption, I'll symmetrically encrypt it with a new 4K encryption key using gpg2.
- checked the differences between hetzner1 & hetzner2 backups. Looks like I used bz2 for hetzner1 (probably because space was limited) and gz on hetzner2.
hancock% ls -R hetzner1/20170701-052001 hetzner1/20170701-052001: home mysqldump public_html hetzner1/20170701-052001/home: home.20170705-052001.tar.bz2 hetzner1/20170701-052001/mysqldump: fef forum openswh osemain wiki hetzner1/20170701-052001/mysqldump/fef: mysqldump-fef.20170705-052001.sql.bz2 hetzner1/20170701-052001/mysqldump/forum: mysqldump-forum.20170705-052001.sql.bz2 hetzner1/20170701-052001/mysqldump/openswh: mysqldump-openswh.20170705-052001.sql.bz2 hetzner1/20170701-052001/mysqldump/osemain: mysqldump-osemain.20170705-052001.sql.bz2 hetzner1/20170701-052001/mysqldump/wiki: mysqldump-wiki.20170705-052001.sql.bz2 hetzner1/20170701-052001/public_html: public_html.20170705-052001.tar.bz2 hancock% ls -R hetzner2/20171101-072001 hetzner2/20171101-072001: etc home log mysqldump root www hetzner2/20171101-072001/etc: etc.20171101-072001.tar.gz hetzner2/20171101-072001/home: home.20171101-072001.tar.gz hetzner2/20171101-072001/log: log.20171101-072001.tar.gz hetzner2/20171101-072001/mysqldump: mysqldump.20171101-072001.sql.gz hetzner2/20171101-072001/root: root.20171101-072001.tar.gz hetzner2/20171101-072001/www: www.20171101-072001.tar.gz hancock%
- the smallest backup on this list is 'hetzner2/20171101-072001', so I'll use that one. But first I confirmed that this comand generates our dir/file list from both the gz & bz2 tarballs from both hetzner1 & hetzner2
hancock% time find 'hetzner2/20171101-072001' -type f -exec tar -tvf '{}' \;
...
11.89s user 2.13s system 69% cpu 20.302 total
- I generated a 4K key file for the client-side symmetric encryption of files before sending them to glacier. I added it to my personal encrypted backups & ensured they were in 3x distinct locations of my personal backups. I'm having packet loss to hetzner now, but this should be copied to hetzner & added to keepass asap.
- tested encryption commands & confirmed that decryption works
gpg --symmetric --cipher-algo aes --armor --passphrase-file "/home/marcin_ose/backups/ose-backups-cron.key" hetzner2_20171101-072001.fileList.txt.bz2 gpg --passphrase-file "/home/marcin_ose/backups/ose-backups-cron.key" --output hetzner2_20171101-072001.fileList.txt.bz2 --decrypt hetzner2_20171101-072001.fileList.txt.bz2.asc bzcat hetzner2_20171101-072001.fileList.txt.bz2 | head
- finished a quick script to generate the encrypted file list & encrypted archive. did a manual upload attempt of the file list before adding it to the script; it worked.
hancock% /home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier upload-archive --account-id - --vault-name 'deleteMeIn2020' --archive-description 'this is a metadata file showing the file and dir list contents of the archive of the same name' --body hetzner2_20171101-072001.fileList.txt.bz2.asc
{
"archiveId": "qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA",
"checksum": "a1301459044fa4680af11d3e2d60b33a49de7e091491bd02d497bfd74945e40b",
"location": "/099400651767/vaults/deleteMeIn2020/archives/qZJWJ57sBb9Nsz0lPGKruocLivg8SVJ40UiZznG308wSPAS0vXyoYIOJekP81YwlTmci-eWETvsy4Si2e5xYJR0oVUNLadwPVkbkPmEWI1t75fbJM_6ohrNjNkwlyWPLW-lgeOaynA"
}
hancock%
- tried to confirm this upload in the aws console, but it showed "-" for "# of Archives" in the "deleteMeIn2020" vault "(as of last inventory)". The last inventory was listed as "Not updated yet"
- this is the script I have so far for automating this process. it is not yet complete.
hancock% cat uploadToGlacier.sh
#!/bin/bash
############
# SETTINGS #
############
backupDirs="hetzner2/20171101-072001"
syncDir="/home/marcin_ose/backups/uploadToGlacier"
encryptionKeyFilePath="/home/marcin_ose/backups/ose-backups-cron.key"
##############
# DO UPLOADS #
##############
for dir in $(echo $backupDirs); do
archiveName=`echo ${dir} | tr '/' '_'`;
timestamp=`date -u --rfc-3339=seconds`
fileListFilePath="${syncDir}/${archiveName}.fileList.txt"
archiveFilePath="${syncDir}/${archiveName}.tar.gz"
#########################
# archive metadata file #
#########################
# first, generate a file list to help the future sysadmin get metadata about the archvie without having to download the huge archive itself
echo "================================================================================" > "${fileListFilePath}"
echo "This file is metadata for the archive '${archiveName}'. In it, we list all the files included in the compressed/encrypted archive (produced using 'ls -lahR ${dir}'), including the files within the tarballs within the archive (produced using 'find "${dir}" -type f -exec tar -tvf '{}' \; ')" >> "${fileListFilePath}"
echo "" >> "${fileListFilePath}"
echo " - Michael Altfield <maltfield@opensourceecology.org>" >> "${fileListFilePath}"
echo "" >> "${fileListFilePath}"
echo " Note: this file was generated at ${timestamp}" >> "${fileListFilePath}"
echo "================================================================================" >> "${fileListFilePath}"
echo "#############################" >> "${fileListFilePath}"
echo "# 'ls -lahR' output follows #" >> "${fileListFilePath}"
echo "#############################" >> "${fileListFilePath}"
ls -lahR ${dir} >> "${fileListFilePath}"
echo "================================================================================" >> "${fileListFilePath}"
echo "############################" >> "${fileListFilePath}"
echo "# tarball contents follows #" >> "${fileListFilePath}"
echo "############################" >> "${fileListFilePath}"
find "${dir}" -type f -exec tar -tvf '{}' \; >> "${fileListFilePath}"
echo "================================================================================" >> "${fileListFilePath}"
# compress the metadata file
bzip2 "${fileListFilePath}"
# encrypt the metadata file
gpg --symmetric --cipher-algo aes --armor --passphrase-file "${encryptionKeyFilePath}" "${fileListFilePath}.bz2"
# delete the unencrypted archive
rm "${fileListFilePath}"
# upload it
/home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier upload-archive --account-id - --vault-name 'deleteMeIn2020' --archive-description 'this is a metadata file showing the file and dir list contents of the archive of the same name' --body "${fileListFilePath}.bz2.asc"
################
# archive file #
################
# generate archive file as a single, compressed file
tar -czvf "${archiveFilePath}" "${dir}/"
# encrypt the archive file
gpg --symmetric --cipher-algo aes --armor --passphrase-file "${encryptionKeyFilePath}" "${archiveFilePath}"
# delete the unencrypted archive
rm "${archiveFilePath}"
# upload it
#/home/marcin_ose/.local/lib/aws/bin/python /home/marcin_ose/bin/aws glacier upload-archive --account-id - --vault-name 'deleteMeIn2020' --archive-description 'this is a compressed and encrypted tarball of a backup from our ose server taken at the time the archive name indicates' --body "${archiveFilePath}.asc"
done
hancock%
- I uploaded both an ascii-armored gpg encrypted version of the file list and one with the --armor arg missing. In the morning I'll test a restore of this small archive. If it works, then I can be confident this backup solution works, and I'll proceed with using it to move all the necessary backups to glacier.
Thr Mar 29, 2018
- Marcin created an aws account & sent me the credentials
- I added the aws credentials to our OSE Keepass
- my greatest fear of using aws is that automation kicks-off costs that spiral out of control. So the first thing I did was check the billing & confirmed it's $0. Then I created a Budget of $1 configure to email ops, me, and Marcin when that is exceeded.
- I also checked the box that says "Receive Free Tier Usage Alerts" so we get alerts when we're approaching the limit of free tier usage limits
- confirmed that there is no way to cap the bill, though it is often requested of Amazon. Probably, they don't want to add this feature! https://forums.aws.amazon.com/thread.jspa?threadID=58127&start=50&tstart=0
- I created a user-specific account for me named 'maltfield'
- I created a Group called 'Administrator' with the preconfigured Managed Policy = AdministratorAccess
- I added my 'maltfield' user to this 'Administrator' group
- created a password policy requiring a minimum of 20 characters
- created a user for Marcin & sent him credentials
- created a policy 'Force_MFA' per this guide & added to the 'Administrator' group https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_users-self-manage-mfa-and-creds.html#tutorial_mfa_step1
- created new group called 'Billing' & attached the default policy 'Billing' to it. Added me & Marcin's users to this group so he could update billing info from his account directly, and so I could monitor our spending.
- this didn't work; the issue was that I needed to enable the checkbox "Activate IAM Access" under the "IAM User and Role Access to Billing Information" section when updating the Account Settings as the root user before federated IAM users could access billing info.
- I removed my 'maltfield' user from the 'Billing' group, and I still had access. So this confirms that the Billing group is superfluous; Administrator is sufficient. I deleted the 'Billing' group.
- went to create a policy for the backup script, but I guess that's only do-able from an instance in ec2
- created a new user 'backup-cron' with pragmatic access only & full access to glacier & s3 services only using new group 'backup-cron'.
- used the aws console to create a new glacier vault in usw2 titled 'deleteMeIn2020' to stash all the backups that predate me & stuff from hetzner1 that we're going to *not* move to hetzner2. This may end up being 1T in size. If there's nothing there that needs to be recovered by 2020, then we should delete the vault entirely. I'll use a different vault for the monthly backups.
- looked into boto & aws-cli. For simplicity, I'll use aws-cli in bash unless there arises a need for boto in python.
Mon Mar 26, 2018
- Marcin asked me about issues with page views on hetzner1's prod wiki site. I responded that these stats are so bad that Mediawiki removed them entirely in 2014 (though we have an older version in prod!). I showed him how to get to hetzner's provided awstats, which should provide the data he wants.
Fri Mar 23, 2018
- Dreamhost very kindly granted my request to extend our deadline to delete our data; they were happy just to hear that we were actively decreasing our usage & working to resolve the issue. They said we have until April 6th not quite the 2-week extension I asked for, but they said "if you need more time, please let me know!"
- confirmed that our daily backups have decreased significantly. hetzner1 is down from 28G to 12G and hetzner2 is down from 25G to 2.8G.
hancock% du -sh hetzner1/* 39G hetzner1/20170701-052001 519M hetzner1/20170718-052001 39G hetzner1/20170801-052001 217M hetzner1/20170802-052001 12G hetzner1/20170901-052001 2.3G hetzner1/20170902-052001 12G hetzner1/20171001-052001 2.4G hetzner1/20171002-052001 12G hetzner1/20171101-062001 2.5G hetzner1/20171102-062001 12G hetzner1/20171201-062001 15G hetzner1/20171201-214116 2.9G hetzner1/20171202-062001 12G hetzner1/20180101-062001 3.1G hetzner1/20180102-062001 27G hetzner1/20180201-062001 241M hetzner1/20180202-062002 28G hetzner1/20180301-062002 28G hetzner1/20180301-150405 254M hetzner1/20180302-062002 0 hetzner1/20180318-062001 28G hetzner1/20180319-062001 28G hetzner1/20180320-062001 12G hetzner1/20180320-220716 12G hetzner1/20180321-062002 12G hetzner1/20180322-062001 12G hetzner1/20180323-062001 hancock% du -sh hetzner2/* 20G hetzner2/20170702-052001 52K hetzner2/20170729-072001 1.7G hetzner2/20170801-072001 1.7G hetzner2/20170901-072001 2.5G hetzner2/20171001-072001 838M hetzner2/20171101-072001 840M hetzner2/20171202-010653 997M hetzner2/20171202-072001 1.1G hetzner2/20180102-072001 14G hetzner2/20180202-072001 26G hetzner2/20180301-150533 25G hetzner2/20180302-072001 0 hetzner2/20180318-072001 25G hetzner2/20180319-072001 25G hetzner2/20180320-072001 2.8G hetzner2/20180320-203228 2.8G hetzner2/20180321-072001 2.8G hetzner2/20180322-072001 2.8G hetzner2/20180323-072001 hancock% <pre> # originally, we had used 587G at the time of their email <pre> We've noticed some content on your account that might run afoul of our Acceptable Use Policy and wanted to draw your attention to it: 572.46G "hancock:/home/marcin_ose/backups" 14.85G "hancock:/home/marcin_ose/hancockCleanupDump_deleteMeIn2018"
- now we're using only 498G
hancock% du -sh /home/marcin_ose/backups 498G /home/marcin_ose/backups hancock% du -sh /home/marcin_ose/hancockCleanupDump_deleteMeIn2018 52K /home/marcin_ose/hancockCleanupDump_deleteMeIn2018 hancock%
- added my email address as a secondary to receive account updates from our dreamhost account
- discovered that amazon breaks uploads to Glacier in chunks. So, when I thought I'd be charged for 1 request for 1 file upload, I was wrong. They charge per 1k requests. So a low chunk size may mean very expensive bills. I should research what the ideal chunk size would be. 32M? 512M? 1G?
- it looks like there might be a surcharge for deleting data <3 months old. That would destroy our nightly backup solution!
- maybe we can use duplicity (which supports s3) to store nightlys in s3, but have first-of-the-months go to glacier instead.
- assuming the worst-case of the bloated backups at 26+28G = 54G per day * 3 days = 162G. usw2 = oregon s3 charges $0.023/G. So 200G in s3 would be $4.6/mo. That's $55/yr. Plus 500G in Glacier = 500G * $0.004 = $2/mo * 12 = $24/yr. So that's $79/yr, and not including the charges per request or charges for retrieval from Glacier.
- researched some options for tools to facilitate storing backups on amazon glacier
- bonus points would be some built-in "cleanup" function to delete all files, except those that meet certain criteria (in our case, it would be: (a) uploaded in the past 3 days, and (b) uploaded on the first of the month)
- https://github.com/basak/glacier-cli
- https://github.com/uskudnik/amazon-glacier-cmd-interface
- https://github.com/vsespb/mt-aws-glacier
Wed Mar 21, 2018
- rsyn'd a copy of hetzner1's files to hetzner2, so we have at least 1 offsite backup of our prod wiki
- Catarina sent me a message that obi has a cert warning. Indeed, the cert expired on 2018-03-17 = 4 days ago
- the cron was running, but it had an error
[root@hetzner2 ~]# date Thu Mar 22 00:09:37 UTC 2018 [root@hetzner2 ~]# cat /etc/cron.d/letsencrypt # once a month, update our letsencrypt cert 20 4 13 * * root /root/bin/letsencrypt/renew.sh &>> /var/log/letsEncryptRenew.log [root@hetzner2 ~]# tail -n 40 /var/log/letsEncryptRenew.log </head><body> <h1>Not Found</h1> <p". Skipping. All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (failure) ------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/opensourceecology.org.conf ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Processing /etc/letsencrypt/renewal/openbuildinginstitute.org.conf ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- The following certs are not due for renewal yet: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem (skipped) All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/openbuildinginstitute.org/fullchain.pem (failure) ------------------------------------------------------------------------------- 1 renew failure(s), 0 parse failure(s) IMPORTANT NOTES: - The following errors were reported by the server: Domain: awstats.openbuildinginstitute.org Type: unauthorized Detail: Invalid response from http://awstats.openbuildinginstitute.org/.well-known/acme-challenge/Fl5qCq0H90b35_6uJJGz7FuR77GPYfV3mFpN6gYxz0A: "<!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p" To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. /root/bin/letsencrypt/renew.sh: line 5: service: command not found ...
- fixed it by changing the awstats dir from '/var/www/html/awstats.openbuildinginstitue.org/htdocs/' to '/var/www/html/certbot/htdocs/'
certbot -nv --expand --cert-name openbuildinginstitute.org certonly -v --webroot -w /var/www/html/www.openbuildinginstitute.org/htdocs/ -d www.openbuildinginstitute.org -w /var/www/html/www.openbuildinginstitute.org/htdocs/ -d openbuildinginstitute.org -w /var/www/html/seedhome.openbuildinginstitute.org/htdocs/ -d seedhome.openbuildinginstitute.org -w /var/www/html/certbot/htdocs/ -d awstats.openbuildinginstitute.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
Tue Mar 20, 2018
- We recieved an email from our free, nonprofit account providers at Dreamhost indicating that storing our backups was not permitted, and that we must delete it before March 26th
Hello,
We hope you're enjoying your DreamHost account!
We've noticed some content on your account that might run afoul of our Acceptable Use Policy and wanted to draw your attention to it: 572.46G "hancock:/home/marcin_ose/backups" 14.85G "hancock:/home/marcin_ose/hancockCleanupDump_deleteMeIn2018"
The storage on our servers is designed specifically for hosting websites and their associated content. It's not well suited to being used as a backup solution or other general-use storage solution. That can include (but is not limited to!) using the web server as method of delivering files not intended for website use.
The bad news is that we have to ask you to delete this content from your server within the next seven days (by the end of March 26, 2018). The good news is that we offer a service, DreamObjects, that's custom-built for storing large amounts of data like backups and other large archives.
You can learn more about DreamObjects here: https://www.dreamhost.com/cloud/dreamobjects/ https://help.dreamhost.com/hc/en-us/articles/214823108-What-is-Object-Storage
As a reference, please see our Acceptable Use and Unlimited storage policies in our terms of service:
Acceptable Use Policy: https://www.dreamhost.com/legal/acceptable-use-policy/
Unlimited Policy: https://www.dreamhost.com/legal/unlimited-policy/
Unfortunately, we cannot allow the data we cited above to continue to live on your current web service. We’d prefer that you move the data off yourself at a time that's convenient for you, but if you take no action we will delete the data above in seven days. Please let us know what you plan to do. Please note, we will try our best to ensure your data is not deleted before the date listed above, and your reply is appreciated.
If you feel that we're wrong about your content (hey, it happens!), please contact us with details so that we can investigate the matter for you and take appropriate action.
We appreciate you hosting with us, so please let us know if you have any questions about this message, our policies, or anything else related to your service. We're always available and happy to help!
Thanks! Elizabeth
- I sent a kind reply asking for a 2-week extension
- I sent an email to marcin asking if he had a large external hdd with ~500G of available space
- I did some quick research on our options for long-term storage of ~600G of backup data
- DreamHost recommended their service = DreamObjects. That would be ~$200/yr
- Amazon Glacier would be ~$50/yr
- storage costs ~$24/yr
- "requests" costs <$1<yr (?)
- tx in costs $0
- tx out costs ~$5 per day's backup restore. realistically, ~$20
- we could also get hetzner's 1T storage box for ~$100/yr, but then there's no geographic distribution of the data https://www.hetzner.com/storage-box?country=us
- mega is $100 per month for 1T too (2 months free with a 1 year plan) https://mega.nz/pro
- got a list of current backup data
hancock% date Tue Mar 20 13:07:40 PDT 2018 hancock% pwd /home/marcin_ose/backups hancock% du -sh hetzner1/* 39G hetzner1/20170701-052001 519M hetzner1/20170718-052001 39G hetzner1/20170801-052001 217M hetzner1/20170802-052001 12G hetzner1/20170901-052001 2.3G hetzner1/20170902-052001 12G hetzner1/20171001-052001 2.4G hetzner1/20171002-052001 12G hetzner1/20171101-062001 2.5G hetzner1/20171102-062001 12G hetzner1/20171201-062001 15G hetzner1/20171201-214116 2.9G hetzner1/20171202-062001 12G hetzner1/20180101-062001 3.1G hetzner1/20180102-062001 27G hetzner1/20180201-062001 241M hetzner1/20180202-062002 28G hetzner1/20180301-062002 28G hetzner1/20180301-150405 254M hetzner1/20180302-062002 0 hetzner1/20180315-062001 28G hetzner1/20180316-062001 28G hetzner1/20180317-062001 28G hetzner1/20180318-062001 28G hetzner1/20180319-062001 28G hetzner1/20180320-062001 hancock% du -sh hetzner2/* 20G hetzner2/20170702-052001 52K hetzner2/20170729-072001 1.7G hetzner2/20170801-072001 1.7G hetzner2/20170901-072001 2.5G hetzner2/20171001-072001 838M hetzner2/20171101-072001 840M hetzner2/20171202-010653 997M hetzner2/20171202-072001 1.1G hetzner2/20180102-072001 14G hetzner2/20180202-072001 26G hetzner2/20180301-150533 25G hetzner2/20180302-072001 0 hetzner2/20180315-072001 25G hetzner2/20180316-072001 25G hetzner2/20180317-072001 25G hetzner2/20180318-072001 25G hetzner2/20180319-072001 25G hetzner2/20180320-072001 hancock%
- so the backups on hetzner2 are 25G currently, but much of that is unnecessary wiki staging data that's already on the prod backups in hetzner1
[maltfield@hetzner2 html]$ du -sh * 4.5M awstats.openbuildinginstitute.org 6.4M awstats.opensourceecology.org 8.0K cacti.opensourceecology.org.old 20K certbot 240M fef.opensourceecology.org 2.9G forum.opensourceecology.org 6.3M munin 8.0K munin.opensourceecology.org 12K openbuildinginstitute.org 124M oswh.opensourceecology.org 75M seedhome.openbuildinginstitute.org 12K SITE_DOWN du: cannot read directory ‘staging.openbuildinginstitute.org/htdocs/wp-content/uploads/2017/12’: Permission denied 490M staging.openbuildinginstitute.org 506M staging.opensourceecology.org 16K varnishTest 31G wiki.opensourceecology.org 493M www.openbuildinginstitute.org 507M www.opensourceecology.org
- I moved that data to /var/tmp temporarily so we could get a small backup to store off dreamhost temporarily so we have at least 1 good, recent offsite backup
[root@hetzner2 ~]# du -sh /var/www/html/* 4.5M /var/www/html/awstats.openbuildinginstitute.org 6.4M /var/www/html/awstats.opensourceecology.org 8.0K /var/www/html/cacti.opensourceecology.org.old 20K /var/www/html/certbot 240M /var/www/html/fef.opensourceecology.org 2.9G /var/www/html/forum.opensourceecology.org 6.3M /var/www/html/munin 8.0K /var/www/html/munin.opensourceecology.org 12K /var/www/html/openbuildinginstitute.org 124M /var/www/html/oswh.opensourceecology.org 75M /var/www/html/seedhome.openbuildinginstitute.org 12K /var/www/html/SITE_DOWN 490M /var/www/html/staging.openbuildinginstitute.org 506M /var/www/html/staging.opensourceecology.org 16K /var/www/html/varnishTest 50M /var/www/html/wiki.opensourceecology.org 493M /var/www/html/www.openbuildinginstitute.org 507M /var/www/html/www.opensourceecology.org [root@hetzner2 ~]#
- discovered that the backups on hetzner1 included the 'tmp/backups_for_migration_to_hetzner2/' dir, which included a copy of all the docroot
osemain@dedi978:~/backups/sync/home/usr/home/osemain$ du -sh ~/tmp/backups_for_migration_to_hetzner2/* 315M /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_20171202 159M /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/fef_20180103 356M /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_20180103 358M /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_20180206 355M /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_20180301 32M /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/oswh_20171220 12G /usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_20180120 osemain@dedi978:~/backups/sync/home/usr/home/osemain$
- moved the 'tmp' dir into the 'noBackup' dir, which is already excluded from backups
- confirmed that the hetzner2 backups shrunk significantly following the wiki docroot move. From 25G to 2.8G!
[root@hetzner2 www]# time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log real 19m1.660s user 3m20.255s sys 0m17.407s [root@hetzner2 www]# [root@hetzner2 www]# # when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding [root@hetzner2 www]# bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*' 20G backups/hetzner2/20170702-052001 52K backups/hetzner2/20170729-072001 1.7G backups/hetzner2/20170801-072001 1.7G backups/hetzner2/20170901-072001 2.5G backups/hetzner2/20171001-072001 838M backups/hetzner2/20171101-072001 840M backups/hetzner2/20171202-010653 997M backups/hetzner2/20171202-072001 1.1G backups/hetzner2/20180102-072001 14G backups/hetzner2/20180202-072001 26G backups/hetzner2/20180301-150533 25G backups/hetzner2/20180302-072001 0 backups/hetzner2/20180315-072001 25G backups/hetzner2/20180316-072001 25G backups/hetzner2/20180317-072001 25G backups/hetzner2/20180318-072001 25G backups/hetzner2/20180319-072001 25G backups/hetzner2/20180320-072001 2.8G backups/hetzner2/20180320-203228 [root@hetzner2 www]#
- uploaded the 2.8G backup to a new mega account
Fri Mar 16, 2018
- Discovered how to get some info out of the api. For example, you can get some useful info from siteinfo:
- http://opensourceecology.org/w/api.php?action=query&meta=siteinfo
- https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo
- unfortunately, this (and many other places in Mediawiki, such as Special:Version) leaks server info
- There was a response to my question about the Chonology Protector cookie preventing varnish caching when Message Caching uses the DB (per default) in MediaWiki on the Support Desk https://www.mediawiki.org/wiki/Topic:U9fys4phj04a85vu
- I was told this may be a bug (I'm not so certain yet), and that most large wikis that use varnish caching also use memcached for object caching.
- I want to avoid having to run a memcached server if possible, but I did some research into it anyway https://www.mediawiki.org/wiki/Manual:Memcached
- the Wikimediawiki:Manual:Memcached article explicitly says that smaller sites should consider using APC with CACHE_ACCEL instead of memcached; that may be a better option for our Message Cache
Memcached is likely more trouble than a small site will need (for a single server, consider storing both code and data in APC - CACHE_ACCEL), but for a larger site with heavy load, like Wikipedia, it should help lighten the load on the database servers by caching data and objects in memory. </blockqoute>
- began researching using a bytecode accelerator for php + using its cache instead of the database in mediawiki
- mediawiki recommends using OPcache, which is built into PHP >=5.5.0 (we're using php v5.6.33) https://www.mediawiki.org/wiki/Manual:Performance_tuning#Bytecode_caching
<blockqoute> PHP works by compiling a PHP file into bytecode and then executing that bytecode. The process of compiling a large application such as MediaWiki takes considerable time. PHP accelerators work by storing the compiled bytecode and executing it directly reducing the time spent compiling code.
OPcache is included in PHP 5.5.0 and later and the recommended accelerator for MediaWiki. If unavailable, APC is a decent alternative. Other supported op code caches are: mmTurck, WinCache, XCache.
Opcode caches store the compiled output of PHP scripts, greatly reducing the amount of time needed to run a script multiple times. MediaWiki does not need to be configured to do PHP bytecode caching and will "just work" once installed and enabled them.
- but we can't use OPcache for the mediawiki caching (ie: message caching) since it is only a opcode cache, not an object cache that mediawiki can utilize https://phabricator.wikimedia.org/T58652
- the second-best option appears to be APC
- apc is currently not installed on hetzner2
[root@hetzner2 includes]# rpm -qa | grep -i apc [root@hetzner2 includes]#
- I went to install APC, but apparently it's not in the repos. Instead, there's APCU. Apparently while (as shown above) OPcache only does opcode caching & no object caching, APC does both opcode caching & object caching. And APCU does no opcode caching & object caching https://wordpress.stackexchange.com/questions/174317/caching-apc-vs-apcu-vs-opcache
- It's my best judgement that the 2018 solution is to use PHP's built-in OPcache for opcode caching + APCU for object caching.
[root@hetzner2 includes]# yum install php56w-pecl-apcu ... Installed: php56w-pecl-apcu.x86_64 0:4.0.11-2.w7 Complete! [root@hetzner2 includes]#
- I stumbled upon this guide "How to make MediaWiki fast" by Aaron Schulz. Jesús Martínez Novo (User:Ciencia_Al_Poder) told me that, if I file a bug report for the cpPosTime cookie issue, I should CC Aaron https://www.mediawiki.org/wiki/User:Aaron_Schulz/How_to_make_MediaWiki_fast
- It suggests using CACHE_ACCEL (so APCU) for MainCache & MessageCache
- surprisingly, it suggests using $wgCacheDirectory so mw will cache "interface message caching" in files
- surprisingly, it suggests using the db (CACHE_DB) for ParserCache
- it does suggest use of memcached, but I think this is unnecessary until there's >1 server.
- It suggests setting $wgJobRunRate to 0 & creating a systemd daemon that executes mw jobs in the background per this guide https://www.mediawiki.org/wiki/Manual:Job_queue
- It suggests increasing the realpath_cache_size in php.ini, possibly to 512k or more. I checked the php documentation, and I found that this defaulted to 16K in PHP <= 7.0.16. Latest versions of php, however, set this to 4M.
- It suggests enabling $wgMiserMode, which disables some features in MediaWiki that are db-intensive
- I'm going to try the above advice instead of stripping the cookie in varnish
- first, I stripped my (failed) set-cookie-removing logic from varnish, reloaded the varnish config, and verified that the cookie was still present
- first, I replaced LocalSettings with the version that was in-use before I started this debugging process. It will enable the extensions again, among other changes
- then I banned the cache & tested again, confirming that the cookie was still present
- now, on the pre-debug config, I disabled the Message Cache & confirmed that the cookie was missing, confirming that I've isolated the issue to this variable
- confirmed that apc appears to be enabled (after I restarted apache)
- I stumbled upon this guide "How to make MediaWiki fast" by Aaron Schulz. Jesús Martínez Novo (User:Ciencia_Al_Poder) told me that, if I file a bug report for the cpPosTime cookie issue, I should CC Aaron https://www.mediawiki.org/wiki/User:Aaron_Schulz/How_to_make_MediaWiki_fast
[root@hetzner2 ~]# php -i | grep -i apc Additional .ini files parsed => /etc/php.d/apcu.ini, apc APC support => Emulated apcu APCu Support => Disabled APCu Debugging => Disabled MMAP File Mask => /tmp/apc.XXXXXX apc.coredump_unmap => Off => Off apc.enable_cli => Off => Off apc.enabled => On => On apc.entries_hint => 4096 => 4096 apc.gc_ttl => 3600 => 3600 apc.mmap_file_mask => /tmp/apc.XXXXXX => /tmp/apc.XXXXXX apc.preload_path => no value => no value apc.rfc1867 => Off => Off apc.rfc1867_freq => 0 => 0 apc.rfc1867_name => APC_UPLOAD_PROGRESS => APC_UPLOAD_PROGRESS apc.rfc1867_prefix => upload_ => upload_ apc.rfc1867_ttl => 3600 => 3600 apc.serializer => default => default apc.shm_segments => 1 => 1 apc.shm_size => 64M => 64M apc.slam_defense => On => On apc.smart => 0 => 0 apc.ttl => 7200 => 7200 apc.use_request_time => On => On apc.writable => /tmp => /tmp [root@hetzner2 ~]#
- since /tmp is necessarily word-readable & word-writeable, we don't use it for services that are publicly accessible, such as our web server. so I updated apc's config (/etc/php.d/apcu.ini) to use /var/lib/php/apcu
mkdir -p /var/lib/php/apcu/writable chown -R root:apache /var/lib/php/apcu chmod -R 0770 /var/lib/php/apcu cp /etc/php.d/apcu.ini /etc/php.d/apcu.ini.`date "+%Y%m%d_%H%M%S"`.bak sed -i 's^/tmp/apc.XXXXXX^/var/lib/php/apcu/tmp.XXXXXX^g' /etc/php.d/apcu.ini echo "apc.writable=/var/lib/php/apcu/writable" >> /etc/php.d/apcu.ini
- updated LocalSettings.php so Message Cache uses APCU with "$wgMessageCacheType = CACHE_ACCEL;"
- tested a query, and saw some apc-related options
[objectcache] The APCu extension is loaded and the apc.serializer INI setting is set to "default". This can cause memory corruption! You should change apc.serializer to "php" instead. See
<https://github.com/krakjoe/apcu/issues/38>.
[objectcache] The APCu extension is loaded and the apc.serializer INI setting is set to "default". This can cause memory corruption! You should change apc.serializer to "php" instead. See
<https://github.com/krakjoe/apcu/issues/38>.
IP: 127.0.0.1
Start request GET /wiki/Michael_Log
HTTP HEADERS:
X-REAL-IP: 138.201.84.223
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.29.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
ACCEPT-ENCODING: gzip
X-VARNISH: 54014
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: APCBagOStuff, session: APCBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[session] SessionBackend "7155kia65vvl9a18q74m7n41u1s0egor" is unsaved, marking dirty in constructor
[session] SessionBackend "7155kia65vvl9a18q74m7n41u1s0egor" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489707345", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489707345", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489707345", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489707345", "/", "", "", "1"
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Michael Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::checkLastModified: client did not send If-Modified-Since header
MediaWikiGadgetsDefinitionRepo::fetchStructuredList: MediaWiki:Gadgets-definition parsed, cache entry should be updated
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is expired/volatile, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: APCBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: local proxy caching; Fri, 16 Mar 2018 22:59:28 GMT **
[DBConnection] Connected to database 0 at 'localhost'.
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
[objectcache] The APCu extension is loaded and the apc.serializer INI setting is set to "default". This can cause memory corruption! You should change apc.serializer to "php" instead. See
<https://github.com/krakjoe/apcu/issues/38>.
[objectcache] The APCu extension is loaded and the apc.serializer INI setting is set to "default". This can cause memory corruption! You should change apc.serializer to "php" instead. See
<https://github.com/krakjoe/apcu/issues/38>.
IP: 127.0.0.1
Start request GET /wiki/Michael_Log
HTTP HEADERS:
X-REAL-IP: 138.201.84.223
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.29.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
ACCEPT-ENCODING: gzip
X-VARNISH: 54017
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: APCBagOStuff, session: APCBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[session] SessionBackend "iognvkk9eue6d21086m46t1jqri1so05" is unsaved, marking dirty in constructor
[session] SessionBackend "iognvkk9eue6d21086m46t1jqri1so05" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489707347", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489707347", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489707347", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489707347", "/", "", "", "1"
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Michael Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::checkLastModified: client did not send If-Modified-Since header
MediaWikiGadgetsDefinitionRepo::fetchStructuredList: MediaWiki:Gadgets-definition parsed, cache entry should be updated
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is expired/volatile, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: APCBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: local proxy caching; Fri, 16 Mar 2018 22:59:28 GMT **
[DBConnection] Connected to database 0 at 'localhost'.
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
- updated 'apc.serializer' from 'default' to 'php' per the warning message's guidance. apache had to be restarted for the error to go away
IP: 127.0.0.1
Start request GET /wiki/Michael_Log
HTTP HEADERS:
X-REAL-IP: 138.201.84.223
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.29.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
ACCEPT-ENCODING: gzip
X-VARNISH: 318849
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: APCBagOStuff, session: APCBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[session] SessionBackend "fu5f7vbkir4v3e4s0dc6ch2osjqr2u8p" is unsaved, marking dirty in constructor
[session] SessionBackend "fu5f7vbkir4v3e4s0dc6ch2osjqr2u8p" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489707577", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489707577", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489707577", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489707577", "/", "", "", "1"
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Michael Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::checkLastModified: client did not send If-Modified-Since header
MediaWikiGadgetsDefinitionRepo::fetchStructuredList: MediaWiki:Gadgets-definition parsed, cache entry should be updated
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is empty, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: APCBagOStuff
Article::view using parser cache: yes
Article::view: doing uncached parse
Saved in parser cache with key osewiki_db-wiki_:pcache:idhash:28902-0!canonical and timestamp 20180316233937 and revision id 164196
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: local proxy caching; Fri, 16 Mar 2018 22:59:28 GMT **
[DBConnection] Connected to database 0 at 'localhost'.
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
- there doesn't appear to be any files in /var/lib/php/apcu, but it does appear to be using APC as a cache
- updated LocalSettings.php with the optimization recommendations discovered above, including creating a cache dir outside the docroot
mkdir /var/www/html/wiki.opensourceecology.org/cache chown apache:apache /var/www/html/wiki.opensourceecology.org/cache chmod 0770 /var/www/html/wiki.opensourceecology.org/cache
- verifed this dir was empty
[root@hetzner2 wiki.opensourceecology.org]# ls -lahR cache cache: total 8.0K drwxrwx--- 2 apache apache 4.0K Mar 16 23:52 . d---r-x--- 4 not-apache apache 4.0K Mar 16 23:55 .. [root@hetzner2 wiki.opensourceecology.org]#
- updated LocalSettings.php to use this dir
## Set $wgCacheDirectory to a writable directory on the web server ## to make your wiki go slightly faster. The directory should not ## be publically accessible from the web. $wgCacheDirectory = "$IP/../cache";
- tried to query the page, and confirmed that it created a file
[root@hetzner2 wiki.opensourceecology.org]# ls -lahR cache cache: total 1.1M drwxrwx--- 2 apache apache 4.0K Mar 16 23:55 . d---r-x--- 4 not-apache apache 4.0K Mar 16 23:55 .. -rw-r--r-- 1 apache apache 1.1M Mar 16 23:55 l10n_cache-en.cdb [root@hetzner2 wiki.opensourceecology.org]#
- I'm going to skip the changes to wgJobRunRate + systemd file. That would be something that would easily slip through the cracks, and I'm not sure how much speed benefit it adds--especially if most traffic is served by the varnish cache
- I updated /etc/php.ini to use "realpath_cache_size = 4M"
- I organized these changes into a well-commented section of LocalSettings.php as follows
################# # OPTIMIZATIONS # ################# # See these links for more info: # * https://www.mediawiki.org/wiki/Manual:Performance_tuning # * https://www.mediawiki.org/wiki/Manual:Caching # * https://www.mediawiki.org/wiki/User:Aaron_Schulz/How_to_make_MediaWiki_fast # INTERNAL MEDIAWIKI CACHE OPTIONS (DISTINCT FROM VARNISH) # MainCache and MessageCache should use APCU per Aaron Schulz $wgMainCacheType = CACHE_ACCEL; # note that if message cache uses the db (per defaults), then it may make every # page load include a db change, which causes mediawiki to emmit a set-cookie # for cpPosTime. The cookie's presence coming from the backend causes varnish # not to cache the page (rightfully so), and the result is that varnish (which # is our most important cache) is rendered useless. For more info, see: # * https://www.mediawiki.org/wiki/Topic:U9fys4phj04a85vu # * https://wiki.opensourceecology.org/wiki/Maltfield_log_2018#Thr_Mar_15.2C_2018 $wgMessageCacheType = CACHE_ACCEL; $wgUseLocalMessageCache = true; # Parser Cache should still use the DB per Aaron Schulz $wgParserCacheType = CACHE_DB; # enable caching navigation sidebar per Aaron Schulz $wgEnableSidebarCache = true; # cache interface messages to files in this directory per Aaron Schulz # note that this should be outside the docroot! $wgCacheDirectory = "$IP/../cache"; # OTHER OPTIMIZATIONS # decrease db-heavy features per Aaron Schulz $wgMiserMode = true; # Causes serious encoding problems $wgUseGzip = false;
- updated the support ticket with my solution to the possible-bug = use APCU for Message Cache https://www.mediawiki.org/wiki/Topic:U9fys4phj04a85vu
- tested a login & was able to login successfully
- confirmed that varnish wasn't hashing my views while logged-in
- my first 2x queries after I clicked logout were cache passes (misses) because my cookie was present, but my 3rd query didn't have the cookie, and I got a cache hit
- began testing purge integration
- made an edit to a page in one vm & loaded it in another vm; it didn't load. There appears to be issues with purge integration
Thr Mar 15, 2018
- ossec is still alerting on wp-login.php attempts, which were going to the 'default' server. so I made the www.opensourceecology.org vhost & www.openbuildinginstitute.org (which are distinct ip addresses) explicitly the "default_server" with server_name = "_" in their configs. those configs reject that query, so it should prevent false-positives https://serverfault.com/questions/527156/setting-nginx-to-catch-all-unhandled-vhosts
- confirmed that hitting the IPs directly gives me a 403 Forbidden
user@personal:~/sandbox/varnishkafka$ curl -ki "https://138.201.84.243/wp-login.php" HTTP/1.1 403 Forbidden Server: nginx Date: Thu, 15 Mar 2018 15:00:43 GMT Content-Type: text/html; charset=iso-8859-1 Content-Length: 214 Connection: keep-alive X-VC-Req-Host: 138.201.84.243 X-VC-Req-URL: /wp-login.php X-VC-Req-URL-Base: /wp-login.php X-VC-Cacheable: NO:Not cacheable, ttl: 0.000 X-Varnish: 4203 Age: 0 Via: 1.1 varnish-v4 <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /wp-login.php on this server.</p> </body></html> user@personal:~/sandbox/varnishkafka$ user@personal:~/sandbox/varnishkafka$ user@personal:~/sandbox/varnishkafka$ user@personal:~/sandbox/varnishkafka$ curl -ki "https://138.201.84.223/wp-login.php" HTTP/1.1 403 Forbidden Server: nginx Date: Thu, 15 Mar 2018 15:01:20 GMT Content-Type: text/html; charset=iso-8859-1 Content-Length: 214 Connection: keep-alive X-VC-Req-Host: 138.201.84.223 X-VC-Req-URL: /wp-login.php X-VC-Req-URL-Base: /wp-login.php X-VC-Cacheable: NO:Not cacheable, ttl: 0.000 X-Varnish: 111304 Age: 0 Via: 1.1 varnish-v4 <!DOCTYPE HTML PUBLIC "-IETFDTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /wp-login.php on this server.</p> </body></html> user@personal:~/sandbox/varnishkafka$
- spent some time uploading graphs to my logs, because a picture speaks a thousand words.
- munin shows the change yesterday (disabling the Fundraising wordpress plugin) had a huge impact on our varnish hit rates. Note that the spike/dip ~6 hours after the change was me restarting varnish (I was debugging an unrelated issue with our staging wiki + varnish integration)
- munin also shows a decrease in cpu usage coorelating to varnish serving more hits. this is the end goal. right now it's not so apparent because our server isn't heavily trafficed (and our biggest site = our wiki hasn't been migrated yet), but this is a big deal for if/when our site gets hug-of-death'd after, for example, something goes viral on reddit. If varnish is just serving cache hits, our site will continue to run, even with a huge spike of traffic coming in (within reason)
- continued work on integrating mediawiki with varnish
- found old version of wikipedia's puppet configs (with varnish config files) in phabricator https://phabricator.wikimedia.org/source/operations-puppet/browse/production/templates/varnish/;b73afc51fedbe1119f4af35c7d9ab331f78357db
- according to this commit, it was removed in 2016-09 to be moved to a module https://phabricator.wikimedia.org/rOPUPc53f1476ab6cddad6150ce85f1441646ba92444b
- a query for "varnish" in the prod tree shows an entry for 'varnishkafka' in .gitmodules https://phabricator.wikimedia.org/source/operations-puppet/browse/production/?grep=varnish
- but the url it references is a 404!
user@personal:~$ curl -i https://gerrit.wikimedia.org/r/operations/puppet/varnishkafka HTTP/1.1 404 Not Found Date: Thu, 15 Mar 2018 14:30:10 GMT Server: Apache Strict-Transport-Security: max-age=106384710; includeSubDomains; preload Content-Type: text/plain;charset=iso-8859-1 Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache Expires: Mon, 01 Jan 1990 00:00:00 GMT Content-Length: 9 Backend-Timing: D=508 t=1521124210965290 Not Founduser@personal:~$
- searched all projects on wikimedia's gerrit for varnish https://gerrit.wikimedia.org/r/#/admin/projects/?filter=varnish
- found a project of the same name https://gerrit.wikimedia.org/r/#/admin/projects/operations/puppet/varnishkafka
- err, it's a 404 on curl, but git understands it
user@personal:~/sandbox$ git clone https://gerrit.wikimedia.org/r/operations/puppet/varnishkafka Cloning into 'varnishkafka'... remote: Total 423 (delta 0), reused 423 (delta 0) Receiving objects: 100% (423/423), 65.54 KiB | 0 bytes/s, done. Resolving deltas: 100% (277/277), done. Checking connectivity... done. user@personal:~/sandbox$
- that git repo doesn't appear to have any vcl configs
user@personal:~/sandbox/varnishkafka$ ls files Gemfile manifests Rakefile README.md templates tox.ini user@personal:~/sandbox/varnishkafka$ find . | grep -i vcl user@personal:~/sandbox/varnishkafka$ grep -irl 'vcl' * user@personal:~/sandbox/varnishkafka$
- it's not totally transparent to me, but I think it's actually varnish (not the backend) that's setting these cookies for wikipedia https://phabricator.wikimedia.org/rOPUPc53f1476ab6cddad6150ce85f1441646ba92444b
- going back to the code that's producing the "set-cookie cpPostTime" in the file 'includes/MediaWiki.php' in function preOutputCommit(), I should figure out what $urlDomainDistance & $lbFactory->hasOrMadeRecentMasterChanges( INF ) are
if ( $urlDomainDistance === 'local' || $urlDomainDistance === 'remote' ) {
...
} else {
// OutputPage::output() is fairly slow; run it in $postCommitWork to mask
// the latency of syncing DB positions accross all datacenters synchronously
$flags = $lbFactory::SHUTDOWN_CHRONPROT_SYNC;
if ( $lbFactory->hasOrMadeRecentMasterChanges( INF ) && $allowHeaders ) {
$cpPosTime = microtime( true );
// Set a cookie in case the DB position store cannot sync accross datacenters.
// This will at least cover the common case of the user staying on the domain.
$expires = time() + ChronologyProtector::POSITION_TTL;
$options = [ 'prefix' => '' ];
error_log( "about to set cpPosTime cookie in spot 2" );
$request->response()->setCookie( 'cpPosTime', $cpPosTime, $expires, $options );
}
}
- ironically, this term "Chronology Protector" comes from late & great Stephen Hawking, who passed away earlier this week :( https://en.wikipedia.org/wiki/Chronology_protection_conjecture
- a very relevant mediawiki script to this is includes/libs/rdbms/lbfactory/LBFactory.php. That defines an object = LBFactory with an instance var = $chronProt to determine if Chronology Protector is enabled or not
/**
* An interface for generating database load balancers
* @ingroup Database
*/
abstract class LBFactory implements ILBFactory {
/** @var ChronologyProtector */
protected $chronProt;
- there's another object = ChronologyProtector which is created by the call to getChronologyProtector()
/**
* @return ChronologyProtector
*/
protected function getChronologyProtector() {
if ( $this->chronProt ) {
return $this->chronProt;
}
$this->chronProt = new ChronologyProtector(
$this->memStash,
[
'ip' => $this->requestInfo['IPAddress'],
'agent' => $this->requestInfo['UserAgent'],
],
isset( $_GET['cpPosTime'] ) ? $_GET['cpPosTime'] : null
);
$this->chronProt->setLogger( $this->replLogger );
if ( $this->cliMode ) {
$this->chronProt->setEnabled( false );
} elseif ( $this->requestInfo['ChronologyProtection'] === 'false' ) {
// Request opted out of using position wait logic. This is useful for requests
// done by the job queue or background ETL that do not have a meaningful session.
$this->chronProt->setWaitEnabled( false );
}
$this->replLogger->debug( METHOD . ': using request info ' .
json_encode( $this->requestInfo, JSON_PRETTY_PRINT ) );
return $this->chronProt;
}
- got a stack trace from within this getChronologyProtector() to see what called it
[Thu Mar 15 18:53:05.906938 2018] [:error] [pid 25589] [client 127.0.0.1:44696] entered getChronologyProtector() function
[Thu Mar 15 18:53:05.907015 2018] [:error] [pid 25589] [client 127.0.0.1:44696] #0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/lbfactory/LBFactor
y.php(517): Wikimedia\\Rdbms\\LBFactory->getChronologyProtector()\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/lbfactory/LBFactorySimple.php(1
28): Wikimedia\\Rdbms\\LBFactory->baseLoadBalancerParams()\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/lbfactory/LBFactorySimple.php(82): Wik
imedia\\Rdbms\\LBFactorySimple->newLoadBalancer(Array)\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/lbfactory/LBFactorySimple.php(91): Wikimed
ia\\Rdbms\\LBFactorySimple->newMainLB(false)\n#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/ServiceWiring.php(62): Wikimedia\\Rdbms\\LBFactorySimple->getMa
inLB()\n#5 [internal function]: MediaWiki\\Services\\ServiceContainer->{closure}(Object(MediaWiki\\MediaWikiServices))\n#6 /var/www/html/wiki.opensourceecology.org/htdoc
s/includes/services/ServiceContainer.php(361): call_user_func_array(Object(Closure), Array)\n#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/services/Service
Container.php(344): MediaWiki\\Services\\ServiceContainer->createService('DBLoadBalancer')\n#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWikiServices
.php(511): MediaWiki\\Services\\ServiceContainer->getService('DBLoadBalancer')\n#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(
180): MediaWiki\\MediaWikiServices->getDBLoadBalancer()\n#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(267): SqlBagOStuff->ge
tDB(0)\n#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(245): SqlBagOStuff->getMulti(Array)\n#12 /var/www/html/wiki.opensourcee
cology.org/htdocs/includes/objectcache/SqlBagOStuff.php(241): SqlBagOStuff->getWithToken('osewiki_db-wiki...', NULL, 0)\n#13 /var/www/html/wiki.opensourceecology.org/htd
ocs/includes/libs/objectcache/CachedBagOStuff.php(56): SqlBagOStuff->doGet('osewiki_db-wiki...', 0)\n#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/ob
jectcache/BagOStuff.php(185): CachedBagOStuff->doGet('osewiki_db-wiki...', 0)\n#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/SessionManager.php(93
8): BagOStuff->get('osewiki_db-wiki...')\n#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/SessionInfo.php(150): MediaWiki\\Session\\SessionManager->
generateSessionId()\n#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/SessionProvider.php(176): MediaWiki\\Session\\SessionInfo->__construct(30, Arra
y)\n#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/SessionManager.php(269): MediaWiki\\Session\\SessionProvider->newSessionInfo(NULL)\n#19 /var/www
/html/wiki.opensourceecology.org/htdocs/includes/session/SessionManager.php(243): MediaWiki\\Session\\SessionManager->getEmptySessionInternal(Object(WebRequest))\n#20 /v
ar/www/html/wiki.opensourceecology.org/htdocs/includes/session/SessionManager.php(193): MediaWiki\\Session\\SessionManager->getEmptySession(Object(WebRequest))\n#21 /var
/www/html/wiki.opensourceecology.org/htdocs/includes/WebRequest.php(735): MediaWiki\\Session\\SessionManager->getSessionForRequest(Object(WebRequest))\n#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/SessionManager.php(129): WebRequest->getSession()\n#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(762): MediaWiki\\Session\\SessionManager::getGlobalSession()\n#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(114): require_once('/var/www/html/w...')\n#25 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(40): require('/var/www/html/w...')\n#26 {main}
- so it looks like the DBLoadBalancer is created in includes/services/ServiceContainer.php after being called by includes/objectcache/SqlBagOStuff.php. But I have "$wgMainCacheType = CACHE_NONE;" in LocalSettings.php. Maybe there's some cached objects in the db post-migration that need to be cleared? Or maybe there's anoter var I need to disable all use of SqlBagOStuff that will in-turn fix this issue?
- umm. I just went to retest the cache ban -> GET query -> GET query, and varnishlog showed a successful miss->hit! wtf
- maybe the db was stuck on something at the time, but I cannot reproduce the cookie issue that was causing varnish to store a hit-for-pass now
- ok, I was able to reproduce it again by commenting-out my error_log() debug lines in MediaWiki.php it was probably caused by writing output to the body of the reply before the headers setting the cookie was attempted to be made, blocking the cookie artificially
- spent some time reading up on optimizing mediawiki & the caches it uses
- stumbled on a built-in custom PHP profiler in Mediawiki. This could be essential for debugging https://www.mediawiki.org/wiki/Manual:Profiling
- I enabled more db-related output by setting "$wgDebugDBTransactions = true;" in LocalSettings.php
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": "true"
}
IP: 127.0.0.1
Start request GET /wiki/Special:Version
HTTP HEADERS:
X-REAL-IP: 198.7.58.245
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.38.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
X-VARNISH: 264293
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
[SQLBagOStuff] Connection 1062367 will be used for SqlBagOStuff
[session] SessionBackend "ei9kj09m8meepl0dnerkvlr8hkd8ku8l" is unsaved, marking dirty in constructor
[session] SessionBackend "ei9kj09m8meepl0dnerkvlr8hkd8ku8l" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489609512", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489609512", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489609512", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489609512", "/", "", "", "1"
[DBConnection] Connected to database 0 at 'localhost'.
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 1):
query-m: REPLACE INTO `wiki_objectcache` (keyname,value,exptime) VALUES ('X')
...
- the output appears to suggest that it's connecting to our one database on localhost twice (is that the cluster that the LB class is thinking it needs replication [and therefore CP] for?)
- the output also says that, yes, we're using one of our db connections for SqlBagOStuf for 'message' and 'session' caching
- tried to test disabling the 'stash' cache (which is set to 'db-replicated' per above, and as is the default in 'includes/DefaultSettings.php' by adding "$wgMainStash = CACHE_NONE;" to LocalSettings.php. This worked, as the next request (after cache ban) output this to the debug file: "[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: 0, message: SqlBagOStuff, session: SqlBagOStuff"
- I probably don't want this permanent; I'm just iterating on all possible options until I can get the cookie to go away.
- discovered that our LocalSettings.php file's "$wgShowIPinHeader = false" should be removed, as this var is deprecated since mediawiki v1.27.0 https://www.mediawiki.org/wiki/Manual:$wgShowIPinHeader
- I added vars to disable both message & session caches (the ones that were mentioned as using SqlBagOStuff in the debug output above with "$wgMessageCacheType = CACHE_NONE;" & "$wgSessionCacheType = CACHE_NONE;" in LocalSettings.php. The debug output showed it as fixed now, but it still says it's using the db connection for SqlBagOStuff!
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": "true"
}
...
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: 0, message: EmptyBagOStuff, session: EmptyBagOStuff
[caches] LocalisationCache: using store LCStoreDB
...
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is empty, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[gitinfo] Computed cacheFile=/var/www/html/wiki.opensourceecology.org/htdocs/gitinfo.json for /var/www/html/wiki.opensourceecology.org/htdocs
[gitinfo] Cache incomplete for /var/www/html/wiki.opensourceecology.org/htdocs
[Preprocessor] Cached preprocessor output (key: osewiki_db-wiki_:preprocess-xml:3a8077569b0d753b8983c7c44d07b5b3:0)
[Preprocessor] Cached preprocessor output (key: osewiki_db-wiki_:preprocess-xml:3a8077569b0d753b8983c7c44d07b5b3:0)
Looking up core head id
[gitinfo] Computed cacheFile=/var/www/html/wiki.opensourceecology.org/htdocs/gitinfo.json for /var/www/html/wiki.opensourceecology.org/htdocs
[gitinfo] Cache incomplete for /var/www/html/wiki.opensourceecology.org/htdocs
[DBConnection] Connected to database 0 at 'localhost'.
[DBPerformance] Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1):
[connect to localhost (osewiki_db)]
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(164): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('masterConns', '[connect to loc...', 1)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/loadbalancer/LoadBalancer.php(678): Wikimedia\Rdbms\TransactionProfiler->recordConnection('localhost', 'osewiki_db', true)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(184): Wikimedia\Rdbms\LoadBalancer->getConnection(-2, Array, false, 1)
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(267): SqlBagOStuff->getDB(0)
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(245): SqlBagOStuff->getMulti(Array)
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(241): SqlBagOStuff->getWithToken('osewiki_db-wiki...', NULL, 0)
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(185): SqlBagOStuff->doGet('osewiki_db-wiki...', 0)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specials/SpecialVersion.php(739): BagOStuff->get('osewiki_db-wiki...')
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specials/SpecialVersion.php(644): SpecialVersion->getCreditsForExtension('skin', Array)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specials/SpecialVersion.php(472): SpecialVersion->getExtensionCategory('skin', NULL)
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specials/SpecialVersion.php(143): SpecialVersion->getSkinCredits()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPage.php(522): SpecialVersion->execute(NULL)
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(NULL)
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Object(Title), Object(RequestContext))
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(861): MediaWiki->performRequest()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#17 {main}
[SQLBagOStuff] Connection 1062529 will be used for SqlBagOStuff
- umm, tried again & hits are working fine; here's the debug log
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": "true"
}
IP: 127.0.0.1
Start request GET /wiki/Michael_Log
HTTP HEADERS:
X-REAL-IP: 198.7.58.245
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.38.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
ACCEPT-ENCODING: gzip
X-VARNISH: 333061
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: 0, message: EmptyBagOStuff, session: EmptyBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[session] SessionBackend "lug2cr578qgttdlit06p4djb0pe30pki" is unsaved, marking dirty in constructor
[session] SessionBackend "lug2cr578qgttdlit06p4djb0pe30pki" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489621803", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489621803", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489621803", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489621803", "/", "", "", "1"
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Michael Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::checkLastModified: client did not send If-Modified-Since header
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is empty, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: SqlBagOStuff
Article::view using parser cache: yes
[DBConnection] Connected to database 0 at 'localhost'.
[DBPerformance] Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1):
[connect to localhost (osewiki_db)]
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(164): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('masterConns', '[connect to loc...', 1)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/loadbalancer/LoadBalancer.php(678): Wikimedia\Rdbms\TransactionProfiler->recordConnection('localhost', 'osewiki_db', true)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(184): Wikimedia\Rdbms\LoadBalancer->getConnection(-2, Array, false, 1)
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(267): SqlBagOStuff->getDB(0)
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(245): SqlBagOStuff->getMulti(Array)
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(241): SqlBagOStuff->getWithToken('osewiki_db-wiki...', NULL, 2)
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(185): SqlBagOStuff->doGet('osewiki_db-wiki...', 2)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/parser/ParserCache.php(181): BagOStuff->get('osewiki_db-wiki...', NULL, 2)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/parser/ParserCache.php(239): ParserCache->getKey(Object(WikiPage), Object(ParserOptions), 0)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(527): ParserCache->get(Object(WikiPage), Object(ParserOptions))
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(861): MediaWiki->performRequest()
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#15 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#16 {main}
[SQLBagOStuff] Connection 1063179 will be used for SqlBagOStuff
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: local proxy caching; Thu, 15 Mar 2018 21:00:18 GMT **
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
- I can't say why it's working now, but I can diff the debug logs to see which lines are relevant to the change, which gives a thread to debug further
# relevant debug lines when the cookie was generated, causing varnish to store a hit-for-pass
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
[SQLBagOStuff] Connection 1057383 will be used for SqlBagOStuff
[session] SessionBackend "dd1bufqqmn58vb0ipofornsdraa2kv5r" is unsaved, marking dirty in constructor
[session] SessionBackend "dd1bufqqmn58vb0ipofornsdraa2kv5r" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489537981", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489537981", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489537981", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489537981", "/", "", "", "1"
[DBConnection] Connected to database 0 at 'localhost'.
...
[caches] parser: SqlBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
[cookie] setcookie: "cpPosTime", "1521073981.6965", "1521074041", "/", "", "1", "1"
[DBReplication] Wikimedia\Rdbms\ChronologyProtector::shutdownLB: DB 'localhost' touched
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: private caching; Wed, 14 Mar 2018 19:33:01 GMT **
Request ended normally
# relevant debug lines when the cookie was *not* generated, causing varnish to actually store the response
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: 0, message: EmptyBagOStuff, session: EmptyBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[session] SessionBackend "lug2cr578qgttdlit06p4djb0pe30pki" is unsaved, marking dirty in constructor
[session] SessionBackend "lug2cr578qgttdlit06p4djb0pe30pki" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489621803", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489621803", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489621803", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489621803", "/", "", "", "1"
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
...
[caches] parser: SqlBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: local proxy caching; Thu, 15 Mar 2018 21:00:18 GMT **
Request ended normally
- note that when it behaves, the "[SQLBagOStuff] Connection 1057383 will be used for SqlBagOStuff" line is absent
- I commented-out the "$wgSessionCacheType = CACHE_NONE;" line in LocalSettings.php, banned the cache, queried the server, and the cookie came back
- then I tried again without changing the config (ban -> query -> check varnish log), and the cookie was not set; I got a hit.
- I uncommented the line above, banned the cache, queried the server, and the cookie came back!!
- then (for the second time) I tried again without changing the config (so still uncommented) = ban -> query -> check varnish log, and the cookie was not set; I got a hit.
- this is very useful information. so the cookie is generated when there is a *change* to our config. Whether this option was true or false didn't matter; the ChronologyProtection cookie (which is intended to notify the LB that a change was recently made to the master DB) appeared whenever there was a change on the first load. Then it disappeared on subsequent loads after the cache was purged (or, I would expect, after the TTL for the hit-for-pass expired). I'm guessing this is because the config itself is stored in the database, and it's not synced until the first user comes along and makes a query following the change.
- I think my testing may be logically flawed; perhaps what I was seeing as the cookie popping up was just routine cron-like db tasks that occur every X minutes such that a well-trafficked site would infrequently trigger a hit-for-pass, but my send-a-query-every-few-minutes testing may just happen to coincide with a master db change (and therefore this CP cookie & hit-for-pass) every time I tested a query. To eliminate that risk, I'd need my testing to a rapid succession of ban, query, query, wait a few seconds, then repeat. Then only check to see if it was a hit/miss/hit-for-pass in the very last query. so I did that:
[root@hetzner2 ~]# for run in {1..3}; do varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'; sleep 1; curl -si https://wiki.opensourceecology.org/wiki/Michael_Log 2>/dev/null | head -n 25 | grep -i 'set-cookie'; curl -si https://wiki.opensourceecology.org/wiki/Michael_Log 2>/dev/null | head -n 25 | grep -i 'set-cookie'; sleep 1; done
Set-Cookie: cpPosTime=1521164025.1436; expires=Fri, 16-Mar-2018 01:34:45 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
[root@hetzner2 ~]#
<pre>
## and the corresponding varnishlog output shows the hit-for-pass caused by the cookie on the first try. Then subsequent calls are miss -> hit = successful
<pre>
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 138.201.84.223'" | grep -Ei 'hit|pass|miss'
- Debug "XXXX MISS"
- VCL_call MISS
- Debug "XXXX HIT-FOR-PASS"
- HitPass 528631
- VCL_call PASS
- Link bereq 446030 pass
- Debug "XXXX MISS"
- VCL_call MISS
- Hit 528634
- VCL_call HIT
- Debug "XXXX MISS"
- VCL_call MISS
- Hit 528637
- VCL_call HIT
- now I began walking-back my disabling of the caching in LocalSettings.php. I uncommented all the new lines so it now reads
#$wgMainStash = CACHE_NONE; #$wgMessageCacheType = CACHE_NONE; #$wgSessionCacheType = CACHE_NONE;
- and I ran the loop; this time I got the set-cookie on *every* call
[root@hetzner2 ~]# for run in {1..3}; do varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'; sleep 1; curl -si https://wiki.opensourceecology.org/wiki/Michael_Log 2>/dev/null | head -n 25 | grep -i 'set-cookie'; curl -si https://wiki.opensourceecology.org/wiki/Michael_Log 2>/dev/null | head -n 25 | grep -i 'set-cookie'; sleep 1; done
Set-Cookie: cpPosTime=1521164321.3098; expires=Fri, 16-Mar-2018 01:39:41 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
Set-Cookie: cpPosTime=1521164321.5951; expires=Fri, 16-Mar-2018 01:39:41 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
Set-Cookie: cpPosTime=1521164324.0022; expires=Fri, 16-Mar-2018 01:39:44 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
Set-Cookie: cpPosTime=1521164324.2661; expires=Fri, 16-Mar-2018 01:39:44 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
Set-Cookie: cpPosTime=1521164326.6755; expires=Fri, 16-Mar-2018 01:39:46 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
Set-Cookie: cpPosTime=1521164326.9391; expires=Fri, 16-Mar-2018 01:39:46 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
[root@hetzner2 ~]#
- and the varnishlog shows hit-for-pass every time too
- Debug "XXXX MISS" - VCL_call MISS - Debug "XXXX HIT-FOR-PASS" - HitPass 409347 - VCL_call PASS - Link bereq 494319 pass - Debug "XXXX MISS" - VCL_call MISS - Debug "XXXX HIT-FOR-PASS" - HitPass 409374 - VCL_call PASS - Link bereq 409381 pass - Debug "XXXX MISS" - VCL_call MISS - Debug "XXXX HIT-FOR-PASS" - HitPass 494364 - VCL_call PASS - Link bereq 409394 pass
- I isolated it specifically to the $wgMessageCacheType variable; If this is *not* set to CACHE_NONE, then the cookie is always present, causing varnish to store the hit-for-pass
- here's the debug output when the $wgMessageCacheType variable is *not* set to CACHE_NONE
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": "true"
}
IP: 127.0.0.1
Start request GET /wiki/Michael_Log
HTTP HEADERS:
X-REAL-IP: 138.201.84.223
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.29.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
X-VARNISH: 494755
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
[SQLBagOStuff] Connection 1064033 will be used for SqlBagOStuff
[session] SessionBackend "lg3t586cdk8n4q9ba89ug1su8ind99ck" is unsaved, marking dirty in constructor
[session] SessionBackend "lg3t586cdk8n4q9ba89ug1su8ind99ck" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489629400", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489629400", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489629400", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489629400", "/", "", "", "1"
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Michael Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::checkLastModified: client did not send If-Modified-Since header
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 1):
query-m: REPLACE INTO `wiki_objectcache` (keyname,value,exptime) VALUES ('X')
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(219): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: REPLAC...', 1)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1037): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion('query-m: REPLAC...', 1521165400.3345, true, 1)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(937): Wikimedia\Rdbms\Database->doProfiledQuery('REPLACE INTO `w...', 'REPLACE /* SqlB...', true, 'SqlBagOStuff::s...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(2263): Wikimedia\Rdbms\Database->query('REPLACE INTO `w...', 'SqlBagOStuff::s...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DatabaseMysqlBase.php(497): Wikimedia\Rdbms\Database->nativeReplace('objectcache', Array, 'SqlBagOStuff::s...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(362): Wikimedia\Rdbms\DatabaseMysqlBase->replace('objectcache', Array, Array, 'SqlBagOStuff::s...')
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(376): SqlBagOStuff->setMulti(Array, 30)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(545): SqlBagOStuff->set('osewiki_db-wiki...', 1, 30)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(418): BagOStuff->add('osewiki_db-wiki...', 1, 30)
#9 [internal function]: BagOStuff->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/vendor/wikimedia/wait-condition-loop/src/WaitConditionLoop.php(92): call_user_func(Object(Closure))
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(429): Wikimedia\WaitConditionLoop->invoke()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(472): BagOStuff->lock('osewiki_db-wiki...', 0, 30, 'MessageCache::g...')
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(762): BagOStuff->getScopedLock('osewiki_db-wiki...', 0, 30, 'MessageCache::g...')
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(420): MessageCache->getReentrantScopedLock('osewiki_db-wiki...', 0)
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): MessageCache->loadFromDBWithLock('en', Array, NULL)
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(991): MessageCache->load('en')
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(921): MessageCache->getMsgFromNamespace('Pagetitle', 'en')
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(Object(LanguageEn), 'pagetitle', true, Array)
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'pagetitle', true)
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get('pagetitle', true, Object(LanguageEn))
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString('text')
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(950): Message->text()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(998): OutputPage->setHTMLTitle(Object(Message))
#25 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(463): OutputPage->setPageTitle('Maltfield Log')
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#29 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(861): MediaWiki->performRequest()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#32 {main}
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 2):
query-m: REPLACE INTO `wiki_objectcache` (keyname,value,exptime) VALUES ('X')
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(219): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: REPLAC...', 2)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1037): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion('query-m: REPLAC...', 1521165400.3374, true, 2)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(937): Wikimedia\Rdbms\Database->doProfiledQuery('REPLACE INTO `w...', 'REPLACE /* SqlB...', true, 'SqlBagOStuff::s...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(2263): Wikimedia\Rdbms\Database->query('REPLACE INTO `w...', 'SqlBagOStuff::s...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DatabaseMysqlBase.php(497): Wikimedia\Rdbms\Database->nativeReplace('objectcache', Array, 'SqlBagOStuff::s...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(362): Wikimedia\Rdbms\DatabaseMysqlBase->replace('objectcache', Array, Array, 'SqlBagOStuff::s...')
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(376): SqlBagOStuff->setMulti(Array, 0)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(690): SqlBagOStuff->set('osewiki_db-wiki...', Array)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(428): MessageCache->saveToCaches(Array, 'all', 'en')
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): MessageCache->loadFromDBWithLock('en', Array, NULL)
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(991): MessageCache->load('en')
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(921): MessageCache->getMsgFromNamespace('Pagetitle', 'en')
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(Object(LanguageEn), 'pagetitle', true, Array)
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'pagetitle', true)
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get('pagetitle', true, Object(LanguageEn))
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString('text')
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(950): Message->text()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(998): OutputPage->setHTMLTitle(Object(Message))
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(463): OutputPage->setPageTitle('Maltfield Log')
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(861): MediaWiki->performRequest()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#26 {main}
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 3):
query-m: DELETE FROM `wiki_objectcache` WHERE keyname = 'X'
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(219): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: DELETE...', 3)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1037): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion('query-m: DELETE...', 1521165400.3392, true, 1)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(937): Wikimedia\Rdbms\Database->doProfiledQuery('DELETE FROM `wi...', 'DELETE /* SqlBa...', true, 'SqlBagOStuff::d...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(2370): Wikimedia\Rdbms\Database->query('DELETE FROM `wi...', 'SqlBagOStuff::d...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(433): Wikimedia\Rdbms\Database->delete('objectcache', Array, 'SqlBagOStuff::d...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(447): SqlBagOStuff->delete('osewiki_db-wiki...')
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(485): BagOStuff->unlock('osewiki_db-wiki...')
#7 [internal function]: BagOStuff->{closure}()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/vendor/wikimedia/scoped-callback/src/ScopedCallback.php(76): call_user_func_array(Object(Closure), Array)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): Wikimedia\ScopedCallback->__destruct()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): MessageCache->loadFromDBWithLock('en', Array, NULL)
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(991): MessageCache->load('en')
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(921): MessageCache->getMsgFromNamespace('Pagetitle', 'en')
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(Object(LanguageEn), 'pagetitle', true, Array)
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'pagetitle', true)
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get('pagetitle', true, Object(LanguageEn))
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString('text')
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(950): Message->text()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(998): OutputPage->setHTMLTitle(Object(Message))
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(463): OutputPage->setPageTitle('Maltfield Log')
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(861): MediaWiki->performRequest()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#27 {main}
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is expired/volatile, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: SqlBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
[cookie] setcookie: "cpPosTime", "1521165400.3692", "1521165460", "/", "", "1", "1"
[DBReplication] Wikimedia\Rdbms\ChronologyProtector::shutdownLB: DB 'localhost' touched
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: private caching; Fri, 16 Mar 2018 01:55:42 GMT **
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
- and here's the debug output when it *is* set to CACHE_NONE.
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": "true"
}
IP: 127.0.0.1
Start request GET /wiki/Michael_Log
HTTP HEADERS:
X-REAL-IP: 138.201.84.223
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.29.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
ACCEPT-ENCODING: gzip
X-VARNISH: 146580
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: EmptyBagOStuff, session: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.29.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
[SQLBagOStuff] Connection 1064073 will be used for SqlBagOStuff
[session] SessionBackend "nhu5kl4t72jhuu3nfcus835sjgpdfpa0" is unsaved, marking dirty in constructor
[session] SessionBackend "nhu5kl4t72jhuu3nfcus835sjgpdfpa0" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489629637", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489629637", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489629637", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489629637", "/", "", "", "1"
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Michael Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::checkLastModified: client did not send If-Modified-Since header
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is empty, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: SqlBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: local proxy caching; Fri, 16 Mar 2018 01:58:47 GMT **
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
- Note the "[caches]" line that shows "message: EmptyBagOStuff" instead of "message: SqlBagOStuff"
- Note that the "[MessageCache]" line is also different
# when it is *not* set to CACHE_NONE. [MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is expired/volatile, loading from database # when it *is* set to CACHE_NONE. [MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is empty, loading from d =Wed Mar 14, 2018= # marked the osemain migration as successful, lifting the content freeze http://opensourceecology.org/wiki/CHG-2018-02-05 # returned to investigating the unnecessarily high hit-for-pass responses in varnish for osemain due to the misbehaving 'fundraiser' wordpress plugin that is injecting cache-control headers, telling varnish not to cache ## found that the image of the qr code for bitcoin was missing https://staging.opensourceecology.org/community/#support ## attempted to update the page using a relative path of the image, rather than "http://..." ### got a modsecurity false-positive ### whitelisted 958008, xss ### whitelisted 973329, xss ## couldn't see any other issues with the differences between the staging site (with the Fundraising plugin disabled) and the production site (with the Fundraising plugin enabled) ## confirmed that the offensive cache-control headers were still present from apache <pre> [root@hetzner2 conf.d]# curl -I "http://127.0.0.1:8000/" -H "Host: www.opensourceecology.org" HTTP/1.1 301 Moved Permanently Date: Wed, 14 Mar 2018 19:21:50 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Location: https://www.opensourceecology.org/ X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=srghpko27hns1894gud5dm8r5tohv2e9hkrth94ges26qpi6pbv749m3i8rst09c9ns5ojp1i90lr0hfu2jgsernm2b2kutu7shdbg3; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 [root@hetzner2 conf.d]#
- disabled the Fundraising app on production
- confirmed that the offending cache-control headers were now absent
[root@hetzner2 conf.d]# curl -I "http://127.0.0.1:8000/" -H "Host: www.opensourceecology.org" HTTP/1.1 301 Moved Permanently Date: Wed, 14 Mar 2018 19:22:49 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Location: https://www.opensourceecology.org/ X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 [root@hetzner2 conf.d]#
- continued work on integrating mediawiki with varnish
- since the rabbit hole investigating the root cause of the cache-control headers causing misses/hit-for-passes, I'm just going to ignroe the backend's requests to avoid caching per official the guide's recommendation https://www.mediawiki.org/wiki/Manual:Varnish_caching#Configuring_Varnish_4.x
- confirmed that repeated queries were not hits
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 198.7.58.245'" | grep -Ei 'hit|pass|miss' - Debug "XXXX MISS" - VCL_call MISS - VCL_return pass - VCL_call PASS - Link bereq 5180433 pass - VCL_return pass - VCL_call PASS - Link bereq 5180436 pass - VCL_return pass - VCL_call PASS - Link bereq 4632277 pass - VCL_return pass - VCL_call PASS - Link bereq 5180444 pass
- updated the varnish config for our wiki site, adding the vcl logic from the official guide linked above to functions: vcl_recv, vcl_backend_response, vcl_pipe, vcl_hit, and vcl_miss
- had to remove the line "set req.backend_hint= default;" from vcl_recv as we already do this in our structured multi-vhost config after checking the req.http.host in every function
- reloaded varnish config
- refreshed a wiki page a couple times & watched the varnishlog output, but it was the same result. no hit.
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 198.7.58.245'" | grep -Ei 'hit|pass|miss' - Debug "XXXX HIT-FOR-PASS" - HitPass 2455036 - VCL_call PASS - Link bereq 5245753 pass - VCL_return pass - VCL_call PASS - Link bereq 4633208 pass - VCL_return pass - VCL_call PASS - Link bereq 4633211 pass - VCL_return pass - VCL_call PASS - Link bereq 5245756 pass - Debug "XXXX MISS" - VCL_call MISS - Debug "XXXX MISS" - VCL_call MISS
- got the full output of the varnishlog
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 198.7.58.245'" * << Request >> 2455213 - Begin req 2455212 rxreq - Timestamp Start: 1521058112.833068 0.000000 0.000000 - Timestamp Req: 1521058112.833068 0.000000 0.000000 - ReqStart 127.0.0.1 38790 - ReqMethod GET - ReqURL /wiki/Maltfield_Log - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate, br - ReqHeader Cookie: cpPosTime=1521058094.075 - ReqHeader DNT: 1 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqUnset X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 198.7.58.245, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - ReqUnset Accept-Encoding: gzip, deflate, br - ReqHeader Accept-Encoding: gzip - VCL_return hash - VCL_call HASH - VCL_return lookup - Debug "XXXX MISS" - VCL_call MISS - VCL_return fetch - Link bereq 2455214 fetch - Timestamp Fetch: 1521058113.024141 0.191072 0.191072 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Wed, 14 Mar 2018 20:08:32 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Content-language: en - RespHeader X-UA-Compatible: IE=Edge - RespHeader Link: </images/ose-logo.png?be82f>;rel=preload;as=image - RespHeader Vary: Accept-Encoding,Cookie - RespHeader Expires: Thu, 01 Jan 1970 00:00:00 GMT - RespHeader Cache-Control: private, must-revalidate, max-age=0 - RespHeader Last-Modified: Wed, 14 Mar 2018 15:08:32 GMT - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Set-Cookie: cpPosTime=1521058112.9626; expires=Wed, 14-Mar-2018 20:09:32 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader X-Varnish: 2455213 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1521058113.024168 0.191099 0.000027 - RespStatus 200 - RespReason OK - RespHeader Date: Wed, 14 Mar 2018 20:08:32 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Content-language: en - RespHeader X-UA-Compatible: IE=Edge - RespHeader Link: </images/ose-logo.png?be82f>;rel=preload;as=image - RespHeader Vary: Accept-Encoding,Cookie - RespHeader Expires: Thu, 01 Jan 1970 00:00:00 GMT - RespHeader Cache-Control: private, must-revalidate, max-age=0 - RespHeader Last-Modified: Wed, 14 Mar 2018 15:08:32 GMT - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Set-Cookie: cpPosTime=1521058112.9626; expires=Wed, 14-Mar-2018 20:09:32 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader X-Varnish: 2455213 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1521058113.024168 0.191099 0.000027 - Debug "RES_MODE 4" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1521058113.038627 0.205559 0.014459 - Debug "XXX REF 2" - ReqAcct 490 0 490 666 16114 16780 - End [root@hetzner2 ~]#
- the varnish book shows that the hit-for-pass is triggered when setting 'beresp.uncacheable' to 'true' https://book.varnish-software.com/4.0/chapters/VCL_Basics.html
- the mediawiki config I adopted from their manual does this in several instances in vcl_backend_response()
sub vcl_backend_response {
if ( beresp.backend.name == "wiki_opensourceecology_org" ){
# set minimum timeouts to auto-discard stored objects
set beresp.grace = 120s;
if (beresp.ttl < 48h) {
set beresp.ttl = 48h;
}
if (!beresp.ttl > 0s) {
set beresp.uncacheable = true;
return (deliver);
}
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
return (deliver);
}
# if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
# set beresp.uncacheable = true;
# return (deliver);
# }
if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") {
set beresp.uncacheable = true;
return (deliver);
}
return (deliver);
}
}
- I tried commenting-out all 4x if blocks, reloaded varnish, and the issue still persisted
- I read the definition of hit-for-pass from the varnish book, and it makes an example of responses that should not be cached as those with set-cookie headers. This makes sense https://book.varnish-software.com/4.0/chapters/VCL_Subroutines.html#hit-for-pass
- saw that our request does include a set-cookie header
Set-Cookie: cpPosTime=1521072364.1071; expires=Thu, 15-Mar-2018 00:07:04 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly
- a quick google shows cpPosTime was added to MediaWiki 1.28; its release notes say this about it https://www.mediawiki.org/wiki/MediaWiki_1.28
After a client performs an action which alters a database that has replica databases, MediaWiki will wait for the replica databases to synchronize with the master database while it renders the HTML output. However, if the output is a redirect to another wiki on the wiki farm with a different domain, MediaWiki will instead alter the redirect URL to include a ?cpPosTime parameter that triggers the database synchronization when the URL is followed by the client. The same-domain case uses a new cpPosTime cookie.
- umm, we only have 1 fucking database.
- cpPosTime appears to be better described in its purpose/use by load balancers in the MediaWiki_architecture article
https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture
MediaWiki also has built-in support for load balancing, added as early as 2004 in MediaWiki 1.2 (when Wikipedia got its second server a big deal at the time). The load balancer (MediaWiki's PHP code that decides which server to connect to) is now a critical part of Wikimedia's infrastructure, which explains its influence on some algorithm decisions in the code. The system administrator can specify in MediaWiki's configuration that there is one master database server, and any number of slave database servers; a weight can be assigned to each server. The load balancer will send all writes to the master, and will balance reads according to the weights. It also keeps track of the replication lag of each slave. If a slave's replication lag exceeds 30 seconds, it will not receive any read queries to allow it to catch up; if all slaves are lagged more than 30 seconds, MediaWiki will automatically put itself in read-only mode.
MediaWiki's "chronology protector" ensures that replication lag never causes a user to see a page that claims an action they've just performed hasn't happened yet. This is done by storing the master's position in the user's session if a request they made resulted in a write query. The next time the user makes a read request, the load balancer read this position from the session, and tries to select a slave that has caught up to that replication position to serve the request. If none is available, it will wait until one is. It may appear to other users as though the action hasn't happened yet, but the chronology remains consistent for each user.
- that makes more sense, but it still doesn't apply to us. why are we setting this cookie? also, why is my simple curl of my log making a 'write' action to the db?
curl -I https://wiki.opensourceecology.org/wiki/Maltfield_Log
- grepped through the mediawiki sourcecode in the includes dir, and saw that 'includes/MediaWiki.php' was setting this cpPosTime cookie
[root@hetzner2 htdocs]# grep -irC10 'cppostime' includes/
...
includes/MediaWiki.php- $output->getRedirect() &&
includes/MediaWiki.php- $lbFactory->hasOrMadeRecentMasterChanges( INF )
includes/MediaWiki.php- ) ? self::getUrlDomainDistance( $output->getRedirect() ) : false;
includes/MediaWiki.php-
includes/MediaWiki.php- $allowHeaders = !( $output->isDisabled() || headers_sent() );
includes/MediaWiki.php- if ( $urlDomainDistance === 'local' || $urlDomainDistance === 'remote' ) {
includes/MediaWiki.php- // OutputPage::output() will be fast; $postCommitWork will not be useful for
includes/MediaWiki.php- // masking the latency of syncing DB positions accross all datacenters synchronously.
includes/MediaWiki.php- // Instead, make use of the RTT time of the client follow redirects.
includes/MediaWiki.php- $flags = $lbFactory::SHUTDOWN_CHRONPROT_ASYNC;
includes/MediaWiki.php: $cpPosTime = microtime( true );
includes/MediaWiki.php- // Client's next request should see 1+ positions with this DBMasterPos::asOf() time
includes/MediaWiki.php- if ( $urlDomainDistance === 'local' && $allowHeaders ) {
includes/MediaWiki.php- // Client will stay on this domain, so set an unobtrusive cookie
includes/MediaWiki.php- $expires = time() + ChronologyProtector::POSITION_TTL;
includes/MediaWiki.php- $options = [ 'prefix' => '' ];
includes/MediaWiki.php: $request->response()->setCookie( 'cpPosTime', $cpPosTime, $expires, $options );
includes/MediaWiki.php- } else {
includes/MediaWiki.php- // Cookies may not work across wiki domains, so use a URL parameter
includes/MediaWiki.php- $safeUrl = $lbFactory->appendPreShutdownTimeAsQuery(
includes/MediaWiki.php- $output->getRedirect(),
includes/MediaWiki.php: $cpPosTime
includes/MediaWiki.php- );
includes/MediaWiki.php- $output->redirect( $safeUrl );
includes/MediaWiki.php- }
includes/MediaWiki.php- } else {
includes/MediaWiki.php- // OutputPage::output() is fairly slow; run it in $postCommitWork to mask
includes/MediaWiki.php- // the latency of syncing DB positions accross all datacenters synchronously
includes/MediaWiki.php- $flags = $lbFactory::SHUTDOWN_CHRONPROT_SYNC;
includes/MediaWiki.php- if ( $lbFactory->hasOrMadeRecentMasterChanges( INF ) && $allowHeaders ) {
includes/MediaWiki.php: $cpPosTime = microtime( true );
includes/MediaWiki.php- // Set a cookie in case the DB position store cannot sync accross datacenters.
includes/MediaWiki.php- // This will at least cover the common case of the user staying on the domain.
includes/MediaWiki.php- $expires = time() + ChronologyProtector::POSITION_TTL;
includes/MediaWiki.php- $options = [ 'prefix' => '' ];
includes/MediaWiki.php: $request->response()->setCookie( 'cpPosTime', $cpPosTime, $expires, $options );
includes/MediaWiki.php- }
includes/MediaWiki.php- }
includes/MediaWiki.php- // Record ChronologyProtector positions for DBs affected in this request at this point
includes/MediaWiki.php- $lbFactory->shutdown( $flags, $postCommitWork );
includes/MediaWiki.php- wfDebug( METHOD . ': LBFactory shutdown completed' );
includes/MediaWiki.php-
includes/MediaWiki.php- // Set a cookie to tell all CDN edge nodes to "stick" the user to the DC that handles this
includes/MediaWiki.php- // POST request (e.g. the "master" data center). Also have the user briefly bypass CDN so
includes/MediaWiki.php- // ChronologyProtector works for cacheable URLs.
includes/MediaWiki.php- if ( $request->wasPosted() && $lbFactory->hasOrMadeRecentMasterChanges() ) {
- quick sanity check: what does the prod site do? I confirmed that it does *not* set this cookie. hmm. But could it be because Cloud Front is altering it? Damn middle boxes obfuscating stuff *shrug*
user@personal:~$ curl -I http://opensourceecology.org/wiki/Maltfield_Log HTTP/1.1 200 OK Date: Thu, 15 Mar 2018 00:28:59 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive Set-Cookie: __cfduid=d9d55ba85a13ac6b64a950ed80e0987b81521073738; expires=Fri, 15-Mar-19 00:28:58 GMT; path=/; domain=.opensourceecology.org; HttpOnly X-Powered-By: PHP/5.4.45 X-Content-Type-Options: nosniff Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Content-Language: en Last-Modified: Mon, 12 Mar 2018 19:04:00 GMT Server: cloudflare CF-RAY: 3fbadcf3e1289f60-IAD user@personal:~$
- checked the debug log file and found some relevant info
- entire log
- checked the debug log file and found some relevant info
IP: 127.0.0.1
Start request HEAD /wiki/Maltfield_Log
HTTP HEADERS:
X-REAL-IP: 198.7.58.245
X-FORWARDED-PROTO: https
X-FORWARDED-PORT: 443
HOST: wiki.opensourceecology.org
USER-AGENT: curl/7.38.0
ACCEPT: */*
X-FORWARDED-FOR: 127.0.0.1
X-VARNISH: 4637894
[caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff
[caches] LocalisationCache: using store LCStoreDB
[CryptRand] openssl_random_pseudo_bytes generated 20 bytes of strong randomness.
[CryptRand] 0 bytes of randomness leftover in the buffer.
[DBReplication] Wikimedia\Rdbms\LBFactory::getChronologyProtector: using request info {
"IPAddress": "127.0.0.1",
"UserAgent": "curl\/7.38.0",
"ChronologyProtection": false
}
[DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection.
[DBConnection] Connected to database 0 at 'localhost'.
[SQLBagOStuff] Connection 1057383 will be used for SqlBagOStuff
[session] SessionBackend "dd1bufqqmn58vb0ipofornsdraa2kv5r" is unsaved, marking dirty in constructor
[session] SessionBackend "dd1bufqqmn58vb0ipofornsdraa2kv5r" save: dataDirty=1 metaDirty=1 forcePersist=0
[cookie] already deleted setcookie: "osewiki_db_wiki__session", "", "1489537981", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_UserID", "", "1489537981", "/", "", "1", "1"
[cookie] already deleted setcookie: "osewiki_db_wiki_Token", "", "1489537981", "/", "", "1", "1"
[cookie] already deleted setcookie: "forceHTTPS", "", "1489537981", "/", "", "", "1"
[DBConnection] Connected to database 0 at 'localhost'.
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
[ContentHandler] Created handler for wikitext: WikitextContentHandler
OutputPage::checkLastModified: client did not send If-Modified-Since header
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 1):
query-m: REPLACE INTO `wiki_objectcache` (keyname,value,exptime) VALUES ('X')
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(219): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: REPLAC...', 1)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1037): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion('query-m: REPLAC...', 1521073981.6619, true, 1)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(937): Wikimedia\Rdbms\Database->doProfiledQuery('REPLACE INTO `w...', 'REPLACE /* SqlB...', true, 'SqlBagOStuff::s...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(2263): Wikimedia\Rdbms\Database->query('REPLACE INTO `w...', 'SqlBagOStuff::s...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DatabaseMysqlBase.php(497): Wikimedia\Rdbms\Database->nativeReplace('objectcache', Array, 'SqlBagOStuff::s...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(362): Wikimedia\Rdbms\DatabaseMysqlBase->replace('objectcache', Array, Array, 'SqlBagOStuff::s...')
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(376): SqlBagOStuff->setMulti(Array, 30)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(545): SqlBagOStuff->set('osewiki_db-wiki...', 1, 30)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(418): BagOStuff->add('osewiki_db-wiki...', 1, 30)
#9 [internal function]: BagOStuff->{closure}()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/vendor/wikimedia/wait-condition-loop/src/WaitConditionLoop.php(92): call_user_func(Object(Closure))
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(429): Wikimedia\WaitConditionLoop->invoke()
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(472): BagOStuff->lock('osewiki_db-wiki...', 0, 30, 'MessageCache::g...')
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(762): BagOStuff->getScopedLock('osewiki_db-wiki...', 0, 30, 'MessageCache::g...')
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(420): MessageCache->getReentrantScopedLock('osewiki_db-wiki...', 0)
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): MessageCache->loadFromDBWithLock('en', Array, NULL)
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(991): MessageCache->load('en')
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(921): MessageCache->getMsgFromNamespace('Pagetitle', 'en')
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(Object(LanguageEn), 'pagetitle', true, Array)
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'pagetitle', true)
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get('pagetitle', true, Object(LanguageEn))
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString('text')
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(950): Message->text()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(998): OutputPage->setHTMLTitle(Object(Message))
#25 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(463): OutputPage->setPageTitle('Maltfield Log')
#26 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#27 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#28 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#29 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(851): MediaWiki->performRequest()
#30 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#31 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#32 {main}
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 2):
query-m: REPLACE INTO `wiki_objectcache` (keyname,value,exptime) VALUES ('X')
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(219): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: REPLAC...', 2)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1037): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion('query-m: REPLAC...', 1521073981.6661, true, 2)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(937): Wikimedia\Rdbms\Database->doProfiledQuery('REPLACE INTO `w...', 'REPLACE /* SqlB...', true, 'SqlBagOStuff::s...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(2263): Wikimedia\Rdbms\Database->query('REPLACE INTO `w...', 'SqlBagOStuff::s...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/DatabaseMysqlBase.php(497): Wikimedia\Rdbms\Database->nativeReplace('objectcache', Array, 'SqlBagOStuff::s...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(362): Wikimedia\Rdbms\DatabaseMysqlBase->replace('objectcache', Array, Array, 'SqlBagOStuff::s...')
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(376): SqlBagOStuff->setMulti(Array, 0)
#7 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(690): SqlBagOStuff->set('osewiki_db-wiki...', Array)
#8 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(428): MessageCache->saveToCaches(Array, 'all', 'en')
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): MessageCache->loadFromDBWithLock('en', Array, NULL)
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(991): MessageCache->load('en')
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(921): MessageCache->getMsgFromNamespace('Pagetitle', 'en')
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(Object(LanguageEn), 'pagetitle', true, Array)
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'pagetitle', true)
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get('pagetitle', true, Object(LanguageEn))
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString('text')
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(950): Message->text()
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(998): OutputPage->setHTMLTitle(Object(Message))
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(463): OutputPage->setPageTitle('Maltfield Log')
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(851): MediaWiki->performRequest()
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#26 {main}
[DBPerformance] Expectation (writes <= 0) by MediaWiki::main not met (actual: 3):
query-m: DELETE FROM `wiki_objectcache` WHERE keyname = 'X'
#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/TransactionProfiler.php(219): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated('writes', 'query-m: DELETE...', 3)
#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(1037): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion('query-m: DELETE...', 1521073981.668, true, 1)
#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(937): Wikimedia\Rdbms\Database->doProfiledQuery('DELETE FROM `wi...', 'DELETE /* SqlBa...', true, 'SqlBagOStuff::d...')
#3 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php(2370): Wikimedia\Rdbms\Database->query('DELETE FROM `wi...', 'SqlBagOStuff::d...')
#4 /var/www/html/wiki.opensourceecology.org/htdocs/includes/objectcache/SqlBagOStuff.php(433): Wikimedia\Rdbms\Database->delete('objectcache', Array, 'SqlBagOStuff::d...')
#5 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(447): SqlBagOStuff->delete('osewiki_db-wiki...')
#6 /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/objectcache/BagOStuff.php(485): BagOStuff->unlock('osewiki_db-wiki...')
#7 [internal function]: BagOStuff->{closure}()
#8 /var/www/html/wiki.opensourceecology.org/htdocs/vendor/wikimedia/scoped-callback/src/ScopedCallback.php(76): call_user_func_array(Object(Closure), Array)
#9 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): Wikimedia\ScopedCallback->__destruct()
#10 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(350): MessageCache->loadFromDBWithLock('en', Array, NULL)
#11 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(991): MessageCache->load('en')
#12 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(921): MessageCache->getMsgFromNamespace('Pagetitle', 'en')
#13 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(888): MessageCache->getMessageForLang(Object(LanguageEn), 'pagetitle', true, Array)
#14 /var/www/html/wiki.opensourceecology.org/htdocs/includes/cache/MessageCache.php(829): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'pagetitle', true)
#15 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(1275): MessageCache->get('pagetitle', true, Object(LanguageEn))
#16 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(842): Message->fetchMessage()
#17 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Message.php(934): Message->toString('text')
#18 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(950): Message->text()
#19 /var/www/html/wiki.opensourceecology.org/htdocs/includes/OutputPage.php(998): OutputPage->setHTMLTitle(Object(Message))
#20 /var/www/html/wiki.opensourceecology.org/htdocs/includes/page/Article.php(463): OutputPage->setPageTitle('Maltfield Log')
#21 /var/www/html/wiki.opensourceecology.org/htdocs/includes/actions/ViewAction.php(68): Article->view()
#22 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(499): ViewAction->show()
#23 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(293): MediaWiki->performAction(Object(Article), Object(Title))
#24 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(851): MediaWiki->performRequest()
#25 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()
#26 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()
#27 {main}
[MessageCache] MessageCache::load: Loading en... local cache is empty, global cache is expired/volatile, loading from database
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache->getParser
Parser: using preprocessor: Preprocessor_DOM
Unstubbing $wgLang on call of $wgLang::_unstub from ParserOptions->__construct
[caches] parser: SqlBagOStuff
Article::view using parser cache: yes
Parser cache options found.
ParserOutput cache found.
Article::view: showing parser cache contents
MediaWiki::preOutputCommit: primary transaction round committed
MediaWiki::preOutputCommit: pre-send deferred updates completed
[cookie] setcookie: "cpPosTime", "1521073981.6965", "1521074041", "/", "", "1", "1"
[DBReplication] Wikimedia\Rdbms\ChronologyProtector::shutdownLB: DB 'localhost' touched
MediaWiki::preOutputCommit: LBFactory shutdown completed
Title::getRestrictionTypes: applicable restrictions to Maltfield Log are {edit,move}
OutputPage::haveCacheVaryCookies: no cache-varying cookies found
OutputPage::sendCacheControl: private caching; Wed, 14 Mar 2018 19:33:01 GMT **
Request ended normally
[session] Saving all sessions on shutdown
[DBConnection] Closing connection to database 'localhost'.
[DBConnection] Closing connection to database 'localhost'.
- relevant lines
MediaWiki::preOutputCommit: primary transaction round committed MediaWiki::preOutputCommit: pre-send deferred updates completed [cookie] setcookie: "cpPosTime", "1521073981.6965", "1521074041", "/", "", "1", "1" [DBReplication] Wikimedia\Rdbms\ChronologyProtector::shutdownLB: DB 'localhost' touched
- this is becoming another rabbit hole, and a quick check confirmed that this is not the issue. I added a vcl line to delete the backend's set-cookie line
unset beresp.http.Set-Cookie;
- and I confirmed that cookie line was now absent
user@personal:~$ curl -I https://wiki.opensourceecology.org/wiki/Maltfield_Log HTTP/1.1 200 OK Server: nginx Date: Thu, 15 Mar 2018 00:53:51 GMT Content-Type: text/html; charset=UTF-8 Content-Length: 0 Connection: keep-alive X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Thu, 15 Mar 2018 00:46:25 GMT X-XSS-Protection: 1; mode=block X-Fuck: Yeah X-Varnish: 5280664 Age: 0 Via: 1.1 varnish-v4 Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" user@personal:~$
- but varnish still did a hit-for-pass
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 198.7.58.245'" | grep -Ei 'hit|pass|miss' - Debug "XXXX HIT-FOR-PASS" - HitPass 2455036 - VCL_call PASS - Link bereq 5154925 pass
- hmm...I just realized that by using the '-I' argument to curl, I don't merely tell curl to output only the headers, but I'm sending a HEAD request to the server instead of a GET request.
- started using `curl -si ... | head` instead
user@personal:~$ curl -si https://wiki.opensourceecology.org/wiki/Maltfield_Log 2>&1 | head -n 25 HTTP/1.1 200 OK Server: nginx Date: Thu, 15 Mar 2018 01:15:22 GMT Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked Connection: keep-alive X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie X-XSS-Protection: 1; mode=block X-Fuck: Yeah X-Varnish: 5281396 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" 1<!DOCTYPE html> <html class="client-nojs" lang="en" dir="ltr"> <head> <meta charset="UTF-8"/> <title>Maltfield Log - Open Source Ecology</title> user@personal:~$
- this produces still produces a hit-for-pass, but the ReqMethod is a GET again
* << Request >> 1923444 - Begin req 1923443 rxreq - Timestamp Start: 1521076586.730409 0.000000 0.000000 - Timestamp Req: 1521076586.730409 0.000000 0.000000 - ReqStart 127.0.0.1 57422 - ReqMethod GET - ReqURL /wiki/Maltfield_Log - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.38.0 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 198.7.58.245, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return hash - VCL_call HASH - VCL_return lookup - Debug "XXXX HIT-FOR-PASS" - HitPass 2455036 - VCL_call PASS - VCL_return fetch - Link bereq 1923445 pass - Timestamp Fetch: 1521076586.923748 0.193339 0.193339 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Thu, 15 Mar 2018 01:16:26 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Content-language: en - RespHeader X-UA-Compatible: IE=Edge - RespHeader Link: </images/ose-logo.png?be82f>;rel=preload;as=image - RespHeader Vary: Accept-Encoding,Cookie - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader X-Fuck: Yeah - RespHeader X-Varnish: 1923444 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1521076586.923774 0.193365 0.000026 - Debug "RES_MODE 4" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1521076586.938319 0.207910 0.014545 - Debug "XXX REF 1" - ReqAcct 232 0 232 417 16115 16532 - End
- I figured out what was happening. It's clear from the above output if you follow it chronologically. First, vcl_recv is called
- VCL_call RECV - ReqUnset X-Forwarded-For: 198.7.58.245, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return hash
- vcl_recv returned 'hash', so that vcl_hash subroutine is called. we don't return anything in ours, so the default varnish subroutine is executed, which simply returns 'lookup'
- VCL_call HASH - VCL_return lookup
- the lookup then returns what's in the cache. And there is an object in the cache. That object is a special 'hit-for-cache' object !! So it has nothing to do with our backend response; we encountered hit-for-pass before we even called the backend! The backend is called when fetch is returned, which calls the vcl_backend_fetch() subroutine
- VCL_return lookup - Debug "XXXX HIT-FOR-PASS" - HitPass 2455036 - VCL_call PASS - VCL_return fetch
- I confirmed the above when I did a `varnish restart`, and the next 2 subsequent GETs resulted in a MISS followed by a HIT
* << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1521077596.856872 0.000000 0.000000 - Timestamp Req: 1521077596.856872 0.000000 0.000000 - ReqStart 127.0.0.1 59084 - ReqMethod GET - ReqURL /wiki/Maltfield_Log - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.38.0 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 198.7.58.245, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return hash - VCL_call HASH - VCL_return lookup - Debug "XXXX MISS" - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1521077597.055543 0.198670 0.198670 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Thu, 15 Mar 2018 01:33:16 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Content-language: en - RespHeader X-UA-Compatible: IE=Edge - RespHeader Link: </images/ose-logo.png?be82f>;rel=preload;as=image - RespHeader Vary: Accept-Encoding,Cookie - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader X-Fuck: Yeah - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1521077597.055605 0.198733 0.000063 - Debug "RES_MODE 4" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1521077597.070207 0.213335 0.014602 - Debug "XXX REF 2" - ReqAcct 232 0 232 411 16115 16526 - End * << Request >> 5 - Begin req 4 rxreq - Timestamp Start: 1521077609.811064 0.000000 0.000000 - Timestamp Req: 1521077609.811064 0.000000 0.000000 - ReqStart 127.0.0.1 59088 - ReqMethod GET - ReqURL /wiki/Maltfield_Log - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.38.0 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 198.7.58.245 - ReqHeader X-Forwarded-For: 198.7.58.245, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 198.7.58.245, 127.0.0.1 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Thu, 15 Mar 2018 01:33:16 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Content-language: en - RespHeader X-UA-Compatible: IE=Edge - RespHeader Link: </images/ose-logo.png?be82f>;rel=preload;as=image - RespHeader Vary: Accept-Encoding,Cookie - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader X-Fuck: Yeah - RespHeader X-Varnish: 5 3 - RespHeader Age: 13 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1521077609.811101 0.000037 0.000037 - RespHeader Content-Length: 16115 - Debug "RES_MODE 2" - RespHeader Connection: close - RespHeader Accept-Ranges: bytes - Timestamp Resp: 1521077609.811124 0.000059 0.000023 - Debug "XXX REF 2" - ReqAcct 232 0 232 437 16115 16552 - End
- so apparently my calls to clear the cache using the ban command were not clearing hit-for-pass objects from the cache
[root@hetzner2 maintenance]# varnishd -Cf /etc/varnish/default.vcl && service varnish reload && varnishadm 'ban req.url ~ "wiki.opensourceecology.org"'
- further confirmation shows that the "req.url ~ ..." match fails
- I found an alternative that does work by matching req.http.host
varnishadm 'ban req.http.host ~ "wiki.opensourceecology.org"'
- updated our documentation on how to clear vhost-specific caches in varnish Web_server_configuration#Varnish
- now that I successfully got a hit, I began undoing my varnish config hacking until I could reproduce the hit-for-pass & isolate the source
- isolated it to this block; when I comment it out, I can get a miss->hit. when I leave it uncommented, I get a miss->hit-for-pass
# if (beresp.http.Set-Cookie) {
# set beresp.uncacheable = true;
# return (deliver);
# }
- that confirms our above assumption that the cpPosTime cookie being set by the backend response was causing varnish to do the hit-for-pass. jumping back down that rabbit hole...
- the cpPosTime logic is inside 'includes/MediaWiki.php' within the function preOutputCommit()
/**
* This function commits all DB changes as needed before
* the user can receive a response (in case commit fails)
* @param IContextSource $context
* @param callable $postCommitWork [default: null]
* @since 1.27
*/
public static function preOutputCommit(
IContextSource $context, callable $postCommitWork = null
) {
- there's 2x instances of the call to setting this cookie in this function; I isolated it to the second one (the error_log() is mine for debugging)
if ( $urlDomainDistance === 'local' || $urlDomainDistance === 'remote' ) {
...
} else {
// OutputPage::output() is fairly slow; run it in $postCommitWork to mask
// the latency of syncing DB positions accross all datacenters synchronously
$flags = $lbFactory::SHUTDOWN_CHRONPROT_SYNC;
if ( $lbFactory->hasOrMadeRecentMasterChanges( INF ) && $allowHeaders ) {
$cpPosTime = microtime( true );
// Set a cookie in case the DB position store cannot sync accross datacenters.
// This will at least cover the common case of the user staying on the domain.
$expires = time() + ChronologyProtector::POSITION_TTL;
$options = [ 'prefix' => '' ];
error_log( "about to set cpPosTime cookie in spot 2" );
$request->response()->setCookie( 'cpPosTime', $cpPosTime, $expires, $options );
}
}
- a stack trace shows that this function was called by index.php's last line calling "$mediaWiki->run();" -> includes/MediaWiki.php:874 = "$this->doPreOutputCommit( $outputWork );" from within private function main()
[Thu Mar 15 03:39:03.257652 2018] [:error] [pid 25129] [client 127.0.0.1:55868] #0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(571): MediaWiki::preOutputCommit(Object(RequestContext), Object(Closure))\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(874): MediaWiki->doPreOutputCommit(Object(Closure))\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/MediaWiki.php(523): MediaWiki->main()\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(43): MediaWiki->run()\n#4 {main}
- curious, this line is just below this block, which mentions our "ChronologyProtector"
// Actually do the work of the request and build up any output
$this->performRequest();
// GUI-ify and stash the page output in MediaWiki::doPreOutputCommit() while
// ChronologyProtector synchronizes DB positions or slaves accross all datacenters.
$buffer = null;
$outputWork = function () use ( $output, &$buffer ) {
if ( $buffer === null ) {
$buffer = $output->output( true );
}
return $buffer;
};
// Now commit any transactions, so that unreported errors after
// output() don't roll back the whole DB transaction and so that
// we avoid having both success and error text in the response
$this->doPreOutputCommit( $outputWork );
// Now send the actual output
print $outputWork();
}
- as a sanity check, I saw that wikipedia also sets cookies (though *different* cookies) for a simple GET by an anon user too!
user@personal:~$ curl -si "https://en.wikipedia.org/wiki/Open_Source_Ecology" | head -n 30 | grep -i cookie Vary: Accept-Encoding,Cookie,Authorization Set-Cookie: WMF-Last-Access=15-Mar-2018;Path=/;HttpOnly;secure;Expires=Mon, 16 Apr 2018 00:00:00 GMT Set-Cookie: WMF-Last-Access-Global=15-Mar-2018;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Mon, 16 Apr 2018 00:00:00 GMT X-Analytics: ns=0;page_id=32407015;https=1;nocookies=1 Set-Cookie: GeoIP=US:VA:Manassas:38.79:-77.54:v4; Path=/; secure; Domain=.wikipedia.org user@personal:~$
- I'll have to research how wikipedia's varnish config handles these WMF-Last-Access, WMF-Last-Access-Global, & GeoIP cookies. It may be translatable to how I should handle this cpPostTime cookie without affecting varnish caching.
Mon Mar 12, 2018
- tuning-out noise on ossec email alerts
- 990012 = bad robots
- 950109 = multiple url encoding detected
- 60912 = failed to parse request body
- 958291 = Range: field exists and begins with 0.
- 960035 = URL file extension is restricted by policy
- continuing to debug the no-cache headers on the wiki
[root@hetzner2 wiki.opensourceecology.org]# for file in $(echo $files); do grep -il 'no-cache, no-store, max-age=0, must-revalidate' $file; done htdocs/includes/OutputPage.php htdocs/includes/OutputPage.phpee htdocs/includes/specials/SpecialRevisiondelete.php htdocs/includes/specials/SpecialUndelete.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/ConfirmAccount_body.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/UserCredentials_body.php [root@hetzner2 wiki.opensourceecology.org]#
- OutputPage appears to default to using the cache. The code that sets our undesired header is encapsulated in an if whoose condition is triggered when 'mEnableClientCache' is not true. It defaults to true. It's flipped to false (causing no cache headers) when the enableClientCache() function is called. This occurs when the $parserOutput->isCacheable() is found to be false and when the page is an error page. Both of those cases are reasonable.
- SpecialRevisiondelete.php appears to use the no-cache headers only in the tryShowFile() function, which is apparently just used when displaying an old article that was deleted (at a specific version). This doesn't appear to be relevant.
- The SpecialUndelete.php file sets the headers in showFile(), but probably only when attempting to restore a deleted article. This doesn't appear to be relevant.
- And the ConfirmAccount scripts shouldn't be relevant as they're disabled.
- so y best guess is that it's still OutputPage.php, caused by the $isCacheable() function triggering a 'no'
- tried adding 'error_log' output into the OutputPage.php script to no avail; it doesn't appear to come from there
- did the error_log() output for debugging purposes on other files above
- couldn't reproduce with any, but then I noticed that the headers are different!
root@hetzner2 ConfirmAccount]# curl -I "http://127.0.0.1:8000/wiki/Main_Page" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Tue, 13 Mar 2018 00:10:05 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Mon, 12 Mar 2018 19:10:05 GMT X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520899806.0221; expires=Tue, 13-Mar-2018 00:11:06 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [root@hetzner2 ConfirmAccount]#
- ok, I pinpointed *this* header being set in OutputPage.php block:
} else {
# We do want clients to cache if they can, but they *must* check for updates
# on revisiting the page.
wfDebug( METHOD . ": private caching; {$this->mLastModified} **", 'private' );
$response->header( 'Expires: ' . gmdate( 'D, d M Y H:i:s', 0 ) . ' GMT' );
$response->header( "Cache-Control: private, must-revalidate, max-age=0" );
}
- so this is within an if statement:
if ( $this->mEnableClientCache ) {
- therefore, we know that mEnableClientCache is true at the time this header is being set
- the else is because this list of conditions equated to false
if (
$config->get( 'UseSquid' ) &&
!$response->hasCookies() &&
!SessionManager::getGlobalSession()->isPersistent() &&
!$this->isPrintable() &&
$this->mCdnMaxage != 0 &&
!$this->haveCacheVaryCookies()
) {
- debugging shows that the issue is the 3rd one, !SessionManager::getGlobalSession()->isPersistent()
- This rabbit hole is too deep; I'll just go with the mediawiki recommendation to configure varnish to ingore mediawiki's cache control headers.
- sent an email to Marcin & Catarina about Munin w/ a screenshot of most graphs for the past week
Tue Mar 06, 2018
- got an ossec alert for brute-force attacks on wp-login.php
- this shouldn't happen, since we're just 403-forbidden blocking this page
- discovered that the httpd config for this block was commented-out in oswh; fixed
- this shouldn't happen, since we're just 403-forbidden blocking this page
- disk usage is higher than I'd like, hovering at 85%
- installed `ncdu` = an ncurses disk usage analyzer
- there's more usage in /var/tmp (61G) than in /var/www (37G)
- most of the space is in /var/tmp/backups_for_migration_from_hetzner1 (58G)
- removed the contents of /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/old/20180214 (30G)
[maltfield@hetzner2 20180214]$ pwd /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/old/20180214 [maltfield@hetzner2 20180214]$ du -sh * 296M core 1.4G db.sql 17G htdocs 8.0K imagesLargerThan20M.txt 191M mysqldump_wiki.20180120.sql.bz2 12G wiki_files.20180120.tar.gz [maltfield@hetzner2 20180214]$ du -sh 30G . [maltfield@hetzner2 20180214]$
- continuing wiki varnish config
- found that mediawiki is producing headers in its response that would prevent varnish from caching the main page!
[maltfield@hetzner2 ~]$ curl -I "http://127.0.0.1:8000/wiki/Main_Page" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Tue, 06 Mar 2018 16:41:16 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Tue, 06 Mar 2018 16:34:44 GMT X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520354476.8255; expires=Tue, 06-Mar-2018 16:42:16 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [maltfield@hetzner2 ~]$
- the recommended/documented varnish config on mediawiki's wiki shows the block respecting any of the "private|no-cache|no-store" Cache-Control headers sent by mediawiki commented-out in order to ignore mediawiki. https://www.mediawiki.org/wiki/Manual:Varnish_caching#Configuring_Varnish_4.x
In this example, the 'no-cache' flag is being ignored on pages served to anonymous-IP users. Such measures normally are only needed if a wiki is making extensive use of extensions which add this flag indiscriminately (such as a wiki packed with random <choose>/<option> Algorithm tags on the main page and various often-used templates).
# Called after a document has been successfully retrieved from the backend.
sub vcl_backend_response {
# set minimum timeouts to auto-discard stored objects
set beresp.grace = 120s;
if (beresp.ttl < 48h) {
set beresp.ttl = 48h;
}
if (!beresp.ttl > 0s) {
set beresp.uncacheable = true;
return (deliver);
}
if (beresp.http.Set-Cookie) {
set beresp.uncacheable = true;
return (deliver);
}
# if (beresp.http.Cache-Control ~ "(private|no-cache|no-store)") {
# set beresp.uncacheable = true;
# return (deliver);
# }
if (beresp.http.Authorization && !beresp.http.Cache-Control ~ "public") {
set beresp.uncacheable = true;
return (deliver);
}
return (deliver);
}
- this issue was exactly what I just encountered on osmain, but in wordpress! There was a plugin named 'fundraising' that was preventing caching. Apparently there's an extension in mediawiki doing something similar.
- found a list of all extensions currently installed: https://wiki.opensourceecology.org/wiki/Special:Version
- disabled all extensions & confirmed that the issue was still present
- note that there's no easy way to do this; you have to just comment-out all the require & wfLoadExtension function calls in LocalSettings.php until Special:Version shows no extensions "installed"
[maltfield@hetzner2 ~]$ curl -I "http://127.0.0.1:8000/wiki/Main_Page?safemode=1" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Wed, 07 Mar 2018 00:19:58 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Wed, 07 Mar 2018 00:18:56 GMT X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520381998.2197; expires=Wed, 07-Mar-2018 00:20:58 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [maltfield@hetzner2 ~]$
- fuck, even when I comment-out all the skins (breaking the site), the issue is still present
[maltfield@hetzner2 ~]$ curl -I "http://127.0.0.1:8000/wiki/Main_Page?safemode=1" -H "Host: wiki.opensourceecology.org" HTTP/1.1 200 OK Date: Wed, 07 Mar 2018 00:22:42 GMT Server: Apache X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1520382163.0842; expires=Wed, 07-Mar-2018 00:23:43 GMT; Max-Age=60; path=/; httponly;HttpOnly Content-Type: text/html; charset=UTF-8 [maltfield@hetzner2 ~]$
- found a list of all the files that have 'cache-control' in them in the wiki dir
[root@hetzner2 wiki.opensourceecology.org]# grep -irl 'cache-control' * htdocs/opensearch_desc.php htdocs/includes/actions/RawAction.php htdocs/includes/OutputPage.php htdocs/includes/AjaxResponse.php htdocs/includes/api/ApiMain.php htdocs/includes/api/i18n/ru.json htdocs/includes/api/i18n/de.jsone htdocs/includes/api/i18n/ru.jsone htdocs/includes/api/i18n/ba.json htdocs/includes/api/i18n/de.json htdocs/includes/api/i18n/ba.jsone htdocs/includes/OutputPage.phpee htdocs/includes/specials/SpecialRevisiondelete.php htdocs/includes/specials/SpecialUndelete.php htdocs/includes/specials/SpecialUploadStash.php htdocs/includes/specials/SpecialJavaScriptTest.php htdocs/includes/HeaderCallback.php htdocs/includes/resourceloader/ResourceLoader.php htdocs/includes/libs/filebackend/HTTPFileStreamer.php htdocs/includes/WebStart.php htdocs/includes/PHPVersionCheck.php htdocs/includes/OutputHandler.php htdocs/thumb.php htdocs/img_auth.php htdocs/extensions/ConfirmEdit/FancyCaptcha/FancyCaptcha.class.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/ConfirmAccount_body.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/UserCredentials_body.php wiki-error.log [root@hetzner2 wiki.opensourceecology.org]#
- found the possible sources
[root@hetzner2 wiki.opensourceecology.org]# for file in $(echo $files); do grep -il 'no-cache, no-store, max-age=0, must-revalidate' $file; done htdocs/includes/OutputPage.php htdocs/includes/OutputPage.phpee htdocs/includes/specials/SpecialRevisiondelete.php htdocs/includes/specials/SpecialUndelete.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/ConfirmAccount_body.php htdocs/extensions/ConfirmAccount/frontend/specialpages/actions/UserCredentials_body.php [root@hetzner2 wiki.opensourceecology.org]#
- OutputPage appears to default to using the cache. The code that sets our undesired header is encapsulated in an if whoose condition is triggered when 'mEnableClientCache' is not true. It defaults to true
Mon Mar 05, 2018
- Updated log & hours
- checked munin, all looks good.
- removed the temp comments in front of the block to deny access to 'wp-login.php' from osemain.
- also noticed that (in addition to the blue "cache misses" graph nearly disappearing) the varnish4 "backend traffic" graph shows a great decrease in backend connections, especially the blue = "success", following my varnish change to cache backend responses with code = 404 not found
- began researching varnish integration with mediawiki
- this great article describes how Mediawiki's core was designed to work with Varnish specifically https://www.mediawiki.org/wiki/Manual:Varnish_caching#cite_note-2
- this is less of an issue for our wiki because we don't allow anonymous edits, but I added these lines to LocalSettings.php to record ip addresses using the "X-Forwarded-For" header instead of the Source IP, which would just be 127.0.0.1 with our varnish->apache architecture
$wgUseSquid = true;
$wgSquidServers = array( '127.0.0.1', 'wiki.opensourceecology.org' );
$wgUsePrivateIPs = true;
//Use $wgSquidServersNoPurge if you don't want MediaWiki to purge modified pages
//$wgSquidServersNoPurge = array('127.0.0.1');
- found a list of squid (relevant to varnish) related options https://www.mediawiki.org/wiki/Manual:Configuration_settings#Squid
Sun Mar 04, 2018
- after enabling caching of 404s, the "Hit rates" graph of varnish4 in munin changed--the "cache misses" shrunk significantly. Now I just need to find & reduce the "Cache hits for pass"
- began testing why a request for 'https://www.opensourceecology.org/robots.txt' produces a HIT-FOR-PASS
root@personal:~# curl -i https://www.opensourceecology.org/robots.txt HTTP/1.1 200 OK Server: nginx Date: Sun, 04 Mar 2018 19:10:42 GMT Content-Type: text/plain; charset=utf-8 Content-Length: 182 Connection: keep-alive Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <https://www.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=ee8abcqulnd7f8e9jbv1gve33cht0fuohmje7bps0iokihfhnah4c5ib4r95vpv9muq2ccqms2vhjf469rfj75gv1bik423vkobovv3; path=/; HttpOnly;HttpOnly X-Varnish: 2109565 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" Sitemap: https://www.opensourceecology.org/sitemap.xml Sitemap: https://www.opensourceecology.org/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php root@personal:~#
- strangely, this file doesn't even exist!
[root@hetzner2 ~]# find /var/www/html/www.opensourceecology.org | grep -i robots.txt [root@hetzner2 ~]#
- it also works if I just hit apache directly, locally
[root@hetzner2 ~]# curl -i 'http://127.0.0.1:8000/robots.txt' -H 'Host: www.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 19:14:38 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <https://www.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=bear8i55veadv2osgrpsg4m6t45n4r1c1gdcveomhbftk26okrgvjmeafge1poj5a0kkrakqn5ghbb7s4r0bqmiic3o9l5djpu5nmc3; path=/; HttpOnly;HttpOnly Content-Length: 182 Content-Type: text/plain; charset=utf-8 Sitemap: https://www.opensourceecology.org/sitemap.xml Sitemap: https://www.opensourceecology.org/news-sitemap.xml User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php [root@hetzner2 ~]#
- indeed, this behaviour differs from fef, another one of our fef wordpress sites. note that the "Expires" & "Cache-Control" headers are absent in Apache's response
[root@hetzner2 ~]# curl -i 'http://127.0.0.1:8000/robots.txt' -H 'Host: fef.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 19:19:56 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <https://fef.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Content-Length: 67 Content-Type: text/plain; charset=utf-8 User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
- I'm beginning to think that it's one of the osemain plugins causing this issue
- it appears that there's some sort of apache-level (or os-level) caching going on too. note that the first request below takes 10 fucking seconds. the second takes less than 1 second.
[root@hetzner2 ~]# time curl -I 'http://127.0.0.1:8000/apple-touch-icon.png' -H 'Host: fef.opensourceecology.org' HTTP/1.1 404 Not Found Date: Sun, 04 Mar 2018 20:34:18 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://fef.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m10.150s user 0m0.002s sys 0m0.001s [root@hetzner2 ~]# time curl -I 'http://127.0.0.1:8000/apple-touch-icon.png' -H 'Host: fef.opensourceecology.org' HTTP/1.1 404 Not Found Date: Sun, 04 Mar 2018 20:34:35 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Wed, 11 Jan 1984 05:00:00 GMT Cache-Control: no-cache, must-revalidate, max-age=0 Link: <https://fef.opensourceecology.org/wp-json/>; rel="https://api.w.org/" X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m0.125s user 0m0.002s sys 0m0.001s [root@hetzner2 ~]#
- if I wait a bit (I've observed this after 3 minutes) & re-run the above, it takes another 10 seconds.
- it looks like the "expires" & "cache-control" headers are set by wp natively (or, at most, a plugin calling this native function) using wp_get_nocache_headers and related functions https://developer.wordpress.org/reference/functions/wp_get_nocache_headers/
- wp-includes/class-wp.php appears to applying the above function for 404s, that's annoying (ie: we want to cache 404s for requests that are slamming our site constantly, such as '/apple-touch-icon.png' https://developer.wordpress.org/reference/classes/wp/handle_404/
- removed google analytics code from osemain via Appearance -> Theme Options -> Footer
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-58526017-1', 'auto');
ga('send', 'pageview');
</script>
<pre>
# created a staging site for osemain so I could tear down plugins & themes to isolate the cause of the call to nocache_headers()
## renamed 'osemain' A record on CF DNS to 'staging' for staging.opensourceecology.org
## created /etc/httpd/conf.d/staging.opensourceecology.org.conf
## created staging vhost files & db. note that the max char limit for a db username is 16, so I used "osemain_s_user" & "osemain_s_db"
<pre>
source /root/backups/backup.settings
prodVhostDir=/var/www/html/www.opensourceecology.org
stagingVhostDir=/var/www/html/staging.opensourceecology.org
prodDbName=osemain_db
stagingDbName=osemain_s_db
stagingDbUser=osemain_s_user
stagingDbPass=CHANGEME
- updated the wp-config.php to be http, not https, so I can just query with curl (otherwise it 301's to https)
- successfully reproduced the issue on the staging site
[root@hetzner2 staging.opensourceecology.org]# time curl -I 'http://127.0.0.1:8000/' -H "Host: staging.opensourceecology.org" HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:45:23 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=b3kuv36bt8fi8j9j07heh3a8epvug5fdq4go0ajrgsv4gq4aevvpi2v5ms212o68pdq4tkrjfde79274bbus3nvs9fvam273i6vohg0; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m10.331s user 0m0.003s sys 0m0.000s [root@hetzner2 staging.opensourceecology.org]# time curl -I 'http://127.0.0.1:8000/' -H "Host: staging.opensourceecology.org" HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:45:35 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=p9gdj5vsrhh18t0ahlv60f68eq35slum7bjv3vb98v3rbplifj9a0e2lpqhh67lsdmi6h1d1krj2pah0ja0vaa8gvrgeokqkp0sha72; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m0.315s user 0m0.002s sys 0m0.001s [root@hetzner2 staging.opensourceecology.org]#
- I removed the plugins directory in the staging docroot & replaced it with an empty dir, and confirmed that the issue went away!
[root@hetzner2 wp-content]# mv plugins plugins.old [root@hetzner2 wp-content]# mkdir plugins [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:49:00 GMT Server: Apache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m12.423s user 0m0.002s sys 0m0.002s [root@hetzner2 wp-content]#
- I copied in my base plugin set from the old plugins dir, and confirmed that the headers look ok still
[root@hetzner2 wp-content]# ls plugins force-strong-passwords google-authenticator google-authenticator-encourage-user-activation rename-wp-login ssl-insecure-content-fixer vcaching [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:52:36 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m13.186s user 0m0.002s sys 0m0.002s [root@hetzner2 wp-content]#
- I iteratively copied-in the plugins one-by-one that were activated on the prod site, until I found the issue crop-up after I added the "fundraising" plugin
[root@hetzner2 wp-content]# rsync -a plugins.old/akismet/ plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:55:27 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m10.166s user 0m0.002s sys 0m0.001s [root@hetzner2 wp-content]# rsync -a plugins.old/brankic-photostream-widget plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:55:45 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m0.134s user 0m0.003s sys 0m0.000s [root@hetzner2 wp-content]# rsync -a plugins.old/duplicate-post plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:56:02 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Content-Type: text/html; charset=UTF-8 real 0m0.133s user 0m0.001s sys 0m0.002s [root@hetzner2 wp-content]# rsync -a plugins.old/fundraising plugins/ [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:56:17 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=d679j00fdalqp4usqe0iuf71vs6jhl29bbdds5e2n56i0p15m1379vnnlj2c1hlunjn061l6rtkij0141mpsvagnei6g8anihk842i2; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m0.144s user 0m0.001s sys 0m0.002s
- I replaced the temp plugins dir with the original dir & confirmed the issue was present
root@hetzner2 wp-content]# rm -rf plugins [root@hetzner2 wp-content]# mv plugins.old plugins [root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:58:51 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=uvjm8sqbdl1uqbmrr27g5qns54sg0enk177m1efqpuournqneir1qtruoqdu6ph3go9vqflrpg58gq1hrn09hoqrrs4a6ttfeukcgs3; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m11.097s user 0m0.001s sys 0m0.002s
- I moved out the 'fundraising' plugin, and confirmed that the issue was fixed by this one change
[root@hetzner2 wp-content]# time curl -I 'http://127.0.0.1:8000/' -H 'Host: staging.opensourceecology.org' HTTP/1.1 200 OK Date: Sun, 04 Mar 2018 22:58:51 GMT Server: Apache X-VC-Enabled: true X-VC-TTL: 86400 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Link: <http://staging.opensourceecology.org/wp-json/>; rel="https://api.w.org/" Link: <http://staging.opensourceecology.org/>; rel=shortlink X-XSS-Protection: 1; mode=block Set-Cookie: OSESESSION=uvjm8sqbdl1uqbmrr27g5qns54sg0enk177m1efqpuournqneir1qtruoqdu6ph3go9vqflrpg58gq1hrn09hoqrrs4a6ttfeukcgs3; path=/; HttpOnly;HttpOnly Content-Type: text/html; charset=UTF-8 real 0m11.097s user 0m0.001s sys 0m0.002s
- I added staging.opensourceecology.org vhost configs to nginx & varnish
- updated cert to include the staging SAN
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/staging.opensourceecology.org/htdocs -d staging.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org -d munin.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- added '950120' [Possible Remote File Inclusion (RFI) Attack: Off-Domain Reference/Link] to the modsec ignore list in /var/ossec/rules/local_rules.xml as it was too spammy
- updated Wordpress documentation for file permissions & wp-cli troubleshooting per what was discovered during the osemain migration (for plugins, themes, & upgrade dirs)
- discovered that the $wgDebugComments variable puts the the mediawiki debug messages (which may contain sensitive user or server info!) directly in the html output! https://www.mediawiki.org/wiki/Manual:Configuration_settings#Debug/logging
- removed this from LocalSettings.php & added a strong warning message telling admins to *only* use $wgDebugLogFile, which exists *outside* the docroot
- added rules to ossec to detect & active response (block via iptables) brute force attempts over mediawiki
<!-- Mediawiki Special:UserLogin brute force --> <rule id="100060" level="3"> <if_sid>31530</if_sid> <url>Special:UserLogin</url> <regex>POST</regex> <description>Mediawiki login attempt.</description> </rule> <!-- If we see frequent mediawiki login POST's, it is likely a bot. --> <rule id="100061" level="8" frequency="6" timeframe="30"> <if_matched_sid>100060</if_matched_sid> <same_source_ip /> <descript
- tested the above rules & verified that I was banned via iptables when I kicked off a bunch of simultanious login attempts with curl
while true; do date; curl --data "wpName=superman&wpPassword1=letmein" "https://wiki.opensourceecology.org/index.php?title=Special:UserLogin"; echo; done
- note that I had to execute this in many terminals at once, as the request<-->response time was too long to actually trigger the ban
- I'm now finished with the general hardening of mediawiki. Next up: varnish integration (and hardening of *that* config)
Sat Mar 03, 2018
- all munin graphs look great. I'll let it populate for 2 weeks before delivering to Marcin.
- updated StatusCake for osemain from 'http://opensourceecology.org' to 'https://www.opensourceecology.org/'
- saw that, indeed, statuscake shows that the site sped up significantly post migration. Before, our load times averaged ~1.3 seconds. Now it's showing ~0.3 seconds
- munin is showing about a 45% hit rate. that's lower than I would expect.
- I found a command to get a list of the cache misses
[root@hetzner2 ~]# varnishtop -i BereqURL list length 38 hetzner2.opensourceecology.org 3.49 BereqURL /apple-touch-icon.png 3.45 BereqURL /apple-touch-icon-precomposed.png 2.31 BereqURL /favicon.png 2.28 BereqURL /apple-touch-icon-120x120.png 2.24 BereqURL /apple-touch-icon-120x120-precomposed.png 1.43 BereqURL /apple-touch-icon-152x152.png 1.40 BereqURL /apple-touch-icon-152x152-precomposed.png 0.57 BereqURL /category/proposals/feed/ 0.48 BereqURL /robots.txt 0.46 BereqURL /localhost/localhost/varnish_expunge-day.png 0.46 BereqURL /static/logo-h.png 0.46 BereqURL /localhost/localhost/varnish_bad-day.png 0.46 BereqURL /localhost/localhost/varnish_uptime-day.png 0.46 BereqURL /localhost/localhost/varnish_objects-day.png 0.46 BereqURL /localhost/localhost/varnish_threads-day.png 0.46 BereqURL /localhost/localhost/varnish_memory_usage-day.png 0.46 BereqURL /localhost/localhost/varnish_request_rate-day.png 0.46 BereqURL /localhost/localhost/varnish_transfer_rates-day.png 0.46 BereqURL /localhost/localhost/varnish_hit_rate-day.png 0.46 BereqURL /static/style-new.css 0.46 BereqURL /localhost/localhost/varnish_backend_traffic-day.png 0.46 BereqURL /varnish4-day.html 0.46 BereqURL /category/ted-fellows/ 0.38 BereqURL / ... [root@hetzner2 ~]#
- the above output shows "BereqURL" occurrences in `varnishlog` = backend requests = cache misses
- We see that the '/apple-touch-icon.png' file is requested about 3 times per minute. It's actually just a 404. This shows that it's not caching this 404, which would be a trivial saver
- We also see that the '/' is being requested ~0.38 times per second. So about every 3 seconds, varnish is troubling apache for the main page.
- why? I want this cached for 24 hours. Looks like we could be better optimized
- dug back in my logs to 2017-11-24, when I configured wordpress' apache no what to cache by response code
# Avoid caching error responses
#if (beresp.status == 404 || beresp.status >= 500) {
if ( beresp.status != 200 && beresp.status != 203 && beresp.status != 300 && beresp.status != 301 && beresp.status != 302 && beresp.status != 304 && beresp.status != 307 && beresp.status != 410 ) {
set beresp.ttl = 0s;
set beresp.grace = 15s;
}
- according to the (still outstanding) bug report I filed 3 months ago, I didn't cache 404 only because the developer of the worpdress plugin didn't cache 404. https://wordpress.org/support/topic/please-dont-cache-403-by-default/
- but varnish, by default, *does* cache 404 https://book.varnish-software.com/4.0/chapters/VCL_Basics.html#the-initial-value-of-beresp-ttl
- therefore, I think I *should* cache 404. Or at least experiment with it.
- I changed it on prod for www.opensourceecology.org to cache 404s too
# Avoid caching error responses
#if (beresp.status == 404 || beresp.status >= 500) {
if ( beresp.status != 200 && beresp.status != 203 && beresp.status != 300 && beresp.status != 301 && beresp.status != 302 && beresp.status != 304 && beresp.status != 307 && beresp.status != 410 && beresp.status != 404 ) {
set beresp.ttl = 0s;
set beresp.grace = 15s;
}
- I noticed that varnishlog shows some HITs with an already-expired header
- ReqURL / - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate, br - ReqHeader Accept-Encoding: gzip - VCL_call HASH - ReqHeader hash: #www.opensourceecology.org - VCL_return lookup - Hit 83893 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Sat, 03 Mar 2018 19:52:48 GMT - RespHeader Server: Apache - RespHeader X-VC-Enabled: true - RespHeader X-VC-TTL: 86400 - RespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT - RespHeader Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 - RespHeader Pragma: no-cache
- why was this "Expires: Thu, 19 Nov 1981 08:52:00 GMT"?
- why was this "Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0"?
- why was this "Pragma: no-cache"?
- a much simpler test is '/robots.txt'. Why is this not caching?
Fri Mar 02, 2018
- Marcin followed-up on osemain
- he said it's running much faster :)
- he found a modesec false-positive when attempting to modify https://www.opensourceecology.org/open-building-institute-nears-launch/
- I whitelisted 959070, sqli
- began doing some ossec tuning
- after osemain migrated to hetzner2, the alert volume has become unreasonably high, so I want to prevent alerts from emailing in a few cases
- rule 30411 = modsecurity where the following strings match
- 960020 = Pragma Header requires Cache-Control Header
- 960009 = Request Missing a User Agent Header
- rule 31123 = Web server 503 error code (Service unavailable)
- but we'll not disable 31163 = Multiple web server 503 error code (Service unavailable).
- rule 30411 = modsecurity where the following strings match
- after osemain migrated to hetzner2, the alert volume has become unreasonably high, so I want to prevent alerts from emailing in a few cases
- returned to configuring cacti
- created nginx, varnish, & http config files
- created log dirs for nginx & apache
- created cacti.opensourceecology.org DNS entry pointing to 138.201.84.243 on CloudFlare
- update /etc/php.ini to include the cacti dir
- created cacti db & user per https://skrinhitam.wordpress.com/2017/04/21/how-to-install-cacti-on-centos-7/
CREATE DATABASE cacti_db; CREATE USER 'cacti_user'@'localhost' IDENTIFIED BY 'CHANGEME'; GRANT ALL PRIVILEGES ON cacti_db.* TO 'cacti_user'@'localhost'; FLUSH PRIVILEGES;
- updated /usr/share/cacti/include/confgi.php with credentials
- confirmed that we're running cacti v1.1.36
[root@hetzner2 cacti]# cat /usr/share/cacti/include/cacti_version 1.1.36 [root@hetzner2 cacti]#
- according to the website, the latest version is 1.1.36 from 2018-02-25. So we're latest, and it was updated within a month ago. That's great. https://www.cacti.net/download_cacti.php
- finally got it connected; sockets was nontrivial; you actually have to define the port as 3306, though it's a misnomer (misnumber?)
$database_type = 'mysql'; $database_default = 'cacti_db'; $database_hostname = 'localhost'; $database_username = 'cacti_user'; $database_password = 'CHANGEME'; $database_port = '3306';
- this time I got an error that the db was not initialized (it's true; there's no tables!), but the sql file they're telling me to use doesn't exist!
The Cacti Database has not been initialized. Please initilize it before continuing. To initilize the Cacti database, issue the following commands either as root or using a valid account. mysqladmin -uroot -p create cacti mysql -uroot -p -e "grant all on cacti.* to 'someuser'@'localhost' identified by 'somepassword'" mysql -uroot -p -e "grant select on mysql.time_zone_name to 'someuser'@'localhost' identified by 'somepassword'" mysql -uroot -p cacti < /pathcacti/cacti.sql Where /pathcacti/ is the path to your Cacti install location. Change someuser and somepassword to match your site preferences. The defaults are cactiuser for both user and password. NOTE: When installing a remote poller, the config.php file must be writable by the Web Server account, and must include valid connection information to the main Cacti server. The file should be changed to read only after the install is completed.
- ah, the distro put it into /usr/share/doc/cacti-1.136/cacti.sql ..of course (?)
[root@hetzner2 cacti]# rpm -ql cacti | grep cacti.sql /usr/share/doc/cacti-1.1.36/cacti.sql [root@hetzner2 cacti]#
- I initialized the db with this file
mysql -u"cacti_user" -p"CHANGEME" cacti_db < /usr/share/doc/cacti-1.1.36/cacti.sql
- set `date.timezone = "UTC"` in /etc/php.ini
- I had to comment-out the default config's block that blocks all non-localhost traffic (my .htpasswd config should suffice)
<Directory /usr/share/cacti/> # <IfModule mod_authz_core.c> # # httpd 2.4 # Require host localhost # </IfModule> </Directory>
- granted the new cacti user SELECT permissions into the mysql.time_zone table
GRANT SELECT ON mysql.time_zone_name TO cacti_user@localhost; flush privileges;
- populated the mysql timezone database, following a backup
sudo su -
source /root/backups/backup.settings
stamp=`date +%Y%m%d_%T`
tmpDir=/var/tmp/dbChange.$stamp
mkdir $tmpDir
chown root:root $tmpDir
chmod 0700 $tmpDir
pushd $tmpDir
service httpd stop
# create backup of all DBs for good measure
time nice mysqldump -u"root" -p"${mysqlPass}" --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz
service httpd start
mysql_tzinfo_to_sql /usr/share/zoneinfo/ | mysql -u"root" -p"${mysqlPass}" mysql
- expanded the cert to include cacti
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org -d cacti.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- Cacti was very unhappy with our mariadb settings. I'm not concerned as this is just for 1 server, and I don't want to fuck with the db backend which is prod on so many sites. But I'll leave this here for records
MariaDB Tuning (/etc/my.cnf) - [ Documentation ] Note: Many changes below require a database restart Variable Current Value Recommended Value Comments version 5.5.56-MariaDB >= 5.6 MySQL 5.6+ and MariaDB 10.0+ are great releases, and are very good versions to choose. Make sure you run the very latest release though which fixes a long standing low level networking issue that was causing spine many issues with reliability. collation_server latin1_swedish_ci utf8_general_ci When using Cacti with languages other than English, it is important to use the utf8_general_ci collation type as some characters take more than a single byte. If you are first just now installing Cacti, stop, make the changes and start over again. If your Cacti has been running and is in production, see the internet for instructions on converting your databases and tables if you plan on supporting other languages. character_set_client latin1 utf8 When using Cacti with languages other than English, it is important to use the utf8 character set as some characters take more than a single byte. If you are first just now installing Cacti, stop, make the changes and start over again. If your Cacti has been running and is in production, see the internet for instructions on converting your databases and tables if you plan on supporting other languages. max_connections 151 >= 100 Depending on the number of logins and use of spine data collector, MariaDB will need many connections. The calculation for spine is: total_connections = total_processes * (total_threads + script_servers + 1), then you must leave headroom for user connections, which will change depending on the number of concurrent login accounts. max_allowed_packet 1048576 >= 16777216 With Remote polling capabilities, large amounts of data will be synced from the main server to the remote pollers. Therefore, keep this value at or above 16M. tmp_table_size 16M >= 64M When executing subqueries, having a larger temporary table size, keep those temporary tables in memory. join_buffer_size 0.125M >= 64M When performing joins, if they are below this size, they will be kept in memory and never written to a temporary file. innodb_file_per_table OFF ON When using InnoDB storage it is important to keep your table spaces separate. This makes managing the tables simpler for long time users of MariaDB. If you are running with this currently off, you can migrate to the per file storage by enabling the feature, and then running an alter statement on all InnoDB tables. innodb_doublewrite ON OFF With modern SSD type storage, this operation actually degrades the disk more rapidly and adds a 50% overhead on all write operations. innodb_additional_mem_pool_size 8M >= 80M This is where metadata is stored. If you had a lot of tables, it would be useful to increase this. innodb_lock_wait_timeout 50 >= 50 Rogue queries should not for the database to go offline to others. Kill these queries before they kill your system. innodb_flush_log_at_trx_commit 1 2 Setting this value to 2 means that you will flush all transactions every second rather than at commit. This allows MariaDB to perform writing less often.
- I setup /etc/snmpd/snmpd.conf with a basic config
- I got cacti to add the device & connect to it
- I couldn't get cacti to add a graph. debugging the apache logs show that it was trying to execute spine, which isn't installed
- I uncommented the poller.php defined in
- the poller was pretty unhappy about our hardened php.ini
[root@hetzner2 snmp]# sudo -u cacti /usr/share/cacti/poller.php PHP Warning: ini_set() has been disabled for security reasons in /usr/share/cacti/include/global.php on line 65 PHP Warning: file_exists(): open_basedir restriction in effect. File(localhost) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/lib/database.php on line 65 PHP Warning: file_exists(): open_basedir restriction in effect. File(/usr/local/spine/bin/spine) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/include/global_arrays.php on line 560 PHP Warning: php_uname() has been disabled for security reasons in /usr/share/cacti/poller.php on line 79 03/02/2018 21:21:41 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 1.168,2 seconds have passed since the last poll! PHP Warning: ini_set() has been disabled for security reasons in /usr/share/cacti/poller.php on line 256 PHP Warning: ini_set() has been disabled for security reasons in /usr/share/cacti/poller.php on line 257 PHP Warning: file_exists(): open_basedir restriction in effect. File(/bin/php) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/lib/poller.php on line 124 PHP Warning: system() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 143 PHP Warning: exec() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 131 PHP Warning: file_exists(): open_basedir restriction in effect. File(/bin/php) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/var/www/html/cacti.opensourceecology.org/:/usr/share/cacti/:/etc/cacti/) in /usr/share/cacti/lib/poller.php on line 124 PHP Warning: system() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 143 PHP Warning: exec() has been disabled for security reasons in /usr/share/cacti/lib/poller.php on line 131 PHP Warning: putenv() has been disabled for security reasons in /usr/share/cacti/lib/rrd.php on line 55 PHP Warning: popen() has been disabled for security reasons in /usr/share/cacti/lib/rrd.php on line 97 ^C03/02/2018 21:23:11 - POLLER: Poller[1] WARNING: Cacti Master Poller process terminated by user
- therefore, I think we need spine
- spine is not in the yum repos :(
- this isn't looking good https://forums.cacti.net/post-91042.html
- ok, I'm abandoning Cacti; let's try Munin. It emphasizes that it works well OOTB, and it appears to run perl on the back-end, only producing static html pages. That's the most ideal. Just give us a whole docroot that is totally static content!
- began attempting to get munin working
- I installed it with `yum install munin` and it automatically created a dir with static files in /var/www/html/munin
- I started the service `service munin-node start`
- It works! That. Was. Fucking. Awesome. Now I just need to wait 30 minutes per https://www.tecmint.com/install-munin-network-monitoring-in-rhel-centos-fedora/
- Holy hell, it includes varnish stats ootb!! What is this awesomeness?!?
- no going back; I'm renaming all the 'cacti' configs 'munin'
- changed the cert SAN from cacti to munin
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org -d munin.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- varnish graphs weren't populating; I think it's because we need the varnish4 plugin https://blog.raika.fr/index.php/2015/06/14/varnish-cache-4-et-munin-node/
pushd /etc/munin/plugins wget https://raw.githubusercontent.com/munin-monitoring/contrib/master/plugins/varnish/varnish4_ chmod +x varnish4_ mv varnish_hit_rate varnish_hit_rate.bak mv varnish_backend_traffic varnish_backend_traffic.bak mv varnish_bad varnish_bad.bak mv varnish_expunge varnish_expunge.bak mv varnish_memory_usage varnish_memory_usage.bak mv varnish_objects varnish_objects.bak mv varnish_request_rate varnish_request_rate.bak mv varnish_threads varnish_threads.bak mv varnish_transfer_rates varnish_transfer_rates.bak mv varnish_uptime varnish_uptime.bak ln -s varnish4_ varnish_hit_rate ln -s varnish4_ varnish_backend_traffic ln -s varnish4_ varnish_bad ln -s varnish4_ varnish_expunge ln -s varnish4_ varnish_memory_usage ln -s varnish4_ varnish_objects ln -s varnish4_ varnish_request_rate ln -s varnish4_ varnish_threads ln -s varnish4_ varnish_transfer_rates ln -s varnish4_ varnish_uptime ln -s varnish4_ varnish_hit_rate cat >> /etc/munin/plugin-conf.d/munin-node << EOF [varnish4_*] user root env.varnishstat /bin/varnishstat EOF munin-node-configure --shell service munin-node restart
- test shows it's working for hit_rate
[root@hetzner2 plugins]# munin-run varnish_hit_rate Line is not well formed (env.name) at /usr/share/perl5/vendor_perl/Munin/Node/Service.pm line 110. at /etc/munin/plugin-conf.d/munin-node line 10. Skipping the rest of the file at /usr/share/perl5/vendor_perl/Munin/Node/Service.pm line 110. client_req.value 726315 cache_miss.value 220489 cache_hitpass.value 17813 cache_hit.value 488013 [root@hetzner2 plugins]#
- confirmed that data is coming in, but under the category "webserver" instead of "varnish". I see no obvious way to change that *shrug*
Thr Mar 01, 2018
- began to prepare for my production migration of the osemain site (www.opensourceecology.org)
- because the domain name is shared by both osemain & the wiki (as opposed to having them on distinct domains or subdomains), the DNS cutover may cause issues with the wiki
- currently, the naked domain name 'opensourceecology.org' points to '78.46.3.178' = hetzner1
- currently, 'www' is a cloud-fronted CNAME to the above naked domain name 'opensourceecology.org'
- I will be deleting the above 'www' CNAME & replacing it with a non-CF'd A record pointing to '138.201.84.243' = hetzner2
- therefore, wiki requests made to 'opensourceecology.org/wiki/...' should be unimpacted. However, anything that links to the wiki at 'www.opensourceecology.org/wiki/...' will be impacted. This is a fairly minor concern that should be fixable with an (nginx?) redirect on hetzner2 for anything '/wiki/...', simply stripping-out the 'www'.
- there may be issues in the other direction, namely that someone wants to get our wordpress site by going to 'opensourceecology.org' but hits our wiki instead. I don't intend to fix this unless we see a ton of 404s or hear an outcry of issues.
- in any case, for good measure, I'll make the wiki read-only during the osemain migration by creating the file at hetzner1:/usr/home/osemain/public_html/w/read-only-message
- I did some research on how to put up a warning for the upcoming maintenance on the wiki as a banner on all pages, and I found this plugin--which is the same plugin that Wikipedia famously uses in their donation campaigns https://www.mediawiki.org/wiki/Extension:CentralNotice
- the above article actually recommended just using Sitenotice for a single wiki https://www.mediawiki.org/wiki/Manual:Interface/Sitenotice
- Sitenotice can be updated using the $wgSiteNotice variable in LocalSettings.php
- tested it on our stating site with success
$wgSiteNotice = '<div style="margin: 10px 0px; padding:12px; color: #9F6000; background-color: #FEEFB3;"><i class="fa fa-warning"></i>NOTICE: This wiki will be temporarily made READ-ONLY during a maintenance window today at 15:00 UTC. Please finish & save any pending changes to avoid data loss.</div>';
- applied the change to the production site at hetzner1. It worked!
- began the production migration of osemain CHG-2018-02-05
- first, I locked the prod wiki by creating this file
osemain@dedi978:~/public_html/w$ cat read-only-message The wiki is currently locked for maintenance. Please check back in a few hours. osemain@dedi978:~/public_html/w$
- I then began the full-system backups on both hetzner 1 & hetzner 2 simultaneously
# hetzner 1 # STEP 0: CREATE BACKUPS source /usr/home/osemain/backups/backup.settings /usr/home/osemain/backups/backup.sh # when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*' # hetzner 2 sudo su - # STEP 0: CREATE BACKUPS # for good measure, trigger a backup of the entire system's database & files: time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log # when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'
- 16:30: ugh, after 1.5 hours, the full backup was still running on hetzner1. I'll proceed anyway with the migration, but try to refrain from any removal of content from hetzner1 until after this backup completion is confirmed
- 16:31: began backup of osemain-specific db & files
- 15:33: osemain-specific backups complete
- confirmed that the system-level backups were done being made on hetzner2, but they were still pending scp to our storage server. eta on the web root file alone was over an hour, so I proceeded
- 15:36: scp'd the osemain backup files from hetzner1 to hetzner2. This completed in <1 min
- 15:38 finished polishing the db.sql dump & replaced the staging db with this fresh copy
- 15:40 finished creating new vhost files
- I got errors when attempting to update the plugins using wp-cli due to the hardened permissions. It's less-than-ideal, but I updated them to grant the apache group write permissions to 'wp-content/themes' & 'wp-content/plugins' as well as 'uploads'
- err, I actually needed to update the permissions in 'wp-content/upgrade'
- 16:04: finished wp-cli updates
- 16:05: destroyed the existing CNAME for 'www' on opensourceecology.org via CloudFlare & re-created it to point to '138.201.84.243' with a 2 min (the min) TTL.
- 16:11 confirmed the domain name change worked
- fixed /etc/nginx/conf.d/www.opensourceecology.org/conf to use 'www' instead of the temp subdomain 'osemain'
- fixed /etc/httpd/conf.d/000-www.opensourceecology.org.conf
- restarted both httpd & nginx
- fixed the wordpress domain in WP_HOME & WP_SITEURL in /var/www/html/www.opensourceecolgy.org/wp-config.php by replacing 'osemain' with 'www'
- got a cert warning from the browser; indeed, 'www' hasn't been added to the cert's SAN yet
- fixed the cert, replacing 'osemain' with 'www'
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- 16:18 successfully loaded the site without
- 16:19 confirmed that both system-wide backups haven't completed yet..
- confirmed that hitting 'opensourceecology.org' doesn't actually take you to the wiki--it still takes you to the old site. After the backups finish, I'll have to move the old wp site's content out of the docroot & install a redirect for queries to the naked root (opensourceecology.org/) to 301 redirect to 'https://www.opensourceecology.org'
- confirmed that the wiki was still accessible at 'http://opensourceecology.org/wiki/'
- confirmed that the wiki was *not* accessible at 'www.opensourceecology.org/wiki/' that now tries to access the wordpress site on hetzner2. I should add a redirect for this--at least for now
- 16:34 finished setting-up the new plugins in the wp dashboard
- when editing a post, I noticed a link pointing to 'blog.opensourceecology.org'. Checking CF, I see that was a CNAME to 'dedi978.your-server.de'. So I deleted this record & added one for 'blog' as a CNAME to 'www'
- 16:47 finished fixing content for knightlab iframes on: /contributors/, /community-true-fans/, & /history-timeline/
- 16:47 confirmed that the dns updated for blog.opensourceecology.org
- attempted to click the 'blog.' link on 'https://www.opensourceecology.org/community-true-fans/', but it resulted in a cert warning of course https://blog.opensourceecology.org/2009/01/towards-1000-global-villages-factor-e-live-distillations-part-1/
- added blog SAN to the cert
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d www.opensourceecology.org -d blog.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- I got an error. I confirmed that let's encrypt supports CNAME, but perhaps I just need to wait longer for propagation https://security.stackexchange.com/questions/134354/why-does-acme-lets-encrypt-require-a-records-not-cname/134389
- 16:56 I confirmed that the system-wide backup on hetzner2 is complete (after 148 minutes), but it's still running on hetzner1
- 17:16 finished updating widgets from "text" to "custom html" on relevant workshops = /cnc-torch-table-workshop/, /eco-tractor_workshop/, /3d-printer-construction-set-workshop-3/
- re-tried the certbot renewal. failed again. actually, I think it's because nginx isn't directing it correctly
- updated the nginx & httpd configs with aliases for 'blog.opensourceecology.org'
- updated the varnish config to use a regex match
- restarted apache, varnish, and nginx
- that worked! the cert now has a SAN for 'blog.opensourceecology.org'
- attempted to click the 'blog.' link on 'https://www.opensourceecology.org/community-true-fans/'; this time it didn't result in a cert error, just a 404 *facepalm* https://blog.opensourceecology.org/2009/01/towards-1000-global-villages-factor-e-live-distillations-part-1/
- got the same on www, *shurg* https://www.opensourceecology.org/2009/01/towards-1000-global-villages-factor-e-live-distillations-part-1/
- 19:03 finished configuring nginx to point wiki requests sent to hetzner2 back to the prod wiki in /etc/nginx/conf.d/www.opensourceecology.org.conf
####################
# REDIRECT TO WIKI #
####################
# this is for backwards-compatibility; before ~ 2018-02, both the wiki and
# this site shared the same domain-name. So, just in case someone sends
# www.opensourceecology.org a query trying to find the wiki, let's send them
# to the right place..
location ~* '^/wiki/' {
# TODO: change this to 'https://wiki.opensourceecology.org' after the wiki
# has been migrated
return 301 http://opensourceecology.org$uri;
}
- 19:11 fixed the facebook iframe on the main page from 'http' to 'https'
- 19:13 the hetzner1 system backup is *still* not finished!
- 19:24 added a redirect from the opensour
# 2018-03-01: osemain was migrated to hetzner2 today; this rule is a catch-all
# to send non-wiki traffic to our hetzner2 server
RewriteCond %{REQUEST_URI} !(/wiki|/w) [NC]
RewriteRule ^(.*)$ https://www.opensourceecology.org/$1 [L,R=301,NC]
- confirmed that correctly requesting on the new www.opensourceecology.org site works https://www.opensourceecology.org/cnc-torch-table-workshop/
- confirmed that correctly requesting on the current wiki works http://opensourceecology.org/wiki/User:Maltfield
- confirmed that incorrectly requesting the wiki from the 'www' works https://www.opensourceecology.org/wiki/User:Maltfield
- confirmed that incorrectly requesting osemain from the naked domain (which is now just the wiki) works http://opensourceecology.org/cnc-torch-table-workshop/
- came back later in the evening & found the upload finished
- moved all of the old osemain wordpress files out of the docroot on on hetzner1:/usr/home/osemain/public_html into hetzner1:/usr/home/osemain/noBackup/deleteMeIn2019/osemain_olddocroot/
- fixed the image on /marcin-jakubowski/ to use 'https://www.opensourceecology.org/...' for Marcin's photo instead of 'http://opensourceecology.org/...'
- confirmed that clicking the 'update' button didn't clear varnish for the page
- confirmed that clicking the "Purge from Varnish" button successfully cleared the single varnish page
- replaced the "Site Logo Image" URL at Appearance -> Theme Options -> Site Logo from 'http://opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png' to 'https://www.opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png'
- first I had to fix a modsec false-positives
- 950001, sqli
- 973336, xss
- 958051, xss
- 973331, xss
- 973330, xss
- first I had to fix a modsec false-positives
- began researching rrdtool-like solution
- rrdtool vs mrtg vs cacti vs zabbix
- I'm leaning to cacti
- began installing cacti
- installed depends
yum install php56w-snmp net-snmp net-snmp-utils cacti
- updated the TODO section of OSE_Server#TODO
Tue Feb 27, 2018
- still no word back from Marcin on the resolution of all outstanding issues on osemain & the wiki
- returned to hardening mediawiki
- $vgVerifyMimeTypes was set to false in a block within LocalSettings.php:
# Workaround for bug in MediaWiki 1.16 # See http://www.mediawiki.org/wiki/Thread:Project:Support_desk/Error:_File_extension_does_not_match_MIME_type $wgVerifyMimeType = false; $wgStrictFileExtensions = false; $wgCheckFileExtensions = false; ##file extensions # Workaround for bug in MediaWiki 1.16 # See http://www.mediawiki.org/wiki/Thread:Project:Support_desk/Error:_File_extension_does_not_match_MIME_type $wgVerifyMimeType = false; $wgStrictFileExtensions = false; $wgCheckFileExtensions = false; $wgFileExtensions = array( 'png', 'gif', 'jpg', 'jpeg', 'ppt', 'pdf', 'psd', 'mp3','xls', 'swf', 'doc', 'odt', 'odc', 'odp', 'ods', 'odg', 'pod', 'mm', 'xls', 'mpp', 'svg', 'dxf', 'stp', 'blend', 'g', 'FCStd', 'dia', 'bz2', 'gz', 'tbz', 'tgz', '7z', 'xz'); # See also: http://www.mediawiki.org/wiki/Manual:$wgMimeTypeBlacklist
- I hardened this, replacing it with
################ # FILE UPLOADS # ################ # MIME $wgVerifyMimeType = true; # EXTENSIONS $wgStrictFileExtensions = true; $wgCheckFileExtensions = true; $wgFileExtensions = array( 'png', 'gif', 'jpg', 'jpeg', 'ppt', 'pdf', 'psd', 'mp3','xls', 'doc', 'odt', 'odc', 'odp', 'ods', 'odg', 'pod', 'mm', 'xls', 'mpp', 'svg', 'dxf', 'stp', 'blend', 'g', 'FCStd', 'dia', 'bz2', 'gz', 'tbz', 'tgz', '7z', 'xz');
- per the mediawiki guide on file uploads, we may still have issues with compressed archives (ie: zip), and require adding them to the wgTrustedMediaFormats array https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads#Configuring_file_types
- reading through all the upload-related variables https://www.mediawiki.org/wiki/Category:Upload_variables
- found the notion of a "Copy Upload" which means a subset of users can upload directly from a URL from a specified set of domains (ie: flickr) it uses the upload_by_url permission (AKA "Sideloading"_ https://www.mediawiki.org/wiki/Manual:$wgAllowCopyUploads
- found there's both an UploadWizard and a Upload dialog. The former is built-into mediawiki and the latter is a popular extension for uploading multiple files at once
- found a cool option for segregating images into dirs by hashing their name (for scaling past FS issues), but apparently we can't enable it after the first image has been uploaded; may need to be revisited if issues are encountered in the future https://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory
- found a good read describing the upload process https://www.mediawiki.org/wiki/Manual:Configuring_file_uploads#Upload_directory
- found another list of upload-related variables https://www.mediawiki.org/wiki/Manual:Configuration_settings#Uploads
- did a deep clean-up of LocalSettings.php
- began researching 2FA for mediawiki
- this guide suggests that the main TOTP extension is OATHAuth https://www.mediawiki.org/wiki/Help:Two-factor_authentication
- added the OATHAuth extension to our meiawiki install
[root@hetzner2 extensions]# date Tue Feb 27 19:11:14 UTC 2018 [root@hetzner2 extensions]# pwd /var/www/html/wiki.opensourceecology.org/htdocs/extensions [root@hetzner2 extensions]# git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/OATHAuth.git Cloning into 'OATHAuth'... remote: Counting objects: 164, done remote: Finding sources: 100% (32/32) remote: Getting sizes: 100% (11/11) remote: Compressing objects: 100% (24018/24018) remote: Total 2210 (delta 13), reused 2200 (delta 12) Receiving objects: 100% (2210/2210), 405.31 KiB | 0 bytes/s, done. Resolving deltas: 100% (1561/1561), done. [root@hetzner2 extensions]#
- that failed as the extension said it required mediawiki >= v 1.31.0...but the latest stable version is 1.30!
==> wiki.opensourceecology.org/error_log <==
[Tue Feb 27 19:19:43.898430 2018] [:error] [pid 21700] [client 127.0.0.1:51702] PHP Fatal error: Uncaught exception 'Exception' with message 'OATHAuth is not compatible with the current MediaWiki core (version 1.30.0), it requires: >= 1.31.0.' in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php:261\nStack trace:\n#0 /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php(148): ExtensionRegistry->readFromQueue(Array)\n#1 /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php(40): ExtensionRegistry->loadFromQueue()\n#2 /var/www/html/wiki.opensourceecology.org/htdocs/includes/WebStart.php(114): require_once('/var/www/html/w...')\n#3 /var/www/html/wiki.opensourceecology.org/htdocs/index.php(40): require('/var/www/html/w...')\n#4 {main}\n thrown in /var/www/html/wiki.opensourceecology.org/htdocs/includes/registration/ExtensionRegistry.php on line 261
- sent an email to the author, Ryan Lane, asking where I can submit a bug report for the extension https://www.mediawiki.org/wiki/User:Ryan_lane
- err, I just noticed that his page lists that extension as "unmaintained" :(
- indeed, 2016-10 was the last update
- actually, that documentation is just out-of-date. The file was updated earlier this month on 2018-02-12 by Jayprakash12345 https://phabricator.wikimedia.org/rEOATd7bdefda31161ad680dd2a88993e3d17734c4bef
- the relevant bug is here https://phabricator.wikimedia.org/T187037
- I OAUTH approved the Phabricator app to access my mediawiki account and was successfully able to submit a reply to the dev that made the relevant breaking change on the relevant bug https://phabricator.wikimedia.org/T187037
- within the hour, Sam Reed (Reedy)--who works for the Wikimedia Foundation--responded stating that the documentation is incorrect, and that I should have instead checked-out the "REL1_30" branch during the clone https://en.wikipedia.org/wiki/User:Reedy
- attempted to download again
[root@hetzner2 extensions]# git clone -b REL1_30 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/OATHAuth.git Cloning into 'OATHAuth'... remote: Counting objects: 164, done remote: Finding sources: 100% (32/32) remote: Getting sizes: 100% (11/11) remote: Compressing objects: 100% (24018/24018) remote: Total 2210 (delta 13), reused 2200 (delta 12) Receiving objects: 100% (2210/2210), 405.31 KiB | 0 bytes/s, done. Resolving deltas: 100% (1561/1561), done. [root@hetzner2 extensions]#
- that fixed the error, but logging-in fails
Database error A database query error has occurred. This may indicate a bug in the software.[WpXJwYmXdCJvWua7xeiBhQAAAAA] 2018-02-27 21:12:33: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- attempted to run `php update.php` from the maintenance dir (as has fixed this previously), but I got a syntax error. Indeed, this was due to my faulty sed find-and-replace of ini_set calls
if (#ini_set( "memory_limit", $newLimit ) === false ) {
- fortunately, that appeared to be a one-time/first-pass issue due my earlier/failed regex arguments when testing the sed command. I removed the comment, then retried the final one that I actually documented, and it worked fine.
- I retried `php update.php, and it ran fine.
- I retried the login, and it also worked fine after the update.
- updated the documentation to include the logic to checkout the appropriate branch when checking out the extensions from git
- removed all git-installed extension dirs & downloaded them all again. verified that they all work. ran update.php, & successfully logged-in again
- went to my user settings & saw a new option to enable 2FA. did it successfully.
- logged-out, attempted to log back in. after entering my username & password, I was sent to another page asking for my TOTP token. I entered it, and I logged in successfully.
- I logged-out, and attempted to log back in again. This time I intentionally entered the wrong password, and it told me this without ever showing me the TOTP field.
- I attempted to login again. This time I used the correct password & an intentionally incorrect TOTP token. I was given an error "Verification failed" on the TOTP "Token" step. I entered the correct token, and I logged in successfully.
- note that I was not asked to type my password again after the totp token was entered incorrectly; that's kinda nice.
- begun researching how to require users (ie: all Administrators!) to use 2FA
- found a list of tasks for this extension https://phabricator.wikimedia.org/tag/mediawiki-extensions-oathauth/
- I set $wgSecureLogin to true in LocalSettings.php per https://phabricator.wikimedia.org/T55197
- found the specific task tracking this ask (forcing users to use 2FA). It was created over a year ago (2016-11) and last updated
Sun Feb 25, 2018
- updated the weekly meeting slides per Marcin's request
- Began investigating (9) Page Information = why the Page Information metadata article's "Display Title" field is blank
- Found relevant documentation on this field
- Discovered at least 1 page where it is *not* blank https://wiki.opensourceecology.org/wiki/CNC_Machine
- I deleted the entire contents of this page, and it still wasn't blank. So it wasn't the contents of the article that filled-in the field
- For comparison, Marcin's biography's article page's "Display title" field is empty https://wiki.opensourceecology.org/wiki/Marcin_Biography
- I tried adding 'DISPLAYTITLE:Marcin Biography' to the head of the article, but it remained blank
- I tried adding 'DISPLAYTITLE:Marcin Biography' to the tail of the article, and it filled-in!
- I removed the above, saved, and it went back to blank so something in the article appears to be blanking it out
- I removed the line 'OrigLang', and the field came back again. That appears to be the culprit.
- So this is the culprit https://wiki.opensourceecology.org/index.php?title=Template:OrigLang
- This was the work of MarkDilley, Venko, & Elifarley
- Confirmed that the spanish version of the article *does* have the filed properly set https://wiki.opensourceecology.org/wiki/Marcin_Biography/es
- I was not able to find an obvious solution to the OrigLang template, though I'm no expert in mediawiki templates
- Sent an email to Marcin describing the issue & asking if it has further implications and if it's a blocker for the migration
- Sent Marcin an email follow-up asking for validation that the (1b) Non-existent File redirects to front page issue has been fixed as well, following the varnish config update
- confirmed that awstats is working as expected for the fef & wiki sites
Sat Feb 24, 2018
- Marcin responded that he got past the 2FA login on osemain, but got a forbidden error when attempting to create new blog posts
- confirmed modsec issue
- whitelisted 973337, xss
- confirmed the fix; emailed Marcin for another validation pass
- Marcin responded to the wiki issues status. I'll list them all for a status update here in my log
- (1a) Can't upload files.
- (1b) Non-existent File redirects to front page = "If I use File:filename.jpg, it takes me to the front page of the wiki."
- (2) Content Missing (Youtube embeds?)
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- And Marcin confirmed that the only issue on this page was in-fact the missing youtube videos
- (3) Youtube Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (4) Issuu Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (5) Scrumy Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (6) Recent Changes issues
- Marcin says there's not many changes, and that's the issue. It is a fork of the other site, so I should confirm if this is really an issue
- (7) Ted Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (8) Vimeo Embeds
- Solved by sed find-and-replace in db.sql file before CREATE DATABASE is issued
- (9) Page information
- Page information for all pages is slightly corrupted, and it never shows number of views - https://wiki.opensourceecology.org/index.php?title=Main_Page&action=info
- The Page Views missing is "won't fix" since this was removed by Mediawiki on 2014-10-20 in their release of mediawiki v1.25
- The "Display Title" field is still blank here
- (10) Popular Pages Missing
- Solved as "won't fix" since this was removed by Mediawiki on 2014-10-20 in their release of mediawiki v1.25
- (11) Page Statistics
- Solved as "won't fix" since this was removed by Mediawiki on 2014-10-20 in their release of mediawiki v1.25
- Therefore, the following wiki issues are outstanding:
- (1a) Can't upload files.
- (1b) Non-existent File redirects to front page = "If I use File:filename.jpg, it takes me to the front page of the wiki."
- (6) Recent Changes issues
- (9) Page information
- Emailed Marcin asking for confirmation of the fix for (1a), which I fixed already via permissions
- Began investigating (1b) again
- Discovered that it's a 301 redirect from our server, but only for the get variable URL. For example:
user@personal:~$ curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload" HTTP/1.1 200 OK Server: nginx Date: Sun, 25 Feb 2018 00:30:28 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive X-Content-Type-Options: nosniff Content-language: en X-UA-Compatible: IE=Edge Link: </images/ose-logo.png?be82f>;rel=preload;as=image X-Frame-Options: DENY Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache, no-store, max-age=0, must-revalidate Pragma: no-cache X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1519518628.2754; expires=Sun, 25-Feb-2018 00:31:28 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly X-Varnish: 1326749 Age: 0 Via: 1.1 varnish-v4 Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" user@personal:~$ curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload&wpDestFile=New.jpg" HTTP/1.1 301 Moved Permanently Server: nginx Date: Sun, 25 Feb 2018 00:30:38 GMT Content-Type: text/html; charset=utf-8 Content-Length: 0 Connection: keep-alive X-Content-Type-Options: nosniff Vary: Accept-Encoding,Cookie Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: private, must-revalidate, max-age=0 Last-Modified: Sun, 25 Feb 2018 00:30:38 GMT Location: https://wiki.opensourceecology.org/wiki/Main_Page X-XSS-Protection: 1; mode=block Set-Cookie: cpPosTime=1519518638.9758; expires=Sun, 25-Feb-2018 00:31:38 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly X-Varnish: 1301183 Age: 0 Via: 1.1 varnish-v4 Strict-Transport-Security: max-age=15552001 Public-Key-Pins: pin-sha256="UbSbHFsFhuCrSv9GNsqnGv4CbaVh5UV5/zzgjLgHh9c="; pin-sha256="YLh1dUR9y6Kja30RrAn7JKnbQG/uEtLMkBgFF2Fuihg="; pin-sha256="C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M="; pin-sha256="Vjs8r4z+80wjNcr1YKepWQboSIRi63WsWXhIMN+eWys="; pin-sha256="lCppFqbkrlJ3EcVFAkeip0+44VaoJUymbnOaEUk7tEU="; pin-sha256="K87oWBWM9UZfyddvDfoxL+8lpNyoUB2ptGtn0fv6G2Q="; pin-sha256="Y9mvm0exBk1JoQ57f9Vm28jKo5lFm/woKcVxrYxu80o="; pin-sha256="EGn6R6CqT4z3ERscrqNl7q7RCzJmDe9uBhS/rnCHU="; pin-sha256="NIdnza073SiyuN1TUa7DDGjOxc1p0nbfOCfbxPWAZGQ="; pin-sha256="fNZ8JI9p2D/C+bsB3LH3rWejY9BGBDeW0JhMOiMfa7A="; pin-sha256="oyD01TTXvpfBro3QSZc1vIlcMjrdLTiL/M9mLCPX+Zo="; pin-sha256="0cRTd+vc1hjNFlHcLgLCHXUeWqn80bNDH/bs9qMTSPo="; pin-sha256="MDhNnV1cmaPdDDONbiVionUHH2QIf2aHJwq/lshMWfA="; pin-sha256="OIZP7FgTBf7hUpWHIA7OaPVO2WrsGzTl9vdOHLPZmJU="; max-age=3600; includeSubDomains; report-uri="http:opensourceecology.org/hpkp-report" user@personal:~$
- I tried the curl again, and found that the second query (with the wpDestFile GET variable appended) caused the *entire* GET variable sequence to be stripped by the time it made it to the mediawiki
# this is when I do `curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload&wpDestFile=New.jpg"` [root@hetzner2 wiki.opensourceecology.org]# tail -f wiki-error.log IP: 127.0.0.1 Start request GET /index.php HTTP HEADERS: X-REAL-IP: 173.234.159.236 X-FORWARDED-PROTO: https X-FORWARDED-PORT: 443 HOST: wiki.opensourceecology.org USER-AGENT: curl/7.38.0 ACCEPT: */* X-FORWARDED-FOR: 173.234.159.236, 127.0.0.1, 127.0.0.1 HASH: #wiki.opensourceecology.org ACCEPT-ENCODING: gzip X-VARNISH: 738663 [caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff ... # this is when I do `curl -I "https://wiki.opensourceecology.org/index.php?title=Special:Upload" [root@hetzner2 wiki.opensourceecology.org]# tail -f wiki-error.log IP: 127.0.0.1 Start request GET /index.php?title=Special:Upload HTTP HEADERS: X-REAL-IP: 173.234.159.236 X-FORWARDED-PROTO: https X-FORWARDED-PORT: 443 HOST: wiki.opensourceecology.org USER-AGENT: curl/7.38.0 ACCEPT: */* X-FORWARDED-FOR: 173.234.159.236, 127.0.0.1, 127.0.0.1 HASH: #wiki.opensourceecology.org ACCEPT-ENCODING: gzip X-VARNISH: 738668 [caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff ...
- so the request mediawiki sees changes form "GET /index.php?title=Special:Upload" to "GET /index.php" when the second variable is added, which is why it just responds with a 301 redirect to the main page.
- confirmed that the correct request does make it to nginx
==> wiki.opensourceecology.org/access.log <== 173.234.159.236 - - [25/Feb/2018:00:39:24 +0000] "HEAD /index.php?title=Special:Upload HTTP/1.1" 200 0 "-" "curl/7.38.0" "-" 173.234.159.236 - - [25/Feb/2018:00:39:30 +0000] "HEAD /index.php?title=Special:Upload&wpDestFile=New.jpg HTTP/1.1" 301 0 "-" "curl/7.38.0" "-"
- confirmed that varnish does see the entire request, but then it appears to strip it!
[root@hetzner2 ~]# varnishlog -q "ReqHeader eq 'X-Forwarded-For: 173.234.159.236'" * << Request >> 1301342 - Begin req 1301341 rxreq - Timestamp Start: 1519519251.915667 0.000000 0.000000 - Timestamp Req: 1519519251.915667 0.000000 0.000000 - ReqStart 127.0.0.1 59862 - ReqMethod HEAD - ReqURL /index.php?title=Special:Upload&wpDestFile=New.jpg - ReqProtocol HTTP/1.0 - ReqHeader X-Real-IP: 173.234.159.236 - ReqHeader X-Forwarded-For: 173.234.159.236 - ReqHeader X-Forwarded-Proto: https - ReqHeader X-Forwarded-Port: 443 - ReqHeader Host: wiki.opensourceecology.org - ReqHeader Connection: close - ReqHeader User-Agent: curl/7.38.0 - ReqHeader Accept: */* - ReqUnset X-Forwarded-For: 173.234.159.236 - ReqHeader X-Forwarded-For: 173.234.159.236, 127.0.0.1 - VCL_call RECV - ReqUnset X-Forwarded-For: 173.234.159.236, 127.0.0.1 - ReqHeader X-Forwarded-For: 173.234.159.236, 127.0.0.1, 127.0.0.1 - ReqHeader X-VC-My-Purge-Key: JOaSAn72IJzrykJp1pfEWaECvUU8KvxZJnxSue3repId3qV8wJOHexjtuhi9r6Wv4FH9y9eFfiMjXX6hvxRrVOEWr2IaBVZMZ7ToEz8nLFdRyjyMkUGMANd6MHOzxiTJ - ReqUnset X-VC-My-Purge-Key: JOaSAn72IJzrykJp1pfEWaECvUU8KvxZJnxSue3repId3qV8wJOHexjtuhi9r6Wv4FH9y9eFfiMjXX6hvxRrVOEWr2IaBVZMZ7ToEz8nLFdRyjyMkUGMANd6MHOzxiTJ - ReqURL /index.php - VCL_return hash - VCL_call HASH - ReqHeader hash: #wiki.opensourceecology.org - VCL_return lookup - Debug "XXXX HIT-FOR-PASS" - HitPass 1301282 - VCL_call PASS - VCL_return fetch - Link bereq 1301343 pass - Timestamp Fetch: 1519519252.049599 0.133932 0.133932 - RespProtocol HTTP/1.0 - RespStatus 301 - RespReason Moved Permanently - RespHeader Date: Sun, 25 Feb 2018 00:40:51 GMT - RespHeader Server: Apache - RespHeader X-Content-Type-Options: nosniff - RespHeader Vary: Accept-Encoding,Cookie - RespHeader Expires: Thu, 01 Jan 1970 00:00:00 GMT - RespHeader Cache-Control: private, must-revalidate, max-age=0 - RespHeader Last-Modified: Sun, 25 Feb 2018 00:40:52 GMT - RespHeader Location: https://wiki.opensourceecology.org/wiki/Main_Page - RespHeader X-XSS-Protection: 1; mode=block - RespHeader Set-Cookie: cpPosTime=1519519252.0366; expires=Sun, 25-Feb-2018 00:41:52 GMT; Max-Age=60; path=/; secure; httponly;HttpOnly - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader X-VC-Req-Host: wiki.opensourceecology.org - RespHeader X-VC-Req-URL: /index.php - RespHeader X-VC-Req-URL-Base: /index.php - RespHeader X-VC-Cacheable: NO:Not cacheable, ttl: 0.000 - RespProtocol HTTP/1.1 - RespHeader X-Varnish: 1301342 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - RespUnset X-VC-Req-Host: wiki.opensourceecology.org - RespUnset X-VC-Req-URL: /index.php - RespUnset X-VC-Req-URL-Base: /index.php - RespHeader X-VC-Cache: MISS - RespUnset X-VC-Cache: MISS - RespUnset X-VC-Cacheable: NO:Not cacheable, ttl: 0.000 - VCL_return deliver - Timestamp Process: 1519519252.049641 0.133974 0.000043 - Debug "RES_MODE 0" - RespHeader Connection: close - Timestamp Resp: 1519519252.049679 0.134012 0.000037 - Debug "XXX REF 1" - ReqAcct 270 0 270 615 0 615 - End
- I checked the varnish config, and it looked like I had used the wordpress varnish config as a place-holder. That must have been stripping the variable in the request somehow. When I reduced the logic to a base config (with vhost support) by copying from the awstats configs, it fixed the issue.
- added php engine off declaration (as was used in mediawiki apache vhost config file) to the seedhome vhost config file
Tue Feb 20, 2018
- began to investigate (10) Popular Pages Missing
- found that the 'includes/specials/SpecialPopularpages.php' file exists on hetzner1 but not the fresh install on hetzner2
# hetzner1 osemain@dedi978:~/public_html/w$ date Tue Feb 20 20:23:00 CET 2018 osemain@dedi978:~/public_html/w$ pwd /usr/home/osemain/public_html/w osemain@dedi978:~/public_html/w$ ls includes/specials | grep -i popular SpecialPopularpages.php osemain@dedi978:~/public_html/w$ # hetzner2 [root@hetzner2 htdocs]# date Tue Feb 20 19:22:45 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/wiki.opensourceecology.org/htdocs [root@hetzner2 htdocs]# ls includes/specials | grep -i popular [root@hetzner2 htdocs]#
- I read that you can choose whether or not to enable counters with 'wgDisableCounter', so I grepped the entire new install dir for this string and found 1 result in includes/Setup.php:
[root@hetzner2 htdocs]# grep -irC 3 'wgDisableCounter' includes/
includes/Setup.php-
includes/Setup.php-// We don't use counters anymore. Left here for extensions still
includes/Setup.php-// expecting this to exist. Should be removed sometime 1.26 or later.
includes/Setup.php:if ( !isset( $wgDisableCounters ) ) {
includes/Setup.php: $wgDisableCounters = true;
includes/Setup.php-}
includes/Setup.php-
includes/Setup.php-if ( $wgMainWANCache === false ) {
[root@hetzner2 htdocs]#
- So it may have been removed. Next question: why?
- Found the merge that removed it from the mediawiki-commits mailing list https://www.mail-archive.com/mediawiki-commits@lists.wikimedia.org/msg193945.html
- Found the RFC that proposed removal of PopularPages https://www.mediawiki.org/wiki/Requests_for_comment/Removing_hit_counters_from_MediaWiki_core
- Some people in there say it's not accurate as it is simply an integer, and cannot provide unique hits. Their solution is to use Google Analytics or Piwik.
- Also found this in the changelog for the mediawiki v1.25 release https://www.mediawiki.org/wiki/MediaWiki_1.25#Hit_counters_removed
- In fact, this would be a useless figure, as mediawiki is only going to increment when varnish hits it. For our setup, this really does need to be on awstats monitoring nginx.
- I updated our awstats config to include our wiki site
[root@hetzner2 awstats]# cat /etc/awstats/awstats.wiki.opensourceecology.org.conf LogFile="/var/log/nginx/wiki.opensourceecology.org/access.log" SiteDomain="wiki.opensourceecology.org" HostAliases="wiki.opensourceecology.org 138.201.84.223" Include "common.conf" [root@hetzner2 awstats]# [root@hetzner2 awstats]# cat /etc/cron.d/awstats_generate_static_files 06 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now 16 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 17 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=seedhome.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 18 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=fef.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ 19 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ 20 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=wiki.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ [root@hetzner2 awstats]#
- emailed marcin saying that (9), (10), & (11) are "won't fix" and that we should use awstats again
- beginning to investigate (1a) Can't upload files.
- tailed the error logs when I attempted to do an upload, and I got this
==> wiki.opensourceecology.org/error_log <== [Tue Feb 20 20:38:29.746565 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.769788 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.770337 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.783048 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.783854 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.798665 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:29.799370 2018] [:error] [pid 18604] [client 127.0.0.1:40818] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [20/Feb/2018:20:38:29 +0000] "GET /api.php?action=query&format=json&formatversion=2&titles=File%3AGi%2Ejpg&prop=imageinfo&iiprop=uploadwarning HTTP/1.0" 200 135 "https://wiki.opensourceecology.org/wiki/Special:Upload" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" ==> wiki.opensourceecology.org/error_log <== [Tue Feb 20 20:38:38.802234 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.826196 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.826646 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.839300 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.840009 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.854396 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.855178 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:38.879773 2018] [:error] [pid 18273] [client 127.0.0.1:40834] PHP Warning: set_time_limit() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 3121, referer: https://wiki.opensourceecology.org/wiki/Special:Upload ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [20/Feb/2018:20:38:38 +0000] "POST /wiki/Special:Upload HTTP/1.0" 429 18407 "https://wiki.opensourceecology.org/wiki/Special:Upload" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" ==> wiki.opensourceecology.org/error_log <== [Tue Feb 20 20:38:49.757596 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.781479 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.781864 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.807448 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.808226 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.853839 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 693, referer: https://wiki.opensourceecology.org/wiki/Special:Upload [Tue Feb 20 20:38:49.854676 2018] [:error] [pid 13256] [client 127.0.0.1:40864] PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/rdbms/database/Database.php on line 705, referer: https://wiki.opensourceecology.org/wiki/Special:Upload ==> wiki.opensourceecology.org/access_log <== 127.0.0.1 - - [20/Feb/2018:20:38:49 +0000] "GET /load.php?debug=false&lang=en&modules=mediawiki.helplink%2CsectionAnchor%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector HTTP/1.1" 200 43047 "https://wiki.opensourceecology.org/wiki/Special:Upload" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
- And the relevant line in the super-verbose 'wiki-error.log'
UploadBase::verifyFile: all clear; passing. [LocalFile] Failed to lock 'Gi.jpg' MediaWiki::preOutputCommit: primary transaction round committed
- that tells me the source of the error is a script named LocalFile
[root@hetzner2 htdocs]# find . | grep -i localfile ./includes/filerepo/file/LocalFile.php ./includes/filerepo/file/OldLocalFile.php ./includes/filerepo/file/UnregisteredLocalFile.php ./tests/phpunit/includes/filerepo/file/LocalFileTest.php [root@hetzner2 htdocs]#
- checked the obvious: permissions
[root@hetzner2 htdocs]# ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images | head -n 2 total 352K d---r-x--- 29 not-apache apache 4.0K Feb 14 03:19 .
- so it doesn't have write permissions to the directory
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
[ -d "${docrootDir_hetzner2}/images" ] || mkdir "${docrootDir_hetzner2}/images"
chown -R apache:apache "${docrootDir_hetzner2}/images"
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
- that's better
[root@hetzner2 htdocs]# ls -lah /var/www/html/wiki.opensourceecology.org/htdocs/images | head -n 2 total 352K drwxrwx--- 29 apache apache 4.0K Feb 14 03:19 .
- I tried the upload again, and it worked fine. That fixes (1a)
- these warnings that our hardened php.ini config is blocking calls to unsafe & disabled functions like ini_set() & putenv() clogging up the logs are annoying. Let's fix it & add it to the migration procedure
find includes/ -type f -exec sed -i 's%^\(\s*\)ini_set\(.*\)%\1#ini_set\2%' '{}' \;
find includes/ -type f -exec sed -i 's%^\(\s*\)putenv\(.*\)%\1#putenv\2%' '{}' \;
- that fixed it
- I documented this on Mediawiki#migrate_site_from_hetzner1_to_hetzner2
- began to investigate (1b)
- what's weird here is that this link totally works:
- https://wiki.opensourceecology.org/index.php?title=Special:Upload
- but as soon as you add the "&wpDestFile=New.jpg" to the URL, it sends me to the Main_Page
- when I tail the wiki-error.log file, I see the very beginning of the message shows it's just getting a "GET /index.php" request. So the problem must exist before mediawiki (apache? varnish? nginx?)
- what's weird here is that this link totally works:
IP: 127.0.0.1 Start request GET /index.php HTTP HEADERS: X-REAL-IP: 209.208.216.133 X-FORWARDED-PROTO: https X-FORWARDED-PORT: 443 HOST: wiki.opensourceecology.org USER-AGENT: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0 ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 ACCEPT-LANGUAGE: en-US,en;q=0.5 REFERER: https://wiki.opensourceecology.org/wiki/Hello UPGRADE-INSECURE-REQUESTS: 1 X-FORWARDED-FOR: 209.208.216.133, 127.0.0.1, 127.0.0.1 ACCEPT-ENCODING: gzip HASH: #wiki.opensourceecology.org X-VARNISH: 947159 [caches] cluster: EmptyBagOStuff, WAN: mediawiki-main-default, stash: db-replicated, message: SqlBagOStuff, session: SqlBagOStuff [caches] LocalisationCache: using store LCStoreDB
- the apache logs show "GET /index.php" as well
127.0.0.1 - - [20/Feb/2018:21:39:50 +0000] "GET /index.php HTTP/1.1" 301 - "https://wiki.opensourceecology.org/wiki/Hello" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" 127.0.0.1 - - [20/Feb/2018:21:39:50 +0000] "GET /wiki/Main_Page HTTP/1.1" 200 71842 "https://wiki.opensourceecology.org/wiki/Hello" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
- Found that I can limit varnishlog output by domain:
varnishlog -q "ReqHeader ~ wiki.opensourceecology.org"
- Better, found that I can limit varnishlog output by the client's (my) ip address
varnishlog -q "X-Forwarded-For ~ 209.208.216.133"
- added the above to the useful commands under our varnish documentation page at Web_server_configuration#Useful_Commands
Mon Feb 19, 2018
- discovered that the bug report I submitted to the Open Office Online Suite to add the functionality of hyperlinks to their Impress Online (Google Docs Presentation replacement) got fixed! https://bugs.documentfoundation.org/show_bug.cgi?id=113361
- sent an email to Marcin, suggesting that we should revisit & submit bug reports, so that this could be a viable solution that meets our needs in a couple years.
- Marcin responded with a list of 11 issues with the Staging wiki
- (1a) Can't upload files.
- (1b) Non-existent File redirects to front page = "If I use File:filename.jpg, it takes me to the front page of the wiki."
- (2) Content Missing (Youtube embeds?)
- Conent missing on https://wiki.opensourceecology.org/wiki/Tractor_User_Manual
- (3) Youtube Embeds
- Videos not embedding - https://wiki.opensourceecology.org/wiki/CEB_Press_Fabrication_videos
- (4) Issuu Embeds
- Issuu not embedding on top - https://wiki.opensourceecology.org/wiki/Civilization_Starter_Kit_DVD_v0.01
- (5) Scrumy Embeds
- https://wiki.opensourceecology.org/wiki/Development_Team_Log . Scrumy doesn't appear to embed anywhere throughout wiki - here's another example - https://wiki.opensourceecology.org/wiki/Tractor_Construction_Set_2017
- (6) Recent Changes issues
- Recent Changes doesn't appear to be displaying properly - https://wiki.opensourceecology.org/index.php?title=Special:RecentChanges&limit=500
- (7) Ted Embeds
- Video embed not working on top of page. https://wiki.opensourceecology.org/wiki/Donate
- (8) Vimeo Embeds
- Video not embedded - https://wiki.opensourceecology.org/wiki/OSEmail
- (9) Page information
- Page information for all pages is slightly corrupted, and it never shows number of views - https://wiki.opensourceecology.org/index.php?title=Main_Page&action=info
- (10) Popular Pages Missing
- Popular Pages is missing from Data and tools section of https://wiki.opensourceecology.org/wiki/Special:SpecialPages
- (11) Page Statistics
- https://wiki.opensourceecology.org/wiki/Special:Statistics doesn't show page view statistics.
- I iterated through the list & attempted to reproduce the issues
- (1a) Can't upload files.
- I was able to reproduce this. Attempting to upload an issue resulted in the error message: 'Action Failed. Could not open lock file for "mwstore://local-backend/local-public/7/78/Image.jpg".'
- (1b) Non-existent File redirects to front page
- Marcin's test page has 2 links to non-existant files. Both redirect to the main page https://wiki.opensourceecology.org/wiki/Hello
- (2) Content Missing (Youtube Embeds?)
- I confirmed that the youtube video embed was missing. Replacing 'https://www.youtube.com/embed//' with 'https://www.youtube.com/embed/' fixed it.
- (3) Youtube Embeds
- Same as above. Replacing 'https://www.youtube.com/embed//' with 'https://www.youtube.com/embed/' fixed one of the videos.
- (4) Issuu Embeds
- confirmed. got a mosec issue when attempting to update the page, replacing the string 'https://static.issuu.com/webembed/' with 'https://static.issuu.com/webembed/'
- whitelisted 950018, generic attack = "universal pdf xss url detected"
- confirmed the fix of the string replacement above fixed this issue
- (5) Scrumy Embeds
- Confirmed that the embed from 'scrumy.com' is not working on the staging wiki
- attempted to make an edit, but got a modesec issue id = 950001
- whitelisted 950001, sqli attack
- Fixed the issue by replacing the string 'https://scrumy.com/' with 'https://scrumy.com/'
- (6) Recent Changes issues
- Marcin's description was vague, but I do see discrepancies between the 2x pages:
- It looks like the difference is that the diff & hist links moved from the left of the article name to the right of the article name. I should confirm with Marcin if this is the only issue && if it's actually OK
- (7) Ted Embeds
- confirmed that the ted talk embed is not loading
- attempted to fix it, but got a modesec forbidden on 958008
- whitelisted 958008, xss
- whitelisted 973329, xss
- fixed with replacing the string 'https://embed.ted.com/' with 'https://embed.ted.com/'
- also discovered that the flattr embed is broken on the Donate page. This is a consequence of us removing the Flattr Mediawiki plugin we were using, which was no longer maintained. I emailed Marcin and asked him to remove all content dependent on the Flattr plugin from the prod wiki.
- (8) Vimeo Embeds
- confirmed that the vimeo video embed is not working
- fixed by replacing the string 'https://player.vimeo.com/' with 'https://player.vimeo.com/'
- (9) Page information
- Confirmed that the Information page does not show the number of views, but it does on the prod site. I'm not sure if this was just removed in newer versions of Mediawiki or what. I'll have to investigate this further..
- Not sure what else is "corrupted" per Marcin's email. I'll have to ask him to clarify.
- (10) Popular Pages Missing
- Confirmed that the Popular Pages link is no longer present, but it is on the prod site. Not sure why. I'll have to investigate this further...
- (11) Page Statistics
- Confirmed that the "View statistics" that's present in the prod wiki is missing. May be related to (9). Further investigation is necessary...
- (1a) Can't upload files.
- Attempted to fix (2), (4), (5), and (8) using a simple string replacement using sed
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
pushd "${backupDir_hetzner2}/current"
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql
# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql
# fix youtube embeds
fromString='https://www.youtube.com/embed/'
toString='https://www.youtube.com/embed/'
sed -i "s^$fromString^$toString^g" db.sql
# fix issuu embeds
fromString='https://static.issuu.com/webembed/'
toString='https://static.issuu.com/webembed/'
sed -i "s^$fromString^$toString^g" db.sql
# fix scrumy embeds
fromString='https://scrumy.com/'
toString='https://scrumy.com/'
sed -i "s^$fromString^$toString^g" db.sql
# fix ted embeds
fromString='https://embed.ted.com/'
toString='https://embed.ted.com/'
sed -i "s^$fromString^$toString^g" db.sql
# fix vimeo embeds
fromString='https://player.vimeo.com/'
toString='https://player.vimeo.com/'
sed -i "s^$fromString^$toString^g" db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT, INSERT, UPDATE, DELETE ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
- unfortunately, that broke the wiki, resulting in an error
[WotzE6X9Jz1lrHTtbDhlLgAAAAI] 2018-02-20 01:00:03: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- running the update script fixed the error above!
pushd "${docrootDir_hetzner2}/maintenance"
php update.php
- bonus: this confirms that updates are not an issue using the hardened SELECT, INSERT, UPDATE, DELETE permissions for the db user
- emailed Marcin asking him to validate that I fixed (2), (4), (5), (7), & (8) above
Sun Feb 18, 2018
- documented a quick-start guide for 2FA http://opensourceecology.org/wiki/2FA#Quick_Start_Guide
- emailed Marcin the above guide/documentation
Thr Feb 15, 2018
- began following the mediawiki hardening guide https://www.mediawiki.org/wiki/Manual:Security
- confirmed that our permissions are ideal, and I updated the documentation with this Mediawiki#Proper_File.2FDirectory_Ownership_.26_Permissions
- confirmed that we're denying the only dir owned by the the web server's user (/images/, per necessity of requiring write access) cannot execute php scripts per the vhost config
[root@hetzner2 conf.d]# date Thu Feb 15 15:24:55 UTC 2018 [root@hetzner2 conf.d]# pwd /etc/httpd/conf.d [root@hetzner2 conf.d]# cat 00-wiki.opensourceecology.org.conf ... # don't execute any php files inside the uploads directory <LocationMatch "/images/"> php_flag engine off </LocationMatch> <LocationMatch "/images/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> Order Deny,Allow Deny from All </LocationMatch> ... [root@hetzner2 conf.d]#
- yes, we're using tls. yes, we use hsts (not to mention hpkp). ssllabs gives us an A+ on our httpsv setup https://www.ssllabs.com/ssltest/analyze.html?d=wiki.opensourceecology.org
- our php.ini config doesn't mention anything about register_globals. I confirmed that it must be off then, since we're running php v 5.6.31, and it's been the default to be off since v4.2.0 https://secure.php.net/manual/en/ini.core.php#ini.register-globals
- allow_url_fopen is already off
- session.use_trans_sid is already off
- the mysql config already has networking disabled (and iptables won't allow it too)
- changed the mediawiki article on our wiki to only grant "SELECT, INSERT, UPDATE and DELETE" permissions to the osewiki_user user per the hardening guide Mediawiki#migrate_site_from_hetzner1_to_hetzner2
- note this may break mediawiki updates from the maintenance scripts, but we can create a new user with all permissions, if necessary, and pass those directly into the update scripts using --dbuser & --dbpass as described here mediawikiwiki:Manual:Maintenance_scripts#Configuration
- added a variable '$wgSecretKey' to the LocalSettings.php file per mediawikwiki:Manual:$wgSecretKey
- I just used keepassx to give me a [a-zA-Z0-9]{128} string.
- the LocalSettings.php file is already located outside the docroot.
- updated the debugging options in LocalSettings.php to write to a log file outside the docroot
[root@hetzner2 ~]# touch /var/www/html/wiki.opensourceecology.org/wiki-error.log [root@hetzner2 ~]# chown apache:apache /var/log/httpd/wiki.opensourceecology.org/wiki-error.log [root@hetzner2 ~]# ls -lah /var/www/html/wiki.opensourceecology.org/wiki-error.log -rw-r--r-- 1 apache apache 0 Feb 15 16:46 /var/www/html/wiki.opensourceecology.org/wiki-error.log [root@hetzner2 ~]#
Wed Feb 14, 2018
- updated mediawiki migration process from findings yesterday Mediawiki#migrate_site_from_hetzner1_to_hetzner2
- Marcin found that clicking a link to a non-existing file on a wiki incorrectly redirects to the main page. it should link to the page to upload the missing file
- note that the following page *does* load
Tue Feb 13, 2018
- no response from Marcin yet about osemain validation
- confirmed that the awstats sites for opensourceecology.org was still working & got data for 2 consecutive days
- the icons were missing for ose awstats but not obi, so I copied them in-place
[maltfield@hetzner2 html]$ sudo cp -r awstats.openbuildinginstitute.org/htdocs/awstatsicons awstats.opensourceecology.org/htdocs/ [sudo] password for maltfield: [maltfield@hetzner2 html]$
- attempted to modify the wiki install to come straight from the tarball. Unfortunately, we can't do the git solution, since it doesn't include the vendor dir & we can't use Composer without destroying our php.ini security
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# cleanup current dir, making backups of old contents
pushd "${backupDir_hetzner2}"
mkdir "old/`date +%Y%m%d`"
mv current/* "old/`date +%Y%m%d`"
mv "${docrootDir_hetzner2}" "old/`date +%Y%m%d`"
# get backups from hetzner1
pushd current
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/
# download mediawiki core source code
wget https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz
tar -xzvf mediawiki-1.30.0.tar.gz
mkdir "${docrootDir_hetzner2}"
rsync -av --progress mediawiki-1.30.0/ "${docrootDir_hetzner2}/"
# copy-in our images from backups
rsync -av --progress "usr/www/users/osemain/w/images/" "${docrootDir_hetzner2}/images/"
# and move the lone image sticking in root into the images directory
rsync -av --progress "usr/www/users/osemain/w/ose-logo.png" "${docrootDir_hetzner2}/images/"
# create LocalSettings.php that just requires the file from outside the docroot
# write multi-line to file for documentation copy & paste
cat << EOF > "${docrootDir_hetzner2}/LocalSettings.php"
<?php
# including separate file that contains the database password so that it is not stored within the document root.
# For more info see:
# * https://www.mediawiki.org/wiki/Manual:Security
# * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening
\$docRoot = dirname( FILE );
require_once "\$docRoot/../LocalSettings.php";
?>
EOF
# extensions
pushd "${docrootDir_hetzner2}/extensions"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd
# skins
pushd "${docrootDir_hetzner2}/skins"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd
# set permissions
chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0040 "${vhostDir_hetzner2}/LocalSettings.php"
[ -d "${docrootDir_hetzner2}/images" ] || mkdir "${docrootDir_hetzner2}/images"
chown -R apache:apache "${docrootDir_hetzner2}/images"
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
# attempt to update
pushd ${docrootDir_hetzner2}/maintenance
php update.php
popd
- mediawiki dumbly appears to hard-code "/tmp" as the system's temporary directory in TempFSFile.php
[Wed Feb 14 01:47:56.364862 2018] [:error] [pid 3152] [client 127.0.0.1:54708] PHP Warning: is_dir(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fsfile/TempFSFile.php on line 90
- the above warning disappears when I tell it to just check the php config--which it really should be doing on its own. I do this by setting the following in LocalSettings.php
# tell mediaiwki to actually ask php what the tmp dir is, rather than assuming # that "/tmp" is the system's temporary directory $wgTmpDirectory = sys_get_temp_dir();
- update.php execution was successful
[root@hetzner2 maintenance]# php update.php ... Done in 0.5 s. [root@hetzner2 maintenance]#
- confirmed that I can login
- attempted to make an edit, but got a modsec forbidden
- 981318, sqli
- confirmed a successful edit!
- the site still looks like shit, probably because the CSS is being 403'd
- https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=site.styles&only=styles&skin=vector
- https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=mediawiki.legacy.commonPrint,shared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cskins.vector.styles&only=styles&skin=vector
- https://wiki.opensourceecology.org/load.php?debug=false&lang=en&modules=jquery.accessKeyLabel,byteLength,checkboxShiftClick,client,getAttrs,highlightText,mw-jump,suggestions,tabIndex,throttle-debounce%7Cmediawiki.RegExp,Title,api,cldr,jqueryMsg,language,notify,searchSuggest,storage,user,util%7Cmediawiki.api.user,watch%7Cmediawiki.language.data,init%7Cmediawiki.libs.pluralruleparser%7Cmediawiki.page.ready,startup%7Cmediawiki.page.watch.ajax%7Csite%7Cskins.vector.js%7Cuser.defaults&skin=vector&version=19n53uv
- confirmed that the page *does* load without the GET vars https://wiki.opensourceecology.org/load.php
- reduced the issue to just the 'modules' GET var, as the rest load fine https://wiki.opensourceecology.org/load.php?debug=false&lang=en&only=styles&skin=vector
- the manual on this script says it could be htaccess https://www.mediawiki.org/wiki/Manual:Load.php
- indeed, the default install came with an htaccess file:
# Protect against bug 28235
<IfModule rewrite_module>
RewriteEngine On
RewriteCond %{QUERY_STRING} \.[^\\/:*?\x22<>|%]+(#|\?|$) [nocase]
RewriteRule . - [forbidden]
</IfModule>
- when I commented-out the above, it loaded fine.
- The bug it mentions (28235) is a XSS vuln in IE 6 https://phabricator.wikimedia.org/T30235
- so it appears that the .htaccess file is triggering a 403 because the query includes a GET variable with a dot (.) in it. Specifically, it's the "modules=site.styles" bit.
- Well, this issue only gives XSS risks to IE6 users. Honestly, I think the best thing to do is to ban IE6 and delete all the .htaccess files with this block
- added this block to the top of LocalSettings.php
# the .htaccess file that ships with mediawiki to prevent XSS attacks on IE6
# clients breaks our CSS loading. OSE's solution = ban IE6 and remove the
# offending .htaccess files
if(strpos($_SERVER['HTTP_USER_AGENT'], 'MSIE 6.') !== false) {
die( "Sorry, you must update your browser to use this site." );
}
- deleted the contents of the htdocs/.htaccess file (the entire file was just this bugfix for 28235
- confirmed that the site css looks good!
- the main page load still triggers a modsec alert
- 981260, sqli
- 950911, generic attack (response splitting?)
- 973302, xss
- 973324, xss
- 973317, xss
- 981255, sqli
- 958057, xss
- 958056, xss
- 973327, xss
- strangely, the sys_get_temp_dir() is wrong! Is this a bug in php?
[root@hetzner2 httpd]# grep -i 'upload_tmp_dir' /etc/php.ini upload_tmp_dir = "/var/lib/php/tmp_upload" [root@hetzner2 httpd]# echo "<?=sys_get_temp_dir();?>" | php /tmp [root@hetzner2 httpd]# echo "<?=php_ini_loaded_file();?>" | php /etc/php.ini [root@hetzner2 httpd]#
- anyway, I just manually set `$wgTmpDirectory = "/var/lib/php/tmp_upload"` in LocalSettings.php
- that fixed all the thumbnail creation issues.
- sent an email to Marcin & Catarina asking for validation
Mon Feb 12, 2018
- updated CHG-2018-02-05 with last week's fixes
- Continuing to fix issues on osemain
- /marcin-jakubowski/
- Marcin pointed out that his headshot is missing on his about page of the staging site
- The offensive html points to http://opensourceecology.org/wp-content/uploads/2014/02/MarcinHeadshot-300x300.png
<div class="center"><div class=" image-box aligncenter clearfix"><a href="#" title="Marcin Jakubowski"><img class="photo_frame rounded-img" src="http://opensourceecology.org/wp-content/uploads/2014/02/MarcinHeadshot-300x300.png" alt="Marcin Jakubowski"></a></div>
- I would expect this to be a mixed content error (http over https), but the firefox developer console doesn't mention this image specificially (though it does mention an image from creativecommons.org, licensebuttons.net, and our OSE_yellow-copy2.png logo.
- Anyway, when I replace the src attribute's 'http:' with 'https:osemain.', it works. Because of the name change, I'll just have to address this post-migration. I'll probably end-up doing a database-wide sed for 'http://opensourceecology.org/' to 'https://www.opensourceecology.org/
- was unable to reproduce Marcin's issue clicking the workshop link
- sent marcin an email following-up on his edit issues
- confirmed that the awstats issues are fixed for obi
- added awstats config files for awstats.opensourceecology.org
[root@hetzner2 awstats.opensourceecology.org]# ls -lah /etc/awstats | grep -i opensourceecology -rw-r--r-- 1 root root 175 Feb 12 21:37 awstats.fef.opensourceecology.org.conf -rw-r--r-- 1 root root 175 Feb 12 21:31 awstats.www.opensourceecology.org.conf [root@hetzner2 awstats.opensourceecology.org]#
- added cron jobs for ose sites:
[root@hetzner2 awstats.opensourceecology.org]# cat /etc/cron.d/awstats_generate_static_files 06 * * * * root /bin/nice /usr/share/awstats/tools/awstats_updateall.pl -configdir=/etc/awstats/ now 16 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 17 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=seedhome.openbuildinginstitute.org -dir=/var/www/html/awstats.openbuildinginstitute.org/htdocs/ 18 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=fef.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ 19 * * * * root /bin/nice /usr/share/awstats/tools/awstats_buildstaticpages.pl -config=www.opensourceecology.org -dir=/var/www/html/awstats.opensourceecology.org/htdocs/ [root@hetzner2 awstats.opensourceecology.org]#
- fixed permissions on the .htpasswd file next to the docroot (one directory above the docroot)
[root@hetzner2 awstats.opensourceecology.org]# date Mon Feb 12 22:10:47 UTC 2018 [root@hetzner2 awstats.opensourceecology.org]# pwd /var/www/html/awstats.opensourceecology.org [root@hetzner2 awstats.opensourceecology.org]# ls -lah total 16K drwxr-xr-x 3 root root 4.0K Feb 9 21:27 . drwxr-xr-x 16 root root 4.0K Feb 9 20:33 .. drwxr-xr-x 3 root root 4.0K Feb 12 21:42 htdocs -r-------- 1 apache apache 138 Feb 9 21:27 .htpasswd [root@hetzner2 awstats.opensourceecology.org]#
- that really shouldn't be owned by apache (it should owned by a group with apache in it with read-only permissions), but I'll fix that later
- began to test fixes to the docroot permissions such that they're not owned by the apache user, but the apache user has read access
- first, I created the new docroot default file/dir owner: not-apache
[root@hetzner2 awstats.opensourceecology.org]# useradd --home-dir '/dev/null' --shell '/sbin/nologin' not-apache [root@hetzner2 awstats.opensourceecology.org]# tail -n1 /etc/passwd not-apache:x:1012:1013::/dev/null:/sbin/nologin [root@hetzner2 awstats.opensourceecology.org]#
- next, I added the apache user to the apache-admins group, so that it can have read-only permissions to the password-containing config files
- tested the permission files updates on seedhome
vhostDir="/var/www/html/seedhome.openbuildinginstitute.org"
wpDocroot="${vhostDir}/htdocs"
chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0060 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0070 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"
- after the above, I still had permission issues; that was fixed by restarting apache (which was necessary following the group membership, I guess)
- I tested logging-in & an edit. it worked.
- attempted to upload an image, but it failed
Unable to create directory wp-content/uploads/2017/12. Is its parent directory writable by the server?
- for one, the 'apache' user wasn't in the apache group. I fixed this:
[root@hetzner2 htdocs]# gpasswd -a apache apache Adding user apache to group apache [root@hetzner2 htdocs]# service ^C [root@hetzner2 htdocs]# httpd -t Syntax OK [root@hetzner2 htdocs]# service httpd restart Redirecting to /bin/systemctl restart httpd.service [root@hetzner2 htdocs]#
- still having issues. I guess to create new files/directories, the directory must be owned by the apache user, not just a group with write permission that the apache user is in (that works for read only)
- also, the above commands wouldn't work as a fresh install doesn't necessarily already have the ;uploads' directory, so I should add a mkdir. Updating the process:
vhostDir="/var/www/html/seedhome.openbuildinginstitute.org"
wpDocroot="${vhostDir}/htdocs"
chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"
[ -d "${wpDocroot}/wp-content/uploads" ] || mkdir "${wpDocroot}/wp-content/uploads"
chown -R apache:apache "${wpDocroot}/wp-content/uploads"
find "${wpDocroot}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content/uploads" -type d -exec chmod 0770 {} \;
- that worked. updated the documentation Wordpress#Proper_File.2FDirectory_Ownership_.26_Permissions
- updated the apache vhost to not execute any php files from within the 'uploads' directory
[root@hetzner2 httpd]# grep -C 3 'uploads' conf.d/00-seedhome.openbuildinginstitute.org.conf </IfModule> </LocationMatch> # don't execute any php files inside the uploads directory <LocationMatch "/wp-content/uploads/.*(?i)\.(cgi|shtml|php3?|phps|phtml)$"> Order Deny,Allow Deny from All </LocationMatch> [root@hetzner2 httpd]#
Thr Feb 09, 2018
- did a theme update. The default themes updated, but the active theme (enigmatic) had no updates (or, at least, it's not available via the api)
[root@hetzner2 www.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme list
...
+------------------------+----------+-----------+---------+
| name | status | update | version |
+------------------------+----------+-----------+---------+
| enigmatic.20170714.bak | inactive | none | 3.1 |
| enigmatic | active | none | 3.5 |
| twentyeleven | inactive | none | 2.7 |
| twentyfifteen | inactive | none | 1.9 |
| twentyfourteen | inactive | available | 1.0 |
| twentyseventeen | inactive | none | 1.4 |
| twentysixteen | inactive | none | 1.4 |
| twentyten | inactive | none | 2.4 |
| twentythirteen | inactive | available | 1.1 |
| twentytwelve | inactive | available | 1.3 |
+------------------------+----------+-----------+---------+
[root@hetzner2 www.opensourceecology.org]# sudo -u wp -i wp --path=${docrootDir_hetzner2} theme update --all
...
Downloading update from https://downloads.wordpress.org/theme/twentyfourteen.2.1.zip...
Using cached file '/home/wp/.wp-cli/cache/theme/twentyfourteen-2.1.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentythirteen.2.3.zip...
Using cached file '/home/wp/.wp-cli/cache/theme/twentythirteen-2.3.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
Downloading update from https://downloads.wordpress.org/theme/twentytwelve.2.4.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the theme...
Theme updated successfully.
+----------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------+-------------+-------------+---------+
| twentyfourteen | 1.0 | 2.1 | Updated |
| twentythirteen | 1.1 | 2.3 | Updated |
| twentytwelve | 1.3 | 2.4 | Updated |
+----------------+-------------+-------------+---------+
Success: Updated 3 of 3 themes.
[root@hetzner2 www.opensourceecology.org]#
- awstats for obi was going to '/var/log/nginx/access.log' instead of the vhost-specific '/var/log/nginx/www.openbuildinginstitute.org/access.log' file
- the permissions were fine; a reload fixed it (note the file size changed from 0 to 940 after the reload & I queried the site again
[root@hetzner2 ~]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/ total 19M drw-r--r-- 2 nginx nginx 4.0K Feb 9 18:20 . drwx------ 9 nginx nginx 4.0K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 0 Feb 9 04:47 access.log -rw-r--r-- 1 nginx nginx 377 Dec 24 01:30 access.log.1.gz -rw-r--r-- 1 nginx nginx 1.9M Dec 18 03:20 access.log-20171210.gz -rw-r--r-- 1 nginx nginx 1.3M Dec 24 01:20 access.log-20171219.gz -rw-r--r-- 1 nginx nginx 2.6M Jan 3 22:03 access.log-20171225.gz -rw-r--r-- 1 nginx nginx 239K Jan 4 21:36 access.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.2M Jan 18 14:43 access.log-20180105.gz -rw-r--r-- 1 nginx nginx 193K Jan 19 13:03 access.log-20180119.gz -rw-r--r-- 1 nginx nginx 288K Jan 20 20:20 access.log-20180120.gz -rw-r--r-- 1 nginx nginx 4.2M Feb 4 20:19 access.log-20180121.gz -rw-r--r-- 1 nginx nginx 1.7M Feb 9 02:08 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 3.3M Feb 9 18:19 access.log-20180209 -rw-r--r-- 1 nginx nginx 224 Dec 24 01:29 access.log.2.gz -rw-r--r-- 1 nginx nginx 510 Dec 24 01:21 access.log.3.gz -rw-r--r-- 1 nginx nginx 0 Feb 6 06:01 error.log -rw-r--r-- 1 nginx nginx 1.1K Nov 30 17:06 error.log-20171201.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 8 13:12 error.log-20171206.gz -rw-r--r-- 1 nginx nginx 1.8K Dec 12 23:31 error.log-20171210.gz -rw-r--r-- 1 nginx nginx 401 Dec 19 13:45 error.log-20171219.gz -rw-r--r-- 1 nginx nginx 388 Dec 24 05:48 error.log-20171224.gz -rw-r--r-- 1 nginx nginx 3.3K Jan 1 09:19 error.log-20171229.gz -rw-r--r-- 1 nginx nginx 359 Jan 3 23:05 error.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.8K Jan 18 13:55 error.log-20180105.gz -rw-r--r-- 1 nginx nginx 888 Feb 2 10:27 error.log-20180124.gz -rw-r--r-- 1 nginx nginx 319 Feb 5 17:04 error.log-20180206 [root@hetzner2 ~]# service nginx reload Redirecting to /bin/systemctl reload nginx.service [root@hetzner2 ~]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/ total 19M drw-r--r-- 2 nginx nginx 4.0K Feb 9 18:20 . drwx------ 9 nginx nginx 4.0K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 940 Feb 9 18:20 access.log -rw-r--r-- 1 nginx nginx 377 Dec 24 01:30 access.log.1.gz -rw-r--r-- 1 nginx nginx 1.9M Dec 18 03:20 access.log-20171210.gz -rw-r--r-- 1 nginx nginx 1.3M Dec 24 01:20 access.log-20171219.gz -rw-r--r-- 1 nginx nginx 2.6M Jan 3 22:03 access.log-20171225.gz -rw-r--r-- 1 nginx nginx 239K Jan 4 21:36 access.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.2M Jan 18 14:43 access.log-20180105.gz -rw-r--r-- 1 nginx nginx 193K Jan 19 13:03 access.log-20180119.gz -rw-r--r-- 1 nginx nginx 288K Jan 20 20:20 access.log-20180120.gz -rw-r--r-- 1 nginx nginx 4.2M Feb 4 20:19 access.log-20180121.gz -rw-r--r-- 1 nginx nginx 1.7M Feb 9 02:08 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 3.3M Feb 9 18:20 access.log-20180209 -rw-r--r-- 1 nginx nginx 224 Dec 24 01:29 access.log.2.gz -rw-r--r-- 1 nginx nginx 510 Dec 24 01:21 access.log.3.gz -rw-r--r-- 1 nginx nginx 0 Feb 6 06:01 error.log -rw-r--r-- 1 nginx nginx 1.1K Nov 30 17:06 error.log-20171201.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 8 13:12 error.log-20171206.gz -rw-r--r-- 1 nginx nginx 1.8K Dec 12 23:31 error.log-20171210.gz -rw-r--r-- 1 nginx nginx 401 Dec 19 13:45 error.log-20171219.gz -rw-r--r-- 1 nginx nginx 388 Dec 24 05:48 error.log-20171224.gz -rw-r--r-- 1 nginx nginx 3.3K Jan 1 09:19 error.log-20171229.gz -rw-r--r-- 1 nginx nginx 359 Jan 3 23:05 error.log-20180104.gz -rw-r--r-- 1 nginx nginx 3.8K Jan 18 13:55 error.log-20180105.gz -rw-r--r-- 1 nginx nginx 888 Feb 2 10:27 error.log-20180124.gz -rw-r--r-- 1 nginx nginx 319 Feb 5 17:04 error.log-20180206 [root@hetzner2 ~]#
- the reload *should* be happening already by the logroate
[root@hetzner2 ~]# cat /etc/logrotate.d/nginx
/var/log/nginx/*log /var/log/nginx/*/*log {
create 0644 nginx nginx
daily
rotate 10
missingok
notifempty
compress
sharedscripts
prerotate
/bin/nice /usr/share/awstats/tools/awstats_updateall.pl now
endscript
postrotate
/bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true
endscript
}
[root@hetzner2 ~]#
- I simulated the rotate & kill, and saw the same undesired behaviour
[root@hetzner2 www.openbuildinginstitute.org]# mv access.log access.log.test [root@hetzner2 www.openbuildinginstitute.org]# touch access.log [root@hetzner2 www.openbuildinginstitute.org]# chown nginx:nginx access.log [root@hetzner2 www.openbuildinginstitute.org]# chmod 0644 access.log [root@hetzner2 www.openbuildinginstitute.org]# /bin/kill -USR1 `cat /run/nginx.pid 2>/dev/null` 2>/dev/null || true [root@hetzner2 nginx]# tail -f /var/log/nginx/access.log /var/log/nginx/www.openbuildinginstitute.org/access.log /var/log/nginx/error.log /var/log/nginx/www.openbuildinginstitute.org/error.log ... ==> /var/log/nginx/access.log <== 98.242.98.106 - - [09/Feb/2018:18:30:05 +0000] "GET /wp-content/uploads/2016/02/kitchen_1.jpg HTTP/1.1" 301 178 "https://www.openbuildinginstitute.org/buildings/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-"
- interseting, when I did the kill again, this log popped-up
[root@hetzner2 nginx]# tail -f /var/log/nginx/access.log /var/log/nginx/www.openbuildinginstitute.org/access.log /var/log/nginx/error.log /var/log/nginx/www.openbuildinginstitute.org/error.log ... ==> /var/log/nginx/error.log <== 2018/02/09 18:31:08 [emerg] 26860#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26861#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26866#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26862#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26865#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26859#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26860#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26861#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26866#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26862#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26865#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26859#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26864#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26864#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26863#0: open() "/var/log/nginx/www.openbuildinginstitute.org/access.log" failed (13: Permission denied) 2018/02/09 18:31:08 [emerg] 26863#0: open() "/var/log/nginx/www.openbuildinginstitute.org/error.log" failed (13: Permission denied)
- but the permissions look fine
[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/www.openbuildinginstitute.org/access.log -rw-r--r-- 1 nginx nginx 0 Feb 9 18:26 /var/log/nginx/www.openbuildinginstitute.org/access.log
- ah, found the issue thanks to this https://stackoverflow.com/questions/37245926/nginx-write-logs-to-the-old-file-after-running-logrotate
[root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/ total 9.0M drwx------ 9 nginx nginx 4.0K Feb 9 04:47 . drwxr-xr-x 13 root root 12K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 2.2M Feb 9 18:39 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 443K Feb 8 07:21 access.log-20180208.gz -rw-r--r-- 1 nginx nginx 4.0M Feb 9 04:46 access.log-20180209 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 249K Feb 9 18:37 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 13K Feb 8 07:21 error.log-20180208.gz -rw-r--r-- 1 nginx nginx 379K Feb 9 04:46 error.log-20180209 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 fef.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 forum.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 oswh.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 seedhome.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 wiki.opensourceecology.org drw-r--r-- 2 nginx nginx 4.0K Feb 9 18:26 www.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 www.opensourceecology.org [root@hetzner2 www.openbuildinginstitute.org]#
- if you look at the above output of the /var/log/nginx contents, it's obvious really. One dir is 0444 while the rest are 0755. We need that containing dir to have execute permissions!
[root@hetzner2 www.openbuildinginstitute.org]# chmod 0755 /var/log/nginx/www.openbuildinginstitute.org [root@hetzner2 www.openbuildinginstitute.org]# ls -lah /var/log/nginx/ total 9.0M drwx------ 9 nginx nginx 4.0K Feb 9 04:47 . drwxr-xr-x 13 root root 12K Feb 9 04:47 .. -rw-r--r-- 1 nginx nginx 2.2M Feb 9 18:41 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 443K Feb 8 07:21 access.log-20180208.gz -rw-r--r-- 1 nginx nginx 4.0M Feb 9 04:46 access.log-20180209 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 250K Feb 9 18:40 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 13K Feb 8 07:21 error.log-20180208.gz -rw-r--r-- 1 nginx nginx 379K Feb 9 04:46 error.log-20180209 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 fef.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 forum.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 oswh.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 seedhome.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 wiki.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 18:26 www.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 9 04:47 www.opensourceecology.org [root@hetzner2 www.openbuildinginstitute.org]#
- that appears to have fixed it!
- I updated the documentation for creating new wordpress vhosts to use these permissions at the get-go Wordpress#Create_New_Wordpress_Vhost
- began investigating how the hell awstats is accessible on port 443 and, more importantly, how certbot is able to refresh its certificate
- I found the logs of attempting to load 'https://awstats.openbuildinginstitute.org/fuck' in '/var/log/nginx/seedhome.openbuildinginstitute.org/access.log'
[root@hetzner2 nginx]# date Fri Feb 9 19:08:10 UTC 2018 [root@hetzner2 nginx]# pwd /var/log/nginx [root@hetzner2 nginx]# tail -f *.log */*.log | grep -i fuck 98.242.98.106 - - [09/Feb/2018:19:04:10 +0000] "GET /fuck HTTP/1.1" 401 381 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" "-" [root@hetzner2 nginx]# grep -irl 'fuck' * seedhome.openbuildinginstitute.org/access.log [root@hetzner2 nginx]#
- I guess that site became our default/catch-all, picking up requests on obi's ip address (138.201.84.223) on port 443. And the nginx config file is generic enough to just pass the host header along to varnish using a dynamic variable ($host), so then apache actually serves the correct page.
- so I think the best thing to do is to explicitly define an nginx listen on port 443 for each awstats site, passing to varnish per usual (but logging in the correct place) && the extensible configs in the proper & expected place. Then I'll configure apache to have 2 distinct docroots for X-Forwarded-Port equal to 4443 vs 443. 4443 will give us the htpasswd-protected awstats files. 443 will go to publicly-accessible docroot where certbot will generate files to negotiate https cert updates.
- successfully tested changing the "X-Forwarded-Port 443" line in the nginx config to "X-Forwarded-Port $server_port"
[root@hetzner2 conf.d]# cat /etc/nginx/conf.d/awstats.openbuildinginstitute.org.conf
################################################################################
# File: awstats.openbuildinginstitute.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################
server {
include conf.d/secure.include;
include conf.d/ssl.openbuildinginstitute.org.include;
listen 138.201.84.223:4443;
listen 138.201.84.223:443;
#listen [::]:443;
server_name awstats.openbuildinginstitute.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
#root /var/www/html/SITE_DOWN/htdocs/;
#index index.html index.htm;
# force all requests to load exactly this page
#location / {
# try_files $uri /index.html;
#}
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
#proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Host $host;
}
}
[root@hetzner2 conf.d]#
- apache isn't very powerful at conditionals, so I decided to move the logic to nginx & varnish
- updated /etc/httpd/conf/httpd.conf to listen on 127.0.0.1:8010 for the certbot vhost
- successfully added 'awstats.opensourceecology.org' to cert SAN with certbot
[root@hetzner2 ~]# certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org -w /var/www/html/certbot/htdocs -d awstats.opensourceecology.org ... IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/opensourceecology.org/privkey.pem Your cert will expire on 2018-05-10. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le [root@hetzner2 ~]# /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* [root@hetzner2 ~]# nginx -t && service nginx reload nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Redirecting to /bin/systemctl reload nginx.service [root@hetzner2 ~]#
- so first, nginx is listening on ports '443' && '4443' It sets the 'X-Forwarded-Port' Header to whichever it heard the request come in on (referenced by $server_port)
[root@hetzner2 conf.d]# date
Fri Feb 9 21:37:33 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/nginx/conf.d
[root@hetzner2 conf.d]# cat awstats.openbuildinginstitute.org.conf
################################################################################
# File: awstats.openbuildinginstitute.org.conf
# Version: 0.2
# Purpose: Internet-listening web server for truncating https, basic DOS
# protection, and passing to varnish cache (varnish then passes to
# apache)
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################
server {
include conf.d/secure.include;
include conf.d/ssl.openbuildinginstitute.org.include;
listen 138.201.84.223:4443;
listen 138.201.84.223:443;
server_name awstats.openbuildinginstitute.org;
#############
# SITE_DOWN #
#############
# uncomment this block && restart nginx prior to apache work to display the
# "SITE DOWN" webpage for our clients
#root /var/www/html/SITE_DOWN/htdocs/;
#index index.html index.htm;
# force all requests to load exactly this page
#location / {
# try_files $uri /index.html;
#}
###################
# SEND TO VARNISH #
###################
location / {
proxy_pass http://127.0.0.1:6081;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
#proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header Host $host;
}
}
[root@hetzner2 conf.d]#
- nginx sends its requests to varnish. The varnish config for the awstats configs defines 2x distinct backends. Both are apache, but the ports are different. The standard 8000 name-based-vhost is used for the 4443 port && there's a new vhost running on 8010 specifically for this certbot vhost.
[root@hetzner2 varnish]# date
Fri Feb 9 21:39:47 UTC 2018
[root@hetzner2 varnish]# pwd
/etc/varnish
[root@hetzner2 varnish]# cat sites-enabled/awstats.openbuildinginstitute.org
################################################################################
# File: awstats.openbuildinginstitute.org.vcl
# Version: 0.3
# Purpose: Config file for awstats for obi
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-23
# Updated: 2018-02-09
################################################################################
vcl 4.0;
##########################
# VHOST-SPECIFIC BACKEND #
##########################
backend awstats_openbuildinginstitute_org {
.host = "127.0.0.1";
.port = "8000";
}
backend certbot_openbuildinginstitute_org {
.host = "127.0.0.1";
.port = "8010";
}
#####################################################################
# STUFF MOSTLY COPIED FROM vcaching WORDPRESS PLUGIN #
# https://plugins.svn.wordpress.org/vcaching/trunk/varnish-conf/v4/ #
#####################################################################
sub vcl_recv {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# is this an admin trying to access awstats or certbot trying to renew the
# https certificate?
if ( req.http.X-Forwarded-Port == "4443" ){
# this is an admin; send them to the auth-protected apache vhost
set req.backend_hint = awstats_openbuildinginstitute_org;
} else {
# this is not an admin accessing the hidden port; send them to the apache
# vhost with no content (that certbot uses)
set req.backend_hint = certbot_openbuildinginstitute_org;
}
}
}
sub vcl_hash {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
sub vcl_backend_response {
if ( beresp.backend.name == "awstats_openbuildinginstitute_org" ){
# TODO
}
}
sub vcl_synth {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
sub vcl_pipe {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
sub vcl_deliver {
if ( req.http.host == "awstats.openbuildinginstitute.org" ){
# TODO
}
}
[root@hetzner2 varnish]# cat default.vcl
################################################################################
# File: default.vcl
# Version: 0.1
# Purpose: Main config file for varnish cache. Note that it's intentionally
# mostly bare to allow robust vhost-specific logic. Please see this
# for more info:
# * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2017-11-12
################################################################################
vcl 4.0;
############
# INCLUDES #
############
#
import std;
include "conf/acl.vcl";
include "lib/purge.vcl";
include "all-vhosts.vcl";
include "catch-all.vcl";
[root@hetzner2 varnish]# cat all-vhosts.vcl
################################################################################
# File: all-hosts.vcl
# Version: 0.8
# Purpose: meta config file that simply imports the site-specific vcl files
# stored in the 'sites-enabled' directory Please see this for more info
# * https://www.getpagespeed.com/server-setup/varnish/varnish-virtual-hosts
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2017-11-12
# Updated: 2018-02-09
################################################################################
include "sites-enabled/staging.openbuildinginstitute.org";
include "sites-enabled/awstats.openbuildinginstitute.org";
include "sites-enabled/awstats.opensourceecology.org";
include "sites-enabled/www.openbuildinginstitute.org";
include "sites-enabled/www.opensourceecology.org";
include "sites-enabled/seedhome.openbuildinginstitute.org";
include "sites-enabled/fef.opensourceecology.org";
include "sites-enabled/oswh.opensourceecology.org";
include "sites-enabled/forum.opensourceecology.org";
include "sites-enabled/wiki.opensourceecology.org";
[root@hetzner2 varnish]#
- finally, here's the 2x distinct vhosts in apache
[root@hetzner2 conf.d]# date
Fri Feb 9 21:41:31 UTC 2018
[root@hetzner2 conf.d]# pwd
/etc/httpd/conf.d
[root@hetzner2 conf.d]# cat awstats.openbuildinginstitute.org.conf
<VirtualHost 127.0.0.1:8000>
ServerName awstats.openbuildinginstitute.org
DocumentRoot "/var/www/html/awstats.openbuildinginstitute.org/htdocs"
Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
</VirtualHost>
<Directory "/var/www/html/awstats.openbuildinginstitute.org/htdocs">
AuthType Basic
AuthName "Authentication Required"
AuthUserFile /var/www/html/awstats.openbuildinginstitute.org/.htpasswd
# it is unnecessary to allow certbot here, since it uses port 80 (which we
# send to port 443), but this server only listens on port 4443 (via nginx)
#Require expr %{REQUEST_URI} =~ m#^/.well-known/acme-challenge/#
Require valid-user
Options +Indexes +Includes
AllowOverride None
Order allow,deny
Allow from all
# Harden vhost docroot by blocking all request types except the 3 essentials
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
# disable mod_security with rules as needed (found by logs in:
# /var/log/httpd/modsec_audit.log
<Location "/">
<IfModule security2_module>
SecRuleEngine On
#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
</IfModule>
</Location>
[root@hetzner2 conf.d]# cat certbot.conf
################################################################################
# File: certbot.conf
# Version: 0.1
# Purpose: localhost-only-listening, http-only, name-based-vhost for serving
# an empty vhost that's to be used by certbot for refreshing our Let's
# Encrypt certificates. This is necessary for admin-only/private vhosts
# that operate on non-standard port with basic auth required. For
# example, see the configs for awstats.
# Author: Michael Altfield <michael@opensourceecology.org>
# Created: 2018-02-09
# Updated: 2018-02-09
################################################################################
<VirtualHost 127.0.0.1:8010>
#ServerName awstats.openbuildinginstitute.org
DocumentRoot "/var/www/html/certbot/htdocs"
#Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org
<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*>
Deny From All
</LocationMatch>
</VirtualHost>
<Directory "/var/www/html/certbot/htdocs">
Options +Indexes +Includes
AllowOverride None
Order allow,deny
Allow from all
# Harden vhost docroot by blocking all request types except the 3 essentials
<LimitExcept GET POST HEAD>
deny from all
</LimitExcept>
</Directory>
# disable mod_security with rules as needed (found by logs in:
# /var/log/httpd/modsec_audit.log
<Location "/">
<IfModule security2_module>
SecRuleEngine On
#SecRuleRemoveById 960015 981173 960024 960904 960015 960017
</IfModule>
</Location>
Thr Feb 08, 2018
- installed our minimal set of security wordpress plugins
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
...
Installing Google Authenticator (0.48)
Downloading installation package from https://downloads.wordpress.org/plugin/google-authenticator.0.48.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/google-authenticator-0.48.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'google-authenticator'...
Plugin 'google-authenticator' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
...
Installing Google Authenticator Encourage User Activation (0.2)
Downloading installation package from https://downloads.wordpress.org/plugin/google-authenticator-encourage-user-activation.0.2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/google-authenticator-encourage-user-activation-0.2.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'google-authenticator-encourage-user-activation'...
Plugin 'google-authenticator-encourage-user-activation' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]# defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
[root@hetzner2 htdocs]# pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
/var/www/html/www.opensourceecology.org/htdocs/wp-content/plugins/google-authenticator /var/www/html/www.opensourceecology.org/htdocs
[root@hetzner2 google-authenticator]# sed -i "s^\$GA_description\s=\s(\s[\"'].*[\"']^\$GA_description = ( '$defaultOtpAccountDescription'^" google-authenticator.php
[root@hetzner2 google-authenticator]# popd
/var/www/html/www.opensourceecology.org/htdocs
[root@hetzner2 htdocs]# # install 'force-strong-passwords' plugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate
...
Installing Force Strong Passwords (1.8.0)
Downloading installation package from https://downloads.wordpress.org/plugin/force-strong-passwords.1.8.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/force-strong-passwords-1.8.0.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'force-strong-passwords'...
Plugin 'force-strong-passwords' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install rename-wp-login plugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate
...
Installing Rename wp-login.php (2.5.5)
Downloading installation package from https://downloads.wordpress.org/plugin/rename-wp-login.2.5.5.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/rename-wp-login-2.5.5.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'rename-wp-login'...
Plugin 'rename-wp-login' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install "SSL Insecure Content Fixer" pugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate
...
Installing SSL Insecure Content Fixer (2.5.0)
Downloading installation package from https://downloads.wordpress.org/plugin/ssl-insecure-content-fixer.2.5.0.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/ssl-insecure-content-fixer-2.5.0.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'ssl-insecure-content-fixer'...
Plugin 'ssl-insecure-content-fixer' activated.
Success: Installed 1 of 1 plugins.
[root@hetzner2 htdocs]#
[root@hetzner2 htdocs]# # install "Varnish Caching" pugin
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate
...
Installing Varnish Caching (1.6.7)
Downloading installation package from https://downloads.wordpress.org/plugin/vcaching.1.6.7.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/vcaching-1.6.7.zip'...
Unpacking the package...
Installing the plugin...
Plugin installed successfully.
Activating 'vcaching'...
Plugin 'vcaching' activated.
Success: Installed 1 of 1 plugins.
- updated existing wordpress plugins on osemain staging site on hetzner2
[root@hetzner2 htdocs]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
...
+----------------------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------------------+-------------+-------------+---------+
| akismet | 3.3 | 4.0.2 | Updated |
| cyclone-slider-2 | 2.12.4 | 3.2.0 | Updated |
| duplicate-post | 3.1.2 | 3.2.1 | Updated |
| insert-headers-and-footers | 1.4.1 | 1.4.2 | Updated |
| jetpack | 4.7.1 | 5.8 | Updated |
| ml-slider | 3.5 | 3.6.8 | Updated |
| post-types-order | 1.9.3 | 1.9.3.6 | Updated |
| recent-tweets-widget | 1.6.6 | 1.6.8 | Updated |
| shareaholic | 7.8.0.4 | 8.6.5 | Updated |
| share-on-diaspora | 0.7.1 | 0.7.2 | Updated |
| w3-total-cache | 0.9.5.2 | 0.9.6 | Updated |
| wp-smushit | 2.6.1 | 2.7.6 | Updated |
+----------------------------+-------------+-------------+---------+
Success: Updated 12 of 12 plugins.
[root@hetzner2 htdocs]#
- updated all the wp dashboard config settings for the new plugins
- discovered that the "Purge ALL Varnish Cache" button resulted in an error
Trying to purge URL : http://127.0.0.1:6081/.* => 403 Forbidden
- there's a cooresponding ossec alert & trigger by modsec:
--388c9d2b-A--
[09/Feb/2018:00:38:31 +0000] WnzthxK3iH4mmnFop@Wm5gAAAAc 127.0.0.1 37192 127.0.0.1 8000
--388c9d2b-B--
PURGE /.* HTTP/1.1
User-Agent: WordPress/4.9.3; https://osemain.opensourceecology.org
Accept-Encoding: deflate;q=1.0, compress;q=0.5, gzip;q=0.5
host: osemain.opensourceecology.org
X-VC-Purge-Method: regex
X-VC-Purge-Host: osemain.opensourceecology.org
Connection: Close, close
X-Forwarded-For: 127.0.0.1
X-Varnish: 45279
--388c9d2b-F--
HTTP/1.1 403 Forbidden
Content-Length: 204
Connection: close
Content-Type: text/html; charset=iso-8859-1
--388c9d2b-E--
--388c9d2b-H--
Message: Access denied with code 403 (phase 1). Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_30_http_policy.conf"] [line "31"] [id "960032"] [rev "2"] [msg "Method is not allowed by policy"] [data "PURGE"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"]
Action: Intercepted (phase 1)
Stopwatch: 1518136711479621 232 (- - -)
Stopwatch2: 1518136711479621 232; combined=97, p1=83, p2=0, p3=0, p4=0, p5=14, sr=18, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"
--388c9d2b-Z--
- added id = 960032 to the httpd config whitelist
- varnish log alerts showed that the purge request was being sent to obi, so I added it to that apache config too
- confirmed this works in seedhome.openbuildinginstitue.org
- confirmed this works in fef.opensourceecology.org
- ah, I think this fails because the varnish config is defined to use 'www.opensourceecology.org' it's temporarily 'osemain.opensourceecology.org' due to a name collision with the prod site. I'm confident this will work once the name is correct after cuttover. ignoring this issue. in the meantime, I'm pretty sure nothing is being cached at all (which is actually nice for testing the staging site).
- the top-left logo shows up fine on my disposable vm, but not in my locked-down https-everywhere browser http://opensourceecology.org/wp-content/uploads/2014/02/OSE_yellow-copy2.png
- should probably change the url to use https instead in Appearance -> Theme Options -> Site Logo to use https instead (note I can't test this until after the name change)
- discovered that the facebook iframe on the main page was missing
- solution was to change 'http' to 'https'
[segment class="first-segment"] <div class="center"><iframe style="border: none; overflow: hidden; width: 820px; height: 435px; background: white; float: center;" src="https://www.facebook.com/plugins/likebox.php?href=http%3A%2F%2Fwww.facebook.com%2FOpenSourceEcology&width=820&colorscheme=light&show_faces=true&border_color&stream=true&header=true&height=435" width="300" height="150" frameborder="0" scrolling="yes"></iframe></div> [/segment]
- confirmed that the old broken pages are still working
- /contributors/
- the top slider that Catarina said we've permenantly lost is still throwing an error; Marcin will remove this from the prod site, so this will be a non-issue at cutover
- the timeline shows up properly
- /community-true-fans/
- this page perfectly matches prod
- /history-timeline/
- this page perfectly matches prod
- /cnc-torch-table-workshop/
- Eventbright on left still working as desired
- videos embed issues are still present as expected, and a non-blocker to the migration
- /about-videos-3/
- missing video content is still (not) present as expected, and a non-blocker to the migration
- /contributors/
- all looks good to me. sent an email to Marcin asking for validation.
- awstats were not updating again
- found a permissions issue
[root@hetzner2 conf.d]# ls -lah /var/log/nginx/ total 11M drwx------ 9 nginx nginx 4.0K Feb 8 07:21 . drwxr-xr-x 13 root root 12K Feb 8 07:21 .. -rw-r--r-- 1 nginx nginx 3.5M Feb 9 02:03 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 143K Jan 30 05:02 access.log-20180130.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 4.8M Feb 8 07:21 access.log-20180208 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 323K Feb 9 02:02 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 7.8K Jan 30 05:00 error.log-20180130.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 433K Feb 8 07:21 error.log-20180208 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 root root 4.0K Feb 8 07:21 fef.opensourceecology.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 forum.opensourceecology.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 oswh.opensourceecology.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 seedhome.openbuildinginstitute.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 wiki.opensourceecology.org drw-r--r-- 2 nginx nginx 4.0K Feb 9 02:01 www.openbuildinginstitute.org drwxr-xr-x 2 root root 4.0K Feb 8 07:21 www.opensourceecology.org [root@hetzner2 conf.d]#
- those probably should all be owned by nginx
[root@hetzner2 nginx]# chown nginx:nginx fef.opensourceecology.org [root@hetzner2 nginx]# chown -R nginx:nginx fef.opensourceecology.org [root@hetzner2 nginx]# chown -R nginx:nginx forum.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx oswh.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx seedhome.openbuildinginstitute.org/ [root@hetzner2 nginx]# chown -R nginx:nginx wiki.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx www.opensourceecology.org/ [root@hetzner2 nginx]# chown -R nginx:nginx www.openbuildinginstitute.org/ [root@hetzner2 nginx]# ls -lah total 11M drwx------ 9 nginx nginx 4.0K Feb 8 07:21 . drwxr-xr-x 13 root root 12K Feb 8 07:21 .. -rw-r--r-- 1 nginx nginx 3.5M Feb 9 02:06 access.log -rw-r--r-- 1 nginx nginx 222 Dec 24 01:33 access.log.1.gz -rw-r--r-- 1 nginx nginx 143K Jan 30 05:02 access.log-20180130.gz -rw-r--r-- 1 nginx nginx 154K Jan 31 07:43 access.log-20180131.gz -rw-r--r-- 1 nginx nginx 186K Feb 1 06:41 access.log-20180201.gz -rw-r--r-- 1 nginx nginx 107K Feb 2 08:54 access.log-20180202.gz -rw-r--r-- 1 nginx nginx 83K Feb 3 05:49 access.log-20180203.gz -rw-r--r-- 1 nginx nginx 81K Feb 4 04:48 access.log-20180204.gz -rw-r--r-- 1 nginx nginx 265K Feb 5 09:12 access.log-20180205.gz -rw-r--r-- 1 nginx nginx 351K Feb 6 06:00 access.log-20180206.gz -rw-r--r-- 1 nginx nginx 458K Feb 7 07:35 access.log-20180207.gz -rw-r--r-- 1 nginx nginx 4.8M Feb 8 07:21 access.log-20180208 -rw-r--r-- 1 nginx nginx 420 Dec 24 01:32 access.log.2.gz -rw-r--r-- 1 nginx nginx 207 Dec 24 01:29 access.log.3.gz -rw-r--r-- 1 nginx nginx 187 Dec 24 01:28 access.log.4.gz -rw-r--r-- 1 nginx nginx 1.2K Dec 24 01:28 access.log.5.gz -rw-r--r-- 1 nginx nginx 271 Dec 24 01:21 access.log.6.gz -rw-r--r-- 1 nginx nginx 324K Feb 9 02:06 error.log -rw-r--r-- 1 nginx nginx 211 Dec 24 01:33 error.log.1.gz -rw-r--r-- 1 nginx nginx 7.8K Jan 30 05:00 error.log-20180130.gz -rw-r--r-- 1 nginx nginx 9.6K Jan 31 07:42 error.log-20180131.gz -rw-r--r-- 1 nginx nginx 9.0K Feb 1 06:41 error.log-20180201.gz -rw-r--r-- 1 nginx nginx 7.8K Feb 2 08:52 error.log-20180202.gz -rw-r--r-- 1 nginx nginx 6.0K Feb 3 05:49 error.log-20180203.gz -rw-r--r-- 1 nginx nginx 6.2K Feb 4 04:48 error.log-20180204.gz -rw-r--r-- 1 nginx nginx 8.5K Feb 5 09:03 error.log-20180205.gz -rw-r--r-- 1 nginx nginx 9.6K Feb 6 06:00 error.log-20180206.gz -rw-r--r-- 1 nginx nginx 13K Feb 7 07:35 error.log-20180207.gz -rw-r--r-- 1 nginx nginx 433K Feb 8 07:21 error.log-20180208 -rw-r--r-- 1 nginx nginx 592 Dec 24 01:33 error.log.2.gz -rw-r--r-- 1 nginx nginx 413 Dec 24 01:29 error.log.3.gz -rw-r--r-- 1 nginx nginx 210 Dec 24 01:28 error.log.4.gz -rw-r--r-- 1 nginx nginx 211 Dec 24 01:27 error.log.5.gz -rw-r--r-- 1 nginx nginx 506 Dec 24 01:24 error.log.6.gz drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 fef.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 forum.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 oswh.opensourceecology.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 seedhome.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 wiki.opensourceecology.org drw-r--r-- 2 nginx nginx 4.0K Feb 9 02:01 www.openbuildinginstitute.org drwxr-xr-x 2 nginx nginx 4.0K Feb 8 07:21 www.opensourceecology.org [root@hetzner2 nginx]#
- that appears to be working. Now I just have to wait another week or so..
- added an 'awstats' record to opensourceecology.org in cloudflare.com
Wed Feb 07, 2018
- Marcin responded to the osemain issues. Here's the status:
- /
- Marcin confirmed that front-page content (the video) this was fixed
- /contributors/
- Catarina will be looking into the "featured collaborators" slider issue on this page which is broken on production & staging
- I'll look into the "collaborators timeline" issue that is fine on prod, but missing on staging
- Marcin said the missing photos on the "collaborators timeline" should be dealt with later & not block this migration
- /community/
- Marcin updated the site, removing the broken Flattr embed http://opensourceecology.org/community/
- /community-true-fans/
- I need to investigate why these 2x sliders are broken in staging, but working in production
- /history-timeline/
- I need to investigate why these 2x sliders are broken in staging, but working in production
- /
/cnc-torch-table-workshop/
- Eventbright is truncated on the left
- The first youtube video shows up as a link, instead of as an embed = https://www.youtube.com/watch?v=JOH03vzDYQg
- The first vemo video shows up as a link, instead of as an embed = https://vimeo.com/23785186
/about-videos-3/
- Marcin said the videos are missing. I asked him to attempt to add them & if he encountered any issues, send me the errors so I can try to reproduce & fix it.
- began debugging osemain staging slider issues on /contributors/
- first, none of the plugins were enabled. This was a permission issue, solved by following the guide at Wordpress
vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"
- next "revolution slider" wasn't enabled not sure why. I enabled it and the one named "Revolution Slider Patch"
- finally, I saw that the last slider wasn't actually a slider; it was just an iframe to an http site = http://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0ArpE5Y9PpJCXdFhleTktTkoyeHNFUXktUXJCMGVkbVE&font=Bevan-PotanoSans&maptype=toner&lang=en&start_at_end=true&hash_bookmark=true&height=650#16
- replacing the above iframe's http protocol with https fixed the issue, but first I had to whitelist some modsec triggers to be able to update the page
- 958057, XSS
- 958056, XSS
- 973327, XSS
- began debugging issues on /community-true-fans/
- same issue, there was an iframe linking to an http page on cdn.knightlab.com; changing to https fixed it https://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0ArpE5Y9PpJCXdEhYb1NkR1dGRnhPV2FGYndseUg2WkE&font=Merriweather-NewsCycle&maptype=toner&lang=en&start_at_end=true&start_at_slide=34&height=650
- began debugging issues on /history-timeline/
- same issue, there was an iframe linking to an http page on cdn.knightlab.com; changing to https fixed it https://cdn.knightlab.com/libs/timeline/latest/embed/index.html?source=0AkNG-lv1ELQvdEMxdzRnU2VFbUllZ2Y0cnZPRld3SXc&font=Bevan-PotanoSans&maptype=toner&lang=en&hash_bookmark=true&start_at_slide=23&height=650
- began troubleshooting /cnc-torch-table-workshop/
- found a section on the page edit called "Choose Custom Sidebar" and a drop-down for "Primary Sidebar Choice" said "CNC Torch Table Workshop". So I opened Appearence -> Widgets in a new tab, and found the "CNC Torcn Table Workshop" widget with this contents:
<a href="http://opensourceecology.org/CNC-torch-table-workshop/#overview">Overview</a> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#learning">Learning Outcomes</a> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#schedule">Schedule</a> <a href="http://opensourceecology.org/CNC-torch-table-workshop#registration">Registration</a> <div style="width:195px; text-align:center;" ><iframe src="https://www.eventbrite.com/countdown-widget?eid=38201763503" frameborder="0" height="322" width="195" marginheight="0" marginwidth="0" scrolling="no" allowtransparency="true"></iframe><div style="font-family:Helvetica, Arial; font-size:12px; padding:10px 0 5px; margin:2px; width:195px; text-align:center;" ><a class="powered-by-eb" style="color: #ADB0B6; text-decoration: none;" target="_blank" href="http://www.eventbrite.com/">Powered by Eventbrite</a></div></div>
- I changed the height of "322" of the iframe to "999", pressed save, and cleared the varnish cache. That didn't help
- I added ";height:999px" to the containing div, pressed save, and cleared the varnish cache.
- I replaced the "CNC Torch Table Workshop" widget of type "Text" with one of the same contents of type "Custom HTML", and it worked.
- the "Custom HTML" widget was added in wordpress v4.8.1 https://wptavern.com/wordpress-4-8-1-adds-a-dedicated-custom-html-widget
- this makes sense, as this migration includes the update of core wp from v4.7.9 to v4.9.2
- I had to add some breaks to make the links on separate lines; I guess that's what "Text" does for you--and also breaks dynamic height? Here's the new contents:
- ugh, fucking hell. I can't copy from the new "Custom HTML widget?" It adds line numbers to my clipboard, breaking copy-and-paste to/from the window. This is horrible!
- I tried switching back to "text" so I could just try to uncheck the "automatically add paragraphs" checkbox, but that's only a "legacy" thing (as the link above points-out. Once you delete a text box & re-add it, that checkbox disappears :(
- it looks like this new "custom html" widget is powered by codemirror
- ok, I found the solution to the annoying codemirror issues, but it can't be fixed site-wide. It must be fixed on a per-user basis by going to the user's profile & checking the "Disable syntax highlighting when editing code" checkbox https://make.wordpress.org/core/tag/codemirror/
- ok, here's the new contents with line breaks:
- the "Custom HTML" widget was added in wordpress v4.8.1 https://wptavern.com/wordpress-4-8-1-adds-a-dedicated-custom-html-widget
<a href="http://opensourceecology.org/CNC-torch-table-workshop/#overview">Overview</a> <br/> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#learning">Learning Outcomes</a> <br/> <a href="http://opensourceecology.org/CNC-torch-table-workshop/#schedule">Schedule</a> </> <a href="http://opensourceecology.org/CNC-torch-table-workshop#registration">Registration</a><div style="width:195px; text-align:center;" > <br/> <div style="width:195px; text-align:center;" ><iframe src="https://www.eventbrite.com/countdown-widget?eid=38201763503" frameborder="0" height="322" width="195" marginheight="0" marginwidth="0" scrolling="no" allowtransparency="true"></iframe><div style="font-family:Helvetica, Arial; font-size:12px; padding:10px 0 5px; margin:2px; width:195px; text-align:center;" ><a class="powered-by-eb" style="color: #ADB0B6; text-decoration: none;" target="_blank" href="http://www.eventbrite.com/">Powered by Eventbrite</a></div></div>
- I emailed marcin; he said that the issue embedding the 2x videos on this page shouldn't be a blocker & confirmed that I should proceed with the migration with the content missing
- began debugging the missing videos on /about-video-3/
- there's literally nothing in the textarea for this page in wordpress. It's empty for both staging & production *shrug*
- I emailed marcin; he said this shouldn't be a blocker & confirmed that I should proceed with the migration with the content missing
- So the osemain issues that remain to be fixed are:
- /contributors/
- Catarina will be looking into the "featured collaborators" slider issue on this page which is broken on production & staging
- /contributors/
/cnc-torch-table-workshop/
Tue Feb 06, 2018
- the osemain staging site was created on 2018-01-03
[maltfield@hetzner2 backups_for_migration_from_hetzner1]$ date Tue Feb 6 15:53:00 UTC 2018 [maltfield@hetzner2 backups_for_migration_from_hetzner1]$ pwd /var/tmp/backups_for_migration_from_hetzner1 [maltfield@hetzner2 backups_for_migration_from_hetzner1]$ ls -lah | grep -i osemain drwxr-xr-x 4 root root 4.0K Jan 3 21:32 osemain_20180103 [maltfield@hetzner2 backups_for_migration_from_hetzner1]$
- I refreshed it for today = 2018-02-06
# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180206"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
dbUser_hetzner2="osemain_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
- I updated the osemain site with a new wp core, but stopped short of installing our new security wp plugins or attempting to update the existing wp plugins. Here's a breakdown of what I found with the site:
- /
- vid fixed w/ update
- /contributors/
- revolution slider is broken on the production site with error message = "Revolution Slider Error: Slider with alias collabslider not found."
- collaborators timeline is missing entirely on the staging site
- many photos 404ing for the collaborators timeline on the production site
- /community/
- /
/community-true-fans/
- 2x sliders missing on staging; working fine on production
/history-timeline/
- 2x sliders missing on staging; working fine on production
/cnc-torch-table-workshop/
- Eventbright is truncated on the left
- The first youtube video shows up as a link, instead of as an embed = https://www.youtube.com/watch?v=JOH03vzDYQg
- The first vemo video shows up as a link, instead of as an embed = https://vimeo.com/23785186
/about-videos-3/
- Marcin said the videos are missing. I asked him to attempt to add them & if he encountered any issues, send me the errors so I can try to reproduce & fix it.
Mon Feb 05, 2018
- DIdn't hear back from Marcin wrt my email about the osemigration, so I decided to delay the migration CHG-2018-02-05
Sun Feb 04, 2018
- Marcin q confirmed that we can migrate the osemain wp site on Monday, Feb 5th.
- I began documenting this CHG process at CHG-2018-02-05
- I found that many of the plugins 3 of the active plugins had pending updates: fundraising, wpmdev-updates, and wp-smush-pro
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------------------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------------------------------+----------+-----------+---------+
| akismet | active | none | 3.3 |
| brankic-photostream-widget | active | none | 1.3.1 |
| cyclone-slider-2 | inactive | none | 2.12.4 |
| duplicate-post | active | none | 3.1.2 |
| flattr | inactive | none | 1.2.2 |
| force-strong-passwords | active | none | 1.8.0 |
| fundraising | active | available | 2.6.4.5 |
| google-authenticator | active | none | 0.48 |
| google-authenticator-encourage-user-activat | active | none | 0.2 |
| ion | | | |
| hello | inactive | none | 1.6 |
| insert-headers-and-footers | inactive | none | 1.4.1 |
| jetpack | active | none | 4.7.1 |
| ml-slider | active | none | 3.5 |
| ml-slider-pro | active | none | 2.6.6 |
| open-in-new-window-plugin | inactive | none | 2.4 |
| patch-for-revolution-slider | active | none | 2.4.1 |
| post-types-order | inactive | none | 1.9.3 |
| really-simple-facebook-twitter-share-button | inactive | none | 4.5 |
| s | | | |
| recent-tweets-widget | active | none | 1.6.6 |
| rename-wp-login | active | none | 2.5.5 |
| revision-control | inactive | none | 2.3.2 |
| revslider | active | none | 4.3.8 |
| shareaholic | inactive | none | 7.8.0.4 |
| share-on-diaspora | inactive | none | 0.7.1 |
| ssl-insecure-content-fixer | active | none | 2.5.0 |
| vcaching | active | none | 1.6.7 |
| w3-total-cache | inactive | none | 0.9.5.2 |
| wp-memory-usage | inactive | none | 1.2.2 |
| wp-optimize | inactive | none | 2.1.1 |
| wp-facebook-open-graph-protocol | inactive | none | 2.0.13 |
| wpmudev-updates | active | available | 4.2 |
| wp-smushit | inactive | none | 2.6.1 |
| wp-smush-pro | active | available | 2.4.5 |
+---------------------------------------------+----------+-----------+---------+
[WPMUDEV API Error] 4.2 | User has blocked requests through HTTP. ((unknown URL) [500])
[root@hetzner2 ~]#
- that update attempt failed
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Warning: Update package not available.
Warning: Update package not available.
Warning: Update package not available.
+-----------------+-------------+-------------+--------+
| name | old_version | new_version | status |
+-----------------+-------------+-------------+--------+
| fundraising | 2.6.4.5 | 2.6.4.9 | Error |
| wpmudev-updates | 4.2 | 4.4 | Error |
| wp-smush-pro | 2.4.5 | 2.7.6 | Error |
+-----------------+-------------+-------------+--------+
Success: Plugins already updated.
[root@hetzner2 ~]#
- after commenting-out the WP_HTTP_BLOCK_EXTERNAL set to 'true' in wp-config.php, however, I see there were many more updates available
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
+---------------------------------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------------------------------+----------+-----------+---------+
| akismet | active | available | 3.3 |
| brankic-photostream-widget | active | none | 1.3.1 |
| cyclone-slider-2 | inactive | available | 2.12.4 |
| duplicate-post | active | available | 3.1.2 |
| flattr | inactive | none | 1.2.2 |
| force-strong-passwords | active | none | 1.8.0 |
| fundraising | active | available | 2.6.4.5 |
| google-authenticator | active | none | 0.48 |
| google-authenticator-encourage-user-activat | active | none | 0.2 |
| ion | | | |
| hello | inactive | none | 1.6 |
| insert-headers-and-footers | inactive | available | 1.4.1 |
| jetpack | active | available | 4.7.1 |
| ml-slider | active | available | 3.5 |
| ml-slider-pro | active | none | 2.6.6 |
| open-in-new-window-plugin | inactive | none | 2.4 |
| patch-for-revolution-slider | active | none | 2.4.1 |
| post-types-order | inactive | available | 1.9.3 |
| really-simple-facebook-twitter-share-button | inactive | none | 4.5 |
| s | | | |
| recent-tweets-widget | active | available | 1.6.6 |
| rename-wp-login | active | none | 2.5.5 |
| revision-control | inactive | none | 2.3.2 |
| revslider | active | none | 4.3.8 |
| shareaholic | inactive | available | 7.8.0.4 |
| share-on-diaspora | inactive | available | 0.7.1 |
| ssl-insecure-content-fixer | active | none | 2.5.0 |
| vcaching | active | none | 1.6.7 |
| w3-total-cache | inactive | available | 0.9.5.2 |
| wp-memory-usage | inactive | none | 1.2.2 |
| wp-optimize | inactive | none | 2.1.1 |
| wp-facebook-open-graph-protocol | inactive | none | 2.0.13 |
| wpmudev-updates | active | available | 4.2 |
| wp-smushit | inactive | available | 2.6.1 |
| wp-smush-pro | active | available | 2.4.5 |
+---------------------------------------------+----------+-----------+---------+
[root@hetzner2 ~]#
- then attempting updates again was partially successful
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
PHP Warning: ini_set() has been disabled for security reasons in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils-wp.php on line 44
PHP Warning: file_exists(): open_basedir restriction in effect. File(man-params.mustache) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pea
r:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.
opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/va
r/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in phar:///home/wp/.wp-cli/wp-cli.phar/php/utils.php on line 454
Enabling Maintenance mode...
Downloading update from https://downloads.wordpress.org/plugin/akismet.4.0.2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/akismet-4.0.2.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/cyclone-slider-2.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/cyclone-slider-2-3.2.0.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/duplicate-post.3.2.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
Downloading update from https://downloads.wordpress.org/plugin/insert-headers-and-footers.1.4.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/jetpack.5.7.1.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/ml-slider.3.6.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from http://wp-updates.com/api/1/download/plugin/sXpq1rDdVNBB4r8Tm-EPBg/ml-slider-pro...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/post-types-order.1.9.3.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/recent-tweets-widget.1.6.8.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/shareaholic.8.6.1.zip...
Using cached file '/home/wp/.wp-cli/cache/plugin/shareaholic-8.6.1.zip'...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/share-on-diaspora.0.7.2.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Downloading update from https://downloads.wordpress.org/plugin/w3-total-cache.0.9.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
Downloading update from https://downloads.wordpress.org/plugin/wp-smushit.2.7.6.zip...
Unpacking the update...
Installing the latest version...
Removing the old version of the plugin...
Plugin updated successfully.
Warning: Update package not available.
+----------------------------+-------------+-------------+---------+
| name | old_version | new_version | status |
+----------------------------+-------------+-------------+---------+
| akismet | 3.3 | 4.0.2 | Updated |
| cyclone-slider-2 | 2.12.4 | 3.2.0 | Updated |
| duplicate-post | 3.1.2 | 3.2.1 | Updated |
| fundraising | 2.6.4.5 | 2.6.4.9 | Updated |
| insert-headers-and-footers | 1.4.1 | 1.4.2 | Updated |
| jetpack | 4.7.1 | 5.7.1 | Updated |
| ml-slider | 3.5 | 3.6.8 | Updated |
| ml-slider-pro | 2.6.6 | 2.7.1 | Updated |
| post-types-order | 1.9.3 | 1.9.3.6 | Updated |
| recent-tweets-widget | 1.6.6 | 1.6.8 | Updated |
| shareaholic | 7.8.0.4 | 8.6.1 | Updated |
| share-on-diaspora | 0.7.1 | 0.7.2 | Updated |
| w3-total-cache | 0.9.5.2 | 0.9.6 | Updated |
| wpmudev-updates | 4.2 | 4.4 | Updated |
| wp-smushit | 2.6.1 | 2.7.6 | Updated |
| wp-smush-pro | 2.4.5 | 2.7.7 | Updated |
+----------------------------+-------------+-------------+---------+
Success: Updated 16 of 16 plugins.
[root@hetzner2 ~]#
- what follows showed everything was updated, except the same 3 plugins that fail. So it's probably those specific plugins' fault. Let's move on..
[root@hetzner2 ~]# sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin list
...
+---------------------------------------------+----------+-----------+---------+
| name | status | update | version |
+---------------------------------------------+----------+-----------+---------+
| akismet | active | none | 4.0.2 |
| brankic-photostream-widget | active | none | 1.3.1 |
| cyclone-slider-2 | inactive | none | 3.2.0 |
| duplicate-post | active | none | 3.2.1 |
| flattr | inactive | none | 1.2.2 |
| force-strong-passwords | active | none | 1.8.0 |
| fundraising | active | available | 2.6.4.5 |
| google-authenticator | active | none | 0.48 |
| google-authenticator-encourage-user-activat | active | none | 0.2 |
| ion | | | |
| hello | inactive | none | 1.6 |
| insert-headers-and-footers | inactive | none | 1.4.2 |
| jetpack | active | none | 5.7.1 |
| ml-slider | active | none | 3.6.8 |
| ml-slider-pro | active | none | 2.7.1 |
| open-in-new-window-plugin | inactive | none | 2.4 |
| patch-for-revolution-slider | active | none | 2.4.1 |
| post-types-order | inactive | none | 1.9.3.6 |
| really-simple-facebook-twitter-share-button | inactive | none | 4.5 |
| s | | | |
| recent-tweets-widget | active | none | 1.6.8 |
| rename-wp-login | active | none | 2.5.5 |
| revision-control | inactive | none | 2.3.2 |
| revslider | active | none | 4.3.8 |
| shareaholic | inactive | none | 8.6.1 |
| share-on-diaspora | inactive | none | 0.7.2 |
| ssl-insecure-content-fixer | active | none | 2.5.0 |
| vcaching | active | none | 1.6.7 |
| w3-total-cache | inactive | none | 0.9.6 |
| wp-memory-usage | inactive | none | 1.2.2 |
| wp-optimize | inactive | none | 2.1.1 |
| wp-facebook-open-graph-protocol | inactive | none | 2.0.13 |
| wpmudev-updates | active | available | 4.2 |
| wp-smushit | inactive | none | 2.7.6 |
| wp-smush-pro | active | available | 2.4.5 |
+---------------------------------------------+----------+-----------+---------+
[root@hetzner2 ~]#
- Found a bunch of content issues on the site when doing a quick spot-check. Emailed Marcin for revalidation & a go/no-go decision for the cutover tomorrow
- error "Revolution Slider Error: Slider with alias collabslider not found."
* http://opensourceecology.org/contributors/ * https://osemain.opensourceecology.org/contributors/
- flattr embed is broken on donate section
* http://opensourceecology.org/community/ * https://osemain.opensourceecology.org/community/
- slider for True Fans isn't appearing
* http://opensourceecology.org/community-true-fans/ * https://osemain.opensourceecology.org/community-true-fans/
- added a CHG for the forum cutover CHG-2018-02-04
- followed the above guide. We're now storing the dynamic (php) vanilla files outside the docroot, next to the static content. Of course, the static content takes up more space, but it's not too bad.
[root@hetzner2 forum.opensourceecology.org]# date Sun Feb 4 18:13:00 UTC 2018 [root@hetzner2 forum.opensourceecology.org]# pwd /var/www/html/forum.opensourceecology.org [root@hetzner2 forum.opensourceecology.org]# du -sh * 2.7G htdocs 173M vanilla_docroot_backup.20180113 [root@hetzner2 forum.opensourceecology.org]#
- updated dns of forum.opensourceecology.org to point to hetzner2 instead of hetzner1
- deleted the CNAME entry for 'forum' pointing to dedi978.your-server.de
- added an A entry for 'forum' pointing to 138.201.84.243
- updated the name 'stagingforum' to 'forum' in /etc/httpd/conf.d/00-forum.opensourceecology.org.conf
- updated the name 'stagingforum' to 'forum' in /etc/nginx/conf.d/forum.opensourceecology.org.conf
- updated the opensourceecology.org certificate
[root@hetzner2 forum.opensourceecology.org]# certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d forum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org ... IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/opensourceecology.org/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/opensourceecology.org/privkey.pem Your cert will expire on 2018-05-05. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le [root@hetzner2 forum.opensourceecology.org]# /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* [root@hetzner2 forum.opensourceecology.org]# nginx -t && service nginx reload nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful Redirecting to /bin/systemctl reload nginx.service [root@hetzner2 forum.opensourceecology.org]#
- Did an unsafe varnish cache clear
service varnish restart
- confirmed that the static-content site was loading on the new url https://forum.opensourceecology.org
- updated our wiki (where the forum links to for info on how to create an account) with a notice that the site was deprecated in 2018 Vanilla_Forums
- began to test sending our encrypted ossec alerts to multiple recipients
- discovered that our spf record is invalid. I looked up the proper way to do this on cloudflare https://support.cloudflare.com/hc/en-us/articles/200168626-How-do-I-add-a-SPF-record-
- I had "spf" in the "Name" column, but it should have been "@". I updated this, and then Cloudflare automatically replaced the "@" that I typed with "opensourceecology.org"
- after the change above, I got a change in the lookup tool; success! https://mxtoolbox.com/SuperTool.aspx?action=spf%3aopensourceecology.org&run=toolpage
- updated the sent_encrypted_alarm.sh script to support multiple recipients
[root@hetzner2 ossec]# date
Mon Feb 5 01:39:24 UTC 2018
[root@hetzner2 ossec]# pwd
/var/ossec
[root@hetzner2 ossec]# cat sent_encrypted_alarm.sh
#!/bin/bash
# store the would-be plaintext email body
plaintext=`/usr/bin/formail -I ""`
# loop through all recipients & send them individually-encrypted mail
recipients="michael@opensourceecology.org marcin@opensourceecology.org"
for recipient in $(echo $recipients); do
# leave nothing unencrypted, including the subject!
echo "${plaintext}" | /usr/bin/gpg --homedir /var/ossec/.gnupg --trust-model always -ear "${recipient}" | /usr/bin/mail -r noreply@opensourceecology.org -s "" "${recipient}"
done
exit 0
- found & documented command to clear varnish cache without actually restarting the process Web_server_configuration
varnishadm 'ban req.url ~ "."'
Thr Feb 01, 2018
- confirmed that the forum wget retry worked
- rsync'd the wget into the document root, and a spot-checked showed that most pages worked
- saw an issue (Forbidden) with all the tagged pages, ie https://stagingforum.opensourceecology.org/discussions/tagged/water.html
[Thu Feb 01 17:29:02.262787 2018] [core:crit] [pid 30854] (9)Bad file descriptor: [client 127.0.0.1:45488] AH00529: /var/www/html/forum.opensourceecology.org/htdocs/discussions/tagged/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/html/forum.opensourceecology.org/htdocs/discussions/tagged/' is executable
- the issue appears to be with the directory named '.htaccess' for the tag of the same name
[root@hetzner2 htdocs]# ls -lah discussions/tagged/.htaccess total 64K drwxr-xr-x 2 root root 4.0K Feb 1 11:36 . drwxr-xr-x 573 root root 36K Feb 1 17:27 .. -rw-r--r-- 1 root root 1.7K Feb 1 10:33 feed.rss -rw-r--r-- 1 root root 17K Feb 1 11:36 p1.html [root@hetzner2 htdocs]#
- removing the above dir worked. The consequence is we will get a 404 for any links to this tag, but that's an acceptable loss IMO; te discussions themselves will still be accessible
- removed the forums dir from /etc/php.ini since we're now just static content
- restarted httpd
- confirmed that awstats has been updating, but--since today happens to be the first of the month--there's no data yet. I'll check again in a week or so.
Wed Jan 31, 2018
- moved the wget'd forums into the docroot & found a lot of Forbidden responses to my requests
- for some reason, the pages are named 'p1' instead of 'index.html' The "Forbidden" is caused by the server's refusal to do a directory listing.
- solution: add a symlink to "p1" from "index.html" in every dir with a "p1" but no "index.html"
[root@hetzner2 htdocs]# for p1 in $(find . -name p1); do dir=`dirname $p1`; pushd $dir; ln -s "p1" "index.html"; popd; done
- that helped, but there's still actually missing content (ie: https://stagingforum.opensourceecology.org/categories/other-languages/)
[root@hetzner2 htdocs]# ls -lah categories/other-languages/ total 12K drwxrwxrwx 2 root root 4.0K Jan 25 16:23 . drwxrwxrwx 55 root root 4.0K Jan 25 20:52 .. -rwxrwxrwx 1 root root 3.5K Jan 25 16:23 feed.rss [root@hetzner2 htdocs]#
- attempting again with a different command from https://www.linuxjournal.com/content/downloading-entire-web-site-wget
[root@hetzner2 oseforum.20180201]# date Thu Feb 1 03:39:34 UTC 2018 [root@hetzner2 oseforum.20180201]# pwd /var/tmp/deleteMeIn2019/oseforum.20180201 [root@hetzner2 oseforum.20180201]# time nice wget --recursive --no-clobber --page-requisites --html-extension --convert-links --domains "forum.opensourceecology.org" "http://forum.opensourceecology.org" <pre> ## if the above doesn't work, I'll try some of the options in the comments (ie: -m, --wait, --limit-rate, etc) =Tue Jan 30, 2018= # Trained Marcin on PGP =Fri Jan 26, 2018= # Emailed Marcin about setting up PGP # Confirmed that the wget of the forums finished <pre> [root@hetzner2 wget]# pwd /var/tmp/deleteMeIn2019/oseoforum.20180125/wget [root@hetzner2 ~]# wget -r -p -e robots=off http://forum.opensourceecology.org ... FINISHED --2018-01-26 00:53:29-- Total wall clock time: 8h 39m 3s Downloaded: 96387 files, 2.2G in 1m 33s (24.5 MB/s) [root@hetzner2 wget]# du -sh forum.opensourceecology.org/* 48K forum.opensourceecology.org/activity 264K forum.opensourceecology.org/applications 300K forum.opensourceecology.org/cache 3.0M forum.opensourceecology.org/categories 27M forum.opensourceecology.org/dashboard 1.4G forum.opensourceecology.org/discussion 15M forum.opensourceecology.org/discussions 873M forum.opensourceecology.org/entry 148K forum.opensourceecology.org/index.html 744K forum.opensourceecology.org/plugins 72M forum.opensourceecology.org/profile 12K forum.opensourceecology.org/search 28K forum.opensourceecology.org/search?Search=#000000&Mode=like 28K forum.opensourceecology.org/search?Search=#0&Mode=like 16K forum.opensourceecology.org/search?Search=#13Lifetime&Mode=like 16K forum.opensourceecology.org/search?Search=#197187&Mode=like 32K forum.opensourceecology.org/search?Search=#1&Mode=like 28K forum.opensourceecology.org/search?Search=#2&Mode=like 16K forum.opensourceecology.org/search?Search=#363636&Mode=like 28K forum.opensourceecology.org/search?Search=#3&Mode=like 16K forum.opensourceecology.org/search?Search=#458&Mode=like 20K forum.opensourceecology.org/search?Search=#4&Mode=like 28K forum.opensourceecology.org/search?Search=#5&Mode=like 20K forum.opensourceecology.org/search?Search=#6&Mode=like 16K forum.opensourceecology.org/search?Search=#7&Mode=like 16K forum.opensourceecology.org/search?Search=#8-High&Mode=like 28K forum.opensourceecology.org/search?Search=#8&Mode=like 16K forum.opensourceecology.org/search?Search=#9-Industrial&Mode=like 16K forum.opensourceecology.org/search?Search=#9&Mode=like 16K forum.opensourceecology.org/search?Search=#Another&Mode=like 16K forum.opensourceecology.org/search?Search=#apollo&Mode=like 16K forum.opensourceecology.org/search?Search=#Cloud&Mode=like 16K forum.opensourceecology.org/search?Search=#Edit&Mode=like 16K forum.opensourceecology.org/search?Search=#Fabiofranca&Mode=like 16K forum.opensourceecology.org/search?Search=#freecad&Mode=like 16K forum.opensourceecology.org/search?Search=#HUBCAMP&Mode=like 16K forum.opensourceecology.org/search?Search=#openhardware&Mode=like 16K forum.opensourceecology.org/search?Search=#opensourceecology&Mode=like 16K forum.opensourceecology.org/search?Search=#qi-hardware&Mode=like 16K forum.opensourceecology.org/search?Search=#qihardware&Mode=like 16K forum.opensourceecology.org/search?Search=#REDIRECT&Mode=like 16K forum.opensourceecology.org/search?Search=#toc&Mode=like 16K forum.opensourceecology.org/search?Search=#toctitle&Mode=like 24K forum.opensourceecology.org/themes 25M forum.opensourceecology.org/uploads 764K forum.opensourceecology.org/vanilla [root@hetzner2 wget]#
Thr Jan 25, 2018
- Marcin said that the forums initial validation is complete
- discovered that there was an 'oseforum' directory inside the forums docroot that had a duplicate copy of all the files in the docroot.
- I moved this director to
- I began a quick investigation on the low-hanging-fruit to harden Vanilla
- I didn't find any good guides. Lots of discussion about permissions, though. https://open.vanillaforums.com/discussion/748/security
- the fucking installation guide tells us to make the conf directory inside the docroot *and* be writeable by the webserver. That's a foundational red-flag! https://github.com/vanilla/vanilla/blob/master/README.md
- probably the most important step to securing the forums would be updating the core Vanilla version
- The version of Vanilla Forums that we're running is Version 2.0.18.1
- The current version of Vanilla Forums is 2.5
- The number of plugins that we're using tells me that an update of the Vanilla Forums core code probably going to be extremely non-trivial
[root@hetzner2 htdocs]# date Thu Jan 25 13:54:16 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/forum.opensourceecology.org/htdocs [root@hetzner2 htdocs]# grep 'EnabledPlugins' conf/config.php // EnabledPlugins $Configuration['EnabledPlugins']['HtmLawed'] = 'HtmLawed'; $Configuration['EnabledPlugins']['embedvanilla'] = 'embedvanilla'; $Configuration['EnabledPlugins']['Tagging'] = 'Tagging'; $Configuration['EnabledPlugins']['Flagging'] = 'Flagging'; $Configuration['EnabledPlugins']['Liked'] = 'Liked'; $Configuration['EnabledPlugins']['OpenID'] = 'OpenID'; $Configuration['EnabledPlugins']['Twitter'] = 'Twitter'; $Configuration['EnabledPlugins']['Sitemaps'] = 'Sitemaps'; $Configuration['EnabledPlugins']['GoogleSignIn'] = 'GoogleSignIn'; $Configuration['EnabledPlugins']['Facebook'] = 'Facebook'; $Configuration['EnabledPlugins']['FileUpload'] = 'FileUpload'; $Configuration['EnabledPlugins']['VanillaStats'] = 'VanillaStats'; $Configuration['EnabledPlugins']['EMailSubscribe'] = 'EMailSubscribe'; $Configuration['EnabledPlugins']['Emotify'] = 'Emotify'; $Configuration['EnabledPlugins']['VanillaInThisDiscussion'] = 'VanillaInThisDiscussion'; $Configuration['EnabledPlugins']['ProxyConnect'] = 'ProxyConnect'; $Configuration['EnabledPlugins']['ProxyConnectManual'] = 'ProxyConnectManualPlugin'; $Configuration['EnabledPlugins']['cleditor'] = 'cleditor'; $Configuration['EnabledPlugins']['TinyMCE'] = 'TinyMCE'; $Configuration['EnabledPlugins']['Categories2Menu'] = TRUE; $Configuration['EnabledPlugins']['SplitMerge'] = TRUE; $Configuration['EnabledPlugins']['Voting'] = TRUE; $Configuration['EnabledPlugins']['FeedDiscussions'] = TRUE; $Configuration['EnabledPlugins']['StopForumSpam'] = TRUE; $Configuration['EnabledPlugins']['Minify'] = TRUE; $Configuration['EnabledPlugins']['Cleanser'] = TRUE; $Configuration['EnabledPlugins']['RegistrationRestrictLogger'] = TRUE; [root@hetzner2 htdocs]#
- I found no obvious way to make the forums read-only from the application-side
- except an old extension that hasn't been updated since 2005 https://open.vanillaforums.com/addon/readonly-plugin
- of course, we could achieve read-only by:
- revoking all access privlidges to the db user = ${dbUser_hetzner2} = 'oseforums_user' && granting them SELECT only access
- And making all of the files/directories read-only
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
dbUser_hetzner2="oseforum_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
time nice mysql -uroot -p${mysqlPass} -sNe "REVOKE ALL ON ${dbName_hetzner2}.* FROM '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"
# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0550 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0440 {} \;
chown apache:apache-admins "${docrootDir_hetzner2}/conf/config.php"
chmod 0440 "${docrootDir_hetzner2}/conf/config.php"
- ran the above commands & confirmed that the permissions changed to SELECT-only in the db
[root@hetzner2 htdocs]# nice mysql -uroot -p${mysqlPass} mysql -sNe "select * from db where Db = 'oseforum_db';"
localhost oseforum_db oseforum_user Y N N N N N N N N N N N N N N N N N N
[root@hetzner2 htdocs]#
- attempted to login, but then I found that the fucking forums are leaking back to the user our server-side error message (in json), that includes the username & hostname of the db server in the response to our login query! https://stagingforum.opensourceecology.org/entry/signin
{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}
- And then it created a popup window leaking all this info!
FATAL ERROR IN: Gdn_Database.Query();
"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'" update GDN_User User set DateLastActive = :DateLastActive, DateUpdated = :DateUpdated, UpdateIPAddress = :UpdateIPAddress where UserID = :UserID LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php > 276: > 277: if (!is_object($PDOStatement)) { > 278: trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR); > 279: } else if ($PDOStatement->execute($InputParameters) === FALSE) { >>> 280: trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR); > 281: } > 282: } else { > 283: $PDOStatement = $this->Connection()->query($Sql); > 284: } BACKTRACE: [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start(); [/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator(); [/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();
- found the source of the logs being sent to the user as the line:
$Configuration['Garden']['Errors']['MasterView'] = 'deverror.master.php';
- fixed the log leaking by commenting-out the above line & replacing it with:
$Configuration['Garden']['Errors']['LogEnabled'] = TRUE; $Configuration['Garden']['Errors']['LogFile'] = '';
- the above setting forced writes to the apache-defined error logfile at /var/log/httpd/forum.opensourceecology.org/error_log
[root@hetzner2 httpd]# date Thu Jan 25 15:41:25 UTC 2018 [root@hetzner2 httpd]# pwd /var/log/httpd [root@hetzner2 httpd]# tail -f forum.opensourceecology.org/access_log forum.opensourceecology.org/error_log ... ==> forum.opensourceecology.org/error_log <== [Thu Jan 25 15:39:48.700738 2018] [:error] [pid 24269] [client 127.0.0.1:56072] [Garden] /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php, 280, Gdn_Database.Query(), UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User', update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID
- ugh, but the response is actually still leaking info
<h1>FATAL ERROR IN: Gdn_Database.Query();</h1>
<div class="AjaxError">"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'"
update GDN_User User set
DateLastActive = :DateLastActive,
DateUpdated = :DateUpdated,
UpdateIPAddress = :UpdateIPAddress
where UserID = :UserID
LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php
> 276:
> 277: if (!is_object($PDOStatement)) {
> 278: trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR);
> 279: } else if ($PDOStatement->execute($InputParameters) === FALSE) {
>>> 280: trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR);
> 281: }
> 282: } else {
> 283: $PDOStatement = $this->Connection()->query($Sql);
> 284: }
BACKTRACE:
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start();
[/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator();
[/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();
</div>
- unfortunately, after a single login attempt, all future queries of any kind produce the same "Bonk" generic error message now. It appears that the fix is to clear the cookie named "Vanilla" for the domain on the client-side. By default, this cookie sticks-around for 1 month.
# I changed the LogEnabled flag to FALSE. That helped, but then the logs disappeared from the apache logfile & the response still contained the error (though a shorter one)!
<pre>
{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}
- I'm pretty fucking over Vanilla. They seem to not give a shit about security.
- Note: the leaks above are nothing more than we've openly disclosed in our specs & docs. However, error messages often contain very sensitive info. Sometimes they contain passwords. Hence, the fact that this is being leaked in general is a huge red flag. The fact that it's considered "normal" behaviour is terrifying. The fact that I googled around for a fix of this issue, and the only thing I found was a developer suggesting to use a browser-side debugger to find the server-side error messages is mortifying. Sending server-side errors to the client should be a opt-in feature. It should not be the default. But Vanilla seems to have it as the baked-in default with not only no way to disable it, but also no obvious way (if at all?) to disable it. This tells me that they're either incompetent or just don't care. That's not what we want from the developers of our tools.## phpbb looks a bit better, they have at least a few blog posts tagged "security" https://blog.phpbb.com/tag/security/
- phpbb has a wiki with a guide recommending to move config.php out of the docroot. Not one of the guides I found for Vanilla mentioned that obvious requirement. They were occupied with permissions, and arguing if taking away the execute bit mattered to php files (read: it doesn't). https://wiki.phpbb.com/Hardening_Tips
- began downloading a static-content snapshot of our forums with wget
[root@hetzner2 wget]# date Thu Jan 25 16:49:39 UTC 2018 [root@hetzner2 wget]# pwd /var/tmp/deleteMeIn2019/oseoforum.20180125/wget [root@hetzner2 wget]# wget -r -p -e robots=off http://forum.opensourceecology.org
- I'm concerned that the output above may be insanely long, since people can permalink to comments within discussions. So effectively we'll store a single thread N times where N is the number of comments in the thread :(
- sent an email to Marcin asking if he was ready to fully archive our fourms to static content
- I discovered that my defined hardened permissions for Wordpress should be improved such that
- All of the files & directories that don't need rw permissions should not have write permissions. That's every file in a wordpress docroot except the folder "wp-content/uploads" and its subfiles/dirs.
- World permissions (not-user && not-group) for all files & directories inside the docroot (and including the docroot dir itself!) should be set to 0 for all files & all directories.
- Excluding 'wp-content/uploads/', these files should also not be owned by the user that runs a webserver (in cent, that's the 'apache' user). For even if the file is set to '0400', but it's owned by the 'apache' user, the 'apache' user can ignore the permissions & write to it anyway.
- Excluding 'wp-content/uploads/', all directories in the docroot (including the docroot dir itself!) should be owned by a group that contains the user that runs our webserver (in cent, that's the apache user). The permissions for this group must be not include write access for files or directories. For even if a file is set to '0040', but the containing directory is '0060', any user in the group that owns the directory can delete the existing file and replace it with a new file, effectively ignoring the read-only permission set for the file.
- currently, our documentation reads:
vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"
chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"
- but I believe the (untested) update should be:
vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"
chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0060 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0070 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"
- ...such that:
- the 'not-apache' user is a new user that doesn't run any software (ie: a daemon such as a web server) and whose shell is "/sbin/nologin" and home is "/dev/null".
- the apache user is now in the apache-admins group
- the result will now be that:
- ...such that:
- a compromised web server can no longer write data to a docroot (ie: adding malicious code to a php script) outside the 'wp-content/uploads/' directory
- for anyone to make changes to any files in the docroot (other than 'wp-content/uploads/'), they must be the root user. I think this is fair if they don't have the skills necessary to become root, they probably shouldn't fuck with the wp core files anyway.
- however, as with before, any user in the 'apache' group can read most files in the docroot. So, if we want an OSE Developer with ssh access to be able to access our server's files read-only, we should add them to the 'apache' group. If we trust them with passwords as well, we should additionally add them to the 'apache-admins' group.
Tue Jan 23, 2018
- found the best way to browse the git sourcecode of the mediawiki core at this URL https://phabricator.wikimedia.org/source/mediawiki/tags/master/
- for example, for v1.30.0 https://phabricator.wikimedia.org/source/mediawiki/browse/REL1_30/
- or for the 'vendor' directory https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/
- using composer to populate the 'vendor' directory does not seem like a practical options with our hardened php config.
- I attempted to download the actual tarball, copy-in the 'vendor' directory, update permissions, and then run the update
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
mkdir /var/tmp/mediawiki-1.30.0
pushd /var/tmp/mediawiki-1.30.0
wget https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz
tar -xzvf mediawiki-1.30.0.tar.gz
cd mediawiki-1.30.0
rm -rf /var/www/html/wiki.opensourceecology.org/htdocs/vendor
cp -r vendor /var/www/html/wiki.opensourceecology.org/htdocs/
# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"
/var/www/html/wiki.opensourceecology.org/htdocs/maintenance
php update.php
popd
- now I got a fatal php error
[Tue Jan 23 22:02:07.182288 2018] [:error] [pid 31837] [client 127.0.0.1:37506] PHP Fatal error: Class 'Wikimedia\\WrappedString' not found in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Gadgets/GadgetHooks.php on line 217
- I changed the require_once() to wfLoadExtension() in LocalSettings.php
- I found that the directory path for the "WrappedString.php" file is distinct for the vendors dirs in [a] the git repo vs [b] the extracted tarball release.
# the tarball has an src dir with the 2 files (WrappedString.php & WrappedStringList.php) [root@hetzner2 htdocs]# ls -lah /var/tmp/mediawiki-1.30.0/mediawiki-1.30.0/vendor/wikimedia/wrappedstring/src/ total 16K drwxr-xr-x 2 501 games 4.0K Dec 8 23:20 . drwxr-xr-x 3 501 games 4.0K Dec 8 23:20 .. -rw-r--r-- 1 501 games 3.5K Dec 8 23:20 WrappedStringList.php -rw-r--r-- 1 501 games 3.3K Dec 8 23:20 WrappedString.php # but the git repo has an src dir with 2 dirs = WrappedString & Wikimedia. Both of *these* dirs have the 2 files https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/wikimedia/wrappedstring/src/ [root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/ total 16K drwxr-xr-x 4 root root 4.0K Jan 22 21:22 . drwxr-xr-x 3 root root 4.0K Jan 22 21:22 .. drwxr-xr-x 2 root root 4.0K Jan 22 21:22 Wikimedia drwxr-xr-x 2 root root 4.0K Jan 22 21:22 WrappedString [root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/WrappedString/ total 16K drwxr-xr-x 2 root root 4.0K Jan 22 21:22 . drwxr-xr-x 4 root root 4.0K Jan 22 21:22 .. -rw-r--r-- 1 root root 98 Jan 22 21:22 WrappedStringList.php -rw-r--r-- 1 root root 90 Jan 22 21:22 WrappedString.php [root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/Wikimedia/ total 16K drwxr-xr-x 2 root root 4.0K Jan 22 21:22 . drwxr-xr-x 4 root root 4.0K Jan 22 21:22 .. -rw-r--r-- 1 root root 3.6K Jan 22 21:22 WrappedStringList.php -rw-r--r-- 1 root root 3.4K Jan 22 21:22 WrappedString.php [root@hetzner2 htdocs]#
Mon Jan 22, 2018
- Marcin found that editing posts on the stagingforum failed with "You don't have permission to access /vanilla/post/editdiscussion/X on this server."
- I coorelated this to a 403 error caused by modsecurity
--e381b03e-A--
[22/Jan/2018:14:02:11 +0000] WmXu49lQF2aUKXkcaav20AAAAAU 127.0.0.1 52050 127.0.0.1 8000
--e381b03e-B--
POST /vanilla/post/editdiscussion/237 HTTP/1.0
X-Real-IP: 173.234.159.236
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: stagingforum.opensourceecology.org
Content-Length: 3125
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: https://stagingforum.opensourceecology.org/vanilla/post/editdiscussion/237
Cookie: Vanilla=20082-1519221520%7C48d7a44f72d9d86b5e70a0da0e4b83c8%7C1516629520%7C20082%7C1519221520; Vanilla-Volatile=20082-1516802320%7C2271357054a693999ba33b7013e7c80e%7C1516629520%7C20082%7C1516802320
DNT: 1
X-Forwarded-For: 173.234.159.236, 127.0.0.1
X-Varnish: 34288
--e381b03e-C--
Discussion%2FTransientKey=82P0S34XB3Q0&Discussion%2Fhpt=&Discussion%2FDiscussionID=237&Discussion%2FDraftID=0&Discussion%2FName=Roof&Discussion%2FCategoryID=1&Discussion%2FBody=I+think+we+need+to+address+what+seems+to+be+a+hole+in+the+GVCS+-+constructing+a+good+roof.+Shelter+is+one+of+the+big+requirements+we+have+to+cover%2C+and+living+in+the+pacific+northwest%2C+I+am+painfully+aware+of+the+importance+and+costs+associated+with+a+roof.+Compressed+earth+blocks+cover%2C+to+a+decent+extent%2C+building+walls.+Floors+are+somewhat+covered+by+the+dimensional+saw+mill%2C+compressed+earth+blocks%2C+and+concrete.+However%2C+what+isn't+taken+into+account+yet+is+the+most+expensive+and+difficult+part+of+a+house+-+the+roof.%3Cbr%3E%3Cbr%3EI+think+figuring+out+a+good+roofing+solution+should+be+one+of+the+primary+focal+points+for+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2Fcomment%2F861%23Comment_861%22%3Efirst+wave%3C%2Fa%3E%3C%2Fu%3E+-+getting+the+basics+of+housing%2Ffood%2Fwater%2Fpower+up+and+running%2C+as+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F227%2Fbootstrapping-the-gvcs-priorities%22%3EGVCS+bootstrapping%3C%2Fa%3E%3C%2Fu%3E+thread.%3Cbr%3E%3Cbr%3EA+good+example+of+this+need+is+the+Factor+E+Farm+workshop+itself.+To+make+a+workshop+to+stage+the+development+of+the+GVCS%2C+they+were+able+to+make+part+of+the+structure+out+of+compressed+earth+block+columns.+However%2C+the+vast+majority+of+the+cost+of+the+structure+was+in+building+the+roof.+They+had+to+use+off-the-shelf+metal+roofing%2C+which+was+paid+for+through+donations%3A+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconstruction-time%2F%22%3Edonations+for+roofing+materials%3C%2Fa%3E%3C%2Fu%3E%2C+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%22%3Ebuilding+the+roof%3C%2Fa%3E%3C%2Fu%3E.%3Cbr%3E%3Cbr%3EAs+you+can+see%2C+building+a+roof+was+one+of+the+first+roadblocks%2C+and+was+solved+through+just+buying+one.%3Cbr%3E%26nbsp%3B%3Cbr%3EPerhaps+we+can+brainstorm+and+come+up+with+some+good+solutions.+I've+already+started+some+sections+and+given+some+background+requirements+on+the+wiki.+One+option+I+find+very+intriguing+myself+is+ferrocement.+You+can+see+it+being+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%23comments%22%3Ecomments%3C%2Fa%3E%3C%2Fu%3E+of+the+building+the+roof+blog+entry+for+the+farm.+I+made+a+thread+about+the+need+for+an+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F214%22%3Eopen+source+metal+lath+making+machine%3C%2Fa%3E%3C%2Fu%3E+with+roofing+as+one+of+its+applications.%3Cbr%3E%3Cbr%3EI've+started+a+wiki+entry+for+the+roof+here%3A+%3Cu%3Ehttp%3A%2F%2Fopenfarmtech.org%2Fwiki%2FRoof%3C%2Fu%3E+As+we+come+up+with+ideas+we+can+add+them.%3Cbr%3E%3Cbr%3EEdit%3A+This+is+Michael+attempting+to+edit+a+roof+post.%3Cbr%3E&Checkboxes%5B%5D=Announce&Checkboxes%5B%5D=Closed&Discussion%2FTags=&DeliveryType=VIEW&DeliveryMethod=JSON&Discussion/Save=Save
--e381b03e-F--
HTTP/1.1 403 Forbidden
Content-Length: 233
Connection: close
Content-Type: text/html; charset=iso-8859-1
--e381b03e-E--
--e381b03e-H--
Message: Access denied with code 403 (phase 2). Pattern match "\\W{4,}" at ARGS:Discussion/Body. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "37"] [id "960024"] [rev "2"] [msg "Meta-Character Anomaly Detection Alert - Repetative Non-Word Characters"] [data "Matched Data: > - found within ARGS:Discussion/Body: I think we need to address what seems to be a hole in the GVCS - constructing a good roof. Shelter is one of the big requirements we have to cover, and living in the pacific northwest, I am painfully aware of the importance and costs associated with a roof. Compressed earth blocks cover, to a decent extent, building walls. Floors are somewhat covered by the dimensional saw mill, compressed earth blocks, and concrete. However, what isn't taken into acc..."] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"]
Action: Intercepted (phase 2)
Stopwatch: 1516629731152710 755 (- - -)
Stopwatch2: 1516629731152710 755; combined=355, p1=104, p2=236, p3=0, p4=0, p5=14, sr=20, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"
--e381b03e-Z--
- realized that most of the whitelisting was done to just the 'wp-admin' dir, so I made it site-wide on the forums
- Catarina reported successfully that she was able to upload 3M images to fef && also connect via sftp using filezilla.
- marked fef migration as complete CHG-2018-01-03
- moved LocalSettings.php out of the document root per these hardening guides
[root@hetzner2 htdocs]# date Mon Jan 22 20:26:16 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/wiki.opensourceecology.org/htdocs [root@hetzner2 htdocs]# cat LocalSettings.php <?php # including separate file that contains the database password so that it is not stored within the document root. # For more info see: # * https://www.mediawiki.org/wiki/Manual:Security # * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening $docRoot = dirname( FILE ); require_once "$docRoot/../LocalSettings.php"; ?> [root@hetzner2 htdocs]#
- this also required me to change the original LocalSettings.php file (now outside the docroot in /var/www/html/wiki.opensourceecology.org) so that IP is inside of 'htdocs'
# If you customize your file layout, set $IP to the directory that contains
# the other MediaWiki files. It will be used as a base to locate files.
if( defined( 'MW_INSTALL_PATH' ) ) {
$IP = MW_INSTALL_PATH;
} else {
$IP = dirname( FILE ) . "/htdocs" ;
}
- decided to move the logo to 'images/' instead of just the base docroot to simplify future updates
- installed composer per the upgrade guide https://www.mediawiki.org/wiki/Manual:Upgrading
- which linked to https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries
- else, I got this error
MediaWiki 1.30 internal error Installing some external dependencies (e.g. via composer) is required. External dependencies MediaWiki now also has some external dependencies that need to be installed via composer or from a separate git repo. Please see mediawiki.org for help on installing the required components.
- and the actual install is
yum install composer
- and I had to add /usr/share/php/ to open_basedir in /etc/php.ini
- the composer steps themselves are included in the code block below
- found that several of our MediaWiki extensions are no longer maintained
- TreeAndMenu https://www.mediawiki.org/wiki/Extension:TreeAndMenu
- RSSReader https://www.mediawiki.org/wiki/Extension:RSS_Reader
- google-coop https://www.mediawiki.org/wiki/Extension_talk:Google_Custom_Search_Engine
- Mtag https://www.mediawiki.org/wiki/MediaWiki_and_LaTeX_on_a_host_with_shell_access
- PayPal https://www.mediawiki.org/wiki/Extension:Paypal
- Flattr https://www.mediawiki.org/wiki/Extension:Flattr
- JSWikiGnatt https://www.mediawiki.org/wiki/Extension:JSWikiGantt
- ProxyConnect https://www.mediawiki.org/wiki/Extension:ProxyConnect
- FreeMind https://www.mediawiki.org/wiki/Extension:FreeMind
- commented-out the entire function fnAddPersonalUrls() a section from LocalSettings.php that caused the following error
[Mon Jan 22 22:25:45.130218 2018] [:error] [pid 26913] [client 127.0.0.1:50368] PHP Fatal error: Call to undefined method User::getSkin() in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 421
- upgraded mediawiki install, including a switch to a git-backed core download to make future updates easier
- removed googleAnalytics require from LocalSettings.php, because privacy matters && awstats works.
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# clone the latest stable version of the Mediawiki core code from git
pushd ${backupDir_hetzner2}/current
time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git
pushd core
latestStableMediawikiVersion=`git tag -l | sort -V | grep -E '^[0-9\.]*$' | tail -n1`
git checkout "${latestStableMediawikiVersion}"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git
# extensions
pushd extensions
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd
#cp -r ../../../old/htdocs/extensions/TreeAndMenu .
#cp -r ../../../old/htdocs/extensions/RSSReader .
#cp -r ../../../old/htdocs/extensions/google-coop.php
#cp -r ../../../old/htdocs/extensions/Mtag.php
#cp -r ../../../old/htdocs/extensions/PayPal.php
#cp -r ../../../old/htdocs/extensions/Flattr
#cp -r ../../../old/htdocs/extensions/JSWikiGantt
#cp -r ../../../old/htdocs/extensions/ProxyConnect
popd
# skins
pushd skins
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd
popd
# copy core into the docroot
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
mkdir "${docrootDir_hetzner2}"
time nice rsync -av --progress core/ "${docrootDir_hetzner2}/"
# copy back essential content
rsync -av --progress "../old/LocalSettings.php" "${vhostDir_hetzner2}/"
rsync -av --progress "../old/htdocs/LocalSettings.php" "${docrootDir_hetzner2}/"
rsync -av --progress "../old/htdocs/images" "${docrootDir_hetzner2}/"
# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"
- having some 404 issues with images
<td><a href="http://www.shuttleworthfoundation.org/"><img alt="Shuttleworth funded-02---web.jpg" src="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg" width="200" height="55" srcset="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/300px-Shuttleworth_funded-02---web.jpg 1.5x, /wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/400px-Shuttleworth_funded-02---web.jpg 2x" /></a>
- the above links to (400) https://wiki.opensourceecology.org/wiki/Images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
- but it should link to (200) https://wiki.opensourceecology.org/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
- I logged into the hetzner1 konsleH config & tried to pull up the apache config there (since we can't see it in ssh on this annoying shared server). In the wui, click 'opensourceecology.org' -> Services -> Server Configuration -> Click the yellow wrench next to public_html -> Enhanced View:
Options All -Indexes
# Site-wide redirects
#Redirect 301 /community http://community.opensourceecology.org
#Redirect 301 /forum http://forum.opensourceecology.org
#Redirect 301 /weblog http://blog.opensourceecology.org
#RedirectMatch 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us
# Redirect 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us
# http://openfarmtech.org/w/index.php?title=CiviCRM_tech_notes&action=edit&redlink=1
# RedirectMatch 301 /CiviCRM http://openfarmtech.org/civicrm
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www\.
RewriteRule ^(.*)$ http://opensourceecology.org/$1 [R=301,L,QSA]
#RewriteRule ^civicrm/(.*)$ /community/civicrm/$1 [PT,L,QSA]
RewriteRule ^wiki/(.*)$ /w/index.php?title=$1 [PT,L,QSA]
#RewriteRule ^wiki/(.*:.*)$ wiki/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/(.*:.*)$ /w/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/*$ /w/index.php [L,QSA,T=text/html]
RewriteRule ^index.php/(.*)$ /wiki/$1?old-url=slash [R=permanent,L,QSA]
# RewriteRule ^/*$ /w/index.php [L,QSA]
# RewriteLog "/home/marcin_ose/openfarmtech.org/rewrite.log"
# RewriteLogLevel 3
# See http://docs.jboss.org/jbossweb/3.0.x/proxy-howto.html
# ProxyPass /OpenKM http://localhost:8080/OpenKM
# ProxyPassReverse /OpenKM http://localhost:8080/OpenKM
# RewriteRule ^OpenKM/(.*)$ http://localhost:8080/OpenKM/$1 [P]
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
- actually, just setting `$wgScriptPath = ""` in LocalSettings.php fixed everything!!
- attempting to login produces an error message
Database error A database query error has occurred. This may indicate a bug in the software.[WmZ1UeWiTZqK5j1xUV@YaQAAAAo] 2018-01-22 23:35:46: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"
- err, I found my password leaked in cleartext to /var/log/httpd/modsec_audit.log
- added "wpPassword" to sanitiseArg list in /etc/httpd/modexecurity.d/do_not_log_passwords.conf
- err, I found my password leaked in cleartext to /var/log/httpd/modsec_audit.log
[root@hetzner2 modsecurity.d]# date Mon Jan 22 23:46:06 UTC 2018 [root@hetzner2 modsecurity.d]# cd /etc/httpd [root@hetzner2 httpd]# cat modsecurity.d/do_not_log_passwords.conf ################################################################################ # File: modsecurity.d/do_not_log_passwords.conf # Version: 0.2 # Purpose: Defines a list of POST arguments that contain passwords and instructs # modsecurity to sanitise-out the values of these variables (with **s) # when logging to files (ie: /var/log/httpd/modsec_audit.log) # Author: Michael Altfield <michael@opensourceecology.org> # Created: 2017-12-16 # Updated: 2018-01-22 ################################################################################ SecAction "nolog,phase:2,id:131,sanitiseArg:password,sanitiseArg:Password,sanitiseArg:wpPassword,sanitiseArg:newPassword,sanitiseArg:oldPassword,sanitiseArg:pwd" [root@hetzner2 httpd]#
- it's possible that the login issue is due to a pending update.php execution required, but I'm getting an error when I attempt to run it
[root@hetzner2 maintenance]# php update.php PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715 PHP Warning: putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53 PHP Warning: is_dir(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/forum.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fsfile/TempFSFile.php on line 90 PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126 PHP Warning: ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127 PHP Notice: Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507 MediaWiki 1.30.0 Updater oojs/oojs-ui: 0.25.1 installed, 0.23.0 required. wikimedia/ip-set: 1.2.0 installed, 1.1.0 required. wikimedia/relpath: 2.1.1 installed, 2.0.0 required. wikimedia/remex-html: 1.0.2 installed, 1.0.1 required. wikimedia/running-stat: 1.2.1 installed, 1.1.0 required. wikimedia/wrappedstring: 2.3.0 installed, 2.2.0 required. Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies [root@hetzner2 maintenance]#
- I can't run composer, since it requires some unsafe, disabled php functions. That's why I chose to grab them via git (`git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git`). But apparently it gave me old versions?
- re-reading the error message, it says I have too new of versions!
- looks like the branches aren't maintained; we want REL1_30 but it isn't there
[root@hetzner2 vendor]# date Tue Jan 23 00:01:57 UTC 2018 [root@hetzner2 vendor]# pwd /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor [root@hetzner2 vendor]# git branch -r origin/HEAD -> origin/master origin/fundraising/REL1_27 origin/master origin/wmf/1.31.0-wmf.15 origin/wmf/1.31.0-wmf.16 origin/wmf/1.31.0-wmf.17 [root@hetzner2 vendor]#
Sat Jan 20, 2018
- the db backup & file backup of the wiki finished in a little over an hour
...
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$ time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
...
real 71m2.031s
user 17m36.700s
sys 1m38.868s
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$
- I initiated an scp of the data to hetnzer2
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
dbUser_hetzner2="osewiki_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# STEP 1: COPY FROM HETZNER1
mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/
- transfer of hetzner1 backups to hetzner2 finished after 8 minutes
- added wiki.opensourceecology.org A record in cloudflare DNS to 138.201.84.243
- disabled CDN/DNS proxy in cloudflare for the 'network' subdomain
- decreased TTL of all DNS entries to lowest time possible = 2 minutes for all entries except 'www', 'opensourceecology.org', and 'blog' (all 3 use the CDN for now, and are thus required to be set to TTL = 'automatic')
- created/updated necessary config files for forum.opensourceecology.org
- /etc/httpd/conf.d/00-wiki.opensourceecology.org.conf
- /etc/varnish/sites-enabled/wiki.opensourceecology.org
- /etc/varnish/all-vhosts.vcl
- /etc/nginx/conf.d/wiki.opensourceecology.org.conf
- created necessary dirs
- /var /www/html/wiki.opensourceecology.org/htdocs
- /var/log/httpd/wiki.opensourceecology.org
- /var/log/nginx/wiki.opensourceecology.org
- updated /etc/php.ini to include "/var/www/html/forum.opensourceecology.org/" in /etc/php.in's "open_basedir"
- added wiki SAN to opensourceecology.org cert
certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d stagingforum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org /bin/chmod 0400 /etc/letsencrypt/archive/*/pri* nginx -t && service nginx reload
- created osewiki_db mysql database on hetzner2 from dump from hetzner1
- copied htdocs files on hetzner2 from backup on hetzner1
- updated LocalSettings.php with new db name & credentials
- updated permissions of docroot files
- commented-out $wgScriptPath & $wgStylePath to remove the "/w/" subdir in LocalSettings.php
- removed the "/w/" subdir prefix from $wgLogo in LocalSettings.php
- commented-out $wgServer & $wgFavicon to prevent a 301 redirect back to the naked domain opensourceecology.org
- commented-out $wgCacheDirectory & $wgUseFileCache. And `rm -rf cache`. This directory is *supposed* to be located outside the docroot for security reasons. But we won't need this feature as we use varnish https://www.mediawiki.org/wiki/Manual:Security#File_permissions
- added "Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php" to /etc/httpd/conf.d/00-wiki.opensourceecology.org
- commented-out debugging lines in LocalSettings.php
################################################################
# Debugging
# error_reporting(E_ALL | E_STRICT);
# error_reporting(E_ALL);
# ini_set("display_errors", 1);
# $wgShowExceptionDetails = true; ## Verbose output to user
#$wgShowSQLErrors = true;
#$wgDebugDumpSql = true;
#$wgDebugLogFile = "/usr/home/osemain/public_html/logs/wiki-error.log";
#$wgDebugLogFile = "/home/oft_site/logs/wiki-error.log";
# $wgDebugRawPage = true;
# $wgDebugComments = true;
################################################################
- saw some errors for generating temporary thumbnails
Error creating thumbnail: Unable to save thumbnail to destination
- attempted setting the image directory to be writeable, to no avail
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
- began investigating the guide to install Mediawiki via git https://www.mediawiki.org/wiki/Download_from_Git
time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git
pushd ${backupDir_hetzner2}/current
- determined that the latest version of Mediawiki is v1.30.0, and that we're currently running v1.24.2 https://wiki.opensourceecology.org/wiki/Special:Version
Thr Jan 18, 2018
- fixed awstats cron job & config files
- confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
- did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site
* https://core.trac.wordpress.org/ticket/17048
- there's no good, robust solution.
* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress * https://wordpress.org/plugins/relative-url/ * https://wordpress.org/plugins/root-relative-urls/ * https://deluxeblogtips.com/relative-urls/ * http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/
- take away is:
- Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
- However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.
- attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
- using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf
[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf # allow large posts for image uploads #client_max_body_size 1k; #client_max_body_size 900k; client_max_body_size 10M;
- next, I got a 403 error from /wp-admin/async-upload.php
- /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:
- next, I got a 403 error from /wp-admin/async-upload.php
==> /var/log/httpd/fef.opensourceecology.org/error_log <== [Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"]
- as above, whitelisted rule IDs:
- 960915, multipart_unmatched_boundary
- 200003, multipart_unmatched_boundary
- as above, whitelisted rule IDs:
- moved '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org' to '/usr/home/osemain/noBackup/deleteMeIn2019/oswh_olddocroot'
- added a 301 redirect from 'http://opensourcewarehouse.org' to 'https://oswh.opensourceecology.org' in new file = '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org/index.php'
Sat Jan 13, 2018
- Meeting with Marcin
Fri Jan 12, 2018
- finished configuring oswh wp plugins
- determined that the oswh hack was responsible for injecting pop-ups for a "windows defender" phishing site on some subset of page loads
- gained access to oseforums, oseblog, osecivi users via ssh on our non-dedicated server (hetzner1)
- checked /etc/passwd & found another 8x org-specific users with home directories that I couldn't access still
- emailed hetzner for advice on how to gain ssh access to these users' home directories
addon:x:1011:1011:addontest.opensourceecology.org:/usr/home/addon:bin/false oseirc:x:1018:1018:irc.opensourceecology.org:/usr/home/oseirc:bin/false oseholla:x:1019:1019:holla.opensourceecology.org:/usr/home/oseholla:bin/false osesurv:x:1020:1020:survey.opensourceecology.org:/usr/home/osesurv:/bin/bash sandbox:x:1021:1021:sandbox.opensourceecology.org:/usr/home/sandbox:/bin/false microft:x:1022:1022:test.opensourceecology.org:/usr/home/microft:/bin/bash zabbix:x:118:121::/var/run/zabbix:/bin/false openswh:x:1023:1023:opensourcewarehouse.org:/usr/home/openswh:/bin/false
- created backup of oseforum's docroot (58M) & db (44M). Both sizes are bzip2'd.
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
dbUser_hetzner2="oseforum_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
Mon Jan 08, 2018
- tried installing the fresh version of the Eventor theme to the website running the old wp core, but it was still broken
- reverted to the docroot I backed-up before attempting to try the outdated wp core && installed _that_ Eventor theme to the fresh version 1.7 that was just downloaded, and the site actually worked!
- finally, attempted the wp-cli commands to update themes & plugins && install our minimal set of sec plugins. The site was still functional after
Fri Jan 05, 2018
- investigation of minor fef issues
Thr Jan 04, 2018
- downloaded the Eventor theme v 1.7, thanks to Simone's contact with Themes Kingdom
- Hetzner responded saying we can use WebFTP to uplaod to $HOME by clicking "the server at the top"
- Marcin responded with some issues with osemain's ephemeral clone
- Catarina found some linking issues in fef
- I brought the site down & did a string replacement for all occurrences of 'http://opensourceecology.org/fef' to '/', brought the site back up, and asked Catarina to check again
- updated documentation at Wordpress#replace_strings_everywhere_in_wp_database_backend
Wed Jan 03, 2018
- migrated fef to hetzner2 CHG-2018-01-03
- updated statuscake for obi to hit 'https://www.openbuildinginstitute.org'
- updated statuscake for fef to hit 'https://fef.opensourceecology.org'
- ensured that ssh was activated for all domains/users on our (apparently dedicated, per hetzner support) hetzner1 server (but without root access) via the konsoleh site -> click on the server -> Account Management -> SSH access -> Select domain (for each) -> Next
- the kosoleh wui only allowed editing files in the docroot, not the user's home-dir, which prevented me from actually adding my ssh pubic key to $HOME/.ssh/authorized_keys file
- I emailed hetzner support back asking if [a] they could just add my pub key to all our user account's authorized_keys files or [b] tell me how I could reset all the user's passwords
- oswh was cannibalized by a virus & is awaiting a fresh version of the theme. the forums is awaiting access to the user account. I'm now going to work on beginning the migration of osemain
- it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
- archive
- w
- logs
- mediawiki-1.24.2.extra
- the entire dir is 23G. Excluding the above, it's ~ 0.7G
- it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
####################
# run on hetzner1 #
####################
# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh
# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'
# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
dbUser_hetzner1="${mysqlUser_osemain}"
dbPass_hetzner1="${mysqlPass_osemain}"
# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}
# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}
- the gz-compressed tarball generated from above was 353M.
# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180103"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
dbUser_hetzner2="osemain_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
- created domain name 'osemain.opensourceecology.org' for testing the osemain site on hetzner2
- using above vars, I followed the guide to migrate the files & db data from hetzner1 to hetzner2 Wordpress#migrate_site_from_hetzner1_to_hetzner2
- created necessary files & dirs:
- /etc/httpd/conf.d/00-www.opensourceecology.org.conf
- /etc/varnish/sites-enabled/www.opensourceecology.org
- /etc/nginx/conf.d/www.opensourceecology.org.conf
- this file has a temporary override for the 'Host' header passed to varnish, since the staging url is going to be 'osemain.opensourceecology.org' but the prod site will be 'opensourceecology.org'
- /var/log/httpd/www.opensourceecology.org
- /var/log/nginx/www.opensourceecology.org
- updated necessary files
- /etc/varnish/all-vhosts.vcl
- /etc/php.ini
- finished setting up ephemeral clone of osemain at https://osemain.opensourceecology.org
- sent email to Marcin & Catarina for validation
Tue Jan 02, 2018
- got an email from Simone Cicero stating that she emailed Themes Kingdom for a clean copy of Eventor 1.7
- emailed back-and-forth with hetzner
- learned that the forums are in /usr/www/users/oseforum/
- learned that we have a bunch of users on this box, and it might even be dedicated just for us (though without root access)
osemain@dedi978:~$ grep 'ose' /etc/group users:x:100:osemain,addon,osecivi,oseblog,oseforum,oseirc,oseholla,osesurv,sandbox,microft,openswh osemain:x:1010: osecivi:x:1014: oseblog:x:1015: oseforum:x:1016: oseirc:x:1018: oseholla:x:1019: osesurv:x:1020:
- but I couldn't actually access the home dirs of the other users through 'osemain'
osemain@dedi978:~$ date Tue Jan 2 16:21:13 CET 2018 osemain@dedi978:~$ ls -lah /usr/home/ ls: cannot open directory /usr/home/: Permission denied osemain@dedi978:~$ ls -lah /usr/home/addon ls: cannot open directory /usr/home/addon: Permission denied osemain@dedi978:~$ ls -lah /usr/home/osecivi ls: cannot open directory /usr/home/osecivi: Permission denied osemain@dedi978:~$ ls -lah /usr/home/oseblog ls: cannot open directory /usr/home/oseblog: Permission denied osemain@dedi978:~$ ls -lah /usr/home/oseirc ls: cannot open directory /usr/home/oseirc: Permission denied osemain@dedi978:~$ ls -lah /usr/home/oseforum ls: cannot open directory /usr/home/oseforum: Permission denied osemain@dedi978:~$ ls -lah /usr/home/osesurv ls: cannot open directory /usr/home/osesurv: Permission denied osemain@dedi978:~$ ls -lah /usr/home/openswh ls: cannot open directory /usr/home/openswh: Permission denied
- so I asked hetzner support to add the 'osemain' user to all the other users groups listed above, and I asked them to find any other accounts that we own that I may have missed