Maltfield Log/2018 Q1

My work log from the year 2018. I intentionally made this verbose to make future admin's work easier when troubleshooting. The more keywords, error messages, etc that are listed in this log, the more helpful it will be for the future OSE Sysadmin.

See Also

1. Maltfield_Log
2. User:Maltfield
3. Special:Contributions/Maltfield

Fri Jan 26, 2018

1. Emailed Marcin about setting up PGP
2. Confirmed that the wget of the forums finished

[root@hetzner2 wget]# pwd
/var/tmp/deleteMeIn2019/oseoforum.20180125/wget
[root@hetzner2 ~]# wget -r -p -e robots=off http://forum.opensourceecology.org
...
FINISHED --2018-01-26 00:53:29--
Total wall clock time: 8h 39m 3s
Downloaded: 96387 files, 2.2G in 1m 33s (24.5 MB/s)
[root@hetzner2 wget]# du -sh forum.opensourceecology.org/*
48K     forum.opensourceecology.org/activity
264K    forum.opensourceecology.org/applications
300K    forum.opensourceecology.org/cache
3.0M    forum.opensourceecology.org/categories
27M     forum.opensourceecology.org/dashboard
1.4G    forum.opensourceecology.org/discussion
15M     forum.opensourceecology.org/discussions
873M    forum.opensourceecology.org/entry
148K    forum.opensourceecology.org/index.html
744K    forum.opensourceecology.org/plugins
72M     forum.opensourceecology.org/profile
12K     forum.opensourceecology.org/search
28K     forum.opensourceecology.org/search?Search=#000000&Mode=like
28K     forum.opensourceecology.org/search?Search=#0&Mode=like
16K     forum.opensourceecology.org/search?Search=#13Lifetime&Mode=like
16K     forum.opensourceecology.org/search?Search=#197187&Mode=like
32K     forum.opensourceecology.org/search?Search=#1&Mode=like
28K     forum.opensourceecology.org/search?Search=#2&Mode=like
16K     forum.opensourceecology.org/search?Search=#363636&Mode=like
28K     forum.opensourceecology.org/search?Search=#3&Mode=like
16K     forum.opensourceecology.org/search?Search=#458&Mode=like
20K     forum.opensourceecology.org/search?Search=#4&Mode=like
28K     forum.opensourceecology.org/search?Search=#5&Mode=like
20K     forum.opensourceecology.org/search?Search=#6&Mode=like
16K     forum.opensourceecology.org/search?Search=#7&Mode=like
16K     forum.opensourceecology.org/search?Search=#8-High&Mode=like
28K     forum.opensourceecology.org/search?Search=#8&Mode=like
16K     forum.opensourceecology.org/search?Search=#9-Industrial&Mode=like
16K     forum.opensourceecology.org/search?Search=#9&Mode=like
16K     forum.opensourceecology.org/search?Search=#Another&Mode=like
16K     forum.opensourceecology.org/search?Search=#apollo&Mode=like
16K     forum.opensourceecology.org/search?Search=#Cloud&Mode=like
16K     forum.opensourceecology.org/search?Search=#Edit&Mode=like
16K     forum.opensourceecology.org/search?Search=#Fabiofranca&Mode=like
16K     forum.opensourceecology.org/search?Search=#freecad&Mode=like
16K     forum.opensourceecology.org/search?Search=#HUBCAMP&Mode=like
16K     forum.opensourceecology.org/search?Search=#openhardware&Mode=like
16K     forum.opensourceecology.org/search?Search=#opensourceecology&Mode=like
16K     forum.opensourceecology.org/search?Search=#qi-hardware&Mode=like
16K     forum.opensourceecology.org/search?Search=#qihardware&Mode=like
16K     forum.opensourceecology.org/search?Search=#REDIRECT&Mode=like
16K     forum.opensourceecology.org/search?Search=#toc&Mode=like
16K     forum.opensourceecology.org/search?Search=#toctitle&Mode=like
24K     forum.opensourceecology.org/themes
25M     forum.opensourceecology.org/uploads
764K    forum.opensourceecology.org/vanilla
[root@hetzner2 wget]# 

Thr Jan 25, 2018

1. Marcin said that the forums initial validation is complete
2. discovered that there was an 'oseforum' directory inside the forums docroot that had a duplicate copy of all the files in the docroot.
   1. I moved this director to
3. I began a quick investigation on the low-hanging-fruit to harden Vanilla
   1. I didn't find any good guides. Lots of discussion about permissions, though. https://open.vanillaforums.com/discussion/748/security
   2. the fucking installation guide tells us to make the conf directory inside the docroot *and* be writeable by the webserver. That's a foundational red-flag! https://github.com/vanilla/vanilla/blob/master/README.md
   3. probably the most important step to securing the forums would be updating the core Vanilla version
      1. The version of Vanilla Forums that we're running is Version 2.0.18.1
      2. The current version of Vanilla Forums is 2.5
      3. The number of plugins that we're using tells me that an update of the Vanilla Forums core code probably going to be extremely non-trivial

[root@hetzner2 htdocs]# date Thu Jan 25 13:54:16 UTC 2018 [root@hetzner2 htdocs]# pwd /var/www/html/forum.opensourceecology.org/htdocs [root@hetzner2 htdocs]# grep 'EnabledPlugins' conf/config.php // EnabledPlugins $Configuration['EnabledPlugins']['HtmLawed'] = 'HtmLawed'; $Configuration['EnabledPlugins']['embedvanilla'] = 'embedvanilla'; $Configuration['EnabledPlugins']['Tagging'] = 'Tagging'; $Configuration['EnabledPlugins']['Flagging'] = 'Flagging'; $Configuration['EnabledPlugins']['Liked'] = 'Liked'; $Configuration['EnabledPlugins']['OpenID'] = 'OpenID'; $Configuration['EnabledPlugins']['Twitter'] = 'Twitter'; $Configuration['EnabledPlugins']['Sitemaps'] = 'Sitemaps'; $Configuration['EnabledPlugins']['GoogleSignIn'] = 'GoogleSignIn'; $Configuration['EnabledPlugins']['Facebook'] = 'Facebook'; $Configuration['EnabledPlugins']['FileUpload'] = 'FileUpload'; $Configuration['EnabledPlugins']['VanillaStats'] = 'VanillaStats'; $Configuration['EnabledPlugins']['EMailSubscribe'] = 'EMailSubscribe'; $Configuration['EnabledPlugins']['Emotify'] = 'Emotify'; $Configuration['EnabledPlugins']['VanillaInThisDiscussion'] = 'VanillaInThisDiscussion'; $Configuration['EnabledPlugins']['ProxyConnect'] = 'ProxyConnect'; $Configuration['EnabledPlugins']['ProxyConnectManual'] = 'ProxyConnectManualPlugin'; $Configuration['EnabledPlugins']['cleditor'] = 'cleditor'; $Configuration['EnabledPlugins']['TinyMCE'] = 'TinyMCE'; $Configuration['EnabledPlugins']['Categories2Menu'] = TRUE; $Configuration['EnabledPlugins']['SplitMerge'] = TRUE; $Configuration['EnabledPlugins']['Voting'] = TRUE; $Configuration['EnabledPlugins']['FeedDiscussions'] = TRUE; $Configuration['EnabledPlugins']['StopForumSpam'] = TRUE; $Configuration['EnabledPlugins']['Minify'] = TRUE; $Configuration['EnabledPlugins']['Cleanser'] = TRUE; $Configuration['EnabledPlugins']['RegistrationRestrictLogger'] = TRUE; [root@hetzner2 htdocs]#

1. I found no obvious way to make the forums read-only from the application-side
   1. except an old extension that hasn't been updated since 2005 https://open.vanillaforums.com/addon/readonly-plugin
   2. of course, we could achieve read-only by:
      1. revoking all access privlidges to the db user = ${dbUser_hetzner2} = 'oseforums_user' && granting them SELECT only access
      2. And making all of the files/directories read-only

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
 dbUser_hetzner2="oseforum_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

 time nice mysql -uroot -p${mysqlPass} -sNe "REVOKE ALL ON ${dbName_hetzner2}.* FROM '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost'; FLUSH PRIVILEGES;"

# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0550 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0440 {} \;
chown apache:apache-admins "${docrootDir_hetzner2}/conf/config.php"
chmod 0440 "${docrootDir_hetzner2}/conf/config.php"

1. ran the above commands & confirmed that the permissions changed to SELECT-only in the db

[root@hetzner2 htdocs]#  nice mysql -uroot -p${mysqlPass} mysql -sNe "select * from db where Db = 'oseforum_db';"
localhost       oseforum_db     oseforum_user   Y       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N       N
[root@hetzner2 htdocs]# 

1. attempted to login, but then I found that the fucking forums are leaking back to the user our server-side error message (in json), that includes the username & hostname of the db server in the response to our login query! https://stagingforum.opensourceecology.org/entry/signin

{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}

1. And then it created a popup window leaking all this info!

FATAL ERROR IN: Gdn_Database.Query();
"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'" update GDN_User User set DateLastActive = :DateLastActive, DateUpdated = :DateUpdated, UpdateIPAddress = :UpdateIPAddress where UserID = :UserID LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php > 276: > 277: if (!is_object($PDOStatement)) { > 278: trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR); > 279: } else if ($PDOStatement->execute($InputParameters) === FALSE) { >>> 280: trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR); > 281: } > 282: } else { > 283: $PDOStatement = $this->Connection()->query($Sql); > 284: } BACKTRACE: [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query(); [/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save(); [/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start(); [/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator(); [/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once(); 

1. found the source of the logs being sent to the user as the line:

$Configuration['Garden']['Errors']['MasterView'] = 'deverror.master.php'; 

1. fixed the log leaking by commenting-out the above line & replacing it with:

$Configuration['Garden']['Errors']['LogEnabled']                = TRUE;                                                                                                     
$Configuration['Garden']['Errors']['LogFile']                   = '';          

1. the above setting forced writes to the apache-defined error logfile at /var/log/httpd/forum.opensourceecology.org/error_log

[root@hetzner2 httpd]# date
Thu Jan 25 15:41:25 UTC 2018
[root@hetzner2 httpd]# pwd
/var/log/httpd
[root@hetzner2 httpd]# tail -f forum.opensourceecology.org/access_log forum.opensourceecology.org/error_log 
...
==> forum.opensourceecology.org/error_log <==
[Thu Jan 25 15:39:48.700738 2018] [:error] [pid 24269] [client 127.0.0.1:56072] [Garden] /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php, 280, Gdn_Database.Query(), UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User', update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID

1. ugh, but the response is actually still leaking info

<h1>FATAL ERROR IN: Gdn_Database.Query();</h1>
<div class="AjaxError">"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'"
update GDN_User User set 
 DateLastActive = :DateLastActive,
 DateUpdated = :DateUpdated,
 UpdateIPAddress = :UpdateIPAddress
where UserID = :UserID
LOCATION: /var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php
> 276: 
> 277:          if (!is_object($PDOStatement)) {
> 278:             trigger_error(ErrorMessage('PDO Statement failed to prepare', $this->ClassName, 'Query', $this->GetPDOErrorMessage($this->Connection()->errorInfo())), E_USER_ERROR);
> 279:          } else if ($PDOStatement->execute($InputParameters) === FALSE) {
>>> 280:             trigger_error(ErrorMessage($this->GetPDOErrorMessage($PDOStatement->errorInfo()), $this->ClassName, 'Query', $Sql), E_USER_ERROR);
> 281:          }
> 282:       } else {
> 283:          $PDOStatement = $this->Connection()->query($Sql);
> 284:       }
BACKTRACE:
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php] PHP::Gdn_ErrorHandler();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.database.php 280] PHP::trigger_error();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1650] Gdn_Database->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/library/database/class.sqldriver.php 1619] Gdn_SQLDriver->Query();
[/var/www/html/forum.opensourceecology.org/htdocs/applications/dashboard/models/class.usermodel.php 865] Gdn_SQLDriver->Put();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.session.php 307] UserModel->Save();
[/var/www/html/forum.opensourceecology.org/htdocs/library/core/class.auth.php 36] Gdn_Session->Start();
[/var/www/html/forum.opensourceecology.org/htdocs/bootstrap.php 168] Gdn_Auth->StartAuthenticator();
[/var/www/html/forum.opensourceecology.org/htdocs/index.php 41] PHP::require_once();
</div>

1. 1. unfortunately, after a single login attempt, all future queries of any kind produce the same "Bonk" generic error message now. It appears that the fix is to clear the cookie named "Vanilla" for the domain on the client-side. By default, this cookie sticks-around for 1 month.

# I changed the LogEnabled flag to FALSE. That helped, but then the logs disappeared from the apache logfile & the response still contained the error (though a shorter one)!
<pre>
{"Code":256,"Exception":"UPDATE command denied to user 'oseforum_user'@'localhost' for table 'GDN_User'|Gdn_Database|Query|update GDN_User User set \n DateLastActive = :DateLastActive,\n DateUpdated = :DateUpdated,\n UpdateIPAddress = :UpdateIPAddress\nwhere UserID = :UserID"}

1. I'm pretty fucking over Vanilla. They seem to not give a shit about security.
   1. Note: the leaks above are nothing more than we've openly disclosed in our specs & docs. However, error messages often contain very sensitive info. Sometimes they contain passwords. Hence, the fact that this is being leaked in general is a huge red flag. The fact that it's considered "normal" behaviour is terrifying. The fact that I googled around for a fix of this issue, and the only thing I found was a developer suggesting to use a browser-side debugger to find the server-side error messages is mortifying. Sending server-side errors to the client should be a opt-in feature. It should not be the default. But Vanilla seems to have it as the baked-in default with not only no way to disable it, but also no obvious way (if at all?) to disable it. This tells me that they're either incompetent or just don't care. That's not what we want from the developers of our tools.## phpbb looks a bit better, they have at least a few blog posts tagged "security" https://blog.phpbb.com/tag/security/
   2. phpbb has a wiki with a guide recommending to move config.php out of the docroot. Not one of the guides I found for Vanilla mentioned that obvious requirement. They were occupied with permissions, and arguing if taking away the execute bit mattered to php files (read: it doesn't). https://wiki.phpbb.com/Hardening_Tips
2. began downloading a static-content snapshot of our forums with wget

[root@hetzner2 wget]# date
Thu Jan 25 16:49:39 UTC 2018
[root@hetzner2 wget]# pwd
/var/tmp/deleteMeIn2019/oseoforum.20180125/wget
[root@hetzner2 wget]# wget -r -p -e robots=off http://forum.opensourceecology.org

1. 1. I'm concerned that the output above may be insanely long, since people can permalink to comments within discussions. So effectively we'll store a single thread N times where N is the number of comments in the thread :(
2. sent an email to Marcin asking if he was ready to fully archive our fourms to static content

1. I discovered that my defined hardened permissions for Wordpress should be improved such that
   1. All of the files & directories that don't need rw permissions should not have write permissions. That's every file in a wordpress docroot except the folder "wp-content/uploads" and its subfiles/dirs.
   2. World permissions (not-user && not-group) for all files & directories inside the docroot (and including the docroot dir itself!) should be set to 0 for all files & all directories.
   3. Excluding 'wp-content/uploads/', these files should also not be owned by the user that runs a webserver (in cent, that's the 'apache' user). For even if the file is set to '0400', but it's owned by the 'apache' user, the 'apache' user can ignore the permissions & write to it anyway.
   4. Excluding 'wp-content/uploads/', all directories in the docroot (including the docroot dir itself!) should be owned by a group that contains the user that runs our webserver (in cent, that's the apache user). The permissions for this group must be not include write access for files or directories. For even if a file is set to '0040', but the containing directory is '0060', any user in the group that owns the directory can delete the existing file and replace it with a new file, effectively ignoring the read-only permission set for the file.
   5. currently, our documentation reads:

vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"

chown -R apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0750 {} \;
find "${vhostDir}" -type f -exec chmod 0640 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0660 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0440 "${vhostDir}/wp-config.php"

1. 1. but I believe the (untested) update should be:

vhostDir="/var/www/html/www.opensourceecology.org"
wpDocroot="${vhostDir}/htdocs"

chown -R not-apache:apache "${vhostDir}"
find "${vhostDir}" -type d -exec chmod 0050 {} \;
find "${vhostDir}" -type f -exec chmod 0040 {} \;
find "${wpDocroot}/wp-content" -type f -exec chmod 0060 {} \;
find "${wpDocroot}/wp-content" -type d -exec chmod 0070 {} \;
chown not-apache:apache-admins "${vhostDir}/wp-config.php"
chmod 0040 "${vhostDir}/wp-config.php"

1. 1. ...such that:
      1. the 'not-apache' user is a new user that doesn't run any software (ie: a daemon such as a web server) and whose shell is "/sbin/nologin" and home is "/dev/null".
      2. the apache user is now in the apache-admins group
   2. the result will now be that:
2. a compromised web server can no longer write data to a docroot (ie: adding malicious code to a php script) outside the 'wp-content/uploads/' directory
3. for anyone to make changes to any files in the docroot (other than 'wp-content/uploads/'), they must be the root user. I think this is fair if they don't have the skills necessary to become root, they probably shouldn't fuck with the wp core files anyway.
   1. however, as with before, any user in the 'apache' group can read most files in the docroot. So, if we want an OSE Developer with ssh access to be able to access our server's files read-only, we should add them to the 'apache' group. If we trust them with passwords as well, we should additionally add them to the 'apache-admins' group.

Tue Jan 23, 2018

1. found the best way to browse the git sourcecode of the mediawiki core at this URL https://phabricator.wikimedia.org/source/mediawiki/tags/master/
   1. for example, for v1.30.0 https://phabricator.wikimedia.org/source/mediawiki/browse/REL1_30/
   2. or for the 'vendor' directory https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/
2. using composer to populate the 'vendor' directory does not seem like a practical options with our hardened php config.
3. I attempted to download the actual tarball, copy-in the 'vendor' directory, update permissions, and then run the update

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

mkdir /var/tmp/mediawiki-1.30.0
pushd /var/tmp/mediawiki-1.30.0

wget https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz
tar -xzvf mediawiki-1.30.0.tar.gz

cd mediawiki-1.30.0
rm -rf /var/www/html/wiki.opensourceecology.org/htdocs/vendor
cp -r vendor /var/www/html/wiki.opensourceecology.org/htdocs/

# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"

/var/www/html/wiki.opensourceecology.org/htdocs/maintenance
php update.php

popd

1. now I got a fatal php error

[Tue Jan 23 22:02:07.182288 2018] [:error] [pid 31837] [client 127.0.0.1:37506] PHP Fatal error:  Class 'Wikimedia\\WrappedString' not found in /var/www/html/wiki.opensourceecology.org/htdocs/extensions/Gadgets/GadgetHooks.php on line 217

1. 1. I changed the require_once() to wfLoadExtension() in LocalSettings.php
   2. I found that the directory path for the "WrappedString.php" file is distinct for the vendors dirs in [a] the git repo vs [b] the extracted tarball release.

# the tarball has an src dir with the 2 files (WrappedString.php & WrappedStringList.php)
[root@hetzner2 htdocs]# ls -lah /var/tmp/mediawiki-1.30.0/mediawiki-1.30.0/vendor/wikimedia/wrappedstring/src/
total 16K
drwxr-xr-x 2 501 games 4.0K Dec  8 23:20 .
drwxr-xr-x 3 501 games 4.0K Dec  8 23:20 ..
-rw-r--r-- 1 501 games 3.5K Dec  8 23:20 WrappedStringList.php
-rw-r--r-- 1 501 games 3.3K Dec  8 23:20 WrappedString.php

# but the git repo has an src dir with 2 dirs = WrappedString & Wikimedia. Both of *these* dirs have the 2 files https://phabricator.wikimedia.org/diffusion/MWVD/browse/master/wikimedia/wrappedstring/src/
[root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/
total 16K
drwxr-xr-x 4 root root 4.0K Jan 22 21:22 .
drwxr-xr-x 3 root root 4.0K Jan 22 21:22 ..
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 Wikimedia
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 WrappedString
[root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/WrappedString/
total 16K
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 .
drwxr-xr-x 4 root root 4.0K Jan 22 21:22 ..
-rw-r--r-- 1 root root   98 Jan 22 21:22 WrappedStringList.php
-rw-r--r-- 1 root root   90 Jan 22 21:22 WrappedString.php
[root@hetzner2 htdocs]# ls -lah /var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor/wikimedia/wrappedstring/src/Wikimedia/
total 16K
drwxr-xr-x 2 root root 4.0K Jan 22 21:22 .
drwxr-xr-x 4 root root 4.0K Jan 22 21:22 ..
-rw-r--r-- 1 root root 3.6K Jan 22 21:22 WrappedStringList.php
-rw-r--r-- 1 root root 3.4K Jan 22 21:22 WrappedString.php
[root@hetzner2 htdocs]# 

Mon Jan 22, 2018

1. Marcin found that editing posts on the stagingforum failed with "You don't have permission to access /vanilla/post/editdiscussion/X on this server."
   1. I coorelated this to a 403 error caused by modsecurity

--e381b03e-A--
[22/Jan/2018:14:02:11 +0000] WmXu49lQF2aUKXkcaav20AAAAAU 127.0.0.1 52050 127.0.0.1 8000
--e381b03e-B--
POST /vanilla/post/editdiscussion/237 HTTP/1.0
X-Real-IP: 173.234.159.236
X-Forwarded-Proto: https
X-Forwarded-Port: 443
Host: stagingforum.opensourceecology.org
Content-Length: 3125
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:48.0) Gecko/20100101 Firefox/48.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
Referer: https://stagingforum.opensourceecology.org/vanilla/post/editdiscussion/237
Cookie: Vanilla=20082-1519221520%7C48d7a44f72d9d86b5e70a0da0e4b83c8%7C1516629520%7C20082%7C1519221520; Vanilla-Volatile=20082-1516802320%7C2271357054a693999ba33b7013e7c80e%7C1516629520%7C20082%7C1516802320
DNT: 1
X-Forwarded-For: 173.234.159.236, 127.0.0.1
X-Varnish: 34288

--e381b03e-C--
Discussion%2FTransientKey=82P0S34XB3Q0&Discussion%2Fhpt=&Discussion%2FDiscussionID=237&Discussion%2FDraftID=0&Discussion%2FName=Roof&Discussion%2FCategoryID=1&Discussion%2FBody=I+think+we+need+to+address+what+seems+to+be+a+hole+in+the+GVCS+-+constructing+a+good+roof.+Shelter+is+one+of+the+big+requirements+we+have+to+cover%2C+and+living+in+the+pacific+northwest%2C+I+am+painfully+aware+of+the+importance+and+costs+associated+with+a+roof.+Compressed+earth+blocks+cover%2C+to+a+decent+extent%2C+building+walls.+Floors+are+somewhat+covered+by+the+dimensional+saw+mill%2C+compressed+earth+blocks%2C+and+concrete.+However%2C+what+isn't+taken+into+account+yet+is+the+most+expensive+and+difficult+part+of+a+house+-+the+roof.%3Cbr%3E%3Cbr%3EI+think+figuring+out+a+good+roofing+solution+should+be+one+of+the+primary+focal+points+for+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2Fcomment%2F861%23Comment_861%22%3Efirst+wave%3C%2Fa%3E%3C%2Fu%3E+-+getting+the+basics+of+housing%2Ffood%2Fwater%2Fpower+up+and+running%2C+as+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F227%2Fbootstrapping-the-gvcs-priorities%22%3EGVCS+bootstrapping%3C%2Fa%3E%3C%2Fu%3E+thread.%3Cbr%3E%3Cbr%3EA+good+example+of+this+need+is+the+Factor+E+Farm+workshop+itself.+To+make+a+workshop+to+stage+the+development+of+the+GVCS%2C+they+were+able+to+make+part+of+the+structure+out+of+compressed+earth+block+columns.+However%2C+the+vast+majority+of+the+cost+of+the+structure+was+in+building+the+roof.+They+had+to+use+off-the-shelf+metal+roofing%2C+which+was+paid+for+through+donations%3A+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconstruction-time%2F%22%3Edonations+for+roofing+materials%3C%2Fa%3E%3C%2Fu%3E%2C+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%22%3Ebuilding+the+roof%3C%2Fa%3E%3C%2Fu%3E.%3Cbr%3E%3Cbr%3EAs+you+can+see%2C+building+a+roof+was+one+of+the+first+roadblocks%2C+and+was+solved+through+just+buying+one.%3Cbr%3E%26nbsp%3B%3Cbr%3EPerhaps+we+can+brainstorm+and+come+up+with+some+good+solutions.+I've+already+started+some+sections+and+given+some+background+requirements+on+the+wiki.+One+option+I+find+very+intriguing+myself+is+ferrocement.+You+can+see+it+being+discussed+in+the+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fweblog%2F2010%2F11%2Fconclusion-of-building-for-2010%2F%23comments%22%3Ecomments%3C%2Fa%3E%3C%2Fu%3E+of+the+building+the+roof+blog+entry+for+the+farm.+I+made+a+thread+about+the+need+for+an+%3Cu%3E%3Ca+href%3D%22http%3A%2F%2Fopenfarmtech.org%2Fforum%2Fdiscussion%2F214%22%3Eopen+source+metal+lath+making+machine%3C%2Fa%3E%3C%2Fu%3E+with+roofing+as+one+of+its+applications.%3Cbr%3E%3Cbr%3EI've+started+a+wiki+entry+for+the+roof+here%3A+%3Cu%3Ehttp%3A%2F%2Fopenfarmtech.org%2Fwiki%2FRoof%3C%2Fu%3E+As+we+come+up+with+ideas+we+can+add+them.%3Cbr%3E%3Cbr%3EEdit%3A+This+is+Michael+attempting+to+edit+a+roof+post.%3Cbr%3E&Checkboxes%5B%5D=Announce&Checkboxes%5B%5D=Closed&Discussion%2FTags=&DeliveryType=VIEW&DeliveryMethod=JSON&Discussion/Save=Save
--e381b03e-F--
HTTP/1.1 403 Forbidden
Content-Length: 233
Connection: close
Content-Type: text/html; charset=iso-8859-1

--e381b03e-E--

--e381b03e-H--
Message: Access denied with code 403 (phase 2). Pattern match "\\W{4,}" at ARGS:Discussion/Body. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_40_generic_attacks.conf"] [line "37"] [id "960024"] [rev "2"] [msg "Meta-Character Anomaly Detection Alert - Repetative Non-Word Characters"] [data "Matched Data: > -  found within ARGS:Discussion/Body: I think we need to address what seems to be a hole in the GVCS - constructing a good roof. Shelter is one of the big requirements we have to cover, and living in the pacific northwest, I am painfully aware of the importance and costs associated with a roof. Compressed earth blocks cover, to a decent extent, building walls. Floors are somewhat covered by the dimensional saw mill, compressed earth blocks, and concrete. However, what isn't taken into acc..."] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"]
Action: Intercepted (phase 2)
Stopwatch: 1516629731152710 755 (- - -)
Stopwatch2: 1516629731152710 755; combined=355, p1=104, p2=236, p3=0, p4=0, p5=14, sr=20, sw=1, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.7.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"

--e381b03e-Z--

1. 1. realized that most of the whitelisting was done to just the 'wp-admin' dir, so I made it site-wide on the forums
2. Catarina reported successfully that she was able to upload 3M images to fef && also connect via sftp using filezilla.
   1. marked fef migration as complete CHG-2018-01-03
3. moved LocalSettings.php out of the document root per these hardening guides
   1. https://www.mediawiki.org/wiki/Manual:Security
   2. https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening

[root@hetzner2 htdocs]# date
Mon Jan 22 20:26:16 UTC 2018
[root@hetzner2 htdocs]# pwd
/var/www/html/wiki.opensourceecology.org/htdocs
[root@hetzner2 htdocs]# cat LocalSettings.php
<?php
# including separate file that contains the database password so that it is not stored within the document root.
# For more info see:
#  * https://www.mediawiki.org/wiki/Manual:Security
#  * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening

$docRoot = dirname( FILE );
require_once "$docRoot/../LocalSettings.php";
?>
[root@hetzner2 htdocs]# 

1. 1. this also required me to change the original LocalSettings.php file (now outside the docroot in /var/www/html/wiki.opensourceecology.org) so that IP is inside of 'htdocs'

# If you customize your file layout, set $IP to the directory that contains                                                                                     
# the other MediaWiki files. It will be used as a base to locate files.                                                                                         
if( defined( 'MW_INSTALL_PATH' ) ) {                                                                                                                            
   $IP = MW_INSTALL_PATH;                                                                                                                                       
} else {                                                                                                                                                        
   $IP = dirname( FILE ) . "/htdocs" ;                                                                                                                      
}

1. decided to move the logo to 'images/' instead of just the base docroot to simplify future updates
2. installed composer per the upgrade guide https://www.mediawiki.org/wiki/Manual:Upgrading
   1. which linked to https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries
   2. else, I got this error

MediaWiki 1.30 internal error

Installing some external dependencies (e.g. via composer) is required.
External dependencies

MediaWiki now also has some external dependencies that need to be installed via composer or from a separate git repo. Please see mediawiki.org for help on installing the required components.

1. 1. and the actual install is

yum install composer

1. 1. and I had to add /usr/share/php/ to open_basedir in /etc/php.ini
   2. the composer steps themselves are included in the code block below
2. found that several of our MediaWiki extensions are no longer maintained
   1. TreeAndMenu https://www.mediawiki.org/wiki/Extension:TreeAndMenu
   2. RSSReader https://www.mediawiki.org/wiki/Extension:RSS_Reader
   3. google-coop https://www.mediawiki.org/wiki/Extension_talk:Google_Custom_Search_Engine
   4. Mtag https://www.mediawiki.org/wiki/MediaWiki_and_LaTeX_on_a_host_with_shell_access
   5. PayPal https://www.mediawiki.org/wiki/Extension:Paypal
   6. Flattr https://www.mediawiki.org/wiki/Extension:Flattr
   7. JSWikiGnatt https://www.mediawiki.org/wiki/Extension:JSWikiGantt
   8. ProxyConnect https://www.mediawiki.org/wiki/Extension:ProxyConnect
   9. FreeMind https://www.mediawiki.org/wiki/Extension:FreeMind
3. commented-out the entire function fnAddPersonalUrls() a section from LocalSettings.php that caused the following error

[Mon Jan 22 22:25:45.130218 2018] [:error] [pid 26913] [client 127.0.0.1:50368] PHP Fatal error:  Call to undefined method User::getSkin() in /var/www/html/wiki.opensourceecology.org/LocalSettings.php on line 421

1. upgraded mediawiki install, including a switch to a git-backed core download to make future updates easier
2. removed googleAnalytics require from LocalSettings.php, because privacy matters && awstats works.

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# clone the latest stable version of the Mediawiki core code from git
pushd ${backupDir_hetzner2}/current
time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git
pushd core
latestStableMediawikiVersion=`git tag -l | sort -V | grep -E '^[0-9\.]*$' | tail -n1`
git checkout "${latestStableMediawikiVersion}"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git

# extensions
pushd extensions
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git

git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd

#cp -r ../../../old/htdocs/extensions/TreeAndMenu .
#cp -r ../../../old/htdocs/extensions/RSSReader .
#cp -r ../../../old/htdocs/extensions/google-coop.php
#cp -r ../../../old/htdocs/extensions/Mtag.php
#cp -r ../../../old/htdocs/extensions/PayPal.php
#cp -r ../../../old/htdocs/extensions/Flattr
#cp -r ../../../old/htdocs/extensions/JSWikiGantt
#cp -r ../../../old/htdocs/extensions/ProxyConnect
popd

# skins
pushd skins
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd

popd

# copy core into the docroot
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
mkdir "${docrootDir_hetzner2}"
time nice rsync -av --progress core/ "${docrootDir_hetzner2}/"

# copy back essential content
rsync -av --progress "../old/LocalSettings.php" "${vhostDir_hetzner2}/"
rsync -av --progress "../old/htdocs/LocalSettings.php" "${docrootDir_hetzner2}/"
rsync -av --progress "../old/htdocs/images" "${docrootDir_hetzner2}/"

# set permissions
chown -R apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0750 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0640 {} \;
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;
chown apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0440 "${vhostDir_hetzner2}/LocalSettings.php"
chown apache:apache-admins "${docrootDir_hetzner2}/LocalSettings.php"
chmod 0440 "${docrootDir_hetzner2}/LocalSettings.php"

1. having some 404 issues with images

<td><a href="http://www.shuttleworthfoundation.org/"><img alt="Shuttleworth funded-02---web.jpg" src="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg" width="200" height="55" srcset="/wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/300px-Shuttleworth_funded-02---web.jpg 1.5x, /wiki/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/400px-Shuttleworth_funded-02---web.jpg 2x" /></a>

1. 1. the above links to (400) https://wiki.opensourceecology.org/wiki/Images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
   2. but it should link to (200) https://wiki.opensourceecology.org/images/thumb/b/be/Shuttleworth_funded-02---web.jpg/200px-Shuttleworth_funded-02---web.jpg
2. I logged into the hetzner1 konsleH config & tried to pull up the apache config there (since we can't see it in ssh on this annoying shared server). In the wui, click 'opensourceecology.org' -> Services -> Server Configuration -> Click the yellow wrench next to public_html -> Enhanced View:

Options All -Indexes

# Site-wide redirects
#Redirect 301 /community http://community.opensourceecology.org
#Redirect 301 /forum http://forum.opensourceecology.org
#Redirect 301 /weblog http://blog.opensourceecology.org
#RedirectMatch 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us
# Redirect 301 /weblog/contact-us http://openfarmtech.org/wiki/Contact_us

# http://openfarmtech.org/w/index.php?title=CiviCRM_tech_notes&action=edit&redlink=1
# RedirectMatch 301 /CiviCRM http://openfarmtech.org/civicrm

RewriteEngine On

RewriteCond %{HTTP_HOST} ^www\.
RewriteRule ^(.*)$ http://opensourceecology.org/$1 [R=301,L,QSA]

#RewriteRule ^civicrm/(.*)$ /community/civicrm/$1 [PT,L,QSA]

RewriteRule ^wiki/(.*)$ /w/index.php?title=$1 [PT,L,QSA]
#RewriteRule ^wiki/(.*:.*)$  wiki/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/(.*:.*)$  /w/index.php?title=$1 [L,QSA]
RewriteRule ^wiki/*$ /w/index.php [L,QSA,T=text/html]
RewriteRule ^index.php/(.*)$ /wiki/$1?old-url=slash [R=permanent,L,QSA]
# RewriteRule ^/*$ /w/index.php [L,QSA]

# RewriteLog "/home/marcin_ose/openfarmtech.org/rewrite.log" 
# RewriteLogLevel 3

# See http://docs.jboss.org/jbossweb/3.0.x/proxy-howto.html
# ProxyPass /OpenKM  http://localhost:8080/OpenKM
# ProxyPassReverse /OpenKM  http://localhost:8080/OpenKM

# RewriteRule ^OpenKM/(.*)$ http://localhost:8080/OpenKM/$1 [P]



# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

1. 1. actually, just setting `$wgScriptPath = ""` in LocalSettings.php fixed everything!!
2. attempting to login produces an error message

Database error
A database query error has occurred. This may indicate a bug in the software.[WmZ1UeWiTZqK5j1xUV@YaQAAAAo] 2018-01-22 23:35:46: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"

1. 1. err, I found my password leaked in cleartext to /var/log/httpd/modsec_audit.log
      1. added "wpPassword" to sanitiseArg list in /etc/httpd/modexecurity.d/do_not_log_passwords.conf

[root@hetzner2 modsecurity.d]# date
Mon Jan 22 23:46:06 UTC 2018
[root@hetzner2 modsecurity.d]# cd /etc/httpd
[root@hetzner2 httpd]# cat modsecurity.d/do_not_log_passwords.conf 
################################################################################
# File:    modsecurity.d/do_not_log_passwords.conf
# Version: 0.2
# Purpose: Defines a list of POST arguments that contain passwords and instructs
#          modsecurity to sanitise-out the values of these variables (with **s)
#          when logging to files (ie: /var/log/httpd/modsec_audit.log)
# Author:  Michael Altfield <michael@opensourceecology.org>
# Created: 2017-12-16
# Updated: 2018-01-22
################################################################################
SecAction "nolog,phase:2,id:131,sanitiseArg:password,sanitiseArg:Password,sanitiseArg:wpPassword,sanitiseArg:newPassword,sanitiseArg:oldPassword,sanitiseArg:pwd"
[root@hetzner2 httpd]# 

1. it's possible that the login issue is due to a pending update.php execution required, but I'm getting an error when I attempt to run it

[root@hetzner2 maintenance]# php update.php 
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 674
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/maintenance/Maintenance.php on line 715
PHP Warning:  putenv() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/Setup.php on line 53
PHP Warning:  is_dir(): open_basedir restriction in effect. File(/tmp) is not within the allowed path(s): (/home/wp/.wp-cli:/usr/share/pear:/var/lib/php/tmp_upload:/var/lib/php/session:/var/www/html/www.openbuildinginstitute.org:/var/www/html/staging.openbuildinginstitute.org/:/var/www/html/www.opensourceecology.org/:/var/www/html/fef.opensourceecology.org/:/var/www/html/seedhome.openbuildinginstitute.org:/var/www/html/oswh.opensourceecology.org/:/var/www/html/forum.opensourceecology.org/:/var/www/html/wiki.opensourceecology.org/:/usr/share/php/Composer) in /var/www/html/wiki.opensourceecology.org/htdocs/includes/libs/filebackend/fsfile/TempFSFile.php on line 90
PHP Notice:  Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 126
PHP Warning:  ini_set() has been disabled for security reasons in /var/www/html/wiki.opensourceecology.org/htdocs/includes/session/PHPSessionHandler.php on line 127
PHP Notice:  Undefined index: SERVER_NAME in /var/www/html/wiki.opensourceecology.org/htdocs/includes/GlobalFunctions.php on line 1507
MediaWiki 1.30.0 Updater

oojs/oojs-ui: 0.25.1 installed, 0.23.0 required.
wikimedia/ip-set: 1.2.0 installed, 1.1.0 required.
wikimedia/relpath: 2.1.1 installed, 2.0.0 required.
wikimedia/remex-html: 1.0.2 installed, 1.0.1 required.
wikimedia/running-stat: 1.2.1 installed, 1.1.0 required.
wikimedia/wrappedstring: 2.3.0 installed, 2.2.0 required.
Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies
[root@hetzner2 maintenance]# 

1. 1. I can't run composer, since it requires some unsafe, disabled php functions. That's why I chose to grab them via git (`git clone https://gerrit.wikimedia.org/r/p/mediawiki/vendor.git`). But apparently it gave me old versions?
   2. re-reading the error message, it says I have too new of versions!
   3. looks like the branches aren't maintained; we want REL1_30 but it isn't there

[root@hetzner2 vendor]# date
Tue Jan 23 00:01:57 UTC 2018
[root@hetzner2 vendor]# pwd
/var/tmp/backups_for_migration_from_hetzner1/wiki_20180120/current/core/vendor
[root@hetzner2 vendor]# git branch -r
  origin/HEAD -> origin/master
  origin/fundraising/REL1_27
  origin/master
  origin/wmf/1.31.0-wmf.15
  origin/wmf/1.31.0-wmf.16
  origin/wmf/1.31.0-wmf.17
[root@hetzner2 vendor]# 

Sat Jan 20, 2018

1. the db backup & file backup of the wiki finished in a little over an hour

...
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$ time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}
...
real    71m2.031s
user    17m36.700s
sys     1m38.868s
osemain@dedi978:~/tmp/backups_for_migration_to_hetzner2/wiki_20180120/current$ 

1. I initiated an scp of the data to hetnzer2

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180120"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

1. transfer of hetzner1 backups to hetzner2 finished after 8 minutes
2. added wiki.opensourceecology.org A record in cloudflare DNS to 138.201.84.243
3. disabled CDN/DNS proxy in cloudflare for the 'network' subdomain
4. decreased TTL of all DNS entries to lowest time possible = 2 minutes for all entries except 'www', 'opensourceecology.org', and 'blog' (all 3 use the CDN for now, and are thus required to be set to TTL = 'automatic')
5. created/updated necessary config files for forum.opensourceecology.org
   1. /etc/httpd/conf.d/00-wiki.opensourceecology.org.conf
   2. /etc/varnish/sites-enabled/wiki.opensourceecology.org
   3. /etc/varnish/all-vhosts.vcl
   4. /etc/nginx/conf.d/wiki.opensourceecology.org.conf
6. created necessary dirs
   1. /var /www/html/wiki.opensourceecology.org/htdocs
   2. /var/log/httpd/wiki.opensourceecology.org
   3. /var/log/nginx/wiki.opensourceecology.org
7. updated /etc/php.ini to include "/var/www/html/forum.opensourceecology.org/" in /etc/php.in's "open_basedir"
8. added wiki SAN to opensourceecology.org cert

certbot -nv --expand --cert-name opensourceecology.org certonly -v --webroot -w /var/www/html/fef.opensourceecology.org/htdocs/ -d fef.opensourceecology.org -w /var/www/html/www.opensourceecology.org/htdocs -d osemain.opensourceecology.org -w /var/www/html/oswh.opensourceecology.org/htdocs/ -d oswh.opensourceecology.org -w /var/www/html/forum.opensourceecology.org/htdocs -d stagingforum.opensourceecology.org -w /var/www/html/wiki.opensourceecology.org/htdocs -d wiki.opensourceecology.org
/bin/chmod 0400 /etc/letsencrypt/archive/*/pri*
nginx -t && service nginx reload 

1. created osewiki_db mysql database on hetzner2 from dump from hetzner1
2. copied htdocs files on hetzner2 from backup on hetzner1
3. updated LocalSettings.php with new db name & credentials
4. updated permissions of docroot files
5. commented-out $wgScriptPath & $wgStylePath to remove the "/w/" subdir in LocalSettings.php
6. removed the "/w/" subdir prefix from $wgLogo in LocalSettings.php
7. commented-out $wgServer & $wgFavicon to prevent a 301 redirect back to the naked domain opensourceecology.org
8. commented-out $wgCacheDirectory & $wgUseFileCache. And `rm -rf cache`. This directory is *supposed* to be located outside the docroot for security reasons. But we won't need this feature as we use varnish https://www.mediawiki.org/wiki/Manual:Security#File_permissions
9. added "Alias /wiki /var/www/html/wiki.opensourceecology.org/htdocs/index.php" to /etc/httpd/conf.d/00-wiki.opensourceecology.org
10. commented-out debugging lines in LocalSettings.php

################################################################                                                                                            
# Debugging                                                                                                                                                 
# error_reporting(E_ALL | E_STRICT);                                                                                                                        
# error_reporting(E_ALL);                                                                                                                                   
# ini_set("display_errors", 1);                                                                                                                             
																																							
# $wgShowExceptionDetails = true; ## Verbose output to user                                                                                                 
																																							
#$wgShowSQLErrors = true;                                                                                                                                   
#$wgDebugDumpSql  = true;                                                                                                                                   
																																							
#$wgDebugLogFile = "/usr/home/osemain/public_html/logs/wiki-error.log";                                                                                     
#$wgDebugLogFile = "/home/oft_site/logs/wiki-error.log";                                                                                                    
# $wgDebugRawPage = true;                                                                                                                                   
# $wgDebugComments = true;                                                                                                                                  
################################################################                                                                                                                          

1. saw some errors for generating temporary thumbnails

Error creating thumbnail: Unable to save thumbnail to destination

1. 1. attempted setting the image directory to be writeable, to no avail

find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;

1. began investigating the guide to install Mediawiki via git https://www.mediawiki.org/wiki/Download_from_Git

time nice git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git

pushd ${backupDir_hetzner2}/current

1. determined that the latest version of Mediawiki is v1.30.0, and that we're currently running v1.24.2 https://wiki.opensourceecology.org/wiki/Special:Version

Thr Jan 18, 2018

1. fixed awstats cron job & config files
2. confirmed that a db dump includes image tags with the domain-name hard-coded in the href. that was put in-place by Wordpress's "Add Media" wui button. That's not good; the links should be relative in the db!
   1. did some research & found that wp core devs decided 7 years ago to keep absolute paths. This is especially bad for continuous integration or even a basic staging site

* https://core.trac.wordpress.org/ticket/17048

1. there's no good, robust solution.

* https://stackoverflow.com/questions/17187437/relative-urls-in-wordpress
* https://wordpress.org/plugins/relative-url/
* https://wordpress.org/plugins/root-relative-urls/
* https://deluxeblogtips.com/relative-urls/
* http://www.456bereastreet.com/archive/201010/how_to_make_wordpress_urls_root_relative/

1. take away is:
   1. Let wordpress do its thing. Don't waste effort fighting wp when it auto-inserts an absolute path.
   2. However, whenever you have to manually type a path in (ie: when configuring a widget, plugin nav bar, etc), please use a relative link.
2. attempted to fix the "Http error" reported by wordpress after attempting to upload a large image
   1. using the browser debugger, I saw that it was nginx that returned a 413 error. I fixed this by increasing 'client_max_body_size' to '10M' in /etc/nginx/nginx.conf

[root@hetzner2 dbChange.20180118_12:22:36]# grep -B 1 'client_max_body_size' /etc/nginx/nginx.conf
		# allow large posts for image uploads
		#client_max_body_size 1k;
		#client_max_body_size 900k;
		client_max_body_size 10M;

1. 1. next, I got a 403 error from /wp-admin/async-upload.php
      1. /var/log/httpd/fef.opensourceecology.org/error_log shows a modsecurity issue:

==> /var/log/httpd/fef.opensourceecology.org/error_log <==
[Thu Jan 18 14:56:25.263164 2018] [:error] [pid 27682] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Match of "eq 0" against "MULTIPART_UNMATCHED_BOUNDARY" required. [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "219"] [id "960915"] [rev "1"] [msg "Multipart parser detected a possible unmatched boundary."] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "8"] [accuracy "8"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"] [hostname "fef.opensourceecology.org"] [uri "/wp-admin/async-upload.php"] [unique_id "WmC1mU7FUiUY6HdrzSWfWgAAAAA"]  

1. 1. 1. as above, whitelisted rule IDs:
         1. 960915, multipart_unmatched_boundary
         2. 200003, multipart_unmatched_boundary
2. moved '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org' to '/usr/home/osemain/noBackup/deleteMeIn2019/oswh_olddocroot'
3. added a 301 redirect from 'http://opensourcewarehouse.org' to 'https://oswh.opensourceecology.org' in new file = '/usr/home/osemain/public_html/archive/addon-domains/opensourcewarehouse.org/index.php'

Sat Jan 13, 2018

1. Meeting with Marcin

Fri Jan 12, 2018

1. finished configuring oswh wp plugins
2. determined that the oswh hack was responsible for injecting pop-ups for a "windows defender" phishing site on some subset of page loads
3. gained access to oseforums, oseblog, osecivi users via ssh on our non-dedicated server (hetzner1)
4. checked /etc/passwd & found another 8x org-specific users with home directories that I couldn't access still
5. emailed hetzner for advice on how to gain ssh access to these users' home directories

addon:x:1011:1011:addontest.opensourceecology.org:/usr/home/addon:bin/false
oseirc:x:1018:1018:irc.opensourceecology.org:/usr/home/oseirc:bin/false
oseholla:x:1019:1019:holla.opensourceecology.org:/usr/home/oseholla:bin/false
osesurv:x:1020:1020:survey.opensourceecology.org:/usr/home/osesurv:/bin/bash
sandbox:x:1021:1021:sandbox.opensourceecology.org:/usr/home/sandbox:/bin/false
microft:x:1022:1022:test.opensourceecology.org:/usr/home/microft:/bin/bash
zabbix:x:118:121::/var/run/zabbix:/bin/false
openswh:x:1023:1023:opensourcewarehouse.org:/usr/home/openswh:/bin/false

1. created backup of oseforum's docroot (58M) & db (44M). Both sizes are bzip2'd.

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp="20180113"
backupDir_hetzner1="/usr/home/oseforum/tmp/backups_for_migration_to_hetzner2/oseforum_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/oseforum_${stamp}"
backupFileName_db_hetzner1="mysqldump_oseforum.${stamp}.sql.bz2"
backupFileName_files_hetzner1="oseforum_files.${stamp}.tar.gz"
dbName_hetzner1='oseforum'
dbName_hetzner2='oseforum_db'
 dbUser_hetzner2="oseforum_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/forum.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

Mon Jan 08, 2018

1. tried installing the fresh version of the Eventor theme to the website running the old wp core, but it was still broken
2. reverted to the docroot I backed-up before attempting to try the outdated wp core && installed _that_ Eventor theme to the fresh version 1.7 that was just downloaded, and the site actually worked!
3. finally, attempted the wp-cli commands to update themes & plugins && install our minimal set of sec plugins. The site was still functional after

Fri Jan 05, 2018

1. investigation of minor fef issues

Thr Jan 04, 2018

1. downloaded the Eventor theme v 1.7, thanks to Simone's contact with Themes Kingdom
2. Hetzner responded saying we can use WebFTP to uplaod to $HOME by clicking "the server at the top"
3. Marcin responded with some issues with osemain's ephemeral clone
4. Catarina found some linking issues in fef
   1. I brought the site down & did a string replacement for all occurrences of 'http://opensourceecology.org/fef' to '/', brought the site back up, and asked Catarina to check again
   2. updated documentation at Wordpress#replace_strings_everywhere_in_wp_database_backend

Wed Jan 03, 2018

1. migrated fef to hetzner2 CHG-2018-01-03
2. updated statuscake for obi to hit 'https://www.openbuildinginstitute.org'
3. updated statuscake for fef to hit 'https://fef.opensourceecology.org'
4. ensured that ssh was activated for all domains/users on our (apparently dedicated, per hetzner support) hetzner1 server (but without root access) via the konsoleh site -> click on the server -> Account Management -> SSH access -> Select domain (for each) -> Next
5. the kosoleh wui only allowed editing files in the docroot, not the user's home-dir, which prevented me from actually adding my ssh pubic key to $HOME/.ssh/authorized_keys file
6. I emailed hetzner support back asking if [a] they could just add my pub key to all our user account's authorized_keys files or [b] tell me how I could reset all the user's passwords
7. oswh was cannibalized by a virus & is awaiting a fresh version of the theme. the forums is awaiting access to the user account. I'm now going to work on beginning the migration of osemain
   1. it looks like the relevant files are heztern1:/usr/home/osemain/public_html/, except the following subdirs:
      1. archive
      2. w
      3. logs
      4. mediawiki-1.24.2.extra
   2. the entire dir is 23G. Excluding the above, it's ~ 0.7G

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
 dbUser_hetzner1="${mysqlUser_osemain}"
 dbPass_hetzner1="${mysqlPass_osemain}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}

1. 1. the gz-compressed tarball generated from above was 353M.

# DECLARE VARIABLES
source /root/backups/backup.settings
#stamp=`date +%Y%m%d`
stamp="20180103"
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
 dbUser_hetzner2="osemain_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

1. created domain name 'osemain.opensourceecology.org' for testing the osemain site on hetzner2
2. using above vars, I followed the guide to migrate the files & db data from hetzner1 to hetzner2 Wordpress#migrate_site_from_hetzner1_to_hetzner2
3. created necessary files & dirs:
   1. /etc/httpd/conf.d/00-www.opensourceecology.org.conf
   2. /etc/varnish/sites-enabled/www.opensourceecology.org
   3. /etc/nginx/conf.d/www.opensourceecology.org.conf
      1. this file has a temporary override for the 'Host' header passed to varnish, since the staging url is going to be 'osemain.opensourceecology.org' but the prod site will be 'opensourceecology.org'
   4. /var/log/httpd/www.opensourceecology.org
   5. /var/log/nginx/www.opensourceecology.org
4. updated necessary files
   1. /etc/varnish/all-vhosts.vcl
   2. /etc/php.ini
5. finished setting up ephemeral clone of osemain at https://osemain.opensourceecology.org
   1. sent email to Marcin & Catarina for validation

Tue Jan 02, 2018

1. got an email from Simone Cicero stating that she emailed Themes Kingdom for a clean copy of Eventor 1.7
2. emailed back-and-forth with hetzner
   1. learned that the forums are in /usr/www/users/oseforum/
   2. learned that we have a bunch of users on this box, and it might even be dedicated just for us (though without root access)

osemain@dedi978:~$ grep 'ose' /etc/group
users:x:100:osemain,addon,osecivi,oseblog,oseforum,oseirc,oseholla,osesurv,sandbox,microft,openswh
osemain:x:1010:
osecivi:x:1014:
oseblog:x:1015:
oseforum:x:1016:
oseirc:x:1018:
oseholla:x:1019:
osesurv:x:1020:

1. 1. but I couldn't actually access the home dirs of the other users through 'osemain'

osemain@dedi978:~$ date
Tue Jan  2 16:21:13 CET 2018
osemain@dedi978:~$ ls -lah /usr/home/
ls: cannot open directory /usr/home/: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/addon
ls: cannot open directory /usr/home/addon: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osecivi
ls: cannot open directory /usr/home/osecivi: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseblog
ls: cannot open directory /usr/home/oseblog: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseirc
ls: cannot open directory /usr/home/oseirc: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/oseforum
ls: cannot open directory /usr/home/oseforum: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/osesurv
ls: cannot open directory /usr/home/osesurv: Permission denied
osemain@dedi978:~$ ls -lah /usr/home/openswh
ls: cannot open directory /usr/home/openswh: Permission denied

1. 1. so I asked hetzner support to add the 'osemain' user to all the other users groups listed above, and I asked them to find any other accounts that we own that I may have missed