Status

2018-05-25 01:00 UTC

A fresh post-migration backup completed successfully. I then updated every user account's password to expire in the past, so they will have to update their password at next login. Then I sent an email to everyone who has created >5 edits (~500 people) telling them to change their passwords & linking to this page for more info.

This change is a success!

2018-05-25 00:20 UTC

I removed the banner from the top, but I went to create a backup before I executed the sql command to force all users to reset their password.

Unfortunately, the disks filled due to a logic error in the backup script after it was restructured to encrypt our backups following the dreamhost hack. The result was that yesterday's backups got included in today backups. Then today's in tomorrow's, etc. Yeah, it exploded & filled our disks. I deleted the backup 'sync' dirs, patched the script, and re kicked-off the backups.

Hopefully the backups will finish in a couple hours, and I can proceed with the password reset, email our users, and call this a successful change.

2018-05-24 23:00 UTC

Marcin & I agreed that the migration is a success. We'll keep it.

I pointed opensourceecology.org to hetzner2 (138.201.84.243), and updated the nginx vhost to redirect queries going to the naked domain starting with '/wiki/' to 'wiki.opensourceecology.org/wiki/...'. That effectively makes our site much more live.

I had to enable 'ini' files for upload per Marcin's requirements. It's some 3d printer "initialization" file, not to be confused with windows-ish the config files.

The one outstanding issue is that Marcin is no longer recieving emails from our server when someone requests an account. We can, in fact, find them in the Special page to approve->create their new user. We confirmed that the user _does_ get an email with their new temp password from the server. It's just that the email notifying Marcin that there was a new request for an account doesn't arrive. This is annoying, but not a blocker. Further investigation is needed.

Now I just need to remove the banner from the top, email all our users, and force them to reset their passwords.

2018-05-24 21:00 UTC

Marcin got back to me. Everything validated properly, except failure to upload freecad files due to a mime type error. I had a syntax issue in my CHG ticket's commands when echo'ing the encoding into the mime.types file. I've fixed that and confirmed that the freecad uploaded without issues.

Marcin ran to a meeting, but I think we can just about call this a success.

2018-05-24 16:40 UTC

The old site is down and the new site is up. I was able to log-in & configure my 2FA settings. I've emailed Marcin asking for validation.

The site at this moment should be recognized as being ephemeral. If we unearth any issues warranting it, we may destroy it & revert to the old site. Thus, I've added a big NOTICE on the top of all pages indicating to users that they should not make changes to the site.

2018-05-24 14:00 UTC

I started this change.

2018-05-23 21:00 UTC

I got a green light from Marcin that we're ready for the validation. There were a few minor "won't fix" items, but we should be ready.

The migration is long, and it's now 16:00 local time, so I'll start first thing tomorrow.

2018-05-23 19:00 UTC

Last night Marcin finished the validation. He found several issues. I've since fixed many of them. We just had a meeting, and we resolved most of the items, but I'm still trying to fix some of the File Upload related issues. We may or may not be able to proceed with the migration today.

2018-05-22

Marcin hasn't had time to finish the wiki validation (I've also stole much of his time for the higher-priority task of getting him an encrypted backup of his secret keys for ose server stuff at FeF), so we're delaying this migration until tomorrow, Wed May 23.

2018-05-18

Marcin set a tentative date of the migration to Tuesday 2018-05-22 pending the completion of validation of the staging site.

Purpose

This change does the following for our wiki

entirely migrate the wiki site from hetzner1 (shared hosting) to hetzner2 (dedicated hosting)
changes the domain from 'http://opensourceecology.org/wiki/' to 'https://wiki.opensourceecology.org/'
update core mediawiki from v1.24.2 (released 2014-09-19) to v1.30.0 (released 2017-12-12)
installs many of the extensions using `git clone ...`, making it easier to update in the future
updates the vhost config to block all access to ".*wp-login.php"
updates the vhost config to block access to git files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
enable https via nginx
enable cache via varnish
disable cache & ddos protection on cloudflare
update extension 'Confirm User Accounts' to 4fe25f7
update extension 'Confrim Edit' from 1.3 to 1.5.0
update extension 'Interwiki' from 3.0 20140719 to 3.1 20160307
update extension 'Nuke' from 1.2.0 to 1.3.0
update extension 'Replace Text' from 1.0 to 1.2 (4426752)
update extension 'User Merge' from 1.9.0 to 1.10.1 (4546537)
update extension 'Widgets' from 0.8.10 to 1.3.0 (fce5acc)
update extension 'CategoryTree' to (850c018)
removed extension 'Flattr'
removed extension 'Google Co-op Extension'
removed extension 'IpbWiki Paypal'
removed extension 'JSWikiGantt'
removed extension 'RSS Reader'
removed extension 'TreeAndMenu'
removed extension 'ProxyConnect'
added new extension 'OATHAuth' version 0.2.2 (bed2e4b)
moves LocalSettings.php outside the docroot, replaces the existing LocalSettings.php with a simple file that does a php include of the LocalSettings.php file outside the docroot.
moves the ose logo into the uploads directory
does a sed text replacement within the db data for all http strings to use https instead for 'https://www.youtube.com/embed/', 'https://static.issuu.com/webembed/', 'https://scrumy.com/', 'https://embed.ted.com/', & 'https://player.vimeo.com/'
reduces the privileges of the wiki user on the db to only SELECT, INSERT, UPDATE, & DELETE
adds an additional "superuser" db user with all permissions on the db for maintenance scripts (creds stored in keepass, not on the server)
hardens the file permissions
prevents the web server from executing php files in the uploads directory
changes LocalSettings.php to ban IE6
changes LocalSettings.php to make the max upload size 1M. It warns > 500k.
changes LocalSettings.php to disable use of imagemagick as we don't let php exec()
changes LocalSettings.php to require all users to have >=10 character passwords and not be a common password or match their username
changes LocalSettings.php to require all sysop users to have >=20 character passwords
changes LocalSettings.php to use varnish
changes LocalSettings.php to not enable error/warning messages sent to user
changes LocalSettings.php to write debugging logs to 'wiki-error.log' outside the docroot
changes LocalSettings.php to use "MiserMode" to decrease db-heavy operations
changes the caching settings of mediawiki to use APCU (via CACHE_ACCEL) for the MainCache & MessageCache. Else it uses the db, and therefore every load includes a cpPosTime cookie, which causes varnish to hit-for-pass on every page.
changes the caching settings of mediawiki to use the DB for ParserCache
changes the caching settings of mediawiki enable the SidebarCache
changes the caching settings of mediawiki cache interface messages to files on disk outside the docroot ($IP/../cache/)

Points of Contact

Change being performed by: Michael Altfield

Service owners: by Catarina Mota & Marcin Jakubowski for Open Building Institute and Open Source Ecology

Apply to Production

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/noBackup/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/w'
dbName_hetzner1='osewiki'
 dbUser_hetzner1="${mysqlUser_wiki}"
 dbPass_hetzner1="${mysqlPass_wiki}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases --single-transaction | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} ${vhostDir_hetzner1}

####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/noBackup/tmp/backups_for_migration_to_hetzner2/wiki_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/wiki_${stamp}"
backupFileName_db_hetzner1="mysqldump_wiki.${stamp}.sql.bz2"
backupFileName_files_hetzner1="wiki_files.${stamp}.tar.gz"
dbName_hetzner1='osewiki'
dbName_hetzner2='osewiki_db'
 dbUser_hetzner2="osewiki_user"
 dbPass_hetzner2="CHANGEME"
 dbSuperUser_hetzner2="osewiki_superuser"
 dbSuperPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/wiki.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
newMediawikiSourceUrl='https://releases.wikimedia.org/mediawiki/1.30/mediawiki-1.30.0.tar.gz'

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

# STEP 2: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql

# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql

# fix youtube embeds
fromString='https://www.youtube.com/embed/'
toString='https://www.youtube.com/embed/'
sed -i "s^$fromString^$toString^g" db.sql

# fix issuu embeds
fromString='https://static.issuu.com/webembed/'
toString='https://static.issuu.com/webembed/'
sed -i "s^$fromString^$toString^g" db.sql

# fix scrumy embeds
fromString='https://scrumy.com/'
toString='https://scrumy.com/'
sed -i "s^$fromString^$toString^g" db.sql

# fix ted embeds
fromString='https://embed.ted.com/'
toString='https://embed.ted.com/'
sed -i "s^$fromString^$toString^g" db.sql

# fix vimeo embeds
fromString='https://player.vimeo.com/'
toString='https://player.vimeo.com/'
sed -i "s^$fromString^$toString^g" db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
 time nice mysql -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbSuperUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbSuperPass_hetzner2}'; FLUSH PRIVILEGES;"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT SELECT, INSERT, UPDATE, DELETE ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"

# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
time nice tar -xzvf ${backupFileName_files_hetzner1}

# set ['Database'] Name/User/Password (use dbUser/dbPass, not dbSuperUser/dbSuperPass)
# add logic to block IE6 so we can safely remove the XSS bugfix 28235 .htaccess that breaks css
# set `$wgTmpDirectory = "/var/lib/php/tmp_upload"`
# set `$wgLogo = "/images/ose-logo.png"`
vim "${vhostDir_hetzner2}/LocalSettings.php"

# download mediawiki core source code (note this must be done instead of using
# git since [a] git does not include the vendor dir contents and [b] we cannot
# use Composer since it would require breaking our hardened php.ini config

# first, do some string analysis to determine the file, version, and branch name
mwSourceFile=`basename "${newMediawikiSourceUrl}"`
mwReleaseName=`echo "${mwSourceFile}" | awk -F'mediawiki-' '{print $2}' | awk -F. '{print "REL" $1 "_" $2}'`
mwExtractedDir="`echo $mwSourceFile | awk -F'.tar.gz' '{print $1}'`"

wget "${newMediawikiSourceUrl}"
tar -xzvf "${mwSourceFile}"
mkdir "${docrootDir_hetzner2}"
rsync -av --progress "${mwExtractedDir}/" "${docrootDir_hetzner2}/"

# mediawiki ships with lots of calls to unsafe php functions that we've
# intentionally disabled in our hardened php.ini config. They're not necessary
# and just flood our log files with warnings; so let's just comment them out now

find "${docrootDir_hetzner2}/includes/" -type f -exec sed -i 's%^\(\s*\)ini_set\(.*\)%\1#ini_set\2%' '{}' \;
find "${docrootDir_hetzner2}/includes/" -type f -exec sed -i 's%^\(\s*\)putenv\(.*\)%\1#putenv\2%' '{}' \;

# copy-in our images from backups
rsync -av --progress "usr/www/users/osemain/w/images/" "${docrootDir_hetzner2}/images/"

# and move the lone image sticking in root into the images directory
rsync -av --progress "usr/www/users/osemain/w/ose-logo.png" "${docrootDir_hetzner2}/images/"

# create LocalSettings.php that just requires the file from outside the docroot
# write multi-line to file for documentation copy & paste
cat << EOF > "${docrootDir_hetzner2}/LocalSettings.php"
<?php
# including separate file that contains the database password so that it is not stored within the document root.
# For more info see:
#  * https://www.mediawiki.org/wiki/Manual:Security
#  * https://wiki.r00tedvw.com/index.php/Mediawiki/Hardening

\$docRoot = dirname( __FILE__ );
require_once "\$docRoot/../LocalSettings.php";
?>
EOF

# extensions
pushd "${docrootDir_hetzner2}/extensions"
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CategoryTree.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmAccount.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ConfirmEdit.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Cite.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ParserFunctions.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Gadgets.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/ReplaceText.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Renameuser.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/UserMerge.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Nuke.git
git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/OATHAuth.git

git clone -b "${mwReleaseName}" https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Widgets.git
pushd Widgets
git submodule init
git submodule update
popd

# skins
pushd "${docrootDir_hetzner2}/skins"
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/CologneBlue.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Modern.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/MonoBook.git
git clone https://gerrit.wikimedia.org/r/p/mediawiki/skins/Vector.git
popd

# add mime types for our weird ose file types
echo "application/zip fcstd" >> "${docrootDir_hetzner2}/includes/libs/mime/mime.types"
echo "text/plain dxf" >> "${docrootDir_hetzner2}/includes/libs/mime/mime.types"

# remove the block that attempts to fix bug 28235, as it breaks css
vim "${docrootDir_hetzner2}/.htaccess"

# set permissions
chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;

chown not-apache:apache-admins "${vhostDir_hetzner2}/LocalSettings.php"
chmod 0040 "${vhostDir_hetzner2}/LocalSettings.php"

[ -d "${docrootDir_hetzner2}/images" ] || mkdir "${docrootDir_hetzner2}/images"
chown -R apache:apache "${docrootDir_hetzner2}/images"
find "${docrootDir_hetzner2}/images" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/images" -type d -exec chmod 0770 {} \;

[ -d "${vhostDir_hetzner2}/cache" ] || mkdir "${vhostDir_hetzner2}/cache"
chown -R apache:apache "${vhostDir_hetzner2}/cache"
find "${vhostDir_hetzner2}/cache" -type f -exec chmod 0660 {} \;
find "${vhostDir_hetzner2}/cache" -type d -exec chmod 0770 {} \;

# attempt to update
pushd ${docrootDir_hetzner2}/maintenance
php update.php --dbuser "${dbSuperUser_hetzner2}" --dbpass "${dbSuperPass_hetzner2}"

popd

After the steps above are finished, proceed with the validation following the test plan at Wiki Validation

CHG-2018-05-22 migrate wiki to hetzner2

Contents

Status

2018-05-25 01:00 UTC

2018-05-25 00:20 UTC

2018-05-24 23:00 UTC

2018-05-24 21:00 UTC

2018-05-24 16:40 UTC

2018-05-24 14:00 UTC

2018-05-23 21:00 UTC

2018-05-23 19:00 UTC

2018-05-22

2018-05-18

Purpose

Points of Contact

Apply to Production

See Also

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Open Source Ecology

Tools