CHG-2018-02-05 migrate osemain to hetzner2

Status

2018-03-13 06:55 UTC

Marcin confirmed that no further issues have been discovered, and we should mark this change as successful. The content freeze is now officially lifted for osemain (www.opensourceecology.org)

There are minor issues causing varnish to hit-for-pass (causing varnish to call the backend) at an unnecessarily high rate, but this is just a minor issue. I'll log the fixing of this issue in my work log.

2018-03-02 05:17 UTC

I confirmed that the backups were complete on hetzner1, and I moved all of the old osemain wordpress files out of the docroot on on hetzner1:/usr/home/osemain/public_html into hetzner1:/usr/home/osemain/noBackup/deleteMeIn2019/osemain_olddocroot/

I sent an email to Marcin & Catarina asking for validation. The site is still under a content freeze for 1 week, excepting trivial changes for testing.

2018-03-01 20:00 UTC

Pausing this change until tomorrow. It's 90% done. All that remains is to actually remove the old files on hetzner1, but the system-wide backup I kicked-off at the start of this CHG is still running. I'll resume this tomorrow after the backup has been verified on our dreamhost server. Then I'll remove the files & get validation from Marcin. In the meantime, my own spotchecks for both the wiki & osemain show that both sites are fully functional.

2018-03-01 15:00 UTC

Started this change.

2018-02-28 02:32 UTC

Marcin confirmed that all outstanding issues are fixed, and that we can proceed with migration. The new migration time is 2018-03-01 15:00 UTC

2018-02-20

Marcin confirmed that enabling 2FA opened up his account, but he got a Forbidden issue when attempting to update or create *blog* posts (as opposed to pages). I fixed this by whitelisting the modsec rule id = 973337, xss.

2018-02-14

Marcin confirmed that all issues were fixed, but couldn't edit. The result was that his account was locked in a demoted state as the Subscriber role because he didn't enable 2FA. I added documentation to 2FA to clarify.

2018-02-12

Last week we discovered issues with several pages. Critical issues were fixed. Minor issues were agreed to be non-blocking of the migration.

1. /
2. /contributors/
3. /community/
4. /community-true-fans/
5. /history-timeline/
6. /cnc-torch-table-workshop/
7. /about-videos-3/

For more info, see my log from 2018-02-07 Maltfield_log_2018#Wed_Feb_07.2C_2018

2018-02-05 15:11 UTC

I had emailed Marcin yesterday asking for re-validation after I updated some plugins & found some issues with the staging site. I haven't heard back from Marcin, so I'm delaying this migration until I do.

2018-02-04 06:02 UTC

Marcin finished validation of the staging site on hetzner2 & gave the green-light for cutover on Monday, February 5th at 10:00 EST, follwed by a 1-week content freeze.

Purpose

This change does the following for osemain

1. entirely migrate the 'osemain' wordpress site from hetzner1 (shared hosting) to hetzner2 (dedicated hosting)
2. changes the domain from 'http://opensourceecology.org/' to 'https://www..opensourceecology.org'
3. update core wp from v4.7.9 to v4.9.2
4. changes the docroot to a subversion repo of the wp codebase, making it easier to update in the future without sacrificing security (ie: permitting wordpress to update itself) for ease-of-use
5. updates the vhost config to block access to svn files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
6. update plugin 'akismet' from v3.3 to v4.0.2
7. update plugin 'cyclone-slider-2' from v2.12.4 to v3.2.0
8. update plugin 'duplicate-post' from v3.1.2 to v3.2.1
9. update plugin 'jetpack' from v4.7.1 to v5.7.1
10. update plugin 'ml-slider' from v3.5 to v3.6.8
11. update plugin 'ml-slider-pro' from v2.6.6 to v2.7.1
12. update plugin 'recent-tweets-widget' from v1.6.6 to v 1.6.8
13. install & activate plugin 'google-authenticator' to provide 2FA
14. install & activate plugin 'google-authenticator-encourage-user-activation' to require users to use 2FA
15. install & activate plugin 'force-strong-passwords' to prevent users from using a poor password, as determined by the built-in wp core password strength feature
16. install & activate plugin 'rename-wp-login' to obfuscate the wordpress login page (reduces server load & alert emails from HIDS blocking brute force attempts)
17. install & activate plugin 'vcaching' so wordpress & varnish communicate to clear the cache when needed
18. updates the vhost config to block all access to ".*wp-login.php"
19. enable https via nginx
20. enable cache via varnish

Points of Contact

Change being performed by: Michael Altfield

Service owners: Catarina Mota & Marcin Jakubowski

Apply to Production

####################
# run on hetzner1 #
####################

# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'

# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
 dbUser_hetzner1="${mysqlUser_osemain}"
 dbPass_hetzner1="${mysqlPass_osemain}"

# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}

# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}

####################
# run on hetzner2 #
####################

sudo su -

# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log

# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'

# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
 dbUser_hetzner2="osemain_user"
 dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"

# STEP 1: COPY FROM HETZNER1

mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/

# STEP 2: ADD DB

# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig

# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql

# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql

 time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};" 
 time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
 time nice mysql -uroot -p${mysqlPass} < "db.sql"
 time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"

# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
tar -xzvf ${backupFileName_files_hetzner1}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`

mkdir -p ${docrootDir_hetzner2}

pushd ${docrootDir_hetzner2}
currentWpVersion=`curl -i https://core.svn.wordpress.org/tags/ | grep '<li>' | tail -n1 | cut -d\" -f2`
svn co https://core.svn.wordpress.org/tags/${currentWpVersion}/ ${docrootDir_hetzner2}
popd

rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/

# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess

# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
# add/replace salts https://api.wordpress.org/secret-key/1.1/salt/
vim ${vhostDir_hetzner2}/wp-config.php

# SET PERMISSIONS

chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;

chown not-apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0040 "${vhostDir_hetzner2}/wp-config.php"

[ -d "${docrootDir_hetzner2}/wp-content/uploads" ] || mkdir "${docrootDir_hetzner2}/wp-content/uploads"
chown -R apache:apache "${docrootDir_hetzner2}/wp-content/uploads"
find "${docrootDir_hetzner2}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content/uploads" -type d -exec chmod 0770 {} \;

# UPDATE EXISTING PLUGINS

sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all

# UPDATE EXISTING THEMES

sudo -u wp -i wp --path=${docrootDir_hetzner2} theme update --all

# INSTALL PLUGINS

# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd

# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate

# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate

# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate

# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate

# SET PERMISSIONS AGAIN

chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;

chown not-apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0040 "${vhostDir_hetzner2}/wp-config.php"

[ -d "${docrootDir_hetzner2}/wp-content/uploads" ] || mkdir "${docrootDir_hetzner2}/wp-content/uploads"
chown -R apache:apache "${docrootDir_hetzner2}/wp-content/uploads"
find "${docrootDir_hetzner2}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content/uploads" -type d -exec chmod 0770 {} \;

# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
# 3. "Settings" -> "SSL Insecure Content" and then [a] uncheck the "WooCOmmerce" checkbox and [b] change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
# 4. "Varnish Caching" and then [a] check the "Enable" checkbox, [b] enter "86400" for the "Homepage cache TTL", [c] enter "86400" for the "Cache TTL", [d] enter "127.0.0.1:6081" for "IPs", [e] check the "Dynamic host" checkbox

Finally, update the various content as needed:

1. /contributors/, /community-true-fans/, /history-timeline/
   1. update iframes to knigtlab.com time sliders to be https instead of http
2. /cnc-torch-table-workshop/, /eco-tractor_workshop/, /3d-printer-construction-set-workshop-3/
   1. update the widgets to use new "Custom HTML" widgets instead of "Text" widgets to prevent the eventbright's height from being truncated