CHG-2018-02-05 migrate osemain to hetzner2
Contents
Status
2018-03-13 06:55 UTC
Marcin confirmed that no further issues have been discovered, and we should mark this change as successful. The content freeze is now officially lifted for osemain (www.opensourceecology.org)
There are minor issues causing varnish to hit-for-pass (causing varnish to call the backend) at an unnecessarily high rate, but this is just a minor issue. I'll log the fixing of this issue in my work log.
2018-03-02 05:17 UTC
I confirmed that the backups were complete on hetzner1, and I moved all of the old osemain wordpress files out of the docroot on on hetzner1:/usr/home/osemain/public_html into hetzner1:/usr/home/osemain/noBackup/deleteMeIn2019/osemain_olddocroot/
I sent an email to Marcin & Catarina asking for validation. The site is still under a content freeze for 1 week, excepting trivial changes for testing.
2018-03-01 20:00 UTC
Pausing this change until tomorrow. It's 90% done. All that remains is to actually remove the old files on hetzner1, but the system-wide backup I kicked-off at the start of this CHG is still running. I'll resume this tomorrow after the backup has been verified on our dreamhost server. Then I'll remove the files & get validation from Marcin. In the meantime, my own spotchecks for both the wiki & osemain show that both sites are fully functional.
2018-03-01 15:00 UTC
Started this change.
2018-02-28 02:32 UTC
Marcin confirmed that all outstanding issues are fixed, and that we can proceed with migration. The new migration time is 2018-03-01 15:00 UTC
2018-02-20
Marcin confirmed that enabling 2FA opened up his account, but he got a Forbidden issue when attempting to update or create *blog* posts (as opposed to pages). I fixed this by whitelisting the modsec rule id = 973337, xss.
2018-02-14
Marcin confirmed that all issues were fixed, but couldn't edit. The result was that his account was locked in a demoted state as the Subscriber role because he didn't enable 2FA. I added documentation to 2FA to clarify.
2018-02-12
Last week we discovered issues with several pages. Critical issues were fixed. Minor issues were agreed to be non-blocking of the migration.
- /
- /contributors/
- /community/
- /community-true-fans/
- /history-timeline/
- /cnc-torch-table-workshop/
- /about-videos-3/
For more info, see my log from 2018-02-07 Maltfield_log_2018#Wed_Feb_07.2C_2018
2018-02-05 15:11 UTC
I had emailed Marcin yesterday asking for re-validation after I updated some plugins & found some issues with the staging site. I haven't heard back from Marcin, so I'm delaying this migration until I do.
2018-02-04 06:02 UTC
Marcin finished validation of the staging site on hetzner2 & gave the green-light for cutover on Monday, February 5th at 10:00 EST, follwed by a 1-week content freeze.
Purpose
This change does the following for osemain
- entirely migrate the 'osemain' wordpress site from hetzner1 (shared hosting) to hetzner2 (dedicated hosting)
- changes the domain from 'http://opensourceecology.org/' to 'https://www..opensourceecology.org'
- update core wp from v4.7.9 to v4.9.2
- changes the docroot to a subversion repo of the wp codebase, making it easier to update in the future without sacrificing security (ie: permitting wordpress to update itself) for ease-of-use
- updates the vhost config to block access to svn files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
- update plugin 'akismet' from v3.3 to v4.0.2
- update plugin 'cyclone-slider-2' from v2.12.4 to v3.2.0
- update plugin 'duplicate-post' from v3.1.2 to v3.2.1
- update plugin 'jetpack' from v4.7.1 to v5.7.1
- update plugin 'ml-slider' from v3.5 to v3.6.8
- update plugin 'ml-slider-pro' from v2.6.6 to v2.7.1
- update plugin 'recent-tweets-widget' from v1.6.6 to v 1.6.8
- install & activate plugin 'google-authenticator' to provide 2FA
- install & activate plugin 'google-authenticator-encourage-user-activation' to require users to use 2FA
- install & activate plugin 'force-strong-passwords' to prevent users from using a poor password, as determined by the built-in wp core password strength feature
- install & activate plugin 'rename-wp-login' to obfuscate the wordpress login page (reduces server load & alert emails from HIDS blocking brute force attempts)
- install & activate plugin 'vcaching' so wordpress & varnish communicate to clear the cache when needed
- updates the vhost config to block all access to ".*wp-login.php"
- enable https via nginx
- enable cache via varnish
Points of Contact
Change being performed by: Michael Altfield
Service owners: Catarina Mota & Marcin Jakubowski
Apply to Production
####################
# run on hetzner1 #
####################
# STEP 0: CREATE BACKUPS
source /usr/home/osemain/backups/backup.settings
/usr/home/osemain/backups/backup.sh
# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /usr/home/osemain/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner1/*'
# DECLARE VARIABLES
source /usr/home/osemain/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
vhostDir_hetzner1='/usr/www/users/osemain/'
dbName_hetzner1='ose_osemain'
dbUser_hetzner1="${mysqlUser_osemain}"
dbPass_hetzner1="${mysqlPass_osemain}"
# STEP 1: BACKUP DB
mkdir -p ${backupDir_hetzner1}/{current,old}
pushd ${backupDir_hetzner1}/current/
mv ${backupDir_hetzner1}/current/* ${backupDir_hetzner1}/old/
time nice mysqldump -u"${dbUser_hetzner1}" -p"${dbPass_hetzner1}" --all-databases | bzip2 -c > ${backupDir_hetzner1}/current/${backupFileName_db_hetzner1}
# STEP 2: BACKUP FILES
time nice tar -czvf ${backupDir_hetzner1}/current/${backupFileName_files_hetzner1} --exclude="${vhostDir_hetzner1}logs" --exclude="${vhostDir_hetzner1}w" --exclude="${vhostDir_hetzner1}archive" --exclude="${vhostDir_hetzner1}mediawiki-1.24.2.extra" ${vhostDir_hetzner1}
####################
# run on hetzner2 #
####################
sudo su -
# STEP 0: CREATE BACKUPS
# for good measure, trigger a backup of the entire system's database & files:
time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log
# when finished, SSH into the dreamhost server to verify that the whole system backup was successful before proceeding
bash -c 'source /root/backups/backup.settings; ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/*'
# DECLARE VARIABLES
source /root/backups/backup.settings
stamp=`date +%Y%m%d`
backupDir_hetzner1="/usr/home/osemain/tmp/backups_for_migration_to_hetzner2/osemain_${stamp}"
backupDir_hetzner2="/var/tmp/backups_for_migration_from_hetzner1/osemain_${stamp}"
backupFileName_db_hetzner1="mysqldump_osemain.${stamp}.sql.bz2"
backupFileName_files_hetzner1="osemain_files.${stamp}.tar.gz"
dbName_hetzner1='ose_osemain'
dbName_hetzner2='osemain_db'
dbUser_hetzner2="osemain_user"
dbPass_hetzner2="CHANGEME"
vhostDir_hetzner2="/var/www/html/www.opensourceecology.org"
docrootDir_hetzner2="${vhostDir_hetzner2}/htdocs"
# STEP 1: COPY FROM HETZNER1
mkdir -p ${backupDir_hetzner2}/{current,old}
mv ${backupDir_hetzner2}/current/* ${backupDir_hetzner2}/old/
scp -P 222 osemain@dedi978.your-server.de:${backupDir_hetzner1}/current/* ${backupDir_hetzner2}/current/
# STEP 2: ADD DB
# create backup before we start changing the sql file
pushd ${backupDir_hetzner2}/current
cp ${backupFileName_db_hetzner1} ${backupFileName_db_hetzner1}.orig
# extract .sql.bz2 -> .sql
bzip2 -dc ${backupFileName_db_hetzner1} > db.sql
# verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db
vim db.sql
time nice mysql -uroot -p${mysqlPass} -sNe "DROP DATABASE IF EXISTS ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} -sNe "CREATE DATABASE ${dbName_hetzner2}; USE ${dbName_hetzner2};"
time nice mysql -uroot -p${mysqlPass} < "db.sql"
time nice mysql -uroot -p${mysqlPass} -sNe "GRANT ALL ON ${dbName_hetzner2}.* TO '${dbUser_hetzner2}'@'localhost' IDENTIFIED BY '${dbPass_hetzner2}'; FLUSH PRIVILEGES;"
# STEP 3: Add vhost files
mv ${vhostDir_hetzner2}/* ${backupDir_hetzner2}/old/
tar -xzvf ${backupFileName_files_hetzner1}
content_dir=`find ${backupFileName_db_hetzner2} -name wp-content -type d | sort | head -n1`
htaccess_file=`find ${backupFileName_db_hetzner2} -name '.htaccess' -type f | sort | head -n1`
wp_config_file=`find ${backupFileName_db_hetzner2} -name 'wp-config.php' -type f | sort | head -n1`
mkdir -p ${docrootDir_hetzner2}
pushd ${docrootDir_hetzner2}
currentWpVersion=`curl -i https://core.svn.wordpress.org/tags/ | grep '<li>' | tail -n1 | cut -d\" -f2`
svn co https://core.svn.wordpress.org/tags/${currentWpVersion}/ ${docrootDir_hetzner2}
popd
rsync -av --progress ${wp_config_file} ${vhostDir_hetzner2}/
rsync -av --progress ${htaccess_file} ${docrootDir_hetzner2}/
rsync -av --progress ${content_dir} ${docrootDir_hetzner2}/
# make sure this is sudomain, not subdir now
vim ${docrootDir_hetzner2}/.htaccess
# update WP_HOME/WP_SITEURL/DB_NAME/DB_USER/DB_PASSWORD/DB_HOST/
# add/replace salts https://api.wordpress.org/secret-key/1.1/salt/
vim ${vhostDir_hetzner2}/wp-config.php
# SET PERMISSIONS
chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0040 "${vhostDir_hetzner2}/wp-config.php"
[ -d "${docrootDir_hetzner2}/wp-content/uploads" ] || mkdir "${docrootDir_hetzner2}/wp-content/uploads"
chown -R apache:apache "${docrootDir_hetzner2}/wp-content/uploads"
find "${docrootDir_hetzner2}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content/uploads" -type d -exec chmod 0770 {} \;
# UPDATE EXISTING PLUGINS
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin update --all
# UPDATE EXISTING THEMES
sudo -u wp -i wp --path=${docrootDir_hetzner2} theme update --all
# INSTALL PLUGINS
# install & configure 2FA plugins
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator --activate
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install google-authenticator-encourage-user-activation --activate
defaultOtpAccountDescription="`basename ${vhostDir_hetzner2}` wp"
pushd ${docrootDir_hetzner2}/wp-content/plugins/google-authenticator
sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php
popd
# install 'force-strong-passwords' plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install force-strong-passwords --activate
# install rename-wp-login plugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install rename-wp-login --activate
# install "SSL Insecure Content Fixer" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install ssl-insecure-content-fixer --activate
# install "Varnish Caching" pugin
sudo -u wp -i wp --path=${docrootDir_hetzner2} plugin install vcaching --activate
# SET PERMISSIONS AGAIN
chown -R not-apache:apache "${vhostDir_hetzner2}"
find "${vhostDir_hetzner2}" -type d -exec chmod 0050 {} \;
find "${vhostDir_hetzner2}" -type f -exec chmod 0040 {} \;
chown not-apache:apache-admins "${vhostDir_hetzner2}/wp-config.php"
chmod 0040 "${vhostDir_hetzner2}/wp-config.php"
[ -d "${docrootDir_hetzner2}/wp-content/uploads" ] || mkdir "${docrootDir_hetzner2}/wp-content/uploads"
chown -R apache:apache "${docrootDir_hetzner2}/wp-content/uploads"
find "${docrootDir_hetzner2}/wp-content/uploads" -type f -exec chmod 0660 {} \;
find "${docrootDir_hetzner2}/wp-content/uploads" -type d -exec chmod 0770 {} \;
# finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings:
# 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login'
# 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
# 3. "Settings" -> "SSL Insecure Content" and then [a] uncheck the "WooCOmmerce" checkbox and [b] change the HTTPS detection from the default "standard WordPress function" to "HTTP_X_FORWARDED_PROTO"
# 4. "Varnish Caching" and then [a] check the "Enable" checkbox, [b] enter "86400" for the "Homepage cache TTL", [c] enter "86400" for the "Cache TTL", [d] enter "127.0.0.1:6081" for "IPs", [e] check the "Dynamic host" checkbox
Finally, update the various content as needed:
- /contributors/, /community-true-fans/, /history-timeline/
- update iframes to knigtlab.com time sliders to be https instead of http
- /cnc-torch-table-workshop/, /eco-tractor_workshop/, /3d-printer-construction-set-workshop-3/
- update the widgets to use new "Custom HTML" widgets instead of "Text" widgets to prevent the eventbright's height from being truncated
