CHG-2017-09-25 migrate obi to hetzner2
Contents
Status
2017-10-02 21:15 UTC
Catarina confirmed that changing a few links from 'http' to 'https' fixed all known issues, confirmed that the change was successful, and that the content freeze can be lifted.
2017-09-25 23:00 UTC
Change completed successfully. There's a few issues with mixed content warning (and refusals to load) from browsers. Catarina confirmed that she will look into them in the morning. A content freeze (for non-trivial content) will remain in effect until Monday Oct 2nd.
2017-09-25 21:25 UTC
Change started after green-light from Catarina.
2017-09-25 19:41 UTC
This change is awaiting a content change by Catarina. Once complete, we will have a content freeze for 1 week, and I will begin applying this change to production.
Purpose
This change does the following for openbuildinginstitute.org
- update core wp from v4.5.9 to v4.8.1
- changes the docroot to a subversion repo of the wp codebase, making it easier to update in the future without sacrificing security (ie: permitting wordpress to update itself) for ease-of-use
- updates the vhost config to block access to svn files, and more generally: ".*\.(svn|git|hg|bzr|cvs|ht)/.*"
- moves '3Dmodels' dir inside 'wp-content' so it won't be orphaned in future updates/migration
- update plugin 'akismet' from v3.3.3 to v3.3.4
- update plugin 'amr-shortcode-any-widget' from v3.3.3 to v3.6
- update plugin 'duplicate-page' from v2.2 to v2.3
- install & activate plugin 'google-authenticator' to provide 2FA
- install & activate plugin 'google-authenticator-encourage-user-activation' to require users to use 2FA
- install & activate plugin 'force-strong-passwords' to prevent users from using a poor password, as determined by the built-in wp core password strength feature
- install & activate plugin 'rename-wp-login' to obfuscate the wordpress login page (reduces server load & alert emails from HIDS blocking brute force attempts)
- updates the vhost config to block all access to ".*wp-login.php"
- enable https
Points of Contact
Change being performed by: Michael Altfield
Service owners: Catarina Mota & Marcin Jakubowski
Apply to Production
sudo su - ################# # set variables # ################# vhostDir=/var/www/html/openbuildinginstitute.org dbName=obi_db dbUser=obi_db_user dbPass=CHANGEME rootDbPass=CHANGEME stamp=`date +%Y%m%d_%T` vhostDirMovedTo="/var/www/html/obi.${stamp}.bak" ################ # make backups # ################ # SYSTEM-SIDE BACKUPS time /bin/nice /root/backups/backup.sh &>> /var/log/backups/backup.log source /root/backups/backup.settings # validate ssh $RSYNC_USER@$RSYNC_HOST du -sh backups/hetzner2/* # LOCAL BACKUPS tmpDir=/var/tmp/chg.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir service httpd stop # create backup of all DBs for good measure time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz # dump wp DB contents time nice mysqldump -u$dbUser -p$dbPass --database $dbName > $dbName.$stamp.sql # files backup rsync -av --progress "/var/www/html/obi" "vhostDir.${stamp}.bak/" ############# # update db # ############# # verify the first 2 (non-comment) occurances of $dbName meet the naming convention of "<siteName>_db" cp $dbName.$stamp.sql $dbName.$stamp.sql.bak vim $dbName.$stamp.sql # update links for 3Dmodels, which we're moving to the 'wp-content' dir sed -i 's^/3Dmodels/^/wp-content/3Dmodels/^g' "$dbName.$stamp.sql" time nice mysql -uroot -p$rootDbPass -sNe "DROP DATABASE IF EXISTS $dbName;" time nice mysql -uroot -p$rootDbPass -sNe "CREATE DATABASE $bName; USE $dbName;" time nice mysql -uroot -p$rootDbPass < "$dbName.$stamp.sql" time nice mysql -uroot -p$rootDbPass -sNe "GRANT ALL ON $dbName.* TO '$dbUser'@'localhost' IDENTIFIED BY '$dbPass'; FLUSH PRIVILEGES;" popd ################ # update files # ################ mv "/var/www/html/obi" "${vhostDirMovedTo}" mkdir -p "${vhostDir}/htdocs" pushd "${vhostDir}/htdocs" yum install subversion svn co https://core.svn.wordpress.org/tags/4.8.1 . find $vhostDir -type d -exec chmod 750 {} \; find $vhostDIr -type f -exec chmod 640 {} \; chown -R apache:apache ${vhostDir} rsync -av --progress ${vhostDirMovedTo}/wp-config.php ${vhostDir}/ chown apache:apache-admins ${vhostDir}/wp-config.php chmod 440 ${vhostDir}/wp-config.php # change WP_HOME, WP_SITEURL, DB_NAME, DB_USER, & DB_PASSWORD as needed vim ${vhostDir}/wp-config.php # we want to copy files that don't exist yet in our new install dir. if a file exists in both, don't overwrite the new from the old rsync -av --progress --ignore-existing "${vhostDirMovedTo}/htdocs/wp-content/" "$vhostDir/htdocs/wp-content/" # delete unnecessary & empty upgrade dir rm -rf "${vhostDir}/htdocs/wp-content/upgrade" # delete unnecessary plugins rm -rf "${vhostDir}/htdocs/wp-content/plugins/hello.php" rm -rf "${vhostDir}/htdocs/wp-content/plugins/password-protected" rm -rf "${vhostDir}/htdocs/wp-content/plugins/simple-301-redirects" # migrate '3Dmodels' dir, but put it in 'wp-content' for future migrations (the wp upgrade docs migrate this dir already) rsync -av --progress "${vhostDirMovedTo}/htdocs/3Dmodels" "${vhostDir}/htdocs/wp-content/" # take the old .htaccess; overwrite existing if needed rsync -av --progress "${vhostDirMovedTo}/htdocs/.htaccess" "${vhostDir}/htdocs/" # set permissions, final pass chown -R apache:apache "${vhostDir}" find "${vhostDir}" -type d -exec chmod 0750 {} \; find "${vhostDir}" -type f -exec chmod 0640 {} \; find "${vhostDir}/htdocs/wp-content" -type f -exec chmod 0660 {} \; find "${vhostDir}/htdocs/wp-content" -type d -exec chmod 0770 {} \; chown apache:apache-admins "${newVhostDir}/wp-config.php" chmod 0440 "${vhostDir}/wp-config.php" # update wp plugins sudo -u wp -i wp --path="${vhostDir}/htdocs" plugin update --all # update wp themes #(we skip this in obi's oshine, since it caused issues) #sudo -u wp -i wp --path="${vhostDir}/htdocs" theme update --all # install & configure 2FA plugins sudo -u wp -i wp --path=${vhostDir}/htdocs plugin install google-authenticator --activate sudo -u wp -i wp --path=${vhostDir}/htdocs plugin install google-authenticator-encourage-user-activation --activate defaultOtpAccountDescription="`basename ${vhostDir}` wp" cd "${vhostDir}/htdocs/wp-content/plugins/google-authenticator" sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php # install 'force-strong-passwords' plugin sudo -u wp -i wp --path="${vhostDir}/htdocs" plugin install force-strong-passwords --activate # install rename-wp-login plugin sudo -u wp -i wp --path="${vhostDir}/htdocs" plugin install rename-wp-login --activate # add deny for 'wp-login.php' _inside_ the VirtualHosts block of the vhost config file # add deny for svn-related files, and a host of other sensitive dot-files. Again, put this _inside_ the VirtualHosts block # change vhost file to change to https mv /etc/httpd/conf.d/obi.conf /etc/httpd/conf.d/obi.conf.$stamp.bak cat << EOF > /etc/httpd/conf.d/00-openbuildinginstitute.org.conf <VirtualHost 138.201.84.223:80> ServerName openbuildinginstitute.org ServerAlias www.openbuildinginstitute.org Redirect permanent / https://openbuildinginstitute.org/ </VirtualHost> Listen openbuildinginstitute.org:443 <VirtualHost openbuildinginstitute.org:443> ServerName openbuildinginstitute.org ServerAlias www.openbuildinginstitute.org DocumentRoot "/var/www/html/openbuildinginstitute.org/htdocs" Include /etc/httpd/conf.d/ssl.openbuildinginstitute.org # disable mod_security with rules as needed # (found by logs in: /var/log/httpd/modsec_audit.log) <Location "/wp-admin/"> <IfModule security2_module> SecRuleRemoveById 960015 981173 960024 960904 960015 960017 970901 950109 </IfModule> </Location> # block dot files, such as svn files from checking out wp core <LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*> Deny From All </LocationMatch> # block access to 'wp-login.php' from brute-forcers; see wp plugin 'rename-wp-login' <LocationMatch .*wp-login.php> Deny From All </LocationMatch> </VirtualHost> <Directory "/var/www/html/openbuildinginstitute.org/htdocs"> Options -Indexes -Includes AllowOverride All Require all granted Order allow,deny Allow from all # Harden vhost docroot by blocking all request types except the 3 essentials <LimitExcept GET POST HEAD> deny from all </LimitExcept> </Directory> #<LocationMatch .*wp-login.php> # Deny From All #</LocationMatch> vim /etc/httpd/conf.d/TODO #<LocationMatch .*\.(svn|git|hg|bzr|cvs|ht)/.*> # Deny From All #</LocationMatch> vim /etc/httpd/conf.d/TODO vim /etc/httpd/conf.d/TODO # test apache config httpd -t # if OK, apply apache configs service httpd start # finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings: # 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login' # 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"
Build Ephemeral Staging
These commands were used to migrate the existing & outdated wordpress installs to a subversion-backed, ephemeral clone of the site. Changes were iterated to these commands documented here as validation by the site owners exposed issues with the upgrades, missing content, etc. Once the clone was validated entirely, the live site could be brought down & confidently updated per the findings with the ephemeral clone.
obi
oldVhostDir=/var/www/html/obi newVhostDir=/var/www/html/obi2 oldDbName=obi_db newDbName=obi2_db newDbUser=obi2_user newDbPass=CHANGEME rootDbPass=CHANGEME cd rm -rf $newVhostDir mkdir -p $newVhostDir/htdocs pushd $newVhostDir/htdocs yum install subversion svn co https://core.svn.wordpress.org/tags/4.8.1 . find $newVhostDir -type d -exec chmod 750 {} \; find $newVhostDIr -type f -exec chmod 640 {} \; chown -R apache:apache $newVhostDir stamp=`date +%Y%m%d_%T` tmpDir=/var/tmp/dbChange.$stamp mkdir $tmpDir chown root:root $tmpDir chmod 0700 $tmpDir pushd $tmpDir time nice mysqldump -uroot -p$rootDbPass --all-databases | gzip -c > preBackup.all_databases.$stamp.sql.gz time nice mysqldump -uroot -p$rootDbPass --databases $oldDbName > $oldDbName.$stamp.sql cp $oldDbName.$stamp.sql $newDbName.$stamp.sql # replace the first 2 (non-comment) occurances of $OldDbName with $newDbName vim $newDbName.$stamp.sql # update links for 3Dmodels, which we're moving to the 'wp-content' dir sed -i 's^/3Dmodels/^/wp-content/3Dmodels/^g' "$newDbName.$stamp.sql" time nice mysql -uroot -p$rootDbPass -sNe "DROP DATABASE IF EXISTS $newDbName;" time nice mysql -uroot -p$rootDbPass -sNe "CREATE DATABASE $newDbName; USE $newDbName;" time nice mysql -uroot -p$rootDbPass < "$newDbName.$stamp.sql" time nice mysql -uroot -p$rootDbPass -sNe "GRANT ALL ON $newDbName.* TO '$newDbUser'@'localhost' IDENTIFIED BY '$newDbPass'; FLUSH PRIVILEGES;" popd rsync -av --progress $oldVhostDir/wp-config.php $newVhostDir/ chown apache:apache $newVhostDir/wp-config.php chmod 400 $newVhostDir/wp-config.php # change WP_HOME, WP_SITEURL, DB_NAME, DB_USER, & DB_PASSWORD vim $newVhostDir/wp-config.php # we want to copy files that don't exist yet in our new install dir. if a file exists in both, don't overwrite the new from the old rsync -av --progress --ignore-existing $oldVhostDir/htdocs/wp-content/ $newVhostDir/htdocs/wp-content/ # delete unnecessary & empty upgrade dir rm -rf $newVhostDir/htdocs/wp-content/upgrade # delete unnecessary plugins rm -rf $newVhostDir/htdocs/wp-content/plugins/hello.php rm -rf $newVhostDir/htdocs/wp-content/plugins/password-protected rm -rf $newVhostDir/htdocs/wp-content/plugins/simple-301-redirects # migrate '3Dmodels' dir, but put it in 'wp-content' for future migrations (the wp upgrade docs migrate this dir already) rsync -av --progress $oldVhostDir/htdocs/3Dmodels $newVhostDir/htdocs/wp-content/ # take the old .htaccess; overwrite existing if needed rsync -av --progress $oldVhostDir/htdocs/.htaccess $newVhostDir/htdocs/ # set permissions chown -R apache:apache "${newVhostDir}" find "${newVhostDir}" -type d -exec chmod 0750 {} \; find "${newVhostDir}" -type f -exec chmod 0640 {} \; find "${newVhostDir}/htdocs/wp-content" -type f -exec chmod 0660 {} \; find "${newVhostDir}/htdocs/wp-content" -type d -exec chmod 0770 {} \; chown apache:apache-admins "${newVhostDir}/wp-config.php" chmod 0440 "${newVhostDir}/wp-config.php" # update wp plugins sudo -u wp -i wp --path=$newVhostDir/htdocs plugin update --all # update wp themes #sudo -u wp -i wp --path=$newVhostDir/htdocs theme update --all # install & configure 2FA plugins sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator --activate sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install google-authenticator-encourage-user-activation --activate defaultOtpAccountDescription="`basename $newVhostDir` wp" cd $newVhostDir/htdocs/wp-content/plugins/google-authenticator sed -i "s^\$GA_description\s=\s__(\s[\"'].*[\"']^\$GA_description = __( '$defaultOtpAccountDescription'^" google-authenticator.php # install 'force-strong-passwords' plugin sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install force-strong-passwords --activate # install rename-wp-login plugin sudo -u wp -i wp --path=$newVhostDir/htdocs plugin install rename-wp-login --activate # add vhosts dir to php.ini 'open_basedir' vim /etc/php.ini # create vhosts apache conf file w/ deny for 'wp-login.php' #<LocationMatch .*wp-login.php> # Deny From All #</LocationMatch> vim /etc/httpd/conf.d/TODO # apply apache configs service httpd reload # finally, log into the new wordpress site (use '/login' instead of '/wp-login.php'. After authenticating, wp will ask you to update, if necessary. Then update settings: # 1. "Settings" -> "Permalinks" -> "Rename wp-login.php" -> "Login url" = 'ose-hidden-login' # 2. "Settings" -> "General" -> "Google Authenticator - Encourage User Activation" = "Force the user"